Building A Scalable Open Source Storage Solution

Building a scalable, open source storage and processing solution for biodiversity dataPhil CryerAnthony GoddardThursday, November 12, 2009> Biodiversity Heritage Library's data • al BHL storage is handled by the Internet Archive• 38,000+ scanned books• approximately 48 terabytes of data• unable to self-hostThursday, November 12, 2009> BHL - Europe• 3 year, EU funded project• 28 major natural history museums, botanical gardens and other cooperating institutions• third file-store of al BHL data• col ecting cultural heritage from al over EuropeThursday, November 12, 2009> Data explosion• more data being created• more data being saved• more data tomorrow• storage has not kept up with Moore’s Law• this presentation wil be saved online, more data!Thursday, November 12, 2009> Data explosion• more data being created• more data being saved• more data tomorrow• storage has not kept up with Moore’s Law• this presentation wil be saved online, more data!Thursday, November 12, 2009> Potential #fail’sThursday, November 12, 2009 > Problem 1 - Data access• file size we can’t store• latency of large files• quality user experience• processing data-mining Thursday, November 12, 2009 > Problem 1 - Data access• file size we can’t store• latency of large filesAccess • quality user experience• processing data-mining denied...Thursday, November 12, 2009 > Problem 2 - Copyright concerns• international copyright concerns• potential related funding issues• we’d rather not let this be an issue©Thursday, November 12, 2009> Problem 3 - RedundancyThursday, November 12, 2009