roque (at) gmail (dot) com
It is the result of a collaboration between Nexedi and Telecom Paristech under the "Wendelin IA" project.
The presentation has four parts.... XXXX
Big Data sharing is essential for research and startups. Building new A.I. models required access to large data sets. Those data sets are available to big platforms such as Google or Alibaba which tend to keep them secret. However, smaller companies or independent research teams have no access to this type of data.
Big Data sharing is also essential inside a large corporation so that production teams which produce big data sets can share it with research teams which produce A.I. models.
A Big Data sharing platform has two solve five problems...
Last, it should protect trade secret against the extra-territorial application of certain Laws. For exemple, any data hosted by a US platform can be accessed by NSA, no matter the location of the data centre. This is the consequence of legal obligation of US companies: the CLOUD Act. Any company willing to enforce trade secret on its data should thus stay away from US platform and use a solution with full access to source code (for auditing purpose), possibly on premise.
There are actually very few solutions to share big data sets efficiently.
Some research teams use Amazon S3. However, Amazon S3 does no provide a tool to manage a catalog of data sets, and is not suitable for data protected by trade secret (unless it is encrypted).
One could also use Github as a data lake thanks to Git Large File Storage. There is even an open source implementation for Gitlab which could be combined with Sheepdog block storage and Nexedi's gitlab acceleration patch (17 times faster) to form a high performance platform.
XXX explain why those solutions are insufficient XXX
We are now going to explain how one can easily build a Big Data sharing platform based on Wendelin framwork.
Wendelin is a big data framework designed for industrial applications based on python, NumPy, Scipy and other NumPy based libraries. It uses at its core the NEO distributed transactional NoSQL database to store petabytes of binary data ...
Wendelin has been implemented mainly in streaming applications where data is ingested in real time from wind turbines, using fluentd middleware by Treasure Data, a pioneer of big data platforms. Fluentd has proven to be useful thanks to its multi-protocol nature which ensure perfect interoperability and to its buffering technology which ensures no data is lost over poor networks.
Wendelin can store petabytes of out-of-core streams or arrays. This is a very important feature because it means that any batch script on server side can process arrays or streams that are much larger than RAM. XXX explain XXX
Based on our success with fluentd, we selected another tool from Treasure Data: embulk. Embulk provides the same kind of interoperability. However, it was a bit complex to use.
Therefore we created a wrapper called ebulk which imitated git syntax yet is based on embulk. We use ebulk to push, pull, add, etc. big data to a data set.
~ $ ebulk init brain_mri ~ $ ebulk push brain_mri
~ $ ebulk pull brain_mri
~ $ ls brain_mri_12
~ $ cp ~/new_mri/2038273.mri brain_mri ~ $ mv brain_mri/Probes.csv brain_mri/initial_probes.csv ~ $ vi brain_mri/Readme.txt ~ $ rm brain_mri/meta.tsv ~ $ ebulk status brain_mri
~ $ ebulk add brain_mri/2038273.mri ~ $ ebulk add brain_mri/initial_probes.csv ~ $ ebulk add brain_mri/Readme.txt ~ $ ebulk remove brain_mri/meta.tsv ~ $ ebulk push brain_mri
~ $ ebulk push brain_mri --directory new_mri/
~ $ ebulk pull brain_mri
~ $ ebulk push brain_mri --storage <protocol>
~ $ ebulk push S3-input --storage s3
ebulk push ebulk pull ebulk add/remove ebulk status ebulk push --storage