NEO powered ERP5 reaches 100 TB of Wendelin out-of-core data

NEO distributed database has reached on November 24th 2017 a storage size of 100 TB consisting of 30 big data streams of about 3.3 TB each stored on a cluster of inexpensive computers running 18 parallel MariaDB and RockDB storage engines.
  • Last Update:2017-11-30
  • Version:001
  • Language:en

NEO, the distributed transactional NoSQL object database of Nexedi, has reached on November 24th 2017 a storage size of 100 TB on a redundant array of inexpensive computers. This success demonstrates that both ERP5 open source ERP/CRM and Wendelin out-of-core Big Data platform can power the biggest commercial bulkloads, both for transactional and data science applications.

The data stored in NEO consists of 30 big data streams of about 3.3 TB, each of which can be accessed sequentially or randomly.

In order to achieve this result, Nexedi has been running a NEO ingestion test drive for 40 days on 6 inexpensive Dedibox computers, each of which with 3 SATA disks of 6TB, 1x Intel® Xeon® E5 1410 v2 and 64 GB RAM. 30 concurrent fluentd streams of data have been ingested into ERP5 / Wendelin platform powered by NEO at an average growth rate of 2.5 TB per day. The NEO cluster was configured with 30 independent Zope application servers and 18 independent replicated storages for a total disk usage of 88.6 TB. A compression factor of 43.5% was observed on the random data that was ingested in this test run.

NEO database relies on an innovative protocol that turns a cluster of independent storage engines into a single transactional storage space. NEO currently supports MariaDB, MySQL, SQLite and POSIX filesystem as possible storage engines. For the current tests, MariaDB has been used with two different storage backends: RocksDB and TokuDB. Both RocksDB and TokuDB have shown similar peformance: RocksDB write performance was more consistent whereas TokuDB read performance was a bit better. Detailed test report will be published soon.

Contact

  • Photo Jean-Paul Smets
  • Logo Nexedi
  • Jean-Paul Smets
  • jp (at) nexedi (dot) com
  • Jean-Paul Smets is the founder and CEO of Nexedi. After graduating in mathematics and computer science at ENS (Paris), he started his career as a civil servant at the French Ministry of Economy. He then left government to start a small company called “Nexedi” where he developed his first Free Software, an Enterprise Resource Planning (ERP) designed to manage the production of swimsuits in the not-so-warm but friendly north of France. ERP5 was born. In parallel, he led with Hartmut Pilch (FFII) the successful campaign to protect software innovation against the dangers of software patents. The campaign eventually succeeeded by rallying more than 100.000 supporters and thousands of CEOs of European software companies (both open source and proprietary). The Proposed directive on the patentability of computer-implemented inventions was rejected on 6 July 2005 by the European Parliament by an overwhelming majority of 648 to 14 votes, showing how small companies can together in Europe defeat the powerful lobbying of large corporations. Since then, he has helped Nexedi to grow either organically or by investing in new ventures led by bright entrepreneurs.