Improving python memory management in a multi-process environment with wendelin.core

Improving python memory management in a multi-process environment with wendelin.core

Jean-Paul Smets

Nexedi Logo

Nexedi

Nexedi World Map
  • Largest OSS Publisher in Europe
  • About 35 staff around the world
  • Mostly using Python | JavaScript
  • Flexible | Sustainable | Resilient

Nexedi - Clients

Nexedi References and Map

Python3 in 2017

A great language if one can accept to:

  • run multiple threads on a single core
  • experience poor performance for arithmetics
  • rewrite all your python2 code
  • learn another language in addition to Javascript

Leaving Python3?

  • Alphabet (Grumpy)
  • Dropbox (dropped pyston)

Good reasons to stay with Python3

  • NumPy
  • scipy
  • scikit-learn
  • pandas
  • jupyter
  • etc.
 

Nexedi Free Software Code Base: 12 Mlc

  • ERP5 ERP and CRM (python2.7)
  • Wendelin Big Data (python2.7)
  • SlapOS Edge Computing (python2.7)
  • re6st reslient network (python2.7)
  • Webrunner web based IDE (python2.7)
  • cloudooo multimedia converter (python2.7)
  • NEO distributed NoSQL database (python2.7)
  • JIO virtual storage (JS)
  • RenderJS light components (JS)
  • OfficeJS web office (JS)
  • NayuOS laptop OS (various)
 

Python3 in 2017

A great language if one can accept to:

 

Multiprocessing

this is how ERP5 works for 15 years

  • scalable
  • reliable
  • clean
  • compute efficient
  • memory inefficient

A Big Array

 

Multithreading

 

Python Multithreading

 

Python Multiprocessing

 

Adding more RAM

 

System memory management

 

Demo

Licensing

GPLv3 with wide exception

  • Free for Free Software (any license)
  • Must sponsor or advertise for Non Free Software

Thank You

Image Nexedi Headquarters, Lille, France
  • Nexedi SA
  • 270 bd ClĂ©menceau
  • 59700 Marcq-en-Baroeul
  • France