WENDELIN combines Scikit Learn machine learning and NEO distributed storage for out-of-core data analytics in python
Table of Contents
Introducing Wendelin
Wendelin Core
NEO
ERP5
SlapOS
res6st
ShaCache
Scikit.Learn et al
- This is Wendelin - stack written in 100% Python
- Integrates with popular libs (Jupyter, scikit.learn, Pandas, ...)
- Wendelin.Core component to compute beyond available RAM
Stack: Wendelin Core
- "Out-Of-Core"
- Allows computation beyond limitation of physical RAM
- Embedded within Wendelin stack, but also usable standalone
Stack: NEO
- Distributed Transactional NoSQL Object Database
- Used as default storage in Wendelin
Stack: ERP5
- Based on 5 Theoretical Principles (Unified Business Model)
- Generic: handles Big Data the same way as millions of ERP-related objects
- ERP5 used under the hood of Wendelin
Stack: SlapOS
- Cloud Provisionment, Deployment, Orchestration for single machines or clusters
- Installable on any architecture
- Uses Ansible recipes, fully automated, everything is a service
- SlapOS is a bit like Gentoo (Linux) in that everything is compiled from scratch every time as described by a recipe. Recipes allow to reuse configurations across a cluster besides enabling a homogenous way to manage installable components.
- Commercial Service available: Vifib.com (40€/4€ per server/vm per month)
- SlapOS Website | Forum
- SlapOS: Gitlab
- -----------
- Photo: Fotolia.fr - TOMO
Stack: re6st
- IPv6-based Resilient Mesh Overlay Network, "internet on top of the internet"
- Monitors possible connections between nodes, channels traffic on demand
- Mitigates risks of unavialable routes/packet loss
- Commercial service available: GrandeNet - Application Delivery Network
- Res6st: Gitlab
- -----------
- Photo: Fotolia.fr - ake1150
Stack: ShaCache
- MD5-signed cache, use signatures to validate cached files
- Allows to reuse trusted compilations by providing key value
- NoSQL storage with REST API, archives files through HTTP PUT method
Stack: Scikit.learn (et al)
- Integrate popular libraries for Data Science and Machine Learning
- Provide familiar interface which connects to ERP5/Wendelin.Core
- Scikit.Learn (project partner), Jupyter, SciPy, NumPy, Pandas
- Tutorial to extend Wendelin with 3rd Party libraries using SlapOS: Coming soon
- Scikit.Learn lectures
- -----------
- Photo: Fotolia.fr - weerapat1003
Wendelin: Advantages
- Single stack to handle big data from creation to utilization
- Computation independent of available RAM
- Fully written in Python, being built to handle Big Data
Wendelin: Challenges
- Simplify processes to make stack more accessable
- Add Wendelin UI on top of ERP5
- Provide tutorials and base documentation to foster adoption
Related Articles