Hyperconvergence: Big Data to Small App

Tutorial for the pydata Paris 2016 conference on how to go use Wendelin to go from Big Data to a small application.
  • Last Update:2017-01-12
  • Version:1
  • Language:en


Pydata logo


  • Nexedi
  • Wendelin
  • Tutorial


Nexedi World Map
  • Largest OSS Publisher in Europe
  • About 30 staff around the world
  • Mostly using Python | JavaScript
  • Lean | Idealistic | Simple

Nexedi - Clients

Nexedi References and Map

Nexedi - Free Software R&D

Nexedi Client List
  • DREAM - Application decision support using discrete event simulation in Python
  • WENDELIN - Big Data stack & Machine Learning, out-of-core computing
  • OPEN PAAS - Next-generation collaboration platform for business

Introducing Wendelin

Wendelin Core
Scikit.Learn et al
  • This is Wendelin - stack written in 100% Python
  • Integrates with popular libs (Jupyter, scikit.learn, Pandas, ...)
  • Wendelin.Core component to compute beyond available RAM

Stack: Wendelin Core

Wendelin Core Computing
  • "Out-Of-Core"
  • Allows computation beyond limitation of physical RAM
  • Embedded within Wendelin stack, but also usable standalone

Stack: NEO

NEO Distributed Storage
  • Distributed Transactional NoSQL Object Database
  • Used as default storage in Wendelin

Stack: ERP5

ERP5 - Open Source Enterprise Application Framework
  • Based on 5 Theoretical Principles (Unified Business Model)
  • Generic: handles Big Data the same way as millions of ERP-related objects
  • ERP5 used under the hood of Wendelin

Stack: SlapOS

SlapOS | Hyperconvergent Platform as a Service
  • Cloud Provisionment, Deployment, Orchestration for single machines or clusters
  • Installable on any architecture
  • Uses Ansible recipes, fully automated, everything is a service
  • SlapOS is a bit like Gentoo (Linux) in that everything is compiled from scratch every time as described by a recipe. Recipes allow to reuse configurations across a cluster besides enabling a homogenous way to manage installable components.
  • Commercial Service available: Vifib.com (40€/4€ per server/vm per month)
  • SlapOS Website | Forum
  • SlapOS: Gitlab
  • -----------
  • Photo: Fotolia.fr - TOMO

Stack: re6st

Res6st | Resilient Mesh Overlay Network
  • IPv6-based Resilient Mesh Overlay Network, "internet on top of the internet"
  • Monitors possible connections between nodes, channels traffic on demand
  • Mitigates risks of unavialable routes/packet loss
  • Commercial service available: GrandeNet - Application Delivery Network
  • Res6st: Gitlab
  • -----------
  • Photo: Fotolia.fr - ake1150

Stack: ShaCache

Shacache | MD5-Signed Cache
  • MD5-signed cache, use signatures to validate cached files
  • Allows to reuse trusted compilations by providing key value
  • NoSQL storage with REST API, archives files through HTTP PUT method

Stack: Scikit.learn (et al)

Wendelin | Extensions
  • Integrate popular libraries for Data Science and Machine Learning
  • Provide familiar interface which connects to ERP5/Wendelin.Core
  • Scikit.Learn (project partner), Jupyter, SciPy, NumPy, Pandas
  • Tutorial to extend Wendelin with 3rd Party libraries using SlapOS: Coming soon
  • Scikit.Learn lectures
  • -----------
  • Photo: Fotolia.fr - weerapat1003

Wendelin: Advantages

Wendelin | Advantages
  • Single stack to handle big data from creation to utilization
  • Computation independent of available RAM
  • Fully written in Python, being built to handle Big Data

Wendelin: Challenges

Wendelin | Challenges
  • Simplify processes to make stack more accessable
  • Add Wendelin UI on top of ERP5
  • Provide tutorials and base documentation to foster adoption

Tutorial Overview

  • 1. Setup: Getting Wendelin
  • 2. Audio: Simulate IOT Sensor
  • 3. Analyse: Work with ingested data
  • 4. Visualize: Display computed data

Setup: Getting Wendelin

Setup: Explained

Traffic routing overview from outside to inside
  • Public Accessable IPv6, IPv4 tutorial fallback
  • Load Wendelin via ShaCache
  • Port forwarding via Socat
  • Internally, Nexedi uses IPv4, front ends configured for IPv6
  • Emulate server or data center in virtual machine
  • Tutorial should be run on a laptop, server
  • For the tutorial we setup 2 machines with 10 Virtual Machines each.
  • IPv6:30001 access to QEMU-KVM IPv6:30001, after that NAT port forwarding to Localhost 20001 IPV4 internal
  • Internally forwarded via SOCAT to slappart Jupyter/Zope...

Setup: Undressed

Machine with Single KVM
  • "Naked" Machine with a KVM
  • Can be hosted anywhere (OVH, AWS, ...) any architecture (ARM, Intel...)
  • Public IPv4 and IPv6 (Res6t)
  • SlapOS is architecture-independent, so any software that can be run on SlapOS could be run on any architecture.
  • In case no IPv6 is available, we use re6st to create IPv6 adresses.
  • Approach for Wendelin is the same, currently however we only test for Debian 8, 64bit machines.

Setup: Tutorial Preparation

Two Machines, 20 partitions and virtual machines
  • We use Slapos to deploy a cluster of VMs for the tutorial
  • Using Two machines, 20 partitions
  • Each partition with KVM Virtual Machine
  • SlapOS = Hyperconvergent PaaS environment
  • Wendelin uses SlapOS to be deployable on large scale
  • Software recipe to create a cluster of 20 VMs on 2 machines: Coming soon
  • Provider: Soyustart. Alternative Vifib.com, (40€/4€ per server/vm per month)

Setup: SlapOS "OUTSIDE" - Manage Access

Partition with KVM, IPv4 and IPv6 overview
  • SlapOS manages partitions and installs virtual machine
  • Port forwarding from Public IPv6 to KVM private IPv4
  • SlapOS simplifies all access by always going through IPv6
  • Machine has a public IPv4 (default) and IPv6 (default/res6st)
  • SlapOS creates partitions with private IPv4, public IPv6
  • Res6t makes all partitions accessible via public IPv6
  • SlapOS recipe installs KVm on partition
  • KVM has only 1 NAT interface to private IPv4 = no internet access by default
  • SlapOS uses port-forwarding: IPv6:10022 to IPv4 same for :80,:443
  • SlapOS "nat-rules" parameter allows additional ports to be forwarded
  • We do this to simplify all access via IPv6, because if re6st is installed IPv6 will always be available
  • Now we have public access to VM, no matter what is on the VM
  • For tutorial we add rewrite rules to have IPv4 on top of this

Setup: SlapOS "INSIDE" - Manage Wendelin

KVM with slapproxy, shacache, slappart instances
  • SlapOS is installed on KVM, too (used to manage cluster and VM)
  • Shortcut: Wendelin download pre-compiled via ShaCache
  • Shortcut: Socat to automatically bind, port forward KVM
  • The VM can be anywhere, in our case it's on SlapOS managed cluster
  • SlapOS is also used to manage the KVM itself
  • SlapOS installed via Ansible script
  • First install Slapproxy, normally manages a cluster, here only a single machine
  • SlapOS components: Binaries and Instances (partitions, running services)
  • Partitions are "mini-containers", services can be ERP5, Jupyter, MariaDB, etc
  • Wendelin only exposes 2 services on IPv6: Apache and Jupyter
  • Normal setup via webrunner and/or res6t will have public IPv6
  • Normal setup set NAT-rules/tunnels between KVM and localhost by hand
  • (SlapOS: use <parameter id="nat-rules">20 80 443 20000 20001 20002 20003</parameter>)
  • More info on VifiB/SlapOS forum

Todo: Access Virtual Machine

Wendelin-ERP5 Shell SSH
  • Browser only tutorial (no IPv6)
  • Go to https://softinst67162.host.vifib.net/shell
  • Don't click "Green Button" please.
  • The webrunner is the interface used to work with SlapOS
  • It includes file browser and editor, terminal, git and and logs section
  • It allows to add/remove/configure different software products on your virtual machine

Todo: Access Terminal

Wendelin-ERP5 Shell SSH
  • Click the terminal tab
  • Enter SSH/VNC provided (like SSH slapos@2001:67c:1254:e:1b::XXXX -p 10022)
  • The webrunner is installed on your virtual machine. We will use it to install Wendelin
  • Get your own resilient KVM through Vifib: Specifications, ViFib Homepage
  • Getting started with Wendelin: Tutorial

Todo: Authenticate and install Wendelin

Wendelin-ERP5 Authenticate and install Wendelin
  • Authenticate
  • Switch to root sudo -su, authenticate again, install via:
  • wget http://deploy.nexedi.cn/wendelin-standalone && bash wendelin-standalone

Todo: Monitor Installaton process

Wendelin-ERP5 Monitor SlapOS Software Log
  • See SlapOS software installation log:
  • tail -f /opt/slapos/log/slapos-node-software.log

Todo: Check Software Installaton Status

Wendelin-ERP5 Check Software Installation Status
  • Check status of Wendelin/ERP5 installation
  • watch -n 30 erp5-show -s
  • Once done it will return you an internal IPv4. Note this down

Todo: Socat Bind

Wendelin-ERP5 Socat Bind Port Forward
  • wget https://lab.nexedi.com/nexedi/wendelin/raw/master/utils/wendelin-standalone-bind.sh && bash wendelin-standalone-bind.sh
  • This will activate urls provided to access Jupyter/ERP5
  • As mentioned before, socat is used to bind to the correct ports to be able to access Wendelin
  • This would normally have to be done by hand inside your SlapOS instance parameters ("nat-rules")

Todo: Test Bindings

Wendelin-ERP5 Socat Bind Port Forward
  • Check if socat works: ps aux | grep socat
  • Check processes: ps xa | grep runzope
  • Check who does what: grep node-id /srv/slapgrid/slappart*/etc/zope-0.conf

Todo: Configure Wendelin - Login

Wendelin-ERP5 Login
  • Go to your Zope 2 Url (vm[x]30002-url)
  • Login using zope/insecure
  • My Favorites > Check Site Consistency

Todo: Fix Consistency

Wendelin-ERP5 Fix Consistencies
  • Select All and Fix Consistencies
  • Checking Site Consitency is always the first step when working with a new ERP5 instance
  • It ensures all components are up to date and installed correctly
  • Next steps are all in Configure Wendelin: Tutorial

Todo: Configure Site

Wendelin-ERP5 Configure Site
  • Once done (!), continue
  • Select Configure Site for Wendelin

Todo: Install Configuration

Wendelin-ERP5 Load Config
  • Select Configuration for Wendelin
  • Click "Install" on next screen
  • Configuring Site will install a predefined set of business templates
  • This could also be done manually, but Wendelin has a predefined configuration, so it can be used directly

Todo: Wait

Wendelin-ERP5 Installed
  • Installation may take a few minutes
  • Once down, click "Start using your ERP5 System"
  • Login again with zope/insecure

Todo: HOTFIX: Add Business Template

Wendelin ERP5 - Manage Business Templates
  • Goto My Favorites > Manage Business Templates
  • HOTFIX are addable in live system via business templates
  • Business Templates are standard way of adding functionality in ERP5/Wendelin
  • Business Templates (bt5) are complete apps or working parts of an application.
  • Full list: ERP5 applications
  • Allows to recreate the same configuration across multiple instances
  • Allows to add hotfixes and patches in case necessary without restarting system
  • Detailed instructions in how to install business templates

Todo: Download Business Template

Wendelin ERP5 - Download Business Template
  • Click the Export Symbol (red/blue arrows)
  • Select Download Business Template from URL

Todo: Enter Business Template URL

Wendelin ERP5 - Enter Business Template URL
  • Enter the HOTFIX Url provided and click to download
  • http://www.wendelin.io/wendelin05_release_hotfix-0.5.bt5/Base_download
  • Click download
  • This is the standard way of manually adding/updating business templates
  • Business templates are made availble by specifing the repositories which contain them
  • On this VM (!), these would be:

Todo: Install Business Template

Wendelin ERP5 - Select Install Business Template Action
  • Once download finishes, go to the business template page
  • From actions, select Install Business Template

Todo: Confirm HOTFIX additions

Wendelin ERP5 - Confirm Installation
  • The dialog will show a list of objects to be added with the installation
  • Confirm and install

Todo: Main Interface

Wendelin ERP5 - All set
  • Go back to the main interface
  • Your instance is now ready to use
  • The start screen shows a list of modules (data-types) directly accessible
  • Modules can be contain anything from Persons, Organizations to Data Streams
  • Modules prefixed with Portal are not displayed (e.g. portal ingestion policies)

Todo: Create Ingestion Policy

Wendelin-ERP5 Ingestion Policies
  • We now setup Wendelin to receive data from fluentd
  • Goto: http://softinstxxxx.host.vifib.net/erp5/portal_ingestion_policies/
  • Security Setting to prevent arbitrary stream from being sent
  • Currently fluentd and Wendelin are setup to receive streams of data
  • A stream is an existing file to which data is continually appended
  • The congestion policy "initializes" a Data Stream (creates a new file)

Todo: Fast Input

Wendelin-ERP5 Create Ingestion Policy
  • Hit the "Green Runner" for fast input
  • Enter "pydata" as our reference name and save
  • This creates a new ingestion policy and corresponding data stream
  • The ingestion policy includes default scripts to be called on incoming data
  • To modify data handling, you could now define your own script

Todo: Check Data Streams

Wendelin-ERP5 Data Stream
  • Switch to Data Stream Module
  • You will now have an empty stream, setup to ingest data from fluentd
  • Click on the stream. You can also upload data by hand if you like to experiment

Audio: Simulate IOT Sensor

Wendelin in Production

Wendelin | Windpark
  • First Wendelin prototype in production to monitor wind turbines.
  • Wendelin used to collect data and manage wind parks.
  • Machine Learning for failure prediction, structural health monitoring.

Wendelin in Production

Wendelin | Wind Turbine
  • Turbine equipped with multiple sensors per blade, PC collecting sensor data
  • PC includes fluentd for ingesting into Wendelin (network availability!)
  • Tutorial: we use same setup, sensor - Fluentd - Wendelin

Basic FluentD

Wendelin | Basic FluentD Flowchart
  • FluentD is open source unified data collector
  • Has a source and destination, generic, easy to extend
  • Can handle data collection even under poor network conditions

Complex FluentD

Wendelin | Complex FluentD Flowchart
  • Setup of FluentD can be much more complex
  • In Wendelin production, every turbine has its own FluentD
  • Tutorial simulates the same, all VM have their own FluentD

Todo: Record Audio

Wendelin-ERP5 Audio Recorder

Todo: Access Monitor using fluentD

Wendelin-ERP5 Monitor Webrunner
  • Access Monitor at: https://softinst67162.host.vifib.net/
  • Go to the Editor, click "This Project"

Todo: Upload to Monitor

Wendelin-ERP5 Monitor Webrunner File Upload
  • We already made a folder for uploads for this tutorial
  • Left click and select upload file

Todo: Forward File to fluentD

Wendelin-ERP5 Webrunner FluentD Configuration
  • In the Editor, open folders project > slapos
  • Create a new file, give it your name
  • Code on next slide (please copy&paste)
  • We are now creating a configuration file to pass to fluentd
  • The file contains all parameters for fluentD regarding data source and destination
  • Normally this is set upfront, but for the tutorial we hardcode

Todo: FluentD Configuration File (Gist)

 @type bin
 format none
 pos_file /srv/slapgrid/slappart9/srv/runner/!!YOURNAMEGOESHERE!!.pos
 enable_watch_timer false
 read_from_head true
 tag pydata

<match pydata>
 @type wendelin
 @id wendelin_out 

 user zope
 password insecure 

 buffer_type memory
 flush_interval 1s
 disable_retry_limit true
  • Note that tag will be the name of your data-ingestion-policy/stream

Todo: Save and send from Terminal

Wendelin-ERP5 Webrunner FluentD File Transfer
  • Switch to the terminal and enter:
  • software/bd283b01bd665a31e41dce567f199491/bin/fluentd -c project/YOUR_NAME.cfg
  • To pass your configuration including what file to send to Wendelin

Todo: Save and send from Terminal

Wendelin-ERP5 - Verify File was received
  • Head back to Wendelin/ERP5
  • In the Data Stream Module, check the file size of the pydata stream
  • It should now show a file size larger than 0

Analyse: Work with Ingested Data


Wendelin Out of Core Computing
  • Wendelin.Core enables computation beyond limits of existing RAM
  • We have integrated Wendelin and Wendelin.Core With Jupyter
  • ERP5 Kernel (out-of-core compliant) vs. Python 2 Kernel (default)

Todo: Head to Jupyter (Notebook)

Wendelin-ERP5 - Juypter Interface
  • Head to Juypter http://[x].pydata-class.erp5.cn
  • Start a new ERP5 Notebook
  • This will make sure you use the ERP5 Kernel
  • The Python 2 Kernel is the default Jupyter Kernel
  • Using Python 2 will disregard Wendelin and Wendelin.Core, so it's basic Jupyter
  • Using ERP5 Kernel will use Wendelin.core in the background
  • To make good use of it, all code written should be Out-of-core "compatible"
  • For example you should not just load a large file into memory (see below)

Todo: Learn ERP5 Kernel (Notebook)

Wendelin-ERP5 - Juypter Interface Introduction
  • Note you have to connect to Wendelin/ERP5
  • The reference you set will store your notebook in the Date Notebook Module
  • Passing login/password will authenticate Juypter with Wendelin/ERP5
  • Note that your ERP5_URL in this case should be your internal url
  • You can retrieve it be running erp5-show -s in your webrunner terminal
  • Note, outside of the tutorial we would set the external IPv6 adress of ZOPE

Todo: Getting Started (Notebook)

Wendelin-ERP5 - Juypter Access Wendelin
  • Connect, set arbitrary reference and authenticate

Todo: Accessing Objects (Notebook)

Wendelin-ERP5 - Juypter Accessing Objects
  • Import necessary libs
  • Type context, this will give you the Wendelin/ERP5 Object
  • Type context.data_stream_module["1"]to get your uploaded sound file
  • Accessing data works the same ways throughout [IPv6]:30002/erp5/[module_name]/[id]
  • All modules you see on the Wendelin/ERP5 start page can be accessed like this
  • Once you have an object you can manipulate it
  • Note that accessing a file by internal id (1) is only one way
  • The standard way would be using the reference of the respective object, which will also allow to user portal_catalog to query

Todo: Accessing Data Itself (Notebook)

Wendelin-ERP5 - Juypter Accessing Data Itself
  • Try to get the length of the file using getData and via iterate
  • Note then when using ERP5 kernel all manipulations should be "Big Data Aware"
  • Just loading a file via getData() works for small files, but will break with volume
  • It's important to understand that manipulations outside of Wendelin.Core need to be Big Data "compatible"
  • Internally Wendelin.Core will run all manipulations "context-aware"
  • An alternative way to work would be to create your scripts inside Wendelin/ERP5 and call them from Juypter
  • Scripts/Manipulations are stored in Data Operations Module

Todo: Compute Fourier (Notebook)

Wendelin-ERP5 - Juypter Compute Fourier Series
  • Proceed to fetch data using getData for now
  • Extract one channel, save it back to Wendelin and compute FFT
  • Note, that ERP5 kernel at this time doesn't support %matplotlib inline
  • Note the way to call methods from Wendelin/ERP5 (Base_renderAsHtml)
  • Wendelin/ERP5 has a system of method acquistion. Every module can come with its own module specific methods and method names are always context specific ([object_name]_[method_name]). Base methods on the other hand are core methods of Wendelin/ERP5 and applicable to more than one object.

Todo: Display Fourier (Notebook)

Wendelin-ERP5 - Juypter Display Fourier Series
  • Check the rendered Fourier graphs of your recorded sound file

Todo: Save Image (Notebook)

Wendelin-ERP5 - Juypter Save Image
  • Save the image back to Wendelin/ERP5.

Todo: Create BigFile Reader (Notebook)

Wendelin-ERP5 - Create Big File Class
  • Add a new class BigFileReader
  • Allows to pass out-of-core objects

Todo: Rerun using Big File Reader (Notebook)

Wendelin-ERP5 - Juypter Rerun using Big File Reader
  • Rerun using the Big File Reader
  • Now one more step is out of core compliant
  • Verify graphs render the same
  • We are now showing how to step by step convert our code to being Out-of-Core compatible
  • This will only be possible for code we write ourselves
  • Whenever we have to rely on 3rd party libraries, there is no guarantee that data will be handled in the correct way. The only option to be truly Out-of-Core is to either make sure the 3rd party methods used are compatible and fixing them accordingly/committing back or to reimplement a 3rd party library completely.

Todo: Redraw from Wendelin (Notebook)

Wendelin-ERP5 - Juypter Recover Plot
  • Redraw the plot directly from data stored in Wendelin/ERP5

Todo: Verify Images are Stored

Wendelin-ERP5 - Image Module
  • Head back to Wendelin/ERP5
  • Go to Image module and verify your stored images are there.

Todo: Verify Data Arrays are Stored

Wendelin-ERP5 - Data Array Module
  • Switch to the Data Array module
  • Verify all computed files are there.

Visualize: Display computed data

Running Web Sites from Wendelin

Wendelin-ERP5 - Start Page
  • Last step is to display results in a web app
  • Head back to main section in Wendelin/ERP5
  • Go to Website Module

WebSite Module

Wendelin-ERP5 - Website Module
  • Website Module contains websites
  • Open renderjs_runner - ERP5 gadget interface
  • Front end components are written with two frameworks, jIO and renderJS
  • jIO (Gitlab) is used to access documents across different storages
  • Storages include: Wendelin, ERP5, Dropbox, webDav, AWS, ...
  • jIO includes querying, offline support, synchronization
  • renderJS (Gitlab) allows to build apps from reusable components
  • Both jIO/renderJS are asynchronous using promises

Renderjs Runner

Wendelin-ERP5 - renderjs Runner
  • Parameters for website module
  • see ERP5 Application Launcher - base gadget
  • Open new tab: http://softinstxxxx/erp5/web_site_module/renderjs_runner/
  • Apps from gadgets are built as a tree structure, the application launcher is the top gadget
  • All other gadgets are child gadgets of this one
  • RenderJS allows to publish/aquire methods from other gadget to keep functionality encapsulated

Renderjs Web App

Wendelin-ERP5 - renderjs Application
  • ERP5 interface as responsive application
  • We will now create an application like this to display our data

Todo: Clone Website

Wendelin-ERP5 - Clone Website
  • Go back to renderjs_runner website
  • Clone the website

Todo: Rename Website

Wendelin-ERP5 - Rename Website
  • Change id to pydata_runner
  • Change name to PyData Runner
  • Save

Todo: Publish Website

Wendelin-ERP5 - Publish Website
  • Select action Publish and publish the site
  • This changes object state from embedded to published
  • Try to access: http://softinstxxxx/erp5/web_site_module/pydata_runner/
  • Wendelin/ERP5 usese workflows to change the state of objects
  • A workflow in this case is to publish a webpage, which means changing its status from Embedded to Published
  • Workflows (among other properties) can be security restricted. This concept applies to all documents in ERP5

Todo: Layout Properties

Wendelin-ERP5 - Modify Layout Properties
  • Change to Tab "Layout Properties tab"
  • Update the router to custom_gadget_erp5_router.html
  • Refresh your app (disable cache), it will be broken, as this file doesn't exist
  • The renderjs UI is also under development, the latest (unreleased) version supports the front pge gadget property
  • We currently do a workaround, which also shows how to work with web pages in ERP5
  • One advantage working with an aync promise-chain based framework like renderJS is the ability to capture errors
  • It is possible to capture errors on client side, send report to ERP5 (stack-trace, browser) and not fail the app
  • Much more fine-grainded control, we currently just dump to screen/console

Todo: Web Page Module

Wendelin-ERP5 - Web Page Module
  • Change to web page module
  • Search for reference "router"
  • The web page module includes html, js and css files used to build the frontend UI
  • The usual way of working with static files is to clone a file, rename its reference and publish it alive (still editable)

Todo: Clone Web Pages

Wendelin-ERP5 - Web Page Module Clone & Publish
  • Open both the html and javascript file in a new tab
  • Clone both, prefix the references with custom_ and publish alive
  • Click edit tab on the html page

Todo: Prefix JS file to display

Wendelin-ERP5 - Web Page Module Edit HTML
  • Prefix the Javascript file to load to the correct reference
  • Save, switch to the Javascript file in the other tab
  • Click edit tab here as well

Todo: Update Default Gadget

Wendelin-ERP5 - Update Default Gadget
  • Look for "worklist" and change it to "pydata"
  • Save, we now have a new default gadget to load
  • Go back to web page module

Todo: Clone Worklist gadgets

Wendelin-ERP5 - Clone Worklist Gadget
  • Search for %worklist%, open both files in new tabs, clone, change title
  • Replace "worklist" in references with "pydata", save and publish
  • We will now edit both files to display our graph

Todo: Graph Gadget HTML (Gist)

  <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
  <meta name="viewport" content="width=device-width, user-scalable=no" />
  <title>PyData Graph</title>
  <script src="rsvp.js" type="text/javascript"></script>
  <script src="renderjs.js" type="text/javascript"></script>
  <script src="gadget_global.js" type="text/javascript"></script>
  <script src="gadget_erp5_page_pydata.js" type="text/javascript"></script>

  • This is a default gadget setup with some HTML.
  • Gadgets should be self containable so they always include all dependencies
  • RenderJS is using a custom version of RSVP for promises (we can cancel promises)
  • The global gadget includes promisified event binding (single, infinite event listener)

Todo: Graph Gadget JS (Gist)

/*global window, rJS, RSVP, URI */
/*jslint nomen: true, indent: 2, maxerr: 3 */
(function (window, rJS, RSVP, URI) {
  "use strict";
    // Init local properties
    .ready(function (g) {
      g.props = {};
    // Assign the element to a variable
    .ready(function (g) {
      return g.getElement()
        .push(function (element) {
          g.props.element = element;
    // Acquired methods
    .declareAcquiredMethod("updateHeader", "updateHeader")

    // declared methods
    .declareMethod("render", function () {
      var gadget = this;
      return gadget.updateHeader({
        page_title: 'PyData'
}(window, rJS, RSVP, URI));

Todo: Save, refresh web app

Wendelin-ERP5 - WebApp Custom Interface
  • Once you saved your files, go back to the web app and refresh
  • You should now have a blank page with header set correctly
  • We will now add fetch our graph and display it

Todo: Update Graph Gadget HTML (Gist)

<!DOCTYPE html>
  <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
  <meta name="viewport" content="width=device-width, user-scalable=no" />
  <title>PyData Graph</title>
  <!-- renderjs -->
  <script src="rsvp.js" type="text/javascript"></script>
  <script src="renderjs.js" type="text/javascript"></script>
  <!-- custom script -->
  <script src="dygraph.js" type="text/javascript"></script>
  <script src="gadget_global.js" type="text/javascript"></script>
  <script src="gadget_erp5_page_pydata.js" type="text/javascript"></script>
  <div class="custom-grid-wrap">
    <div class="custom-grid ui-corner-all ui-body-inherit ui-shadow ui-corner-all"></div>
   <div data-gadget-url="gadget_ndarray.html"
  • Took from existing project, HTML was created to fit a responsive grid of graphs
  • Added JS library for multidimensional arrays: NDArray
  • Added JS libarary for displaying graphs: Dygraph

Todo: Graph Gadget JS (1) (Gist)

/*global window, rJS, console, RSVP, Dygraph */
/*jslint indent: 2, maxerr: 3 */
(function (rJS) {
  "use strict";

      start_date: 0,
      time_factor: 1000,
      resolution: 1,
      xlabel: 'x',
      ylabel: 'y',
      key_list: ["Channel 1", "Date"],
      label_list: ["Date", "Channel 1"],
      series_dict: {
        "Channel 1": {
          axis : "y",
          color: "#00884B",
          pointSize: 1,
          visible : true,
          connectSeparatedPoints: true
      axis_dict: {
        y: {position : "left", axisLabelColor: "grey", axisLabelWidth : 40, pixelsPerLabel : 30},
        x: {drawAxis : true, axisLabelWidth : 60, axisLabelColor: "grey", pixelsPerLabel : 30}
      connectSeparatedPoints: true
  • First we only defined options for the Dygraph plugin
  • In production system these are either set as defaults or stored along with respective data

Todo: Graph Gadget JS (2) (Gist)

  function generateInitialGraphData(label_list) {
    var i,
      data = [[]];
    for (i = 0; i < label_list.length; i += 1) {
    return data;

  function convertDateColToDate(gadget, array) {
    var label_list = gadget.property_dict.option_dict.label_list,
      time_factor = gadget.property_dict.option_dict.time_factor,
      time_offset = gadget.property_dict.option_dict.time_offset || 0,

    for (k = 0; k < label_list.length; k += 1) {
      if (label_list[k] === "Date") {
        for (i = 0; i < array.length; i += 1) {
          array[i] = [i, array[i]];
    return array;

  • Add methods outside of the promise chain
  • Simplified (removed actual creation of date objects)

Todo: Graph Gadget JS (3) (Gist)


    .ready(function (gadget) {
      gadget.property_dict = {};
      return gadget.getElement()
        .push(function (element) {
          gadget.property_dict.element = element;
          gadget.property_dict.option_dict = OPTION_DICT;

    .declareAcquiredMethod("jio_getAttachment", "jio_getAttachment")

    // render gadget
    .declareMethod('render', function () {
      var gadget = this,
        interaction_model = Dygraph.Interaction.defaultModel,
        option_dict = {},

      url = "http://softinstxxxx/erp5/web_site_module/pydata_runner/hateoas/data_array_module/[your_id]";

  • "ready" triggered once gadget is loaded
  • define gadget specific parameters
  • "render" called by parent gadget or automatically
  • we hardcode url parameter, by default it would be URL based

Todo: Graph Gadget JS (4) (Gist)

      return new RSVP.Queue()
        .push(function () {
          return gadget.jio_getAttachment("erp5", url, {
            start : 0,
            format : "array_buffer"
        .push(function (buffer) {
          var array_length,
          array_width = 1;

        array_length = Math.floor(
          buffer.byteLength / array_width / ARRAY_VALUE_LENGTH
        length = buffer.byteLength - (buffer.byteLength % ARRAY_VALUE_LENGTH);
        if (length === buffer.byteLength) {
          array = new Float64Array(buffer);
        } else {
          array = new Float64Array(buffer, 0, length);
        return nj.ndarray(array, [array_length, array_width]);
  • Orchestrated process starting with a cancellable promise queue
  • First step requesting the full file (NOT OUT-OF-CORE compliant - we load the whole file)
  • Return file converted into ndarray

Todo: Graph Gadget JS (5) (Gist)

    .push(function (result) {
        var i,
          data = [],
          ndarray = result,
          label_list =  gadget.property_dict.option_dict.label_list,
          key_list = gadget.property_dict.option_dict.key_list;

        for (i = 1; i < label_list.length; i += 1) {
          data = data.concat(
        data = convertDateColToDate(gadget, data);
        gadget.property_dict.data = data;
        return gadget
    .declareService(function () {
      var gadget = this;
      return gadget.property_dict.graph = new Dygraph(
  • Convert data into graph compatible format, store onto gadget
  • "declareService" triggered once UI is built
  • Graph will be rendered there.

Todo: Refresh Web Application

Wendelin-ERP5 - WebApp FFT Curve
  • Not out-of-core compliant, but jIO already allows to fetch ranges
  • Example computes client-side as project requires to work offline "in the field"
  • Ongoing process to fix libraries to work asynchronously and Big Data aware

Summary: Hyperconvergence?

Wendelin-ERP5 - Hyperconvergence"
  • Generic Hardware, specialized software services
  • No more files, only services for compute, storage, networking, virtualization
  • Manage through common toolset, scale by adding nodes

Summary: What did we do?

Wendelin-ERP5 - Hyperconvergence
  • SlapOS manages everything as a Service
  • Including Deployment of VM Cluster, hosting of Audio Recording website
  • Everything is a process


Image Nexedi Headquarters, Lille, France
  • Nexedi SA
  • 270 bd Clémenceau
  • 59700 Marcq-en-Baroeul
  • France
  • We are hiring!