Wendelin Exanalytics Libre

WENDELIN combines Scikit Learn machine learning and NEO distributed storage for out-of-core data analytics in python

Table of Contents

  • Notes on Extending the File Manager of Jupyter and Jupyter Lab

    In this document my findings about extending Jupyter classic and Jupyter Lab are summarized.

    ${table_of_content}

    Notes on Extending the File Manager of Jupyter and Jupyter Lab

    This document summarizes my research of Jupyter and Jupyter Lab integration into ERP5. In particular building a plugin/extension to allow notebooks to be stored in ERP5.

    The Task

    The desired outcome is to store Jupyter notebooks in ERP5 instead of the server's file system. In particular this means that the Jupyter (Lab) server needs to be able to

    • list: get a list of all notebooks (stored in ERP5) from ERP5. It then displays this list in the filemanager component of Jupyter (Lab)
    • get: get meta data and cells content of a notebook from the ERP5 storage
    • save: save modifications made within Jupyter (Lab) to the stored version in ERP5
    • create/delete/rename: typical filemanager operations on notebooks stored in ERP5

    Additional (non-core) functionallity may include: upload notebooks, manage checkpoints, ...

    Essentially, a simple file manager has to implemented such that it interfaces with ERP5 via hateoas. This modification/extension should be applicable for both Jupyter classic and Jupyter Lab as both will be relevant for Nexedi.

    Extending Jupyter and Jupyter Lab

    To achieve this task Jupyter (Lab) needs to be extended to either

    1. replace the default file manager with a new version that interfaces with ERP5 instead of the local filesystem
    2. extend the default file manager to show notebooks stored in ERP5 in addition to the local filesystem

    The second approach has the advantage, that local storage is also possible. This might be especially relevant for Jupyter Lab, as it is a multi-function tool and not exclusively used for Jupyter Notebooks.

    Extending Classic Jupyter

    For the classic Jupyter there exist two possibilities.

    1. A frontend nbextension written in Javascript (Klaus' approach) (undocumented). Reference implementation: jupyter-drive
    2. An ipython extension implementing the contentsmananger interface (reference). Reference Implementation: PGContents

    Note, that these two are extending the built-in file manager of the Jupyter classic and not the notebook functionality itself.

    Extending Jupyter Lab

    Jupyter Lab is composed of plugins which are written in TypeScript and based on phosphorjs (reference). In case of the notebook plugin an inteface layer is written in TypeScript to communicate with the jupyter server (as far as I understand) (reference).

    The default file manager (or file browser) of Jupyter Lab is such a plugin (see here).

    There exist two relevant (or three) (documentation is somewhat confusing, see here vs here) ways to extend Jupyter Lab.

    1. Jupyter Lab extensions: Extensions to plugins, written in TypeScript (reference)
    2. Ipython widget extensions: These seem specific for the notebook plugin and are most likely similar to the ipython extension of the classic Jupyter Notebook (reference). They do not seem to be implemented yet (see note at the bottom of reference).

    The Problem

    Based on this information it seems impossible to make a single extension for both Jupyter and Jupyter Lab. The file managers of Jupyter and Jupyter Lab are completely different. One is built into classic Jupyter and one is a phosphor/typescript widget (Jupyter Lab), which can not be extended with the same tools.

    Another point of evidence for this is, that there are two completely seperate implementations of their google drive file manager for classic Jupyter and Jupyter Lab:

    In other words: Jupyter Lab has decoupled file management from Jupyter Notebook and extensions to the file manager are different from extensions to the Notebook plugin which no longer needs to provide file management itself.

    Thus writing a single extension for both classical Jupyter and Jupyter Lab to enable storage of notebooks in ERP5 seems impossible to me.

    Moving forward

    I still need to find out if ipython extensions to the Jupyter plugin still work in Jupyter Lab. My guess would be: yes. Still, this does not solve the problem. In my opinion two different extensions need to be written to support both Jupyter (classic) and Jupyter Lab.

    ${related_subject_list}