Wendelin Exanalytics Libre

WENDELIN combines Scikit Learn machine learning and NEO distributed storage for out-of-core data analytics in python

Table of Contents

  • Setup a KVM for Uploads Packages

    Setup a KVM for Uploads Packages

    This post is aiming to explain how to create a KVM and configure it for uploads packages to the Shacache system. Shacache is a cache system that caches packages like Python packages in PyPI(Shacache Network Cache Concept), variety source codes(from GitHub,etc). When running a software release on a machine, slapos will try to fetch packages or source code from shacache instead of the original site. If slapos didn't find the corresponding package from the Shacache, it will try to get it from the original site. On a non-configured machine, slapos will just fetch the following packages. On a configured machine, slapos will upload the missing packages with predefined keys. This post will explain how to add keys to configure a machine. But how to get such keys is another topic. The user should contact Julien.

    This post assumes the reader has some basic background, so it omitted some details regarding how to install KVM, Debian, etc. 

    1. Create a KVM

    Create a service on the panel.rapid.sapce. Select KVM with version 1.0.232. (https://handbook.rapid.space/user/rapidspace-HowTo.Request.A.KVM) The disk space needs 100GB. Set the RAM size to 16384. Click "Show Parameter XML", then copy the following XML to the editor:

    <?xml version="1.0" encoding="utf-8"?>
    <instance>
        <parameter id="ram-size">16384</parameter>
        <parameter id="disk-size">100</parameter>
    </instance>

    2. Install Debian

    Normal procedure. Just make sure you installed the SSH component.

    2.1 Enable IPv6

    On the instance page which you requested the KVM, you can see there has a description about how to enable IPv6 on your Debian system. See below:

    PERMANENT SOLUTION: in your VM, add the lines below in /etc/network/interfaces and then run: "ifup ens4"
    auto ens4
    iface ens4 inet6 static
        address 2001:67c:1254:xxxx:xxxx:xxxx:0:1
        netmask 96
        gateway 2001:67c:1254:xxxx:xxxx:xxxx:ffff:ffff

    After this operation, you can use ssh to login into the machine:

    ssh YOUR_USER_NAME@2001:67c:1254:xxxx:xxxx:xxxx:0:1

    3. Install slapos on the Debian

    Run the following command to install Slapos:

    wget https://deploy.erp5.net/vifib; bash vifib

    The script will do some processing, after that. It will ask for some information, provide it as follow:

    SlapOS Master API
    https://slap.vifib.com

    SlapOS Master Web UI
    https://panel.rapid.space

    Your Token
    20220126-XXXXXX

    For the first two options, just use the default one. For the token, you need to go to the panel.rapid.space, in the "Server" tab, request a token.

    And then set up a name for this computer, eg: Shacache-testnode

    Note: The most important part is, don't forget to set ens4 network as the IPv6 provider. The default one is "lo". We need to change it to "ens4".

    4 Get the certificate files

    Get YOUR_NAME.crt  YOUR_NAME.key  ca.crt, these three files were given by the shacache manager. You should contact him.

    5 Place the cerfiticate files

    Put the given key and certificate in /etc/opt/slapos/shacache/YOUR_NAME.key and /etc/opt/slapos/shacache/YOUR_NAME.cert 

    6 Modify the slapos configuration file

    In your slapos configuration file (/etc/opt/slapos/slapos.cfg), add/modify the following section:

    [networkcache]
    # Define options for binary cache, used to download already compiled software.
    download-binary-cache-url = http://www.shacache.org/shacache
    download-binary-url = http://www.shacache.org/shadir

    # More options for 'networkcache-download' command
    download-cache-url = http://www.shacache.org/shacache
    download-dir-url = http://www.shacache.org/shadir

    # Upload Configuration
    signature-private-key-file = /etc/opt/slapos/shacache/signature.key
    signature-certificate-file = /etc/opt/slapos/shacache/signature.crt
    upload-binary-dir-url = https://www.shacache.org/shadir
    upload-binary-cache-url = https://www.shacache.org/shacache
    upload-cache-url = https://www.shacache.org/shacache
    upload-dir-url = https://www.shacache.org/shadir

    # Options for HTTPS URLs
    shacache-ca-file = /etc/opt/slapos/shacache/ca.crt
    shacache-cert-file = /etc/opt/slapos/shacache/
    YOUR_NAME.crt
    shacache-key-file = /etc/opt/slapos/shacache/
    YOUR_NAME.key
    shadir-ca-file = /etc/opt/slapos/shacache/ca.crt
    shadir-cert-file = /etc/opt/slapos/shacache/
    YOUR_NAME.crt
    shadir-key-file = /etc/opt/slapos/shacache/
    YOUR_NAME.key

    That's is add the path of YOUR_NAME.crt and YOUR_NAME.key to the shacache-cert-file and shacache-key-file.

    7 Generate SSL key

    Create a directory: mkdir /etc/opt/slapos/shacache, run this command: "/opt/slapos/bin/generate-signature-key /etc/opt/slapos/slapos.cfg".

    If you encountered an error like "ConfigParser.NoOptionError: No option 'signature-certificate-file' in section: 'networkcache'". Which means in step 6, your slapos configuration file (/etc/opt/slapos/slapos.cfg) is still using the field name 'signature_certificate_file'. All you need to do is modify the configuration file, use the dash to replace the underscore, then rerun the command.

    This command will generate a file signature.crt, this is YOUR certificate. Put the content of signature.crt to the /etc/opt/slapos/slapos.cfg, in the signature-certificate-list field. This means you are trusted, and the file that you want to upload to the Shacache will be considered trusted.

    Change the permission of signature.key

    chmod 644 /etc/opt/slapos/shacache/signature.key

    You can refer to the step in "How to upload to cache" section of this article.

    Then build and install SR as a normal machine. One thing worth to mentioning: when providing SR, it has to come from the branch that has this signature.cert content.

    5 Further Configuration for TestNode.

    If we had set up a testnode on this KVM, and we want to upload the missing packages during the testnode running SR. In addition to step 4, we need to do something more.

    1. We need to commit the content of signature.cert, which is generated in step 4, to stack/slapos.cfg. And when setting up a testnode, we need to use this testnode to use the slapos branch which contains the modified stack/slapos.cfg.

    2. If you are set up with more than one machine. No need to generate the certificate for each machine, always use ONE certification generated based on your key on the first machine. This means on different test nodes you had set up, always use the same slapos branch.

    3. If you want to download the packages that you uploaded by yourself, you have to add your certificate to two files:

    - signature-certificate-list in /etc/opt/slapos/slapos.cfg -> download binary from cache
    - signature-certificate-list in slapos/stack/slapos.cfg -> downloda source from cache