Welcome to SiLA2_Manager’s documentation!

Introduction

Laboratory management and information systems (LMS, LIMS) have developed in recent years to support and automate more functions that are encountered in the laboratory on a regular basis. Commercial software facilitates experiment planning, analysis and other lab related activities such as audit trail, storage and user management. However, they always exclude device management and device control. A user is always required to to run the experiments step by step, transfer data, and translate input. Device integration is time-consuming and requires the development of expensive custom solutions.

The main reason for the lack of this functionality is more obvious than one might think. It all boils down to a lack of standardization. Devices in a laboratory environment cannot be integrated easily into a control software making it near impossible to conduct experiments that require many devices at once. Experiments that rely on inter-device communication in real-time, experiments that need to be flexible, adjustable to new findings, require a framework for easy integration, management, data access and workflow design.

This software is building on top of the SiLA 2 standard. SiLA 2 is a mission to establish international standards in laboratory automation and to enable open connectivity and thus rapid integration capability. Goals are interoperability, flexibility, resource optimization for laboratory instrument integration and software services based on standardized communication protocols and content specifications. SiLA 2 is an open standard and is maintained by the not-for-profit SiLA consortium. The SiLA 2 standard is based on well established communication protocols (Http2, gRPC). SiLA 2 is based on an inherent server-client architecture. Connection is established over a TCP/IP network. For more information visit the SiLA website or check out the SiLA_Base repository on GitLab.

SiLA uses auto-discovery to broadcast services on the network. Accessing, managing and integrating these services into workflows in the lab environment is not part of the SiLA standard. A software framework on the middleware level is needed to connect and manage available lab devices. The proposed software provides the user with these additional features.

Installation

This software depends on third-party software. At the time of this release, all third-party software is free-to use and open-source. The respective version numbers are indicated. It cannot be guaranteed that this software will remain compatible with future releases of these packages. Hence, we recommend installing the appropriate software version as specified in this guide or as specified in the pipfile. The repository and this document will be updated to reflect major version updates of third-party software.

Guide

Download

Download the installation files using wget or download them directly from the SiLA 2 manager repository website. Extract the compressed tar.gz file using tar:

sudo wget https://gitlab.com/lukas.bromig/sila2_manager/-/archive/master/sila2_manager-master.tar.gz
sudo tar -xvf sila2_manager-master.tar.gz

or clone the git repository with:

git clone https://gitlab.com/lukas.bromig/sila2_manager.git

Setting up the python environment

This project requires python3.7 or higher. The latest python distribution can be downloaded here.

Before running any code in this project, all required python packages must be installed. It is strongly recommended to set up a virtual environment. The project is shipped with a pipfile that contains information on all required packages and their respective versions and dependencies. To set up the python environment in the most user friendly way, pipenv is recommended. Pipenv provides both, a virtual environment and the well known pip python package management. If you want to install your virtual environment and manage your packages with pip separately, feel free to use the supplied requirements.txt

1. Install pipenv

   To install the pipenv package visit the pipenv project page for more installation options or run:

pip install pipenv

2. Install the project pipfile

   Move to the project main directory, where the pipfile and the pipfile.lock are located. Install the python packages and the virtual environment by running:

pipenv install

3. Entering the virtual environment

   Most IDE’s support automatic detection of virtual environments and will start the console in this environment. For some IDE’s this requires some manual changes in the settings menu. If no IDE is used you can enter the environment by entering the following code from the project main directory:

pipenv shell

4. [Windows only] Manual re-installation of protobuf

   The standard wheel installation of protobuf doesn’t allow the use of multiple files with the same name in the same pool. All SiLA devices implement the standard features, thus every device will add a file with that feature name to the pool. The protobuf –no-binary installation is different to the pre-compiled wheel installation and allows multiple files with the same name in the pool. Run the protobuf_no_binary_install.bat in the main directory to replace the existing wheel installation. The issue is discussed in this thread in the protobuf GitHub repository issue #3002.

protobuf_no_binary_install.bat

Setting up the javascript run-time environment

The frontend is written in typescript. A javascript run-time environment is needed to compile the code. It is recommended to install node.js. To download node.js visit the node.js download website (Windows) or install using apt-get (Linux):

sudo apt-get install nodejs

Make sure you have a current version of node.js (v12.18.4 or greater). Run the following command to check your node.js version:

node -v

The node.js package manager npm can downloaded from the npm download website (Windows) or using apt-get:

sudo apt-get install npm

Make sure you have a current version of npm (v6.14.6 or greater). Run the following command to check your npm version:

npm -v

The node.js packages can be installed by executing the following code from within the frontend directory:

cd frontend
npm install

To compile the frontend files from source, move into the frontend directory and run:

cd frontend
npm start

Installing docker

Docker containers are used for the execution of experiments. Furthermore, they are used in the development version for running the postgreSQL and the redis database. In the deployment version, these are replaced with a system wide installation. You can download docker (v2.3.0.5) on the docker website here.

1. Create the user-script docker image.

   You can modify the docker container that is used for experiments by changing the dockerfile in ‘user_script_env’ to include packages that you want to use in the scripting environment.To create the container run. If you encounter an error, try a different docker base image such as python:3 or python:3.8.0-slim by changing the first line in the Dockerfile “FROM python:3.8.3-alpine” to your respective choice On Linux run:

cd user_script_env
create_container_image.sh

 On windows run:

cd user_script_env
create_container_image.bat

2. For the development version the containers for the postgrSQL and redis DB need to be downloaded:

docker run --name postgres -e POSTGRES_PASSWORD=1234 -d -p 5432:5432 postgres
docker run --name redis -d -p 6379:6379 redis

3. Once downloaded, the containers can be started:

docker start postgres
docker start redis

4. [Optional] Experiments are run in docker containers. The container can be customized. To re-create the container, run the create_container_image script* in the user_script_env folder:

cd user_script_env
[Unix] create_container_image.sh
[Windows] create_container_image.bat

You can modify the container image by editing the Dockerfile or by adding new python packages to the requirements.txt.

Setup of a development server

The development servers scan the code base and will restart if changes to the source code of the frontend or backend are detected.

1. Activate the development mode

   Run the following code from inside your pipenv environment (Linux):

export DEVICE_MANAGER_ENV_PRODUCTION=0

For Windows: .. code-block:: console

set DEVICE_MANAGER_ENV_PRODUCTION=0

2. Set up a test database

   A test database is created that includes pre-defined users, devices, scripts and experiments. Run the following code in your pipenv shell from the main directory:

pipenv run python setup_db.py

2. Create a configuration file

   The configuration file specifies the secret key for the encryption between the frontend and the backend, as well as the database connection details for the postgreSQL database. To create the file run the supplied script ‘generate_config.py’ in your pipenv environment.

pipenv run python generate_config.py

Starting the device manager in development mode

To start the device manager in the development mode, the respective modules must be started individually.

1. Start the backend development server

   On Windows:

   ./run_backend_server.bat
   

   On Linux:

   ./run_backend_server.sh
   

2. Start the frontend server

   If you have already set up your javascript run-time environment and run npm install in the frontend folder, you can start the frontend in a separate process with:

   cd frontend
   ng serve
   

3. Start the scheduler application.

   The scheduler application is responsible for the experiment execution using docker containers. In a new process run:

   python scheduler.py
   

Setup of a deployment server

This documentation will guide you through the installation process of the SiLA 2 Manager. Server deployment is explained for systems running Ubuntu 12.04.

First Install

1. Install nginx

   To run the device manager web-service, nginx is required. Nginx is an open-source webserver-software. On Linux systems it can be installed using apt (Linux):

sudo apt install nginx

in this project nginx v.1.18.0 is used.

2. Install PostgreSQL

   Download the PostgreSQL database and install it (Windows and others). PostgreSQL can also be installed using apt (Linux):

sudo apt install postgresql-12
sudo apt install postgresql-client-12

In this project postgreSQL v.13 is used.

3. Install Docker

   Use the [official instructions](https://docs.docker.com/engine/install/ubuntu/)

4. Install Redis

   Download the redis in-memory database and install it. Redis can be installed using apt as well:

sudo apt install redis-server

In this project redis v.6.0.9 is used.

5. Install supervisor

sudo apt install supervisor

6. Install and run pipenv

sudo apt install pipenv
sudo mkdir .venv
sudo pipenv sync

7. Fix protobuf installation Uninstall protobuf and reinstall it using the –no-binary flag.

pipenv shell
sudo pipenv uninstall protobuf

Check that protobuf has been uninstalled (Replace <usr> with your username!):

pip3 list
sudo pip3 install --no-binary=:all: -t /home/<usr>/sila2_device_manager/.venv/lib/python3.8/site-packages protobuf==3.15.0
[sudo pip3 install --no-binary=:all: protobuf==3.15.0]

Check that protobuf has been reinstalled.

8. Replace some files in the sila2lib of the virtual environment:

sudo pipenv run python3.8 replace_files.py

9. Install and enable nginx config

sudo cp server-config/device-manager.conf /etc/nginx/sites-available/
sudo ln -s /etc/nginx/sites-available/device-manager.conf
/etc/nginx/sites-enabled/device-manager.conf

10. Install supervisor config

sudo cp server-config/device-manager-backend.supervisor.conf /etc/supervisor/conf.d
sudo cp server-config/device-manager-scheduler.supervisor.conf /etc/supervisor/conf.d

11. Create the device-manager user and group and add yourself

sudo adduser --system --no-create-home --group --ingroup docker device-manager
sudo gpasswd -a your-user-name device-manager

12. Create www directory

sudo mkdir /var/www/html/device-manager-frontend
chmod -R device-manager /var/www/html/device-manager-frontend
chgrp -R device-manager /var/www/html/device-manager-frontend
chmod -R 775 /var/www/html/device-manager-frontend

13. Create backend config directory

sudo mkdir /etc/device-manager/

14. Start and enable PostgreSQL

sudo systemctl enable postgresql.service
sudo systemctl start postgresql.service

15. Set PostgreSQL password

sudo -u postgres psql postgres
\password postgres
<enter password>
\q

Fill the database with some initial values and example/ test information:

python setup_db.py

16. Start and enable Docker

sudo systemctl enable docker.service
sudo systemctl start docker.service

17. Enable and configure Redis edit /etc/redis/redis.conf and change supervised no to supervised systemd

sudo systemctl enable redis.service
sudo systemctl start redis.service

18. Create the user-script docker image

cd user_script_env
sudo docker build -t user_script .
cd ..

19. Deploy backend service

sudo pipenv run ./deploy_backend.sh

20. Edit Device-Manager Configuration File The configuration files are located in the main directory under:

./server-config/device-manager.conf
./server-config/device-manager-backend.supervisor.conf
./server-config/device-manager-scheduler.supervisor.conf

21. Build and install frontend

cd frontend
sudo make
sudo make install
cd ..

22. Start and enable Nginx

sudo systemctl enable nginx.service
sudo systemctl start nginx.service

23. Start and enable Supervisor

sudo systemctl enable supervisor.service
sudo systemctl start supervisor.service

Deploying new versions To deploy a new version its often enough to repeat step 19 and 21. Then restart nginx and supervisor by using:

sudo systemctl restart nginx.service
sudo systemctl restart supervisor.service

If you made changes to the PostgreSQL database entries, you need to delete old entries and setup a new one. Don’t forget to adjust the database setup script according to the changes made. Run the following code from within the root directory of the repository/ your installation.

pipenv shell
python3.8 delete_db.py
python3.8 setup_db.py

Server management You can use supervisorctl to manage the backend and scheduler processes separately. The logs of the backend and the scheduler can be viewed under: /var/log/device-manager. The logs of the docker containers are located here: /$TEMPDIR/device-manager/container To restart the backend or the scheduler service, use supervisorctl. Enter supervisorctl:

sudo supervisorctl

and run the restart command for the respective service:

restart backend:device-manager-backend-0
restart scheduler:device-manager-scheduler-0

Important paths The most important paths are listed below. If you have trouble with the services, check out the log files! You have to replace <your_username> with the username respective username you are using (Either your username or the username you created for the SiLA Manager.

Under Linux

Content	File Path
The installation directory of the SiLA 2 Manager	/home/<your_username>/sila2_device_manager
The directory of the virtual environment	/home/<your_username>/sila2_device_manager/.venv
The logs of the backend and scheduler service	/var/log/device-manager
The logs of the supervisord service software	/var/log/supervisor
The SiLA client files generated by the dynamic client	/tmp/device-manager/SiLA
The container logs of the experiments	/tmp/device_manager/container

Under Windows

Content	File Path
The directory of the virtual environment	<your_install_directory>sila2_device_manager.venv or C:Users<your_username>.virtualenvs
The logs of the backend and scheduler service	Todo: Add Windows equivalent for: /var/log/device-manager
The SiLA client files generated by the dynamic client	C:Users<your_username>AppDataLocalTempdevice-managerSiLA
The container logs of the experiments	C:Users<your_username>AppDataLocalTempdevice-managercontainer

How-To

Connecting to the SiLA 2 Manager

If your device manager server is running on your local machine or on a computer within your network, you can connect to the frontend from any internet capable device with a modern browser. Simply connect using one of the following URLs:

If the server is running locally:

localhost:4200 or 127.0.0.1:4200

If the server is running on another computer within your network:

<host-server-ip>:4200

Login page

The log-in view is a security feature, that ensures that only registered users have access to the services in your network. The Open Authorization protocol 2 (OAuth2) is used to secure communication between the backend and frontend. Once logged-in, you can add new users, delete users and reset passwords. The device manager frontend uses the Angular Authentication tool (AuthInterceptor) which relies on a HttpInterceptor interface to grant access to permitted users. If accessed from the host machine, the url of the login-page is localhost:4200/login. The default username is admin and the default password is 1234. It is strongly recommended to change the default password of the admin account.

Main page - Services

The devices page is the SiLA 2 Managers main view. Registered services are listed here and some useful detail is provided on first sight. This includes the services server name, address, port. Furthermore, the connection status is indicated. In a future version other devices types than SiLA 2, such as offline devices, custom device or OPC-UA types could be supported. The software structure allows for such adaptations. Several buttons allow the user to expand the visible detail of the service, change its current name, or remove the service from the SiLA 2 Manager.

Clicking on the service name or the information icon expands the view of the selected service, showing the implemented features and the respective descriptions. Each feature can be expanded even further to investigate which (observable) commands and (observable) properties are implemented by the feature. Exploring individual commands and properties shows the user useful information on functionality and usage. Required parameters and responses are displayed with the attributed SiLA-datatype.

Note

In this documentation and the code base the word SiLA Device and SiLA Service are often used unanimously. However, a SiLA Device is just a special case of a SiLA Service. Both are always special implementations of a SiLA Server. A SiLA Server can implement a broad variety of soft- and hardware, such as laboratory device, virtual machines, software solution or API wrappers.

Service discovery

The SiLA 2 Manager uses the SiLA 2 auto-discovery functionality which relies on multicast DNS service discovery (zeroconf) to register its services in the network. New services can be added by clicking the “plus”-button on the top right of the service table. Service discovery is started from within a new pop-up window. The discovery mode scans for SiLA 2 services in the network and displays the basic information it was registered with by the server. This information is used to connect to the server using a dynamic SiLA 2 python client. The client files are stored in the local temporary folder named after the services server-UUID: Relative path to the directory: […]temp/device-manager/SiLA/<device-UUID>/

SiLA Explorer - The service tree

Each service that is added to the SiLA 2 Manager is assigned an internal UUID. This way services with the same server name can be uniquely identified. The service tree enables the user to run commands and request properties interactively from within the browser. On the lowest level of the service tree, the command/property level, a run button can be clicked to execute the function. For functions that require user input, the parameters can be entered in the corresponding text box. The syntax by which the call can be incorporated into python scripts in the scripting environment is shown.

The data handler

InfluxDB databases can be registered and linked to a service. InfluxDB is a time-series database that is well suited for experimental data. To be able to use this feature, an InfluxDB server must be running within your network. Providing the connection details to the SiLA 2 Manager is sufficient. A username and password can be added optionally for additional security. A registered database can be linked to a service to setup automatic data transfer. Data transfer is started as soon as the booking of a service commences, i.e. the experiment the service is used in is started. The database-service link can be deleted by selecting the empty database in the dropdown menu.

The data handler will execute the configured calls in the user-specified polling intervals and store the responses in the linked database with experiment name, service name, and user name as tags. To activate the data acquisition for a selected service, the “active”-checkbox must be ticked. If responses of certain functions, or features all together, should not be stored, further checkboxes can be found on the lower levels of the service tree to deactivate data transfer. This is crucial to disable the execution of set commands for example.

Most types of data can be classified as either meta-data or measurement data. Typically, meta-data doesn’t need to be queried on a continuous basis. In most cases, requesting meta data (device ID, calibration data, etc. etc.) once at the beginning of an experiment is sufficient. Measurement data (Temperature, pressure, etc. etc.) on the contrary is usually queried on a more frequent basis. The data handler distinguishes between the two data types. Since there is no way to distinguish the type of data queried by a call automatically in a reliable fashion, the user can specify the type for each command using the meta-checkbox. Depending on the selection, a default value is implemented (1h for meta-data, 30s for measurement data). Obviously, different users have different needs regarding polling intervals, thus the defaults can be overwritten to transfer data according to a custom polling interval.

Only one configuration can be stored at a time. Future releases will include the possibility to upload and download configuration files and select configuration files for a specific booking. The data handler simplifies data-acquisition and encourages collection of all data and meta-data for improved data integrity. The separation of the data acquisition from the user script used in the experiment has several advantages:

1. The query calls are not part of the user-script, improving readability and making the script shorter.

2. Reduces the amount of code that needs to be written by the operator.

3. Data-acquisition is out-sourced to a separate process. This way data-acquisition is guaranteed to continue in case an experiment crashes.

4. The data can be easily accessed from within the user script. An example script is provided in the scripts-section of the application.

Scripting environment - Scripts

This page allows the user to upload, create and edit scripts. The main view shows a list of all saved scripts. Clicking on the script name or the <>-icon opens the script editor. The code editor is based on the Monaco Editor and includes syntax highlighting. Auto-completion is not supported. Registered scripts can be assigned to experiments in the experiment section. A script assigned to an experiment is executed in a docker container. The docker image is created based on the provided dockerfile which is stored in the folder user_script_env. If non-standard python packages are required for the script execution, they must be specified in the requirements.txt.

Warning

Scripts are not checked for programming errors. Check your code in an IDE before scheduling any experiments!

Tutorial 1: Hello SiLA 2 Manager!

The Hello device! example is one of three pre-installed example scripts. You can find this example among the others in the 'Scripts'-tab. Assign this script to a new experiment and schedule it for execution. The output should be printed to the experiment console view.

"""
TUTORIAL 1: Hello_SiLA_2_Manager
---------------------------------------------

1.1 You can use this code editor like a regular scripting environment.
    If you require specific python packages for your script, you can import them here.

Hint 1: Packages you want to import must be specified in the dockerfiles requirements.txt!
    The file is located in your SiLA 2 Manager Installation directory. The default location
    on Linux is /home/<your_username>/sila2_device_manager/user_script_env/requirements.txt.
    If you change the requirements, you need to rerun the create_container.sh to update the
    docker container image.
"""
import sys
import time
import logging
import numpy as np

"""
1.2 You can use the python logging package and configure the output format here. Logging statements
    are transferred via the stderr of the docker container and are flushed by default by the logging function.
    All output is forwarded to the SiLA 2 Manager frontend. You can display the logs in the "experiments"
    tab by clicking on an experiment.
    When an script crashes straight-away, the logs may fail to arrive at the frontend so you have to open
    the files directly.
    The log files are stored locally on your computer:
    Linux:   /tmp/device_manager/container
    Windows: C:\\Users\\<your_username>\\AppData\\Local\\Temp\\device-manager\\container
"""

logging.basicConfig(format='%(levelname)-8s| %(module)s.%(funcName)s: %(message)s', level=logging.DEBUG)
logger = logging.getLogger(name=__name__)

"""
1.3 When the experiment is started, the function run() is called. Therefore, every script must
    contain a run() function. The run function requires one argument: services. This argument
    is used to pass the SiLA Server clients information into the script. You have to supply this
    argument, even if you don't use it!

Hint 2: Use the flush argument when using print statements or add a newline character (\n) to the
    end of your string. Logging statements are flushed automatically.

"""


def run(services):
    """ Required to import and instantiate devices """

    print('Hello SiLA2 Manager')
    # The above print statement will not be shown in the experiment terminal before the statement below is executed and flushed.
    time.sleep(5)
    print('Yay me, i got flushed!', flush=True)
    print(f'A random number: {np.random.rand()}', flush=True)

    write_logging_statement()
    write_to_output()
"""
1.4 You can call other functions from within thr run() function. The function "write_logging_statement"
    writes logging statements of all available log_levels.
"""


def write_logging_statement():
    """Writes logging statements"""
    time.sleep(1)
    logger.debug('A debug statement')
    time.sleep(1)
    logger.info('An info statement')
    time.sleep(1)
    logger.warning('A warning statement')
    time.sleep(1)
    logger.critical('A critical warning statement')
    time.sleep(1)
    logger.error('An error statement\n')
    time.sleep(3)

"""
1.5 If direct calls to stdout and stderr are made, they won't get flushed either. Output has to be flushed explicitly.

Hint 2: Use the flush argument when using print statements and the sys.stderr.flush and sys.stdout.flush function for
    write operations with sys.
"""


def write_to_output():
    """Writes message to stderr"""
    sys.stderr.write('Error\n')
    sys.stderr.flush()
    time.sleep(1)
    sys.stdout.write('All Good\n')
    sys.stdout.flush()

Tutorial 2: Incorporating SiLA Services

All registered services can be accessed in the scripting environment. However, used services should be selected in the experiment setup phase. A dictionary with all service clients can be imported. Instantiating the client enables the user to execute all functions the service offers. Further information on the python syntax for the service object access can be found in the ‘service example’ in the scripting environment. It is recommended to select all used services during the experiment setup phase to avoid multi-access and interference with other experiments. Selecting a service will reserve the service for exclusive use for that script.

"""
TUTORIAL 2: Incorporating SiLA 2 Clients
---------------------------------------------

This example uses the SiLA Python HelloSiLA_Full example server.
You can download it from the repository at https://gitlab.com/SiLA2/sila_python/-/tree/master/examples/HelloSiLA2/HelloSiLA2_Full

To run this example follow these steps:

2.1. Add a SiLA Server to your Services (Ideally the HelloSiLA example from the SiLA Python or Tecan repository)
2.2. Go to the Data Handler tab and deactivate the "Active" checkmark for the device you want to use
2.3. Set up an experiment with and select this script and the device you want to use
2.4. Hit the run button or wait for the scheduled execution time (You can click on the experiment
    name to get the docker container stdout, i.e the output of your script)

Hint 3: The command/property call syntax is displayed in the "Services" tab. It is shown under
    "Usage" on the lowest level of the device tree for every command and property.
"""
import time


def run(services):
    """ Instantiates selected devices for this experiment """
    client = services[0]
    print(f'Service instantiated: {client.name}@{client.ip}:{client.port}', flush=True)
    client.connect()
    # A GET command. A call to the SiLAService feature. Request the server name.
    response = client.call_property("SiLAService\n", "ServerName")
    ServerName = response
    print(response, flush=True)

    for i in range(10):
        response = client.call_property("SiLAService\n", "ServerName")
        print(f'{i}. call:', response, flush=True)
        time.sleep(1.5)
    OldServerName = response

    # A Set command. A call to the SiLAService feature. Change the server name.
    client.call_command("SiLAService\n","SetServerName", parameters={"ServerName/constrained/String": "MyNewName"})
    response = client.call_property("SiLAService\n", "ServerName")
    print('Changed name to: ', response['servername/constrained/string'], flush=True)
    # Change the ServerName back to the original one
    client.call_command("SiLAService\n","SetServerName", parameters={
        "ServerName/constrained/String": OldServerName['servername/constrained/string']})
    response = client.call_property("SiLAService\n", "ServerName")
    print('Changed name back to:', response['servername/constrained/string'], flush=True)

Note

The syntax of the command call is shown in the service tree on the lowest level of each function call. You need to replace the “yourObject” part of the displayed call with the client object of that service!

Tutorial 3: Using the Data Handler

The data handler allows the user to link a SiLA Service to a time-series database (InfluxDB). The data handler can be configured for every SiLA Service to poll data in a user specified interval for selected commands and roperties. If data transfer is activated for a Service, the data acquisition is started automatically for as soon as the respective experiment is started.

"""
TUTORIAL 3: Using the Data Handler
---------------------------------------------
3.1 The data handler can be used without writing a script. Go to the data handler tab and setup a database. Influx
    databases are supported. The overview will show you whether a connection has been established between the SiLA 2
    Manager and the specified database. Depending on the security settings of your database, supplying a username or
    password may not be necessary.

3.2 You can link a database to a specific SiLA service. Click on the link button and select the desired database. You
    can unlink a database by selecting the empty option [] in the drop-down menu.

3.3 There are several levels of customization. Expand the Service tree and activate the SiLA Features you want the data-
    acquisition to be active for. Selecting "Data Transfer" on the top level will automatically activate all features.
    Start de-selecting.

3.4 The data handler distinguishes between two types of data: Meta data and process/experimental data. They differ in
    the time interval they are queried in. Both types have a set default polling interval. However, you may customize
    the time interval for both of them for each SiLA Command and Property on the lowest level of the tree.

3.5 If a command requires a parameter, you can set the parameter here.

3.6 The data handler is active for the full duration of the experiment and does not stop when the script is finished.
    It will only stop at the specified time or if the experiment is stopped manually.

Hint 4: If your parameter changes over time, you should exclude the command query from the data handler and add a
    respective function and command call to your experimental script.
"""


def run(services):
    """ Required to import and instantiate devices """
    return

Tutorial 3: Incorporating Databases

This example shows you how to include read and write operations to and from databases into your script. The client package of the database must be included in the docker requirements.txt, so you can access the client object in the scripting environment. Specify the database connection details when instantiating the client object.

As a first step, we ping the database to check whether we can establish a connection. The ping operation should return the version number of the used database. Afterwards, a datapoint is written to the database and queried subsequently. This is repeated a hundred times with an in-built delay time of 10 seconds. Check the experiment output console and the chronograf interface.

"""
TUTORIAL 4: Incorporating Databases
---------------------------------------------
4.1 Import the InfluxDBClient and other packages you will need
"""
from influxdb import InfluxDBClient
from datetime import datetime
import numpy as np
import time

"""
4.2 Instantiate the InfluxDBClient with the connection details of the respective database and check the connection by
    pinging the database server. Make sure to change the default connection details below to your database details!
"""


def run(services):
    influx_client = InfluxDBClient(host='127.0.0.1', port=8086, username='root', password='root', database='SiLA_2_Manager')
    print(f'Checking connectivity. DB server version: {influx_client.ping()}', flush=True)

    """
    4.3 If you do not already have a database, creat a new one:
    """

    influx_client.create_database(dbname='SiLA_2_Manager')

    """
    4.4 Create a datapoint to write to the database. Add adequate tags so you can filter your data efficiently in
        chronograf. If your script is running, you can check your data live in your browser, if your chronograf server
        is running: <ip-of-the-chronograf/influxDB-server>:8888 .
    """

    for i in range(0, 25, 1):
        # This is an example write operation
        random_number = np.random.rand()
        data_point = {
            "measurement": "testMeasurement",
            "tags": {
                "experiment_name": "influxDB_test",
                "device": "experiment_docker_container"
            },
            "time": datetime.now().strftime("%Y-%m-%dT%H:%M:%SZ"),
            "fields": {
                "test_number": random_number
            }
        }

        try:
            influx_client.write_points([data_point])
            print(f'A random number was written to the database: {random_number}', flush=True)
        except:
            print("This did not work...")

        """
        4.5 Query your data using the SQL-like Influx Query Language (InfluxQL). You can find information on the syntax at:
            https://docs.influxdata.com/influxdb/v1.8/query_language/ . The following query will read the value that was
            just written to the database.

        Hint 4: Copy and paste the query below to visualize the data in chronograf. Change the LIMIT to display the
            number of last data points. Remove the escape character (backslash) around the influxDB_test in the query.
            You can leave out the last part of the query, starting at ORDER BY, to display all available measurements
            of this type.
        """

        # This is an example query.
        results = influx_client.query(
            'SELECT test_number FROM "SiLA_2_Manager"."autogen"."testMeasurement" WHERE experiment_name = \'influxDB_test\' GROUP BY position ORDER BY DESC LIMIT 1')
        print('The latest random number was queried from the database: ',  flush=True)
        print(results, flush=True)
        time.sleep(5)

Process monitoring

Scripts are executed in a docker container. Interaction with a running docker container is limited. The stdout of the docker container is transferred to the frontend by WebSockets. For real-time visualization of process data we recommend using chronograf. Chronograf offers a complete interface for the influxDB database. All data collected by the data handler can be visualized using the chronograf IoT frontend.

Note

[WIP] Interaction with running docker containers is currently not possible. This feature is planned for a future release. Feel free to help us out!

Experiments

Automated experiments (or processes) require access to services via a well-specified interface, a script that orchestrates service operation and a database to store relevant information. The experiments view enables the user to setup such workflows and schedule their execution. On the experiments page new experiments can be created by pressing the the ‘plus’-symbol. The experiment execution time, the script to be executed, as well as the services required for execution can be defined in a pop-up form. Creation of experiments that require the booking of services unavailable during the desired time-frame is not possible. If two experiments must access the same service simultaneously, a booking should be avoided. The device is still accessible to the script.

Warning

This is a safety measure to avoid potentially detrimental consequences caused by multiple access to state-sensitive laboratory devices.

The main view of the experiments tab shows a list of scheduled experiments, accompanied by the most important information such as the start and end time of the experiment, booked services, the user script and the current status of the experiment. Scheduled experiments automatically create a booking entry in the calendar view. The experiment status may be one of the following:

Status	Meaning
unknown	The experiment has not been started yet
finished	Docker container finished with exit code 0
running	Docker container is still running. The scheduled end time has not yet elapsed
error	Docker container finished with exit code 1

It is possible to start experiments before their scheduled starting time by pressing the ‘play’-icon. A running experiment can be aborted prematurely by pressing the ‘stop’-button.

Note

In development mode the scheduler.py script must be running for experiments to be executed.

Service calendar

If a service is assigned to a specific experiment, a booking is automatically created for the entire timeframe of the experiment. A service can only be assigned to an experiment if it is available throughout the experiments start and end time. The calendar page visualizes these bookings and provides the user with additional information on the booking, such as the respective experiment, script and the user who created it. Furthermore, it is possible to create bookings manually. This allows the reservation of services. In a future release, non-SiLA services and “offline”-services will be available as well. The calendar could be used as a laboratory-wide booking system for other legacy services such as autoclaves or other resources.

It is possible to delete bookings manually. Even automatically created bookings can be deleted. However, this would circumvent the in-built security mechanism and may lead to services being accessed by multiple experimental scripts at the same time.

Software Architecture

General architecture

The SiLA 2 Manager is based solely on freely available, open-source code. Used packages were selected with long-term support and respective low maintenance considerations.

The current implementation is mainly focused on the SiLA_python repository but is interoperable with most other implementations as well (like C++). The SiLA-python repository, and all other repositories for that matter, currently don’t implement the full standard yet. The SiLA 2 implementations are still being actively developed. We work closely together with the SiLA Working group and some of our members are actively contributing to the programming language specific repositories. As the SiLA implementation evolve, the SiLA 2 Manager will be updated to incorporate the new changes.

The software system level shows the interaction between the SiLA 2 Manager and other connected software systems.

The container view shows the internal structure of the SiLA 2 Manager.

Discovery of SiLA devices

All SiLA servers implement Multicast DNS (mDNS) and DNS-based Service Discovery (DNS-SD). The SiLA2 specifications for service discovery are defined in the SiLA Part (B) - Mapping Specification. The SiLA 2 Manager uses the python-zeroconf implementation to discover registered services.

Note

The service description multicasted by mDNS is not the same for all SiLA implementations. At the moment, only the services implemented with SiLA 2 python will show the correct name of the service. However, a change to the SiLA standard has been made to obtain the description details from all implementations. This change will be included in future releases (09.03.2021). For implementations other than SiLA 2 python, the discovery will display the service with the default “unnamed” service name. The displayed name for SiLA python servers is equal to the SiLA Server Name.

The backend code used for the discovery feature is located in the folder source.device_manager.sila_auto_discovery.

class source.device_manager.sila_auto_discovery.sila_auto_discovery.Listener

Bases: zeroconf.ServiceListener

add_service(zeroconf, type, name)

remove_service(zeroconf, type, name)

update_service(zeroconf, type, name)

class source.device_manager.sila_auto_discovery.sila_auto_discovery.SilaServerInfo(uuid: str, name: str, ip: str, port: int, hostname: str)

Bases: object

Stores SiLA device information displayed by the server in the network via mDNS

hostname: str

ip: str

name: str

port: int

uuid: str

source.device_manager.sila_auto_discovery.sila_auto_discovery.find() → List[source.device_manager.sila_auto_discovery.sila_auto_discovery.SilaServerInfo]

Return a list of SilaServerInfo objects of which each entry represents the information about a server discovered on the network.

Returns

List of SiLa server information

Return type

List[SiLaServerInfo]

Discovery Example

The discovery functionality can easily be explored by running the find() function from the root directory of this project:

import os
import sys
sys.path.insert(0, os.path.abspath('.'))
from source.device_manager.sila_auto_discovery.sila_auto_discovery import find


servers = find()
print(servers)

Dynamic client

The dynamic client is capable of connecting to a server without any prior knowledge of the servers functionality. In SiLA2 this is made possible by standard features. The SiLA Service feature contains the necessary functions (Get_ImplementedFeatures(), and Get_FeatureDefinition()) to query the information necessary to construct the client once a connection has been established.

The SiLA2 Device Manager uses the SiLA_python dynamic client. The created client files are stored as temporary data on the host machine the device machine is running on. The client files are deleted if the device is deleted within the application. When a device is added to application, a UUID is assigned for internal reference. This UUID is displayed in the expandable device detail information on the frontend main page and is used. This UUID is also used as storage name for the device client files.

Dynamic client Example

class source.device_manager.device_layer.dynamic_client.DynamicSiLA2Client(name: str, description: str = '', server_name: Optional[str] = None, client_uuid: Optional[str] = None, version: str = '0.0', vendor_url: str = '', server_hostname: str = 'localhost', server_ip: str = '127.0.0.1', server_port: int = 50051)

Bases: sila2lib.sila_client.SiLA2Client

The dynamic client class

SiLAService_stub: SiLAService_feature_grpc

Access to the SiLAService stubs to obtain primary server information

SimulationController_stub: SimController_feature_grpc

Access to the SimulationController stubs to control the simulation/real mode behaviour of the server

call_command(feature_id: str, command_id: str, parameters: Dict[str, Any]) → Dict[str, Any]

call_property(feature_id: str, property_id: str) → Dict[str, Any]

channel: grpc.Channel

Store the grpc channel which is used to communicate with the server

client_uuid: str

The uuid of the client

count_commands(feature_id: str) → int

count_properties(feature_id: str) → int

data_storage: str

Path where the general information about the server is stored

description: str

Client description

encryption: bool

Store whether encryption is used in the connection

generate_files()

Creates the Feature Definition (FDL) xml-files, proto, pb2 and pb2_grpc files in the temporary files directory

get_feature_path(feature_id) → str

list_command_names(feature_id: str) → List[str]

list_commands(feature_id: str)

list_features() → List[str]

list_properties(feature_id: str)

list_property_names(feature_id: str) → List[str]

name: str

Client name

run()

main gRPC client routine

server_cert: bytes

The server certificate which is used in an encrypted channel

server_hostname: Union[str, None]

The server hostname to which to connect

server_port: Union[int, None]

The server port to which to connect

sila2_config: ConfigParser

Access to the configuration file used to store/read data in for this client

stop(force: bool = False)

Stop SiLA client routine

Parameters

force – If set True, the client is supposed to disconnect and stop immediately. Otherwise it can first try to finish what it is doing.

Returns

Whether the client could be stopped successfully or not.

vendor_url: str

The vendors URL

version: str

Client software version

class source.device_manager.device_layer.dynamic_client.FunctionParameterDescription(identifier)

Bases: object

identifier: str

input_data_path: List[str]

input_data_type: List[str]

output_data_path: List[str]

output_data_type: List[str]

source.device_manager.device_layer.dynamic_client.delete_dynamic_client(uuid: uuid.UUID)

Delete the files generated by the dynamic client from the temporary files folder

Parameters

uuid (str) – The uuid of the SiLA server

source.device_manager.device_layer.dynamic_client.get_sila_device_directory(uuid: uuid.UUID) → str

Get the applications temp files directory path

Parameters

uuid (str) – The uuid of the SiLA server

source.device_manager.device_layer.dynamic_client.get_sila_device_lock_file(uuid: uuid.UUID) → str

Get the lock file of the files

Parameters

uuid (str) – The uuid of the SiLA server

The dynamic client is located at source.device_manager.device_layer. The dynamic client can be executed freely without the application. If invoked directly, the respective code snippet at the bottom of the file can be un-commented. The basic connection and code generation functionality can be achieved with the code snippet below. Further examples can be found in the file itself.

if __name__ == "__main__":
     # Add source to path to enable imports
     import os
     import sys
     sys.path.insert(0, os.path.abspath('.'))

     # or use logging.INFO (=20) or logging.ERROR (=30) for less output
     # logging.basicConfig(format='%(levelname)-8s| %(module)s.%(funcName)s: %(message)s', level=logging.INFO)
     client = DynamicSiLA2Client(name="DynamicClient", server_ip='127.0.0.1', server_port=50051)

     # create the client files
     client.generate_files()
     # start the client, which will load all data from the server
     client.run()

Backend API

The python backend uses the FastAPI web framework. The source code is open-source and available in the fastapi repository.

PROJECT

SiLA2_device_manager

details

The API of the backend.

file

backend.py

authors

Lukas Bromig, David Leiter, Alexandru Mardale

date

(creation) 2021-01-27

date

(last modification) 2021-01-27

Copyright: This file is provided “AS IS” with NO WARRANTY OF ANY KIND, INCLUDING THE WARRANTIES OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

For further Information see LICENSE file that comes with this distribution.

class backend.LoginCredentials(*, username: str, password: str)

Bases: pydantic.main.BaseModel

Required for user authentication during login

class backend.ResetPasswordData(*, newPassword: str, oldPassword: str = None)

Bases: pydantic.main.BaseModel

Required for resetting of passwords

class backend.Status(*, running: bool)

Bases: pydantic.main.BaseModel

Class containing the status of a running experiment

class backend.User(*, id: int = None, name: str, fullName: str, role: str, newPassword: str = None, oldPassword: str = None)

Bases: pydantic.main.BaseModel

Contains user information for user management and user authentication

backend.add_database(database: source.backend.device_manager_service.NewDatabaseModel, username: str = Depends(decode_token))

Add a new database to the system. The database information is stored in the postgreSQL database.

Parameters

• database (NewDatabaseModel) – An object containing the information of the new database, i.e. connection details and name.

• username (str) – The name of the executing user

Returns

None

backend.add_device(device: source.backend.device_manager_service.NewDeviceModel, username: str = Depends(decode_token))

Addition of a new device to the postgreSQL database

Parameters

• device (NewDeviceModel) – Device information object including name, type, address, port etc. etc. but without an assigned internal UUID4

• username (int) – The name of the executing user

Returns

None

backend.add_user(new_user: backend.User, username: str = Depends(decode_token))

Saves a new user to the postgreSQL database

Parameters

• new_user (User) – Information of the new user

• username (str) – The name of hte executing user

Returns

None

backend.book_device(bookingInfo: source.backend.device_manager_service.BookingModel, username: str = Depends(decode_token))

Store a booking for a device in the postgreSQL database

Parameters

• bookingInfo (BookingModel) – The booking information

• username (str) – The name of the executing user

Returns

None

backend.call_feature_command(uuid: str, feature_originator: str, feature_category: str, feature_identifier: str, feature_version_major: str, command_id: str, parameterList: source.backend.device_manager_service.DeviceCommandParameters, username: str = Depends(decode_token))

Executes the specified command and returns the response

Parameters

• uuid (str) – Internally assigned device uuid

• feature_originator (str) – The SiLA 2 Originator of the feature the command belongs to

• feature_category (str) – The SiLA 2 Category of the feature the command belongs to

• feature_identifier (str) – The SiLA 2 Feature Identifier of the feature the command belongs to

• feature_version_major (str) – The SiLA 2 Major Feature Version of the feature the command belongs to

• command_id (str) – the id of the command to be called

• parameterList (DeviceCommandParameters) – A list of parameters required by the command

• username (str) – The name of the executing user

Returns

The response of the call

async backend.control_experiment(status: backend.Status, experimentID: int, username: str = Depends(decode_token))

Asynchronous function to keep track of the experiment status. Updates the experiment status in the databse.

Parameters

• status (bool) – The status whether the experiment is running or not

• experimentID (int) – The id of the experiment

• username (str) – The name of the executing user

Returns

None

backend.create_experiment(experiment: source.backend.device_manager_service.ExperimentBookingModel, username: str = Depends(decode_token))

Store a new experiment in the postgreSQL database

Parameters

• experiment (ExperimentBookingModel) – The new experiment

• username (str) – The name of the executing user

Returns

None

backend.create_token(username: str, expiration: int)

Creates an authentication token for the current user

Parameters

• username (str) – The name of the current user

• expiration (int) – Expiration date of the user token

Returns

Returns username, expiration date, secret key and JWT algorithm

Return type

dict

backend.database_status(id: int, username: str = Depends(decode_token))

Check the connection status of the database by pinging the database server

Parameters

• id (int) – Internally assigned id of the database

• username (str) – The name of the executing user

Returns

An object containing the status information

Return type

DatabaseStatus

backend.decode_token(token: str = Depends(OAuth2PasswordBearer))

Decodes and verifies token using the JWT algorithm HS256

Parameters

token (str) – The user token

Returns

user key/ payload

Return type

str

backend.delete_booking(bookingID: int, username: str = Depends(decode_token))

Delete a booking from the postgreSQL by id

Parameters

• bookingID (int) – The booking id

• username (str) – The name of the executing user

Returns

None

backend.delete_database(id: int, username: str = Depends(decode_token))

Delete a registered database from the postgreSQL database

Parameters

• id (int) – Internally assigned database id

• username (str) – The name of the executing user

Returns

None

backend.delete_device(uuid: str, username: str = Depends(decode_token))

Delete the device from postgreSQL database and delete generated dynamic client files in temporary data

Parameters

• uuid (str) – Internally assigned device uuid

• username (str) – The name of the executing user

Returns

None

backend.delete_experiment(experimentID: int, username: str = Depends(decode_token))

Delete an experiment from the postgreSQL database.

Parameters

• experimentID (str) – The internal id of the experiment to be deleted

• username (str) – The name of the executing user

Returns

None

backend.delete_user(id: int, username: str = Depends(decode_token))

Delete a user from the postgreSQL

Parameters

• id (int) – Internally assigned user id

• username (int) – The name of the executing user

Returns

None

backend.delete_user_script(scriptID: int, username: str = Depends(decode_token))

Delete a specific user-script from the postgreSQL database

Parameters

• scriptID (str) – The id of the script

• username (str) – The name of the executing user

Returns

None

backend.device_features(uuid: str, username: str = Depends(decode_token))

Get all features and associated information of the requested device.

Parameters

• uuid (str) – Internally assigned device uuid

• username (str) – The name of the executing user

Returns

List of SiLA feature objects that include all associated information

backend.device_features_for_datahandler(uuid: str, username: str = Depends(decode_token))

Get all features and associated information of the requested device.

Parameters

• uuid (str) – Internally assigned device uuid

• username (str) – The name of the executing user

Returns

List of SiLA feature objects that include all associated information

backend.device_log(start: int = 0, end: int = 1629723128.209646, excludeInfo: bool = False, excludeWarning: bool = False, excludeCritical: bool = False, excludeError: bool = False, username: str = Depends(decode_token))

Get the device logs.

Parameters

• start (int, optional) – Time of the first log entry to be gathered. Defaults to 0.

• end (int, optional) – Time of the last log entry to be gathered. Defaults to current time.

• excludeInfo (bool, optional) – Filter Info log level messages

• excludeWarning (bool, optional) – Filter Warning log level messages

• excludeCritical (bool, optional) – Filter Critical log level messages

• excludeError (bool, optional) – Filter Error log level messages

• username (str) – The name of the executing user

Returns

A dictionary containing the requested log messages

Return type

dict

backend.device_status(uuid: str, username: str = Depends(decode_token))

Get the availability status of the device by pinging the device server and retrieving the booking information

Parameters

• uuid (str) – Internally assigned device uuid

• username (str) – The name of the executing user

Returns

An object containing status information

Return type

DeviceStatus

backend.edit_experiment(experimentID: int, experiment: source.backend.device_manager_service.ExperimentBookingModel, username: str = Depends(decode_token))

Edit an already existing experiment

Parameters

• experimentID (int) – The internal experiment id

• experiment (ExperimentBookingModel) – The experiment information object

• username (str) – The name of the executing user

Returns

None

async backend.experiment_logs_websocket(websocket: starlette.websockets.WebSocket)

Asynchronous function that forwards the experiment logs of the docker container via websocket

Parameters

websocket (Websocket) – The websocket the information is transferred by

Returns

None

async backend.experiment_status_websocket(websocket: starlette.websockets.WebSocket)

Asynchronous function that forwards the experiment status via websocket

Parameters

websocket (Websocket) – The websocket the information is transferred by

Returns

None

backend.get_booking(bookingID: int, username: str = Depends(decode_token))

Get booking information for a booking id

Parameters

• bookingID (int) – The id of the requested booking

• username (str) – The name of the executing user

Returns

Information of the booking

Return type

BookingInfo

backend.get_booking_list(start: int = 0, end: int = 4294967295, username: str = Depends(decode_token))

Fetches the registered bookings from the postgreSQL database

Parameters

• start (int, optional) – The time of the first booking to be gathered. Defaults to 0.

• end (int, optional) – The time of the last booking to be gathered.

• username (str) – The name of the executing user

Returns

A list of all bookings

Return type

List[booking_info]

backend.get_current_user(username: str = Depends(decode_token))

Fetches the details of the current user.

Parameters

username – The name of the current user

Type

str

Returns

An object containing information on the current user

Return type

User

backend.get_database(id: int, username: str = Depends(decode_token))

Get information on a specific database

Parameters

• id (int) – Internally assigned id of the database

• username (str) – The name of the executing user

Returns

An object containing information on the database

Return type

DatabaseInfo

backend.get_databases(username: str = Depends(decode_token))

Retrieves all information on the registered databases

Parameters

username (str) – The name of the executing user

Returns

Returns a list of objects containing database information

Return type

List[DatabaseInfo]

backend.get_device(uuid: str, username: str = Depends(decode_token))

Get device information for a provided device-uuid

Parameters

• uuid (str) – Unique identifier (UUID4) of the device used for internal reference

• username (str) –

Returns

Device information including name, type, address, port etc. etc.

Return type

DeviceInfo

backend.get_device_booking_list(uuid: str, start: int = 0, end: int = 1629723128.211518, username: str = Depends(decode_token))

Fetches a list of device bookings

Parameters

• uuid (str) – Internally assigned device uuid

• start (int, optional) – The time of the first booking entry to be gathered. Defaults to 0.

• end (int, optional) – The time of the last booking entry to be gathered. Defaults to current time.

• username (str) – The name of the executing user

Returns

List of booking information objects for the device

Return type

List[BookingInfoWithNames]

backend.get_experiments(username: str = Depends(decode_token))

Get information of all stored experiments

Parameters

username (str) – The name of the executing user

Returns

Returns a list of experiments with the corresponding information

Return type

List[Experiment]

backend.get_feature_property(uuid: str, feature_originator: str, feature_category: str, feature_identifier: str, feature_version_major: str, property_id: str, username: str = Depends(decode_token))

Requests the specified property and returns the response

Parameters

• uuid (str) – Internally assigned device uuid

• feature_originator (str) – The SiLA 2 Originator of the feature the property belongs to

• feature_category (str) – The SiLA 2 Category of the feature the property belongs to

• feature_identifier (str) – The SiLA 2 Feature Identifier of the feature the property belongs to

• feature_version_major (str) – The SiLA 2 Major Feature Version of the feature the property belongs to

• property_id (str) – The id of the property

• username (str) – The name of the executing user

Returns

The response of the call

backend.get_user(id: int, username: str = Depends(decode_token))

Fetches detailed information of a user from the postgreSQL database

Parameters

• id (int) – Internally assigned user id

• username (int) – The name of the executing user

Returns

An object containing the user information

Return type

User

backend.get_user_script(scriptID: int, username: str = Depends(decode_token))

Get the information of a specific user-scripts

Parameters

• scriptID (int) – The internally assigned script id

• username (str) – The name of the executing user

Returns

The script object containing the scripts content and information

Return type

Script

backend.get_user_scripts_info(username: str = Depends(decode_token))

Get the information of all registered user-scripts

Parameters

username (str) – The name of the executing user

Returns

A list of script objects that contain the script content and information

Return type

List[Script]

backend.link_database(uuid: str, id: int = Body(Ellipsis), username: str = Depends(decode_token))

Link a database to a device. Required for the data-handler functionality.

Parameters

• uuid (str) – Internally assigned device-uuid (uuid4)

• id (int) – Internally assigned database id

• username (str) – The name of the executing user

Returns

None

backend.login(form: fastapi.security.oauth2.OAuth2PasswordRequestForm = Depends(NoneType))

Transfer authentication details for Auth2 authentication. Compare authentication data with registered users. Create access token and assign expiration date.

Parameters

form (OAuth2PasswordRequestForm) – A form containing the necessary authentication details

Returns

A dictionary of authentication elements such as tokens and expiration date

Return type

dict

backend.reset_password(id: int, password_data: backend.ResetPasswordData, username: str = Depends(decode_token))

Changes the password of the user to a new one.

Parameters

• id (int) – Internally assigned user id

• password_data (ResetPasswordData) – Contains the old and the new password

• username (str) – The name of the executing user

Returns

None

backend.set_command_attributes_for_data_handler(uuid: str, feature_id: str, command_id: str, parameters: List[source.backend.device_manager_service.DeviceCommandParameter], active: bool = Body(Ellipsis), meta: bool = Body(Ellipsis), nonMetaInterval: int = Body(None), metaInterval: int = Body(None), username: str = Depends(decode_token))

Set the state of the data acquisition mode on a command level. De/Activate data acquisition and switch from meta to non-meta polling interval.

Parameters

• uuid (str) – Internally assigned device-uuid

• feature_id (str) – The SiLA id of the feature the command belongs to

• command_id (str) – The SiLA id of the command

• parameters (List[DeviceCommandParameter]) – The parameter to be used by the data-handler

• active (bool) – The state of the data-handler data acquisition mode this command of the device. To be set!

• meta (bool) – The kind of polling interval that shall be used. To be set! Defaults to meta (True).

• nonMetaInterval (int) – Polling interval for non-meta data acquisition

• metaInterval (int) – Polling interval for meta data acquisition

• username (str) – The name of the executing user

Returns

None

Returns

backend.set_database(id: int, database: source.backend.device_manager_service.DatabaseInfoModel, username: str = Depends(decode_token))

Parameters

• id (int) – Internally assigned id of the database

• database (DatabaseInfoModel) – An object containing database information

• username (str) – The name of the executing user

Returns

backend.set_device(uuid: str, device: source.backend.device_manager_service.DeviceInfoModel, username: str = Depends(decode_token))

Parameters

• uuid (str) – Internally assigned device uuid

• device (DeviceInfoModel) – A device information object

• username (str) – The name of the executing user

Returns

None

backend.set_device_attributes_for_data_handler(uuid: str, active: bool = Body(Ellipsis), username: str = Depends(decode_token))

Sets the available attribute of the device status to the specified state.

Parameters

• uuid (str) – Internally assigned device-uuid

• active (bool) – The new state setting of the device

• username (str) – The name of the executing user

Returns

None

backend.set_feature_attributes_for_data_handler(uuid: str, feature_id: str, active: bool = Body(Ellipsis), meta: bool = Body(Ellipsis), username: str = Depends(decode_token))

Set the state of the data acquisition mode on a feature level. De/Activate data acquisition and switch from meta to non-meta polling interval.

Parameters

• uuid (str) – Internally assigned device-uuid

• feature_id (str) – The SiLA id of the feature

• active (bool) – The state of the data-handler data acquisition mode of this feature of the device. To be set!

• meta (bool) – The kind of polling interval that shall be used. To be set! Defaults to meta (True).

• username (str) – The name of the executing user

Returns

None

backend.set_property_attributes_for_data_handler(uuid: str, feature_id: str, property_id: str, active: bool = Body(Ellipsis), meta: bool = Body(Ellipsis), nonMetaInterval: int = Body(None), metaInterval: int = Body(None), username: str = Depends(decode_token))

Set the state of the data acquisition mode on a property level. De/Activate data acquisition and switch from meta to non-meta polling interval.

Parameters

• uuid (str) – Internally assigned device-uuid

• feature_id (str) – The SiLA id of the feature the property belongs to

• property_id (str) – The SiLA id of the property

• active (bool) – The state of the data-handler data acquisition mode this property of the device. To be set!

• meta (bool) – The kind of polling interval that shall be used. To be set! Defaults to meta (True).

• nonMetaInterval (int) – Polling interval for non-meta data acquisition

• metaInterval (int) – Polling interval for meta data acquisition

• username (str) – The name of the executing user

Returns

None

backend.set_user_script(scriptID: int, script: source.backend.device_manager_service.ScriptModel, username: str = Depends(decode_token))

Parameters

• scriptID (int) – The id of the script

• script (ScriptModel) – The object containing the script content and information

• username (str) – The name of the executing user

Returns

None

backend.set_user_script_info(scriptID: int, info: source.backend.device_manager_service.ScriptInfoModel, username: str = Depends(decode_token))

Parameters

• scriptID (int) – The id of the script

• info (ScriptInfoModel) – The object containing information and content of the script

• username (str) – The name of the executing user

Returns

None

backend.sila_discovery()

Searches for all registered SiLA devices on the network and returns the information

Returns

Returns a list containing all discovered devices and the corresponding information available

Return type

List[List[SilaServerInfo]]

backend.unlink_database(uuid: str, username: str = Depends(decode_token))

Unlink a database from a device. The link is deleted.

Parameters

• uuid (str) – Internally assigned device-uuid (uuid4)

• username (str) – The name of the executing user

Returns

None

backend.update_user(id: int, new_user: backend.User, username: str = Depends(decode_token))

Updates the stored information of an existing user. Can be used to change the user role or name.

Parameters

• id (int) – Internally assigned user id

• new_user (User) – The updated user information

• username (str) – The name of the executing user

Returns

None

backend.upload_user_script(script: source.backend.device_manager_service.ScriptModel, username: str = Depends(decode_token))

Add a new script object to the postgreSQL database

Parameters

• script (ScriptModel) – The script object containing content and additional information

• username (str) – The name of the executing user

Returns

None

FAQ

Error 500: Internal Server Error

Error:

Error 500: Internal Server Error

The device manager frontend issues a token to a user upon login. The token has an expiration date and is only valid for X min. If the user is inactive for a longer period of time, the token is not refreshed and access is denied. A token may invalidate for other reasons as well. Access from multiple browsers by the same user may also cause this issue.

Solution: Logout and log back in again. This will renew your token. Close device manager tabs in other browsers.

Note

You can set the expiration time and the extension time for a token-refresh in the backend.py file in line 87-88.

Protobuf: A file with this name is already in the pool

Error:

TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "messages.proto":
  messages.proto: A file with this name is already in the pool.

This error will show in the backend console and it’s logs. It is a known error on windows machines and related to the protobuf package. The standard wheel installation of protobuf doesn’t allow the use of multiple files with the same name in the same pool. All SiLA devices implement the standard features, thus, this is problematic. The protobuf –no-binary installation is different to the wheel and allows just that.

Solution: Run the protobuf_no_binary_install.bat script which is located in the root directory of this software. The script uninstalls the standard protobuf installation and replaces it with the binary build. Pipenv doesn’t implement the –no-binary flag, thus pip is used. Protobuf is added to the pipfile afterwards for completeness sake.

Psycopg2 (PostgreSQL client): Connection Pool Exhausted

Error:

The number of connections to the connection pool (SimpleConnectionPool) is exhausted. The error message reads:

Traceback (most recent call last):
File "scheduler.py", line 380, in <module>
  main()
File "scheduler.py", line 374, in main
  schedule_future_experiments_from_database()
File "scheduler.py", line 258, in schedule_future_experiments_from_database
  for exp in experiment.get_scheduling_info():
File "/usr/device-manager/source/device_manager/experiment.py", line 125, in get_scheduling_info
  conn = get_database_connection()
File "/usr/device-manager/source/device_manager/database.py", line 34, in get_database_connection
  return storage['pool'].getconn()
File "/usr/device-manager/.venv/lib/python3.8/site-packages/psycopg2/pool.py", line 92, in _getconn
   raise PoolError("connection pool exhausted")
psycopg2.pool.PoolError: connection pool exhausted

Solution: Generally, a connection is returned and closed after use. A maximum number of connections is allowed. This number is specified in the file /source/device_manager/database.py in the function get_database_connection. Each user requires several connections. If multiple users access the device manager, the total number of connections may get exhausted. Increasing the number of maxconn of the SimpleConnectionPool will solve this problem.

Update If the maximum number of connections is exhausted, the SiLA 2 Manager will close all idle connections fully and put them back into the pool. The above error should not arise anymore.

SiLA 2 Python (sila2lib) Error: KeyError ‘HOME’

Error:

The SiLA 2 Manager is run as a service. This service doesn’t have access to the environmental variables of the user. The SiLA 2 library is trying to get the path of the home directory. The command os.environ returns a dictionary of the environmental variables. The key [‘HOME’] is not contained and hence the operation raises a KeyError:

Traceback (most recent call last):
File "./source/device_manager/device_layer/sila_device.py", line 14, in create_and_init_dynamic_client
  client = DynamicSiLA2Client(name=f'{name}-client',
File "./source/device_manager/device_layer/dynamic_client.py", line 93, in __init__
  super().__init__(name, description, server_name, client_uuid, version,
File "/home/david/sila2_device_manager/.venv/lib/python3.8/site-packages/sila2lib/sila_client.py", line 117, in __init__
  self.sila2_config = read_config_file('client', name)
File "/home/david/sila2_device_manager/.venv/lib/python3.8/site-packages/sila2lib/_internal/config.py", line 22, in read_config_file
  config_dir = get_config_dir(subdir=name)
File "/home/david/sila2_device_manager/.venv/lib/python3.8/site-packages/sila2lib/_internal/config.py", line 12, in get_config_dir
  path = os.path.join(os.environ['HOME'], '.config', 'sila2')
File "/usr/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'HOME'
None

Solution: Replace the following line in file: sila2_device-manager/.venv/lib/python3.8/site-packages/sila2lib/_internal/config.py Line 13: path = os.path.join(os.environ[‘HOME’], ‘.config’, ‘sila2’) with the explicit path of your home directory or the home directory that you created for the SiLA 2 Manager: path = os.path.join(‘/home/device-manager’, ‘.config’, ‘sila2’)

To-do:

• Incorporate SiLA client meta-data in python repository

• Incorporate observable commands in device manager

• Implement lock/authorization feature

• Edit experiment and update bookings and experiments in backend properly

• Change project nomenclature to SiLA 2 nomenclature

• Full SiLA 2 release 1.0 support

• Out-of-scope: Support SiLA 2 release 1.1

Contact

Contact

Please direct all contact to the following address:

Lukas Bromig

Technical University of Munich

Chair of Biochemcial Engineering

Boltzmannstraße 15

DE-85748 Garching

Tel. +49 89 289 15712

Mail. lukas.bromig@tum.de

List of authors

Name	Role	E-Mail
Lukas Bromig	Maintainer	lukas.bromig@tum.de
David Leiter	Developer	david.leiter@tum.de
Alexandru Mardale	Developer	alexandru.mardale@tum.de
Nikolas von den Eichen	Observer	nikolas.eichen@tum.de

License

All work of this project is licensed under the MIT License. The license can be found in the GitLab repository.

Indices and tables

• Index

• Module Index

• Search Page