How to write an install Job Package

Introduction

This document covers how to write a job package in Python and Bash that performs a flash installation of an RTOS operating system onto an ESP32 SoC board using the ESP IoT Developer Framework.

The ESP32 board is a very inexpensive, light weight System on Chip that is popular for IoT applications. However, due to very limited memory and storage, it can only host a very small set of applications at any one time and any application intended to run on the board must be flashed to it prior to execution. For this example, we will show you how to write a Job Package that builds and installs a Real Time OS (RTOS) with a MicroPython (µPython) runtime onto an ESP32 SoC utilizing the ESP IoT Developer Framework is based on FreeRTOS, and the add-on MicroPython package for the IDF.

Creating the ESP32 RTOS with µPython Job Package

Prerequisites

For this example, it is expected that C developer toolchain and Python3.x have been installed onto the development PC and hosts running the LabScale agent. To install the toolchain, execute the following commands in a terminal for the given hosts below:

Ubuntu Linux: sudo apt install git wget flex bison gperf python3 python3-pip python3-venv cmake ninja-build ccache libffi-dev libssl-dev dfu-util libusb-1.0-0
MacOS: brew install cmake ninja dfu-util

This example also assumes that there is an ESP32 DIP that declares the serial_port environment variable from the device, and that the ESP32 device has defined the serial port, for example /dev/tty.usb1234 on MacOS or /dev/ttyUSB0 on Linux.

Building the ESP32 platform

Most if not all IoT devices require applications to be built and flashed onto their hardware prior to execution. We will first want to create a script that builds our application prior to flashing. For this, we will write a very simple Bash script and let the ESP IDF do most of the heavy lifting. This script will be executed as part of the Job defined within the package we will create.

The first thing is to make sure that the ESP32 IDF is present in the job's working directory. We can guarantee this by fetching it from its GitHub repository and set up its environment prior to building, for example:

#!/bin/bash -ex
# Fetch the ESP32 IDF from its GitHub repository
git clone -b v5.0.2 --recursive https://github.com/espressif/esp-idf.git > install.log
pushd ./esp-idf
./install.sh esp32
source export.sh
popd

Next, we also want Micropython to be available on the ESP32 board and provide a simple shell, so get the Micropython add-on package from its GitHub repo:

#!/bin/bash -ex
# Fetch the ESP32 IDF from its GitHub repository
git clone -b v5.0.2 --recursive https://github.com/espressif/esp-idf.git > install.log
# Set up the build environment
pushd ./esp-idf
./install.sh esp32
source export.sh
popd

# Fetch the Micropython add-on from its GitHub repository
git clone 'https://github.com/micropython/micropython.git'
pushd ./micropython
# Build the Micropython package and submodules
make -C mpy-cross
pushd ./ports/esp32
make submodules
popd
popd

Finally, build RTOS for a generic ESP32 board linking the MicroPython static object file created previously (this is done transparently by the IDF):

#!/bin/bash -ex
# Fetch the ESP32 IDF from its GitHub repository
git clone -b v5.0.2 --recursive https://github.com/espressif/esp-idf.git > install.log
# Set up the build environment
pushd ./esp-idf
./install.sh esp32
source export.sh
popd

# Fetch the Micropython add-on from its GitHub repository
git clone 'https://github.com/micropython/micropython.git'
pushd ./micropython
# Build the Micropython package and submodules
make -C mpy-cross
pushd ./ports/esp32
make submodules
popd
popd

# Build RTOS (linking in µPython)
pushd ./micropython/ports/esp32
make BOARD=ESP32_GENERIC_S3
popd

Now create a folder named esp_rtos_mpy, save this to a file in that directory named esp_rtos_mpy/esp32_rtos_mpy_build.sh, and make it executable chmod +x esp_rtos_mpy/esp32_rtos_mpy_build.sh.

Installing the ESP32 platform

Now that we have a built RTOS with MicroPython, it is ready for installation by our Job Package. Because the ESP32 IDF comes complete with tools necessary to flash the hardware, we will again employ the IDF to do the heavily lifting. The only requirement is that the ESPPORT and PORT environment variables are set to the serial port of connected the ESP32 board, in this case we will leverage an ESP DIP that exports a serial_port environment variable (this variable is set by the user when first creating the ESP32 device in LabScale). The ESP32 IDF provides a deploy make target that will take the most recent built platform object in the current working directory and flash onto the ESP32.

# Install RTOS with µPython onto the ESP32
pushd ./micropython/ports/esp32
export ESPPORT="${LS_DIP_serial_port}"
export PORT="${ESPPORT}"
make deploy
popd

Save this to a file named esp_rtos_mpy/esp32_rtos_mpy_deploy.sh and make it executable chmod +x esp_rtos_mpy/esp32_rtos_mpy_deploy.sh

Writing the Job Package main script

Now that we have build and install scripts, we can write a script that utilizes these scripts during job execution. For this, we will used Python. One thing to keep in mind is that the LabScale service may be intermittently accessing the hardware concurrent to the execution of this script. To avoid stepping on each other's toes, we will need to test and lock the serial port prior to flashing. An example of how this may work in Python is as follows:

import os

# the serial_port environment variable is defined by
# the ESP32 device DIP and set at the time the user
# created the ESP32 device in the LabScale service.
PORT = os.environ["LS_DIP_serial_port"]

if __name__ == "__main__":
    from filelock import FileLock

    devname = os.path.basename(PORT)
    lockpath = os.path.join("/tmp/labscale_agent/run", f"{devname}.lock")

    # Build RTOS and µPython
    assert os.system("./esp32_rtos_mpy_build.sh") == 0

    # Create a lockfile to protect the serial port
    # during flash installation. This lock should
    # be defined in the ESP32 DIP.
    with FileLock(lockpath):
        # Flash the application onto the ESP32 board
        assert os.system("./esp32_rtos_mpy_deploy.sh") == 0

Save this to a file named esp_rtos_mpy/esp32_rtos_mpy_install.py.

Writing the exec.sh script

Because we are using Python with a package that does not come with the standard distribution, we will want to set up a virtual environment and install the dependencies beforehand. The easiest way to do this is to wrap the execution of our scripts within an exec script that performs the setup prior to execution. An example for this may look like this Bash script:

#!/bin/bash -e
if ! python3 -m venv .
then
  if ! python3 -c "import virtualenv" 2>/dev/null
  then
    if ! python3 -m pip install virtualenv
    then
      echo "Failed to install Python virtualenv, please install it manually"
      exit 1
    fi
  fi
  python3 -m virtualenv .
fi
source ./bin/activate
./bin/python3 -m pip install --upgrade pip
./bin/python3 -m pip install filelock
exec $@

This script will do its best to install a virtual environment and install the filelock Python package, but if these packages are already installed onto the host that will execute this Job, then this script is not necessary.

Now save this to a file named exec.sh and make it executable chmod +x exec.sh.

Creating the config.yaml file

Now that we have our scripts, we need to tell the job runner what the main entry point is to call. This is done with the config.yaml file that is provided in every Job Package. For our example, we will write it like this:

name: esp32_rtos_mpy_install
version: 1.0.0
cmd: "${DIR}/exec.sh python esp32_rtos_mpy_install.py"

Create the Job Package bundle

Once all of the files are saved, we now want to bundle them together into a Job Package. The Job Package is imply a compressed archive file, specifically a gzipped tar file. To do this, run the following on the command line:

tar  -C esp_rtos_mpy -czf esp32_rtos_mpy_install.tgz .

All that is left to do is upload it to the desired project's Job Packages page in the LabScale service.

Introduction​

Creating the ESP32 RTOS with µPython Job Package​

Prerequisites​

Building the ESP32 platform​

Installing the ESP32 platform​

Writing the Job Package main script​

Writing the exec.sh script​

Creating the config.yaml file​

Create the Job Package bundle​