Skip to main content

How to Write a Test

The LabScale service does little to distinguish a job package from a test suite, they are essentially same thing with the only difference being that a test suite reports additional data in the form of test results. Once reported, these test results are available on the LabScale project dashboard and can be aggregated for project level analysis.

This guide will focus on writing a new test suite that executes within the LabScale service.

Terminology​

  • Agent - A locally running service that executes jobs.
  • Device - The hardware under test.
  • Host - The machine that an agent is running on.
  • Job - A job is the description of a task that can be queued up for executed against devices.
  • Job Package - An archive file containing a configuration file and 0 or more resource files and/or executables that are delivered to an agent for execution.
  • Test - A job that executes a test against a target device and produces test results.
  • Test Suite - A job package that contains one or more tests.
  • YAML - A standardized, human readable, data serialization file format. Visit the YAML page on The Wikipedia for more information.

Anatomy of a Job Package​

The LabScale automation platform does not limit tests to any particular programming language or test framework. The only requirements is that the test provide a YAML file, labeled config.yaml, that describes the main entry point into the test which the Agent will call for execution, 0 or more executables and/or supplementary resources required by the job, and all all packaged within a standard UNIX gzipped tar archive file. The package is how jobs and tests are delivered to remote agents for execution.

The config.yaml File​

Every Job Package must contain a config.yaml file. This is necessary because it contains vital information required by the agent to invoke the job on the command line. The most basic YAML configuration file could look something like this:

name: helloworld
version: 1.0.0
cmd: "echo hello world"

The YAML name field contains the name of the job or test suite. The version field is simply the version of the job package. And finally, the cmd is the command line to be invoked by the agent. The YAML configuration can also contain some additional environment variables, e.g. DIR which is replaced by the current working directory in which the job is running, typically the same fold as the config.yaml file.

This is a fully self-contained configuration and when invoked by the agent, it will simply print "hello world".

The Package​

Before this job can be executed by the agent, the YAML will need to be saved to a file named config.yaml and then bundled into an archive, for example, like this:

tar -czf helloworld.tgz config.yaml

Packages are necessary for distribution of jobs, tests and resources to remote agents. Once an agent receives a job package, it will be unarchived into its own work folder, and executed by calling command line defined in the *cmd line of the config.yaml file.

Writing a New Test Suite​

The LabScale automation platform is very flexible and does not have any dependency on any particular test framework. However, at the moment, LabScale does require that tests be cross platform, that is, they can execute on a variety of host operating systems or CPU architectures without recompilation. However, tests targeting an embedded system can install platform specific tests onto the embedded device and invoke them via SSH or serial from the Agent's command line or from another test executable called by the Agent.

To begin, we will first need a test. In this example we will use Python as the runtime as it is installed by default on a wide variety of host systems, but also because it has a clean coding style and simple to use. We will write the test for a Virtual Device meaning that it will execute in a regular shell on the host, but cat be scheduled for tests as if it were a physically distinct device.

# The simplest single line test one can write
print("example_test.py::example_test::Test01 PASSED")
Note

This example will output its result in Pytest format.

Save this to a file, named example_test.py.

Next, we need a config.yaml file that tells the Agent how to invoke our test. Copy the following YAML example and save it to a file named config.yaml in the same folder as the test file from the previous section.

name: example_test
version: 1.0.0
cmd: "python3 ${DIR}/example_test.py"
adapter:
  name: pytest_stdout_adapter
Note

Because this is not an executable Python script, we must pass the script to the Python interpreter in the cmd property. Also declared is the pytest_stdout_adapter output adapter used to parse Pytest output.

The config.yaml and example_test.py must now be put into an job package. In a terminal, change directory to the location of the example_test.py and config.yaml files, and then type the following:

tar czf example_test.tgz config.yaml example_test.py

This will create a job package named exampletest.tgz. The archive will now need to be registered with LabScale. To do this, upload the _example_test.tgz file to the LabScale service.

Adding a Job Package​

Before LabScale can allocate a test to the agent the test must first be registered with the service and made available to agents. This requires uploading the job package to the service, the following instructions guide you through this:

  1. In the LabScale home screen, select the Projects menu item in the navbar on the left side of the screen.
  2. Then select the project you would like this test to be associated with.
  3. Once at the project dashboard, select the Job Packages menu in the navbar on the left.
  4. Click on the ADD JOB PACKAGE in the upper right corner of the page.
  5. In the dialog, enter the name of the package, and optionally, a description.
  6. Once back at the Job Packages page, click on the package name.
  7. Now, click on the UPLOAD PACKAGE button.
  8. Drag the job package file to the Drag and drop box within the dialog, or click on the box and select the package in the file explorer.

Executing the Test​

Once a job package is registered it is now available for execution by an Agent. The following instructions will guide you through the process:

Note

The following instructions assume you have already created a Lab and Device. If not, please review the Quick Start guide.

  1. In the LabScale home screen, select the Projects menu item in the navbar on the left side of the screen.
  2. Then select the project the test is associated to in the previous section.
  3. Select the Run Tests menu item in the side navbar on the left.
  4. Click on the checkbox next to the name of your test. Then click the NEXT STEP button.
  5. Select the Release number and Build version of the product. For demo purposes, this can be simply the first item in the list, v1.0 and 0001 for release and build respectively. Then click the NEXT STEP button.
  6. Select the type of device you want to run the test. For demo purposes, select Virtual Device. Select Devices, and the device you want to run

Viewing the Results​

  1. In the LabScale home screen, select the Projects menu item in the navbar on the left side of the screen.
  2. Select the Jobs menu in the side nav bar on the left.
  3. If the job has not executed yet, it will be in a pending state. Select the Pending tab at the top of the Jobs table. If the job is running, it can be found by selecting the Running tab. Finally, if the job has finished, it can be found under the Completed tab.
  4. The pass|fail|skip numbers for the tests within the suite executed by the job will appear with the status of the job.