How to Upload Artifacts from a Job

Jobs typically generate logs and LabScale will always upload those, however there are times when something other than a log file is necessary. LabScale provides a mechanism where a job can upload objects created during execution.

There are two ways to do this:

• Declaring an artifacts stanza in the job package's config.yaml. The agent watches a folder you nominate and uploads files individually, both periodically while the job is still running and once more after it finishes. This is the recommended method.
• Writing to the LS_JOB_artifacts_path folder. Everything in that folder is collected into a single archive and uploaded after the job finishes. This is the original method and is still supported.

The two mechanisms are independent, so a job may use either or both.

Declaring artifacts in the job package config

Add an artifacts stanza to the config.yaml in the root of your job package:

name: the-package-name
version: 1.0.0
cmd: ${DIR}/main_script

artifacts:
  location: ${DIR}/results
  frequency: 60

location is the folder the agent watches, and frequency is the number of seconds between upload passes. With the configuration above, the agent uploads anything new in ${DIR}/results once a minute while the job runs, then makes a final pass after the job's script exits so that nothing written near the end is missed.

Each pass uploads only the files that were created or modified since the last successful pass, so a file is not re-uploaded on every tick. Files are uploaded individually rather than being collected into an archive.

The ${DIR} variable expands to the root of the unpacked job package, and any environment variable available to the job may also be used in the path. The full set of options for the stanza — including type and call_hook — is described in the Job Package config.yaml reference.

Why this is preferred over the legacy method

• Artifacts appear on the job page while the job is still running, instead of only after it ends. On a long-running job this is the difference between watching results accumulate and waiting hours to see anything.
• If a job is canceled, times out, or the host dies, everything uploaded by the passes that already completed is preserved. The legacy method uploads only at the very end, so an interrupted job produces nothing.
• There is no per-file size limit. The legacy archive silently skips any individual file larger than 100MB, which is easy to hit with video captures or memory dumps.
• Uploading steadily throughout the run spreads the network cost out, rather than concentrating it into one large transfer at the end while the device is still held.

Creating the folder

The agent does not create the location folder for you. If the folder does not exist when an upload pass runs, that pass is skipped and a warning is written to the agent log; later passes pick it up once it appears.

Have your job script create the folder before it writes to it, or create it up front with a setup_cmd in the package config.

caution

Do not point location at the LS_JOB_artifacts_path folder. Both mechanisms would then process the same files, and every artifact would be uploaded twice — once individually during the run and again inside the archive at the end. Use a separate folder.

Choosing a frequency

frequency is a trade-off between how quickly artifacts show up and how much work the agent does mid-job:

• A small value (say 15 seconds) suits jobs producing a steady trickle of small files that you want to watch in near real time.
• A larger value (several minutes) suits jobs producing large files, where uploading too eagerly competes with the job itself for bandwidth.
• A value of 0 disables periodic uploading, and artifacts are uploaded only in the final pass after the job's script exits.

The agent never runs two upload passes at once. If a pass is still in flight when the next one is due, that tick is skipped and the interval starts over, so a slow upload cannot pile up behind itself.

Preparing files before each upload

If your artifacts have to be gathered from somewhere else before they can be uploaded — pulled off a device, for example — use the call_hook option to nominate a script the agent runs immediately before each upload pass:

artifacts:
  location: ${DIR}/results
  frequency: 60
  call_hook: ${DIR}/collect_artifacts.sh

The hook is called before every pass, including the final one. On the final call, and only then, the agent passes the argument final to the script. This lets the script do cheap work on every call and expensive work only once at the end — pulling a log file each time, for instance, but pulling crash dumps only as the job wraps up. A worked example of such a script is included in the Job Package config.yaml reference.

note

If your call_hook script needs the target device during the final pass, set early_device_release to false in the package config. Otherwise the device may be handed to the next job while your hook is still using it.

The legacy method

The original mechanism requires no configuration. In your job script:

1. Get the artifacts folder path from the LS_JOB_artifacts_path environment variable.
2. Create the folder in that location.
3. Copy or move your generated objects into that folder.

When the job completes, the contents of that folder are archived into a single file and uploaded to the LabScale service, where it can be downloaded to your desktop from the job page. Example code of how to do this within a job may look like the following:

import os

# First create the artifacts folder
artifactsDir = os.environ["LS_JOB_artifacts_path"]
os.mkdir(artifactsDir)

# Write a file into the folder
artifactPath = os.path.join(artifactsDir, "test-artifact.txt")
with open(artifactPath, "w") as f:
    f.write("example artifact")

# When the job completes, the agent will
# automatically archive and upload the
# artifacts to the backend.  Once there
# they can be downloaded from the job page.

Be aware of the limitations of this method when deciding whether to use it:

• Nothing is uploaded until the job has finished, so an interrupted job yields no artifacts at all.
• Any single file larger than 100MB is skipped when the archive is built. This is recorded as a warning in the agent log, but the job itself still reports success, so it is easy to miss.
• Everything arrives as one archive that must be downloaded and unpacked in full, even to retrieve a single file from it.

Where to find the artifacts

Once the Job has completed, if artifacts are placed in the correct location, the Job page will show an artifacts field that reveals a link to download the archived set of artifacts.