Skip to main content

Run the Services

This section includes information to run the LabScale services.

Run the Main Services​

The LabScale main stack includes

  • LabScale services
    • Webapp
    • Auth Server
    • Messenger
    • Scheduler
  • Third party services
    • Postgres: Relational database
    • ClickHouse: Column-oriented database for metrics
    • EMQX: MQTT broker
    • Nginx: Reverse proxy
    • Loki: Log aggregation
Note

If using an external Postgresql database, be sure it is version 16.6 or higher.

To start the main stack, take the following steps:

  1. Check out the deployment artifacts from the repository labscale-release 
    git clone git@bitbucket.org:labscale/labscale-release.git
    git checkout <release_branch>
  2. Create <repo>/config/labscale/run-config.sh based on the template <repo>/config/labscale/run-config-template.sh.
    • Set the location of SSL certificate file and key file, which are obtained in the Setup process.
    • Set EXTERNAL_DATABASE to false for using the Postgres inside the docker stack, or true to use an external Postgres database.
  3. Configure the manual secrets and credentials.
    • Put the aws_secret_access_key of the AWS IAM smtp-user to $HOME/.labscale/secrets/smtp-pwd
    • (Only for external database) Put the password to $HOME/.labscale/secrets/postgres-pwd
  4. Run the initialization script. The script will automatically generates some of the secrets/credentials for the services and install the Loki Docker driver client. 
    <repo>/scripts/initialize.sh
  5. Create LabScale service configs
    • Webapp: Create <repo>/config/labscale/webapp.env based on the template <repo>/config/labscale/webapp-template.env. Most of the environment variables work out-of-the-box except for the ones with value <TO_BE_SET>:
      • NEXTAUTH_URL: URL of the Web UI, e.g., "https://app.labscale.com"
      • BROKER_URL: URL of the broker, e.g., "tls://broker.labscale.com:8883"
      • SMTP_USER: aws_access_key_id of the smtp-user
      • EMAIL_FROM: From field of the emails sent from the Webapp The following variables might need to be updated for some advance users:
      • (Only for external database) DB_USER: username of the database user
      • (AWS) REGION: Change it if your services are not hosted in us-west-2
      • (AWS) SMTP_HOST: Change it if the REGION is not in us-west-2
      • Note that LABSCALE_BUCKET should be kept as files.labscale.com. This is the LabScale owned bucket that hosts the initial example job packages, which will be copied to the team specific bucket during team creation.
    • Backend: Create <repo>/config/labscale/backend.yaml based on the template <repo>/config/labscale/backend-template.yaml. Most of the environment variables work out-of-the-box except for the ones with value <TO_BE_SET>:
      • webapp.domain: Webapp URL that will be used to construct the Slack notification. e.g. app.labscale.com
      • Note that storage.s3_file_bucket and storage.s3_file_bucket_url should be kept as is.They point to the LabScale owned S3 bucket for getting the agent binaries.
  6. Start the services. 
    <repo>/scripts/run-labscale.sh up
  7. (Only for self-hosted Postgres) Update Postgres DB password and update the secret file in the local file system.
    • The initial username and password is labscale:changeme, update the password through SQL query using PGAdmin or psql 
      ALTER USER labscale WITH PASSWORD 'xxx';
    • Update the new password to $HOME/.labscale/secrets/postgres-pwd
  8. Update EMQX dashboard password in https://<domain>/emqx
    • The initial username and password is admin:public, login and the dashboard will prompt for setting a new password.
  9. (Optional) Disk Cleanup
    • Create a Cron job for docker image clean up: 0 0 * * 0 docker system prune -af
  10. (Optional) Data backup
    • Until the LabScale backup tool is available, the manual ways to back up the data are:

To take down the service (data will be persisted in the docker named volume), run

<repo>/scripts/run-labscale.sh down

Run the Monitoring Services​

The monitoring stack provides metrics, dashboards, alerts, etc for the LabScale stack, it includes

  • Prometheus: Metrics database
  • Grafana: Metrics visualization
  • cAdvisor: Containers metrics collector
  • Node Exporter: Host metrics collector
  • Nginx Prometheus Exporter: Nginx metric collector
  • Postgres Exporter: Postgres metric collector

To start the monitoring stack, take the following steps:

  1. Start the services. 
    <repo>/scripts/run-monitor.sh up
  2. Configure Grafana: https://<domain>/grafana
    • Update dashboard user password, the initial username and password is labscale:changeme
    • Set up ClickHouse data source
      • Server address: host.docker.internal
      • Server port: 9000
      • Username: default
      • Password: Check $HOME/.labscale/secrets/clickhouse-pwd
    • Add alert contact points and update notification policies, e.g. set up Slack using webhook

Update the Stacks​

To update the stacks when there are new releases available, checkout new artifacts and run the start scripts.

cd <repo>
git pull
git checkout <new_release_branch>
<repo>/scripts/run-labscale.sh up
<repo>/scripts/run-monitor.sh up