Collecting metrics with the LSST Verification Framework¶

The LSST Verification Framework (lsst.verify) is the DM framework for collecting metrics from the Science Pipelines. It aims to generalize the process of defining metrics and specifications, persist the results in verification jobs and send them to SQuaSH (see SQR-019 for a demonstration).

Currently, the following pipelines run in our Jenkins CI system:

• validate_drp
• ap_verify

Storing results in SQuaSH¶

The SQuaSH REST API is a web app implemented in Flask for managing the SQuaSH metrics dashboard.

When a verification job is sent to the SQuaSH API, the context information like metrics definition and specifications, execution environment metadata, etc are stored in the context database. The SQuaSH context database is a relational database using MySQL (see The SQuaSH context database).

The metric values and associated metadata (the time-series data) are stored primarily in a time-series database using InfluxDB. The resulting data model is exposed through the Chronograf UI, and is used for querying the data and visualizing the time series.

Time-series visualization with Chronograf¶

The SQuaSH data model is presented to the users through the UI which is based on Chronograf. From the Chronograf UI, the user can query the metric values, the associated metadata, aggregate results and present them in interactive dashboards.

Figure 2 The figure shows how the SQuaSH data model is presented to the user through the Chronograf UI. This particular query displays a time series for the validate_drp.AM1 metric, selecting the validation_data_hsc dataset and grouping the results by filter.

Figure 3 The same query as shown in the previous figure, but now adding the run_id and visualizing the results in a table view.

Alert rules and notifications with Kapacitor¶

Chronograf is also the user interface for Kapacitor, a native data processing engine that can process both stream and batch data from InfluxDB. Although, the user can create alerts through the UI the goal is to create alert rules programmatically from the metric specifications stored in SQuaSH.

Using SQuaSH with the LSST Science Platform¶

Supporting multiple execution environments¶

To be generally useful for the verification activities, SQuaSH must support multiple execution environments.

Examples of metadata for different execution environments:

• Jenkins CI

  • Look up key: ID of the CI run
  • Environment metadata: ci_id, ci_name, ci_dataset, ci_url, stack packages.

• LSST Data Facility (LDF)

  • Look up key: ID of the pipeline run.
  • Environment metadata: run ID, pipeline name, pipeline configuration, butler repository

• User local environment

  • Look up key: ID of the user run
  • Environment metadata: run ID

The SQuaSH API provides a generic resource to interact with verification jobs, /job/<id>, and specific resources to interact with runs on different execution environments. A run may contain results of multiple verification jobs. For example a GET request to /jenkins/<run_id> or to /local/<username>/<run_id> will retrieve the corresponding jobs.

The SQuaSH context database¶

We adopted a relational database for the SQuaSH context database. The motivation for this choice is mainly for the deploy of the SQuaSH context database to the LSST consolidated database, and the common TAP interface to access the SQuaSH metrics.

In its current deployment, SQuaSH uses a MySQL 5.7 instance in Google Cloud SQL. MySQL 5.7 offers support to JSON data types which are used to make the database schema more flexible. We store verification job metadata, environment metadata as well as metric definitions and specifications as JSON data types.

Current SQuaSH context database:

• Entities:

  • env, user, job, package, blob, measurement, metric, spec

• Relationships:

  • 1 env : N jobs
  • 1 job : N packages
  • 1 job : N measurements
  • M measurements : N data blobs
  • 1 metric : N specs
  • 1 metric : N measurements

Figure 4 The figure shows the relational schema for the SQuaSH context database.

In the Google platform deployment, the Cloud SQL manages the backups of the SQuaSH context database.

Acronyms¶