Collecting metrics with the LSST Verification Framework¶

The LSST Verification Framework (lsst.verify) is the DM framework for collecting metrics from the Science Pipelines. It aims to generalize the process of defining metrics and specifications, persist the results in verification jobs and dispatch them to SQuaSH (see SQR-019 for a demonstration of the LSST Verification Framework).

Currently, the following pipelines run in our Jenkins CI system:

• validate_drp
• ap_verify

Storing results in SQuaSH¶

The SQuaSH REST API is a web app implemented in Flask for managing the verification jobs.

When a verification job is sent to the SQuaSH REST API, context information like metrics definition and specifications, execution environment metadata, etc are stored in the SQuaSH context database. The SQuaSH context database is a relational database using MySQL (see The SQuaSH context database).

The metric values and associated metadata (time series data) are stored primarily in InfluxDB. The resulting data is exposed through Chronograf, that is used for querying and visualizing the time series data.

Time-series visualization with Chronograf¶

The SQuaSH data model is presented to the users through the Chronograf UI. From the Chronograf UI, the user can query the metric values, the associated metadata, aggregate results and present them in interactive dashboards.

Figure 2 The figure shows how the SQuaSH data model is presented to the user through the Chronograf UI. This particular query displays a time series for the validate_drp.AM1 metric, selecting the validation_data_hsc dataset and grouping the results by filter.

Figure 3 The same query as shown in the previous figure, but now adding the run_id and visualizing the results in a table view.

Alert rules and notifications with Kapacitor¶

Chronograf is also the UI for Kapacitor, a native data processing engine that can process both stream and batch data from InfluxDB. Although, the user can create alerts through the UI the goal is to create alert rules programmatically from the metric specifications stored in SQuaSH.

Supporting multiple execution environments¶

To be generally useful for the verification activities, SQuaSH must support multiple execution environments.

Examples of metadata for different execution environments:

• Jenkins CI

  • Look up key: ID of the CI run
  • Environment metadata: ci_id, ci_name, ci_dataset, ci_url, stack packages.

• LSST Data Facility (LDF)

  • Look up key: ID of the pipeline run.
  • Environment metadata: run ID, pipeline name, pipeline configuration, butler repository

The SQuaSH API provides a generic resource to interact with verification jobs, /job/<id>, and specific resources to interact with runs on different execution environments. A run may contain results of multiple verification jobs. For example a GET request to /jenkins/<run_id> or to /local/ldf/<run_id> will retrieve the corresponding jobs.

The SQuaSH context database¶

We adopted a relational database for the SQuaSH context database. The main motivations for this choice is to deploy the SQuaSH context database at the US DF consolidated database (Postres), and to access the SQuaSH metrics through a common TAP interface.

In its current deployment, SQuaSH uses a MySQL 5.7 instance in Google Cloud SQL. MySQL 5.7 offers support to JSON data types which are used to make the database schema more flexible. We store verification job metadata, environment metadata as well as metric definitions and specifications as JSON data types.

Current SQuaSH context database:

• Entities:

  • env, user, job, package, blob, measurement, metric, spec

• Relationships:

  • 1 env : N jobs
  • 1 job : N packages
  • 1 job : N measurements
  • M measurements : N data blobs
  • 1 metric : N specs
  • 1 metric : N measurements

Figure 4 The figure shows the relational schema for the SQuaSH context database.

In the Google platform deployment, the Cloud SQL manages the backups of the SQuaSH context database.

Acronyms¶