Introduction¶

This document describes the implementation of a prototype dashboard for the Science Quality Analysis Harness (SQUASH) system.

As stated in the SQUASH QA Database Design the verification data sets use case gives us the opportunity to leverage the QA tests done in the past with pipeQA and more recently with HSC, DECam and CFHT in a comprehensive infrastructure preserving the code and practices developed by the verification datasets group.

For QA-0, SQUASH run the QA tests as an afterburner an pushes the results to a QA dashboard. That is integrated to the CI system for checking the stability of the LSST software against the science requirements for single image processing (see LPM-17), also known as key performance metrics (KPMs).

The development will follow a rapid prototype workflow to reach this goal more efficiently. The goal for X16 was to have a minimal viable product (MVP) in production computing a few metrics for one dataset (chft). For F16 we extended the dashboard adding support to multiple datasets and implemented the Code Changes feature to monitor the commits that had impact on the KPMs.

The current implementation can be found at https://squash.lsst.codes/

We expect to get early feedback from users and iterate to improve the system. The main goal is to anticipate SQUASH needs for commissioning, and leverage the production SDQA system based on the experience of testing the LSST software on precursor datasets.

Selecting the technology stack¶

The selected technologies prioritize the use of Python as the main development language, a mature framework like Django DRF and the Bokeh plotting library to create interactive visualization.

We also are considering to follow Vega-lite specification for plot data.

The QA analysis is being refactored to use the Measurements API being developed by SQuaRE.

Another component called post_qa is responsible to shim the information from the CI run and push all the results to the QA dashboard.

For QA-1 and QA-2, we plan to add FITS image visualization using FFTools JS API or opensource third party tools like Aladin Lite + HiPS image format or visiOmatic + PTIF image format.

Other project visualization needs are also being considered.

Architecture¶

The architecture of the SQUASH dashboard is shown in Figure 1. The QA-0 DB is modeled based on the current LSST baseline database schema adapted for this prototype.

The implemented uses the object-relational mapper (ORM) built in the Django framework and exposed to the dashboard through a REST API.

This choice makes it easy to replace the QA-0 DB and Django REST API by WebServ and QAserv later on.

Figure 1 Main components of SQUASH dashboard prototype.

Implementation Epics¶

• SQUASH MVP for X16 (DM-5555)
• SQUASH extended MVP for F16 (DM-6196)
• SQUASH extended MVP II

The dashboard API¶

For QA-0, the databade model includes the Job, Measurement, Metric and VersionedPackage tables which are sufficient to characterize a metric and its measurement by the CI job and keep track of the version of all packages in the LSST software stack.

Figure 2 QA-0 database schema.

The metrics table is initialized with the values specified in the science requirements document LPM-17, example:

Figure 3 API endpoint for listing and creating metrics.

>>> import requests
>>> response = requests.get('http://localhost:8000/dashboard/api/')
>>> response.status_code
200
>>> api = response.json()
>>> api['metric']
'http://localhost:8000/api/metric/'
>>>
>>> metric = {
               "metric": "PA1",
               "description": "Photometric Repeatability",
               "units": "millimag",
               "minimum": 8,
               "design": 5,
               "stretch": 3,
               "user": 10,
               }
>>> response = requests.post(api['metric'], json=metric, auth=(TEST_USER, TEST_PASSWD))
>>> response.status_code
201

A job with a list of measurements and versioned packages can be inserted with a single request given the metric name, example:

>>> job = {
             "ci_id": "1",
             "ci_name": "validate_drp",
             "ci_dataset": "cfht",
             "ci_label": "centos-7",
             "ci_url": "https://ci.lsst.codes/job/validate_drp/1/",
             "status": 0,
             "measurements": [
                 {
                     "metric": "PA1",
                     "value": 1.0
                 },
                 {
                     "metric": "AM1",
                     "value": 2.0
                 },
                 {
                     "metric": "AM2",
                     "value": 3.0
                 }
             ],
             "packages": [
                 {
                     "name": "afw",
                     "git_url": "http://github.com/lsst/afw.git",
                     "git_commit": "a7aa450f60375473c010319e56db559457b43f9a",
                     "git_branch": "master",
                     "build_version": "b1"
                 }
             ]
           }

>>> response = requests.post(api['job'], json=job, auth=(TEST_USER, TEST_PASSWD))
>>> response.status_code
201

References¶

• LSE-63 Data Quality Assurrance Plan
• LPM-17 Science Requirements Document
• LDM-135: Database Design
• LSST baseline Database Schema
• HiPS
• Django REST Framework

APPENDIX A - Making of the squash project¶

In this appendix we document the initial setup to create the Django project (tickets/DM-5728) and its integration with the bokeh server (tickets/DM-5750).

$ django-admin.py startproject squash

$ python manage.py startapp dashboard

# Application definition

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'dashboard',
)

brew install mysql  (or mariadb)
mysql.server start
mysql -u root -e "CREATE DATABASE squash"

django-mysql==1.1.0
mysqlclient==1.3.9

INSTALLED_APPS = (
     ...
     'django_mysql',
 )

DATABASES = {
     'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME' : 'squash',
        'USER': 'root',
        'PASSWORD': '',
        'HOST': 'localhost',
        'PORT': '3306',
        'OPTIONS': {
              'charset': 'utf8mb4',
        },
        'TEST': {
            'CHARSET': 'utf8mb4',
            'COLLATION': 'utf8mb4_unicode_ci',
        },
    }

from django.db import models
from django_mysql.models import JSONField, Model

class Measurement(Model):
"""Store metric measurements associate to a job"""
    metric = models.ForeignKey(Metric, null=False)
    job = models.ForeignKey(Job, null=False, related_name='measurements')
    value = JSONField()

 def __float__(self):
     return self.value

measurement.value = "a string"
measurement.value = 1
measurement.value = 0.3
measurement.value = ["a", "list"]
measurement.value = {"a": "dict"}

# Check your settings first
python manage.py check

# The password created here is used to access the django admin interface
export TEST_USER=<user for the development database>
export TEST_USER_EMAIL="$TEST_USER@example.com"

python manage.py createsuperuser --username $TEST_USER --email $TEST_USER_EMAIL

# Implement the models
python manage.py makemigrations
python manage.py migrate

STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
    )

APPENDIX B - JSON schema for SQUASH¶

Adapted from https://community.lsst.org/t/json-schema-for-squash/777

The purpose of this schema is to consistently describe results from validate_drp so that they can be reliably consumed by the dashboard application and database. Having a schema means:

• Every new metric will be presented in a way that is consistent with the metrics already provided by validate_drp.
• Databases and HTTP APIs can be built to accept this schema. That is, the SQuaSH dashboard’s database schema and serialization code must understand and be compatible with this schema.
• The validate_drp output will be self-describing.

Our goal is to have validate_drp output a single JSON file that fully describes the QA run settings and measurements. Some of this information won’t be known directly by validate_drp, and will instead be inserted into the JSON by the testing harness. Ultimately this JSON data structure would be POST’d to the SQuaSH Dashboard API with a single HTTP request. Likewise, when an API consumer GETs a QA Job, this is the data structure that will be returned.

{
   "date": date time when the job was run (ISO 8601, e.g., 2016-05-13T18:27:53+00:00)
   "measurements": [{measurement document}, ...]
   "packages": [{ versioned package document}, ...]
   "ci_name": Jenkins CI job name.
   "ci_id": Jenkins CI job number
   "ci_url":  URL of build on Jenkins dashboard
   "status": Job status (0 = success, 1 = failure)
}

{
  "name": EUPS package name, e.g. "afw",
  "git_url": Git URL of package, e.g. "http://github.com/lsst/afw.git",
  "git_commit": Git commit SHA1 for package
  "git_branch": Git branch that commit resides on (e.g., 'master')
  "build_version": EUPS build version identifier for package
}

{
  "metric": metric slug (e.g., AM1)
  "metric_url": URL linking to this metric's definition in documentation
  "value": { datum with measurement's scalar value }
  "parameters": { blob with measurement code parameters }
  "blob": { blob with extra data about measurement; for plotting }
}

{
  ... blob datum fields
  "schema_id": "metric-blob-number"
}

{
   "value": numeric value (scalar or array)
   "units": astropy unit string (http://docs.astropy.org/en/v1.1.2/units/format.html)
   "label": label suitable for a plot axis (minus units)
   "description": a longer description
}