Developer Documentation

Quickstart

This project uses Docker containers for development and testing, and make to provide a consistent UI.

Build a new Docker image and execute the test suite with:

make test

Launch an interactive shell inside a container, with your workspace visible inside the container, with:

make interactive

To list all available targets, execute make without any arguments, e.g.,

equuleus:cdm-shared-library $ make
build                          build the application image
down                           stop develop/test environment and any interactive session
help                           show this help.
interactive                    start an interactive session using the project image (caution: R/W mounts source directory to /app)
lint                           lint the application (static code analysis)
piplock                        overwrite Pipfile.lock with the image version
pull                           download the application image
push                           push the image to the Docker registry
test                           test the application
up                             start develop/test environment

Background

SKA Tango devices have commands that accept structured arguments and/or return structured responses. These structured data are often expressed as JSON-formatted strings.

The Configuration Data Model (CDM) is a data model used to describe subarray resource allocations and the subsequent configuration of those resources. It is effectively the superset of the configurations used by receptors, correlators, and data processing systems. The CDM is one such example of structured data delivered to TMC Tango devices.

This project defines object representations of the structured data passed to and from Tango devices, and serialisation schema used to convert the structured data to and from JSON. This project defines:

  1. a Python object model of the CDM;
  2. a Python object model for the structured arguments sent to TMC Tango devices and the structured responses received in return;
  3. serialisation schema to convert the Python object model instances to and from JSON.
  4. validation of the JSON strings sent between devices are compliant with the agreed interfaces.

The primary users of this shared library are the OET, SubArrayNode, and CentralNode. The OET uses this library to construct object representations of telescope configurations and resource allocation instructions, to convert those object representations to JSON-formatted payloads for TMC devices, and finally to convert the JSON responses returned by TMC devices back into Python objects.

It is intended that TMC devices also use this library to guarantee correct data exchange with the OET. TMC can also use this library to marshall and unmarshall its arguments to CSP and SDP Tango devices, which accept the appropriate subset of the JSON.

Project layout

The CDM project contains three top-level packages, ska.cdm.messages, ska.cdm.schemas and ska.cdm.jsonschema as shown in the figure below. The ska.cdm.messages package contains Python object models for the JSON command arguments agreed in the ICDs. The ska.cdm.schemas package contains code to transform the classes defined in ska.cdm.messages to and from JSON. The ska.cdm.jsonschema package contains code to verify that the JSON strings sent between devices are compliant with the agreed interfaces.

project layout

Project layout and naming conventions.

The project layout and naming conventions are:

  • Each Tango device has a corresponding Python sub-package in ska.cdm.messages and ska.cdm.schemas.
  • Code and schema for each Tango device command are located in Python modules inside their respective package.
  • Structured input for the Tango command is modelled by a Request object.
  • Structured output from the command is modelled by a Response object.
  • Marshmallow schema are created to transform Python Request and Response instances to an from JSON, along with any other content they contain.

Messages

The Python object model for the JSON defined in the ICD is located in the ska.cdm.messages package. In general, each CDM JSON entity is represented as a Python class and each CDM attribute presented as a class property.

CDM attributes can be typed as plain Python data types (strings, floats, etc.) or, where appropriate, represented by rich objects if this provides additional value to the client. For example, while astronomical coordinates are represented by floats and strings in the JSON schema, in the object model they are defined as Astropy SkyCoord instances to ensure ensure correct coordinate handling and permit easier manipulation downstream. Similarly, quantities with units could be defined as instances of Astropy Quantity to provide additional functionality.

For details on the device messages modelled by this library, see:

Marshmallow Schemas

Classes to marshall the ska.cdm.messages objects to and from JSON are defined in the ska.cdm.schemas package. The cdm-shared-library project uses Marshmallow for JSON serialisation. Classes in the ska.cdm.schemas define Marshmallow schemas which are used by Marshmallow during JSON conversion.

CentralNode schema

Schema mapping for objects used to communicate with TMC CentralNode device.

SubArrayNode schema

Schema mapping for objects used to communicate with TMC SubArrayNode device.

MCCSController schema

Schema mapping for objects used to communicate with MCCSController device.

MCCSSubArray schema

Schema mapping for objects used to communicate with MCCSSubarray device.

JSON Schemas

The CDM library uses the SKA Telescope Model to ensure the JSON accepted and JSON generated by the library are compliant with the schema declared by the data.

The entry points for code handling JSON schema validation is located in the ska.cdm.jsonschema module. This module contains methods for fetching version-specific JSON schemas using interface URI and validating the structure of JSON against these schemas. Json Schema validation functionality is enabled by default with the parameter validate=True when converting a JSON string to CDM using ska.cdm.schemas.CODEC.loads() and when converting CDM to a JSON string using ska.cdm.schemas.CODEC.dumps().

JSON schema Validation

Extending the CDM

Additional devices and applications cay use this library to communicate CDM elements wherever useful. Developers are encouraged to extend the cdm-shared-library project, adding object models and schemas for the structured arguments for their Tango devices.

The steps to extend the CDM are:

  1. Create a new package for the Tango device in ska.cdm.messages.
  2. For each device command, create a new module in the new package.
  3. If the command accepts structured input, define a Request class in the module.
  4. If the command returns a structured response, define a Response class in the module.
  5. With the Python object model defined, create a corresponding package and module structure in ska.cdm.schemas.
  6. In the schema module, define Marshmallow schemas to convert the object model classes and any structure to JSON.
  7. If this is a major entity, register the schema with the ska.cdm.schemas.CODEC object using the @CODEC.register_mapping decorator.