Extensible Data Sources (XDS) walk-through

A narrative walk-through of the XDS capability

Define the Schema Attributes - The first step is craft an XDS Schema to adequately describe our deployment domain space and the events we expect to send to Orchestrate. The submission of the data type schema is itself governed by an API, which is documented at XDS Schema API.

1. API version The version number of the XDS Schema API (not your new schema's version number).
2. Source association key This value is obtained by creating a new "XDS Source" in Orchestrate's web client. Supplying a source association key is optional during schema registration. If provided, the source will be associated with the schema when the schema is received. Otherwise, the XDS source will be associated to a registered schema when the first activity message is received. There is no "preferred" option; this flexibility exists purely to suit various types of integration models.
3. Schema name Should be a human-friendly identifier of the data source.
4. Schema ID Takes a UUID as the value and the value should be unique per schema. If UUID generation is not possible, characters are limited to alphanumerics and the dash symbol.
5. Schema version Allows for the same schema to be updated without sending a separate schema ID. This is an integer field, best practice is to start at 1 and increment.
6. Event time The time at which your newly crafted schema is submitted to Orchestrate. Requires UTC RFC3339 format.

   The schema ID is user-defined; we suggest a long "UUID" style string. The schema name is also user-defined. In our deploy example, we can add a schema ID and name as follows:

   // Example Deploy XDS Schema
   {
     "api_version": "1",
     "source_association_key" : "8578c900-f8df-0131-84ff-3c07547a48b0",
     "custom_schema": {
        "name": "Faux Schema",
        "schema_id": "fc358dd0-11dc-0132-b9fe-3c07547a48b0",
        "schema_version": 2,
        "event_time": "2015-02-02T17:15:32.320Z",
      }
   }

   Note, the source association key value was obtained from our instance of Orchestrate. It is an attribute of the parent XDS Collection with which you intend to register the new schema.

Defining the fields expected by your schema - Since you are defining an API at this stage, the next step is to define the list of fields expected by your API. Fields are the attributes you wish to supply with each event message sent by an adapter. We'll address optional vs. required fields in the next section, but for now we want a comprehensive list of all fields represented in your new activity message format. For instance, in our deploy example you may want to supply deploy environment, time, duration of deploy activities, etc.

The XDS Activity API categorizes fields into three classes: mandatory fields, stock fields, and custom fields. "Mandatory fields" are required in all registered schemas and thus must be present in all activity messages. "Stock fields" are a set of non-mandatory fields that have special relevance to the Orchestrate application, such as appearing in specific places in the user interface. "Custom fields" are truly arbitrary and may be used to capture desired data not encompassed by mandatory or stock fields.

7. Mandatory fields - all of these fields are required in every XDS schema.
   1. schema_id Activity messages will need to send this to identify themselves as governed by a specific schema.
   2. schema_version Should match the same schema_version as sent by the schema. This makes sure that the message is displayed with the correct version of the schema.
   3. event_time The UTC RFC3339 time of the event.
   4. remote_id Represents the source tool's identifier for the activity. For instance, if we were modeling Subversion commits, the remote_id would be the revision number. Messages with the same remote_id are treated as part of the same activity and are aggregated as "sub-events".
8. Stock fields - Optional fields with special relevance in the Orchestrate application.
   1. associated_remote_id If your activity has an association this field is used to specify one or more IDs for purposes of association.
   2. created_by The user who instantiated the activity.
   3. summary The primary heading or title of the activity.
   4. description A detailed description of the activity.
   5. status_name The current status of the activity, corresponds to the "status_type" field.
   6. status_type The current status class, either "open", "success", or "fail" each of which maps to status icons displayed in the user interface.
   7. link_to Creates a link back to the activity on the source system. The link_to field can take an array with a link label along with the URL to customize the display text in the user interface. The default label is "See More". Following our example, one could display "Deployment Details" in the user interface in place of the default "See More".

      Let's return to our deployment example and see how fields are defined in the Data Type Schema:

      // Example Deploy XDS Schema
      {
        "api_version": "1",
        "source_association_key" : "8578c900-f8df-0131-84ff-3c07547a48b0",
        "custom_schema": {
           "name": "Faux Schema",
           "schema_id": "fc358dd0-11dc-0132-b9fe-3c07547a48b0",
           "schema_version": 2,
           "event_time": "2015-02-02T17:15:32.320Z",
           "fields": ["schema_id", "schema_version", "event_time", "remote_id", 
             "associated_remote_id", "summary", "status_name", "status_type", 
             "link_to", "rule", "deploy_url", "deploy_message"],
         }
      }

      Note that we've included the four mandatory fields, a couple of "stock fields", and some custom fields modeling our "deploy" specific domain data.

Requiring specific fields - You may impose required fields. Remember that some fields are always required, so this section is not optional for that reason. But you can extend the set of required fields beyond those required by the API.

9. required Pass in an array of required fields. Any activity message without all of these fields will be discarded.

   Let's return to our deploy example and see how fields are defined in the Data Type Schema:

   // Example Deploy XDS Schema
   {
     "api_version": "1",
     "source_association_key" : "8578c900-f8df-0131-84ff-3c07547a48b0",
     "custom_schema": {
        "name": "Faux Schema",
        "schema_id": "fc358dd0-11dc-0132-b9fe-3c07547a48b0",
        "schema_version": 2,
        "event_time": "2015-02-02T17:15:32.320Z",
        "fields": ["schema_id", "schema_version", "event_time", "remote_id", 
          "associated_remote_id", "summary", "status_name", "status_type", 
          "link_to", "rule", "deploy_url", "deploy_message"], 
        "required": ["schema_id", "schema_version", "event_time", "remote_id", "rule"],
      }
   }

   The four fields required by the API are represented, along with one additional field we added above named "rule". Any activity message that fails to supply all five of these fields will be discarded.

Select "important" fields that result in activity stream posts - Not every event that triggers an activity message is necessarily worthy of a post in the activity stream. List only those fields that you deem worth of an activity stream update in the "important" field.

10. important Array of field names. Only changes to these fields will trigger a new entry in the activity stream.

    In this case, we only want changes to the "summary" or "status_name" field to trigger activity stream posts:

    // Example Deploy XDS Schema
    {
      "api_version": "1",
      "source_association_key" : "8578c900-f8df-0131-84ff-3c07547a48b0",
      "custom_schema": {
         "name": "Faux Schema",
         "schema_id": "fc358dd0-11dc-0132-b9fe-3c07547a48b0",
         "schema_version": 2,
         "event_time": "2015-02-02T17:15:32.320Z",
         "fields": ["schema_id", "schema_version", "event_time", "remote_id", 
           "associated_remote_id", "summary", "status_name", "status_type", 
           "link_to", "rule", "deploy_url", "deploy_message"], 
         "required": ["schema_id", "schema_version", "event_time", "remote_id", "rule"],
         "important": ["summary", "status_name"],
       }
    }

Parse certain fields for TeamForge object references - Say you have a field that often contains a reference to a tracker artifact and you want Orchestrate to automatically convert those into links. An association will also be created if Orchestrate is aware of the referenced object.

In addition, any URL reference will be converted to a HTML link in the Orchestrate user interface. Say, www.example.com is referenced in a "parse" field, then it will be converted to a HTML link, e.g. www.example.com

11. parse Array of field names. Converts any TeamForge object references to links, converts URL references to links.

    // Example Deploy XDS Schema
    {
      "api_version": "1",
      "source_association_key" : "8578c900-f8df-0131-84ff-3c07547a48b0",
      "custom_schema": {
         "name": "Faux Schema",
         "schema_id": "fc358dd0-11dc-0132-b9fe-3c07547a48b0",
         "schema_version": 2,
         "event_time": "2015-02-02T17:15:32.320Z",
         "fields": ["schema_id", "schema_version", "event_time", "remote_id", 
           "associated_remote_id", "summary", "status_name", "status_type", 
           "link_to", "rule", "deploy_url", "deploy_message"], 
         "required": ["schema_id", "schema_version", "event_time", "remote_id", "rule"],
         "important": ["summary", "status_name"],
         "parse": ["deploy_message"],
      }
    }

    By adding "deploy_mesage" to the "parsed" field, any CTF object will be linked and possibly associated. Any web addresses referenced will also be converted to a full HTML link.

Registered schemas may be used site-wide. In our deploy example, perhaps we've written an extension to our deploy product and have registered its schema. The same schema may be reused by multiple sources (e.g., multiple instances of the deploy product) across the site.

Note also that successive submissions of the same schema ID and version will be ignored and it is a safe practice to send the same schema multiple times if it is programmatically convenient to do so.

To modify a schema, increment the schema_version value and re-register the schema with Orchestrate using the methods described above.

12. Success You can use the XDS Schema HTTP API (see /api/1/docs/http_api_xds_schema) to inspect registered schemas and their versions.
13. Troubleshooting Check the server log "custom_queue.log" to see if the schema was rejected and why it was rejected.

Send Activity messages - Now that you've defined an API for your tool/domain you can start supplying activity messages to Orchestrate that coincide with your new schema. For a technical reference on crafting activity messages, see the XDS Schema API documentation.

14. Submit XDS Activity messages to orchestrate.custom queue

    Note that the activity stream will only reflect new entries when an "important" field is modified or set in an activity message.

    // Example Deploy XDS Activity Message
    {
      "api_version": "1",
      "source_association_key" : "8578c900-f8df-0131-84ff-3c07547a48b0",
      "custom_data": {
        //required fields
        "name": "Faux Schema",
        "schema_id": "fc358dd0-11dc-0132-b9fe-3c07547a48b0",
        "schema_version": 2,
        "event_time": "2015-02-02T17:15:32.320Z",
        "remote_id": "deploy-1234",
        "rule": "1 comes before 2",
        // stock fields
        "summary": "Deploy at 4:52pm on a Friday",
        "deploy_message": "Deploying to fix [artf12345] and [artf12346] as critical bugs.",
        "associated_remote_id": "b4c8e50c908ade5c6f053e1ef6422f88",
        "link_to": ["Deploy Details", "http://www.example.com/deploy/deploy-1234"],
        "status_type": "open",
        // other custom fields
        "environment": "Dev Stage 1"
      }
    }

Optional Features - The XDS API offers some optional features to help pretty things up and make life a bit easier.

15. Interpolate TeamForge URLs

    It may be desirable to create URL references to TeamForge pages or functions (like SSO authentication). There are two interpolation functions in XDS:

    Reference TeamForge base URLs: %CTF_BASE_URL%

    Reference TeamForge "Go" URLs: %CTF_GO_URL%

    In our example, we may wish to reference a TeamForge Go URL like so:

    // Example Deploy XDS Activity Message
    ...
    "summary": "Deploy at 4:52pm on a Friday. See %CTF_GO_URL%/page1001",

16. Control field visibility Say you wanted to load specific data into Orchestrate's database for reporting purposes, but you don't want to expose all of those fields in the user interface. The solution is to add those fields to the hidden array in the XDS Schema message. The hidden array takes only custom fields (optional, non-stock). Custom fields are visible by default.

    Continuing our example, the newly introduced "environment" and "authorized_by" custom fields will be stored in the Orchestrate database, but not exposed in the user interface:

    // Example Deploy XDS Schema
    {
      "api_version": "1",
      "source_association_key" : "8578c900-f8df-0131-84ff-3c07547a48b0",
      "custom_schema": {
         "name": "Faux Schema",
         "schema_id": "fc358dd0-11dc-0132-b9fe-3c07547a48b0",
         "schema_version": 2,
         "event_time": "2015-02-02T17:15:32.320Z",
         "fields": ["schema_id", "schema_version", "event_time", "remote_id", "associated_remote_id", 
           "summary", "status_name", "status_type", "rule", "deploy_url", "deploy_message",
           "environment", "authorized_by"],
         "required": ["schema_id", "schema_version", "event_time", "remote_id", "rule"],
         "important": ["summary", "status_name"],
         "parse": ["deploy_message"],
         "link_to": ["deploy_url"],
         "hidden": ["environment","authorized_by"]
       }
    }