Migration Center Pipeline Overview

The Migration Center pipeline is the process of using the Migration Center APIs and client-developed software to take ANS documents temporarily stored in Arc XP's system, validate them against the Arc XP ANS Schema, and move them to be permanently part of an appropriate Arc XP application for the ANS content type.

Restated, the Migration Center pipeline is for moving content from a source CMS into Arc XP. The parts of the pipeline processes include:

We refer to the software an organization develops that collects source content from a legacy CMS, transforms it into ANS, and directs the transformed ANS into the Arc XP applications as an Adapter. Before you build an adapter, you will go through an extensive content mapping exercise. At the end of content mapping, you will have a list, a spreadsheet, a matrix, or some document that correlates the legacy CMS’s source data fields with the ANS fields they will populate. The content mapping exercise will also identify complex content that requires more in-depth solution designs to plan how these can be stored properly using the fields ANS offers. The adapter will follow this mapping when transforming the legacy content into ANS. The adapter will also include processes to push ANS to Arc XP using Migration Center APIs and query for reports on the process of the ANS sent through the pipeline. 

Archiving source content from a legacy CMS creates an inventory of the content in the Migration Center pipeline.  You can then access the stored copy in the pipeline for use in the ANS transformation you develop. However, archiving is optional. If you choose not to archive in the pipeline, you would query the legacy CMS directly to transform the source content to ANS. If you expect to lose access to your old CMS before your migration to Arc XP is fully complete, archiving source content to Migration Center would offer you flexibility to take more time to develop your transformation process and the adapter software to perform the migration. 

Once you map your source content to ANS fields, you will encode the mappings into software, called an adapter, that will apply transformations to the source content such that the result is a new document in Arc XP ANS format.  Your adapter will collect source content from the legacy CMS, or from the Migration Center inventory, and transform the source content to ANS. The adapter will need at least one transform method per Arc XP Content type, and depending on the outcome of the content mapping exercise multiple methods may also be called for.

Your transformed ANS Content must be able to pass validation against the Arc Native Specification (ANS Schema). Migration Center will test the ANS when the ANS ingestion endpoints are used.  Only successfully tested ANS will be moved through the pipeline.

After your adapter has transformed the legacy source content into ANS, it will send it to the pipeline through the POST /ans endpoint, which accepts a JSON object.  We will refer to this as a Migration Center JSON object:

{
  “ANS": {},           // the ANS object
  “circulations": [], // a list of circulations, only used for the story ANS type 
  “operations": [],  // any scheduled operations, only used for the story ANS type
  // special pipeline instructions to direct aspects of the ingestion process, broken down by ANS type
  “arcAdditionalProperties": { 
    story: {},
    video: {}
  }
}

The ANS key of the Migration Center JSON object must contain full ANS object representing your piece of content.  Every time you post ANS to Migration Center, it must contain all of the ANS fields the object needs to use in their entirety. You can post the same ANS to Migration Center many times, however you can not post a partial ANS object. If you want to post the same object a second time, but only change one key of the ANS, you will change that key but still post the entire ANS to the endpoint.

The circulations key of the Migration Center JSON object contains the data describing a story’s website, section and URL information. Migration Center takes the circulation data in the circulations key and will send it to Draft API. The circulations key can be omitted from the Migration Center JSON if the ANS data is not of type: story.

The operations key holds the object that will creates story publishing operations at a future date. For example, you may need to ingest a story to be unpublished or deleted at a specific date in the future. Migration Center will use this information to communicate to the Content Operations API. This key is not used to indicate that a migrated story is already published

The arcAdditionalProperties key holds commands that Migration Center will use to modify the requests it makes to other Arc XP APIS. For example, to publish a story when it is created, you add a specific key arcAdditionalproperties.story.publish: true. 

Migration Center will validate the ANS against the ANS schema. If validation passes, the pipeline handles the rest of the process necessary to send the ANS into the appropriate Arc XP application.

Once content is transformed and sent through the pipeline to make its way into the Arc XP applications, it will go through several different pipeline statuses. You find out what status an item of ANS or a group of ANS items have in the pipeline by querying the Reporting API. 

When a piece of content has a fail status, more details about the error can be found in the error_message field that is returned as part of the Migration Center Reporting API. 

The Migration Center Reporting API can be used to understand the successes and failures happening in the pipeline. The reporting endpoints can provide a count of the number of records in a particular status, query for a list of matching records matching a filter value, query for a list of records sent to the pipeline on a date, or return detailed information on a specific record in the pipeline found by querying for its ANS id, content type, or original source id.

Migration Center makes the following SLA commitments for the APIs.

*Migration Center JSON containing story ANS is picked up off of the queue at this rate unless your organization has acquired a higher Draft API rate limit. If Arc XP acquired a higher Draft API rate limit for your organization, that higher rate limit can be communicated to Migration Center and results in more story ingestions per minute.