How to Ingest Wire Content Using an Inbound Wires Adapter

Wires are brought into Arc XP through an adapter, very much like when clients migrate content from an old CMS. The steps involved are:

The questions you’ll want to answer about the wire are:

You will require sample content of the wire’s source material for mapping. Save a sample response for each product type that returns a unique source schema into a file for use in development and testing. Ideally try to gather as many unique samples as possible, for example if sometimes stories have images and sometimes they do not, collect at least one of each. Things to consider when planning your mapping code:

You will need to map each field from the wire to ANS, but only those fields that are needed to create a valid ANS document. You will require sample content of the wire’s source material for mapping. You may also find it useful to have some examples of ANS content, so you have examples of the target. To get samples of ANS content, create some stories in Composer that look like what you expect to see out of an ingested wire. Then use the APIs and query for those examples.

Create a table where you list out each of the wire’s fields and to which ANS fields the data within them should be saved. 

Be aware there are some fields in ANS that are inserted automatically by Content API, so not every ANS field that will be in the Composer samples needs to be in the ANS you post.  Focus on mapping source values and things that are unique to the organization’s publishing requirements or publishing workflow, and ANS fields that have been identified as specifically used by the front end implementation of the website.

Below is a representative, though incomplete, example of what the mapping table might look like were you planning to ingest content from the Associated Press.

The form of the adapter is difficult to dictate. How complex a form it takes depends on clients’ priorities of scale, cost and ease of maintenance. Clients could write a python (or server side engine of your choice) application to do this, and run it off a physical server. They could use AWS services and engineer a combination of Serverless Framework, Lambda, Dynamo DB, Parameter Store, API Gateway and Step Functions to break the application into micro services and deploy them to operate in the cloud.

This process of ingesting wire content into Arc XP is composed of the same steps as when writing an adapter to migrate content from a historical CMS into Arc XP. The shape the units code will take will be highly determined by the methods and costs with which the development team are comfortable. What follows is a high level treatment of the processes an adapter for importing wire content either “will need” or “may find useful” to include.

Clients will need to store and retrieve sensitive values used to access the provider's API.  It is not advisable to do this in the adapter code. This “store and retrieve” could be by hard coding the credentials into an environment file that is stored locally on the server where the adapter runs, or it could be kept in a parameter store service that is accessible through an HTTP request.

Clients will need to access the wire content. They may do this by making HTTP requests to the wire endpoint. Perhaps they will know the id of a single wire story and fetch just that piece of content from an endpoint. Or, perhaps they will access a large feed of wire stories, and will need to first retrieve the group of story ids and then loop through to process every story in the group into Arc. If the wire content has links to additional content, the adapter may need to make additional requests to fetch the content of those references for processing.

The adapter should process a wire story once. If the wire has been verifiably updated since it was previously fetched, then it can be updated. Devising ways to keep track of new content versus updates versus repeated content that can be ignored is tricky. But this should be worked out, or the consequences include:

You may want to keep an inventory database of the wire content that you have already processed. During the retrieval process you would check this inventory first to decide if further work is even necessary. If you want to avoid unnecessary processing but still allow for processing updates to existing wire content, you may want to store in the inventory database a SHA-1 hash of the wire content that has already been processed. If a SHA-1 hash is available, you can query the Arc XP version of the wire content, process it into a SHA-1 hash and compare it to your inventory’s stored copy. If the SHA-1 hashes are different, then the adapter could proceed to the next step in the process.

The adapter will need a process that reforms the wire content from its source into ANS. Clients will take the mapping document you created and turn it into a method, class, lambda, or other unit of work that will take as input the wire content source and return a JSON document that is valid ANS, or possibly a Migration Center object (which also contains ANS).

The wire will provide story content and so you will need a converter that creates valid story ANS. If the wire provides photo content as well, then clients will need a converter that creates valid photo ANS, and if the wire provides video content then clients will need a converter that creates valid video ANS.

They will also need to prepare a separate circulation document for the story. They will need to have decided where the wire content will live in your website hierarchy. If the client organization has multiple websites, they will need to take into account in which websites and section(s) the wire content will live.

Clients will have one or more units of ANS to be sent to the appropriate Arc application. They will create multiple HTTP requests to to send the story and associated references to Draft API, Photo Center API, and Video Center API, or possibly instead a single HTTP request to Migration Center API should you decide to pay the extra costs associated with that option.

In order to set the story content to expire at a future time, clients need to access the Content Operations API.

In order to set wires photo content to expire at a future time, clients need to set specific values in the Photo ANS during the transformation process. No other special endpoints or APIs are needed, other than the same Photo Center API call that sends the Photo ANS in to Arc XP.

Video content cannot be set to expire at a future time.

Unlike stories that are migrated from a historical CMS, it is strongly recommended that wire content needs to come into Arc XP unpublished. If you wish wire content to come into Arc XP published, the adapter will need to make an HTTP request to the Draft API endpoint that publishes a document. This will add to the use of your Draft API rate limit, reducing the number of stories you can bring in to Arc XP without experiencing an HTTP 429 over-the-rate-limit error.  You will also need to ensure that if your adapter is publishing wire content, that the same content item is not getting published over and over again.  Not only will this be a waste of your Draft API rate limit, but it could result in a delay in the publishing operations that are generated from the Arc XP editorial applications used by the newsroom.

In order to have wires photo content come into Arc XP published, the adapter will need to set specific values in the Photo ANS during the transformation process. No other special endpoints or APIs are needed, other than the same Photo Center API call that sends the Photo ANS in to Arc XP.

To ensure stable performance, the adapter must control the rate at which it sends content to Arc XP APIs:

When making API calls to create Draft API stories, set the ingestion header Arc-Priority: ingestion.

curl --request POST \
  --url https://api.sandbox.arctesting1.arcpublishing.com/draft/v1/story/ \
  --header 'Arc-Priority: ingestion' \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ans}'

There are two lanes of API traffic that come into Arc XP: a standard lane and an ingestion lane. Standard traffic receives priority and it is the lane that content created with the editorial tools uses. Ingestion traffic is everything else created by the Draft API. Traffic created by ingestion adapters should not come through on the standard lane unless there is a valid use case for this. If you believe you have a use case, contact Arc XP Customer Support. Arc XP reserves the right to disable an integration abusing the standard priority lane.

Content in the ingestion lane is not sent to WebSked and is therefore not available to be added to Collections.

Unless the intent is to request content from the wire source manually, clients will need a way to kick off the start of the adapter process on a schedule. If clients are hosting the adapter on an internal server, they might use a cron job. AWS has the ability trigger events on automated schedules.