Protecting your content sources from bad actors, crawlers, and bots

Bots and crawlers are expected consumers of content-based digital experiences. Search engines operates with sophisticated crawlers. Content-based businesses welcome these known crawlers. You may receive unexpected volume of traffic from unrecognized crawlers that may cause high volume of uncached renders to overwhelm the system.

These crawlers want to read as much content as they can without knowing the render stack’s limits. Or its' developers haven’t considered or aren’t concerned about the impact. Another common scenario for these crawlers is attempting to read old and archived content that is not receiving much traffic, which falls under the “uncached renders” case where it causes more content source and render volume.

Bad actors can also use crawler-like scripts, but not for the sake of reading the content like search engines. Instead, bad actors use many resources, which overwhelms the system. Bad actor volume is similar to a DDoS attack, but they don’t necessarily have to be in extreme volumes.

Oftentimes, bad actors discover the content source API calls (/pf/api/content/fetch) with finding a parameter that accepts a wide-range of enumerated values and sends the calls at a relatively high volume. Their goal is to bypass multiple layers of the Arc XP cache (CDN and Engine) and cause either of the systems to become overwhelmed. When the render systems start throttling requests, the actual user render requests begin to fail as well.

Here is an example how an HTTP request gets constructed from your feature custom fields:

GET /pf/api/v3/content/fetch/my-content-source?query=%7B%22feature%22%3A%22top-stories-list%22%2C%22offset%22%3A0%2C%22query%22%3A%22type%3Astory%22%2C%22size%22%3A5%2C%22resizer_size%22%3A%22500x300%22%7D HTTP/1.1

If we URL decode this request URL:

GET/pf/api/v3/content/fetch/my-content-source?query={"feature":"top-stories-list","offset":0,"query":"type:story","size":5,"resizer_size":"500x300"}HTTP/1.1

You can see the PageBuilder serializes all custom field values into a JSON serialized object. If we format the JSON object that is serialized and pass in a "query" parameter:

{
  "feature": "top-stories-list",
  "offset": 0,
  "query": "type:story",
  "size": 5,
  "resizer_size": "500x300"
}

Two parameters in this example are dangerous: "query" and "resizer_size". Both parameters are widely defined, possibly not validated in the content source code, and can be highly open to enumeration. This article covers open query parameters and their risk of sensitive information leakage in the following section, but let's look at simpler example between these two parameters: the resizer size. To visualize an example of enumerated variations of valid request URL, they may look like this:

Determine if a content source should use a sever-side render or a client-side content refresh.

When does a content source can be private, or public? Can it be private, and only used on server-side render?

PageBuilder Engine has the capability to let you control if a content source is exposed as an API endpoint or not. By default, all your features are rendered isomorphically. That means they are rendered on both server-side render as well as client-side re-renders (hydrated). On the client-side render, if the content used to render that particular feature has expired, PageBuilder calls its back-end APIs to refresh the content. You can learn more about render modes in Rehydration vs. Server-Side vs. Client-Side (SPA) Rendering.

If there is no need for client-side render, you could make it fully server-side. To do that, add the .http property to your content source definition before exporting it in your code, and set the value of it to false. See Content Source API documentation.

When changing content source visibility, it’s important to keep another PageBuilder feature, Static, in mind. When a content source is paired with a Static feature, content source responses are used only in server-side render runtime. That means the response object of that content source does not make it to server-side render, or it does not cause any /pf/api calls. In this case, you can make a content source server-side only. If a content source is changed to become private-only; you must ensure all calls from features should be Statically served; otherwise, the client-side render breaks for the instances of the features that use this content source, and it still renders dynamically on the client-side.

When setting http=false, beware that the content source becomes fully private from all API calls to PageBuilder Engine, including Debugger tools, as well as non-PageBuilder consumers if you utilize /pf/api endpoints in third-party applications. The most common scenario is consuming these endpoints from a mobile app or a back-end system. These are acceptable use cases, but when switching an existing content source to http=false, this process impacts these scenarios.

We talked about how important the “request” (or input) parameters for a content source are. PageBuilder Engine has an opt-in parameter for Content Source objects that lets you enforce parameter sanitation before cache-key gets generated.

By default, everything passed from the HTTP call within the query object gets used in cache key generation. And cache-key is a crucial part of how many requests the cache responds to quickly without executing the actual content source logic. If a bad actor or a crawler can enumerate the parameters, they can inject almost anything to the query object.

The most simple strategy is to enable .strict=true and make sure you define all parameters in the params object that a content source works with. Anything that is not recognized gets ignored before the cache-key gets generated. This minimizes the enumeration opportunities for a bad actor. It doesn’t completely eliminate them, but it minimizes then and helps you narrow down your content source inputs.

You can read more about strict property and how to use it in Content Source documentation.

In some cases, doing simple parameter and type recognization (with .strict=true) is not enough. You can harden your content source parameters even further by implementing parameter validation at the top of your content source before the HTTP requests gets executed.

Clients often create or use a helper method to do schema and value validation to ensure the incoming content source parameters (and values) are expected and are valid.

Again, this does not catch every enumeration scenario, but it can narrow it even further to much more predictable values.

In a web form, each field (including hidden fields) deserves deliberate consideration. Consider when readers will complete them, who completes them, and what are the expected values. More importantly, do the values need to be open, could they be dangerous formats, or are they a simple checkbox or drop-down option format. A best practice is to create fields with as narrow responses as possible. Avoid open, large, ambiguous values.

Let’s look at two examples (that we gave as examples in the bad actors section previously) to explain and demonstrate how to intentionally and more narrowly define two example parameters.

const possibleResizerSizes = {
  "hero":   "1000x400"
  "large":  "800x600"
  "medium": "400x300"
  "small":  "200x150"
  "thumb":  "100x40"
}

mysite.com/pf/api/v3/content/fetch/my-content-source?query={..., "resizer": "large"}

{
  "feature": "top-stories-list",
  "offset": 0,
  "query": "type:story",
  "size": 5,
  "resizer_size": "500x300"
}

{
  ...
  "query": "type:story",
  ...
}

{
  ...
  "content_type": "stories",
  ...
}

const possibleQueries = {
  "stories": "type:story",
  "featured_stories": "type:story AND taxonomy.primary_section.name:featured",
  ...
  
  // Also can be templatized and accepts separate content source parameter
  "stories_by_author": "type:story AND credits.by._id:%ATHOR_ID%",
  
  ...
}

The underlying theme of the suggestions in this article is protecting your downstream API activity from the public traffic changes. One of the effective methods to shield downstream APIs independent from the upstream volume is to use partial caching within content sources. Partial caching is a PageBuilder Engine capability that allows you to wrap your business logic and API calls inside content sources to group them and control cache windows separate from the parent content sources.

Partial caching is used for various use cases, including controlling a specific API’s call volume across content sources, so it can be used as a shared cache. But more importantly, you can control the cache key generation parameters for partial cache calls.

This means you have one more layer where your developers can control what dynamic values constitute different API calls, and these calls get cached at the lower level inside content sources before the actual API calls are executed to downstream services.

A benefit to using partial caching is an optimized execution speed of content sources. For example, if you have three services you orchestrate together in a content source, without partial caching, all three services are executed at the same time. But you can isolate each API call and/or group them in partially cached calls. This way, you can control each one of them to be cached at different TTLs.

import axios from 'axios'

const fetch = async ({ 'arc-site': arcSite }, { cachedCall }) => {
  try {
    const topPostsAPIUrl = 'https://google-analytics-api.com/pageviews?url=/posts/*'
    const topPosts = await cachedCall(
      'top-story', // content source name
      () => axios(topPostsAPIUrl), // child fetch function
      {
        query: topPostsAPIUrl, // Will be used to generate partial cache-key
        independent: true,
        ttl: 60 * 60 * 1000 // 1 hour
      }
    )
    const topPostId = topPosts[0].id

    const postsResponse = await axios(`https://jsonplaceholder.typicode.com/posts/${topPostId}`)
    const post = postsResponse.data

    return {
      title: post.title,
      body: post.body
    }
  }
  catch (error) {
    throw new Error('Content fetch error')
  }
}

export default {
  fetch
}

One of the biggest motivations for bad actors creating and using crawlers to seek information is to find private, sensitive, or unpublished information. Your content sources most likely use the Arc XP content platform as its source (although not required). It’s important for your developer team to be aware of the possible private, internal information that resides in your ANS objects.