Is it possible to block CDN URLs and non-production domains
Question
My CDN and non-production URLs (sandbox, staging, dev, etc..) are accessible to the public, allowing the URLs to be accessed and indexed. How do I prevent the public from accessing these URLs directly?
Example:
https://org-site-prod.web.arc-cdn.net/story-of-the-day/
https://sandbox.site.com/story-of-the-day/
Answer
To block search engine indexing:
Add CDN protections like cookie authentication, header authentication, IP whitelist, etc...
Add a redirect.
Add a no-index meta tag to the individual HTML pages.
Disallow the indexing of all the paths through robots.txt (not supported for CDN URL).
Important
Options #1 and #2 for blocking public access must be implemented by ACS.
Submit a ticket to ACS for assistance applying the following options to block all public access:
Cookie authentication (recommended)
Header authentication
Parameter authentication
Use an IP address whitelist
Add a redirect