Get robots.txt Content

GET

https://api.webflow.com/beta/sites/:site_id/robots_txt/content

Returns the exact text value of the robots.txt field under Site settings > SEO > Indexing. When the field is empty, content is an empty string.

The response doesn’t include the lines that Webflow adds to the published robots.txt file when you publish the site:

A Disallow: / block for each known crawler when Allow Search Engine Crawlers or Allow Artificial Intelligence Bots is turned off under Traffic controls.
A Sitemap: line when the site has a sitemap, unless Remove sitemap.xml from robots.txt is turned on.

Enterprise Only

This endpoint requires an Enterprise workspace.

Required scope | site_config:read

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Path parameters

site_idstringRequiredformat: "objectid"

Unique identifier for a Site

Response

Request was successful

contentstring

The robots.txt text.

Errors

400

Bad Request Error

401

Unauthorized Error

403

Forbidden Error

404

Not Found Error

429

Too Many Requests Error

500

Internal Server Error