Replace robots.txt Content
Replaces the text value of the robots.txt field under Site settings > SEO > Indexing with the exact text you send, and returns the new value. Comments, blank lines, line endings, and directives such as Crawl-delay are kept.
The published robots.txt file doesn’t change until you publish the site. A successful change marks the site as having unpublished changes, unless you send the text that the field already holds. Single-page publishing doesn’t update the robots.txt file.
Each site publish rebuilds the published file from this field and replaces any changes made through the robots.txt endpoints. Use one set of endpoints for a site, not both.
When Webflow subdomain indexing is turned off, the robots.txt file on the webflow.io subdomain blocks all crawlers, whatever the robots.txt field says.
Required scope | site_config:write
Authentication
Bearer authentication of the form Bearer <token>, where token is your auth token.
Path parameters
Request
The new robots.txt text, up to 512,000 bytes (500 KiB) when encoded as UTF-8. Some crawlers, including Google, ignore robots.txt text past 500 KiB. Send an empty string to clear the text.