New: robots.txt Content API (beta)

The new robots.txt content endpoints read and write the exact text value of the robots.txt field under Site settings > SEO > Indexing. The text is sent and returned in a content field, and Webflow keeps comments, blank lines, and directives such as Crawl-delay.

Get robots.txt Content returns the field’s text.
Replace robots.txt Content replaces the field’s text. The text must be 512,000 bytes (500 KiB) or smaller.
Delete robots.txt Content clears the field.

GET    /beta/sites/{site_id}/robots_txt/content
PUT    /beta/sites/{site_id}/robots_txt/content
DELETE /beta/sites/{site_id}/robots_txt/content

Scopes: site_config:read to read, site_config:write to replace or clear
Plan: the site must be on an Enterprise plan
Publishing: the published robots.txt file changes only after you publish the site

The robots.txt endpoints write the published robots.txt file directly. The next site publish replaces that file with the text from the robots.txt field, so changes made through the robots.txt endpoints are lost. Use one set of endpoints for a site, not both.

New: Sitemap Indexing API (beta)

The new Sitemap Indexing endpoints read and update whether CMS items and static pages are listed in a site’s sitemap.xml file. Each endpoint works with the includeInSitemap setting only, so you can change sitemap settings without sending item field data or page settings.

CMS items

List Item Sitemap Settings lists the staged setting of each item in a collection.
List Live Item Sitemap Settings lists the live setting of each item in a collection.
Get Item Sitemap Setting returns the staged setting of one item.
Get Live Item Sitemap Setting returns the live setting of one item.
Update Item Sitemap Setting updates one item.
Update Item Sitemap Settings updates up to 100 items in one request.

Requirements

Scopes: cms:read and cms:write for CMS items, pages:read and pages:write for pages
Plan: the site must be on an Enterprise plan
Beta access: the endpoints return 404 for workspaces that aren’t in the sitemap indexing beta
Publishing: item updates apply to staged items, and publishing an item doesn’t rebuild sitemap.xml. The live sitemap.xml file changes after you publish the site. Publishing a single page from the Designer updates only that page’s entry.

Changelog