> This page is for Home.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://developers.webflow.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://developers.webflow.com/_mcp/server.

# New: robots.txt Content API (beta)

> Read, replace, and clear the exact text value of the robots.txt field in a site's SEO settings, including comments and directives that the structured robots.txt endpoints don't support.

The new robots.txt content endpoints read and write the exact text value of the **robots.txt** field under **Site settings > SEO > Indexing**. The text is sent and returned in a `content` field, and Webflow keeps comments, blank lines, and directives such as `Crawl-delay`.

* **[Get robots.txt Content](/data/v2.0.0-beta/reference/enterprise/site-configuration/robots-txt-content/get)** returns the field's text.
* **[Replace robots.txt Content](/data/v2.0.0-beta/reference/enterprise/site-configuration/robots-txt-content/replace)** replaces the field's text. The text must be 512,000 bytes (500 KiB) or smaller.
* **[Delete robots.txt Content](/data/v2.0.0-beta/reference/enterprise/site-configuration/robots-txt-content/delete)** clears the field.

```http
GET    /beta/sites/{site_id}/robots_txt/content
PUT    /beta/sites/{site_id}/robots_txt/content
DELETE /beta/sites/{site_id}/robots_txt/content
```

* **Scopes:** `site_config:read` to read, `site_config:write` to replace or clear
* **Plan:** the site must be on an Enterprise plan
* **Publishing:** the published robots.txt file changes only after you publish the site

The [robots.txt endpoints](/data/v2.0.0-beta/reference/enterprise/site-configuration/robots-txt/get) write the published robots.txt file directly. The next site publish replaces that file with the text from the **robots.txt** field, so changes made through the robots.txt endpoints are lost. Use one set of endpoints for a site, not both.