Replace robots.txt | Webflow Developer Documentation

Replace the robots.txt configuration for various user agents.

Enterprise Only

This endpoint requires an Enterprise workspace.

Required scope | site_config:write

Replace the `robots.txt` configuration for various user agents. <Warning title="Enterprise Only">This endpoint requires an Enterprise workspace.</Warning> Required scope | `site_config:write`

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Path parameters

site_idstringRequiredformat: "objectid"

Unique identifier for a Site

Request

The robots.txt file for a given site

ruleslist of objectsOptional

List of rules for user agents.

sitemapstringOptional

URL to the sitemap.

Response

Request was successful

ruleslist of objects

List of rules for user agents.

sitemapstring

URL to the sitemap.

Errors

400

Bad Request Error

401

Unauthorized Error

404

Not Found Error

429

Too Many Requests Error

500

Internal Server Error

$	curl -X PUT https://api.webflow.com/v2/sites/580e63e98c9a982ac9b8b741/robots_txt \
>	-H "Authorization: Bearer <token>" \
>	-H "Content-Type: application/json" \
>	-d '{
>	"rules": [
>	{
>	"userAgent": "googlebot",
>	"allows": [],
>	"disallows": [
>	"/vogon-poetry",
>	"/total-perspective-vortex"
>	]
>	}
>	],
>	"sitemap": "https://heartofgold.ship/sitemap.xml"
>	}'

1	{
2	"rules": [
3	{
4	"userAgent": "googlebot",
5	"allows": [],
6	"disallows": [
7	"/vogon-poetry",
8	"/total-perspective-vortex"
9	]
10	}
11	],
12	"sitemap": "https://heartofgold.ship/sitemap.xml"
13	}

$	curl -X PUT https://api.webflow.com/v2/sites/580e63e98c9a982ac9b8b741/robots_txt \
>	-H "Authorization: Bearer <token>" \
>	-H "Content-Type: application/json" \
>	-d '{
>	"rules": [
>	{
>	"userAgent": "googlebot",
>	"allows": [],
>	"disallows": [
>	"/vogon-poetry",
>	"/total-perspective-vortex"
>	]
>	}
>	],
>	"sitemap": "https://heartofgold.ship/sitemap.xml"
>	}'

1	{
2	"rules": [
3	{
4	"userAgent": "googlebot",
5	"allows": [],
6	"disallows": [
7	"/vogon-poetry",
8	"/total-perspective-vortex"
9	]
10	}
11	],
12	"sitemap": "https://heartofgold.ship/sitemap.xml"
13	}