Delete robots.txt | Webflow Developer Documentation

Remove specific rules for a user-agent in your `robots.txt` file. To delete all rules for a user-agent, provide an empty rule set. This will remove the user-agent's entry entirely, leaving it subject to your site's default crawling behavior. **Note:** Deleting a user-agent with no rules will make the user-agent's access unrestricted unless other directives apply. <Warning title="Enterprise Only">This endpoint requires an Enterprise workspace.</Warning> Required scope: `site_config:write`

Remove specific rules for a user-agent in your robots.txt file. To delete all rules for a user-agent, provide an empty rule set. This will remove the user-agent’s entry entirely, leaving it subject to your site’s default crawling behavior.

Note: Deleting a user-agent with no rules will make the user-agent’s access unrestricted unless other directives apply.

Enterprise Only

This endpoint requires an Enterprise workspace.

Required scope: site_config:write

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Path parameters

site_idstringRequiredformat: "objectid"

Unique identifier for a Site

Request

The robots.txt file for a given site

ruleslist of objectsOptional

List of rules for user agents.

sitemapstringOptional

URL to the sitemap.

Response

Request was successful

ruleslist of objects

List of rules for user agents.

sitemapstring

URL to the sitemap.

Errors

400

Bad Request Error

401

Unauthorized Error

404

Not Found Error

429

Too Many Requests Error

500

Internal Server Error

Note: Deleting a user-agent with no rules will make the user-agent’s access unrestricted unless other directives apply.

Enterprise Only

This endpoint requires an Enterprise workspace.

Required scope: site_config:write

Bearer authentication of the form Bearer <token>, where token is your auth token.

Unique identifier for a Site

The robots.txt file for a given site

List of rules for user agents.

Request was successful

List of rules for user agents.

$	curl -X DELETE https://api.webflow.com/v2/sites/580e63e98c9a982ac9b8b741/robots_txt \
>	-H "Authorization: Bearer <token>" \
>	-H "Content-Type: application/json" \
>	-d '{
>	"rules": [
>	{
>	"userAgent": "*",
>	"allows": [],
>	"disallows": [
>	"/bubbles"
>	]
>	}
>	]
>	}'

1	{
2	"rules": [
3	{
4	"userAgent": "googlebot",
5	"allows": [],
6	"disallows": [
7	"/vogon-poetry",
8	"/total-perspective-vortex"
9	]
10	}
11	],
12	"sitemap": "https://heartofgold.ship/sitemap.xml"
13	}

$	curl -X DELETE https://api.webflow.com/v2/sites/580e63e98c9a982ac9b8b741/robots_txt \
>	-H "Authorization: Bearer <token>" \
>	-H "Content-Type: application/json" \
>	-d '{
>	"rules": [
>	{
>	"userAgent": "*",
>	"allows": [],
>	"disallows": [
>	"/bubbles"
>	]
>	}
>	]
>	}'

1	{
2	"rules": [
3	{
4	"userAgent": "googlebot",
5	"allows": [],
6	"disallows": [
7	"/vogon-poetry",
8	"/total-perspective-vortex"
9	]
10	}
11	],
12	"sitemap": "https://heartofgold.ship/sitemap.xml"
13	}