Delete robots.txt

DEL

Remove specific rules for a user-agent in your robots.txt file. To delete all rules for a user-agent, provide an empty rule set. This will remove the user-agent’s entry entirely, leaving it subject to your site’s default crawling behavior.

Note: Deleting a user-agent with no rules will make the user-agent’s access unrestricted unless other directives apply.

Required scope: site_config:write

Path parameters

site_idstringRequired

Unique identifier for a Site

Request

This endpoint expects an object.

ruleslist of objectsOptional

List of rules for user agents.

sitemapstringOptional

URL to the sitemap.

Response

Request was successful

ruleslist of objectsOptional

List of rules for user agents.

sitemapstringOptional

URL to the sitemap.

Errors

Update robots.txt

Up Next

Built with

1	curl -X DELETE https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt \
2	-H "Authorization: Bearer <token>" \
3	-H "Content-Type: application/json" \
4	-d '{
5	"rules": [
6	{
7	"userAgent": "*",
8	"allows": [
9	"/public"
10	],
11	"disallows": [
12	"/bubbles"
13	]
14	}
15	]
16	}'

1	{
2	"rules": [
3	{
4	"userAgent": "googlebot",
5	"allows": [
6	"/public"
7	],
8	"disallows": [
9	"/vogon-poetry",
10	"/total-perspective-vortex"
11	]
12	}
13	],
14	"sitemap": "https://heartofgold.ship/sitemap.xml"
15	}