Get robots.txt

GET

Retrieve the robots.txt configuration for various user agents.

Required scope: site_config:read

site_idstringRequired

Unique identifier for a Site

Request was successful

ruleslist of objectsOptional

List of rules for user agents.

sitemapstringOptional

URL to the sitemap.

1	{
2	"rules": [
3	{
4	"userAgent": "googlebot",
5	"allows": [
6	"/public"
7	],
8	"disallows": [
9	"/vogon-poetry",
10	"/total-perspective-vortex"
11	]
12	}
13	],
14	"sitemap": "https://heartofgold.ship/sitemap.xml"
15	}

1	curl https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt \
2	-H "Authorization: Bearer <token>"