For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Resources
Get started
ReferenceGuidesExamplesChangelog
ReferenceGuidesExamplesChangelog
  • Data API
    • Introduction
  • Token
    • GETGet Authorization User Info
    • GETGet Authorization Info
  • Sites
    • GETList Sites
    • GETGet Site
    • GETGet Custom Domains
    • POSTPublish Site
  • Pages and Components
  • CMS
  • Forms
  • Custom Code
  • Assets
  • Custom fonts
  • Comments
    • GETList Comment Threads
    • GETGet Comment Thread
    • GETList Comment Replies
  • Ecommerce
  • Webhooks
    • GETList Webhooks
    • GETGet Webhook
    • POSTCreate Webhook
    • DELRemove Webhook
  • Site Configuration
  • Enterprise
        • GETGet robots.txt
        • PUTReplace robots.txt
        • PATCHUpdate robots.txt
        • DELDelete robots.txt
LogoLogo
Resources
Get started
EnterpriseSite Configurationrobots.txt

Get robots.txt

GET
https://api.webflow.com/v2/sites/:site_id/robots_txt
GET
/v2/sites/:site_id/robots_txt
$curl https://api.webflow.com/v2/sites/580e63e98c9a982ac9b8b741/robots_txt \
>     -H "Authorization: Bearer <token>"
1{
2  "rules": [
3    {
4      "userAgent": "googlebot",
5      "allows": [],
6      "disallows": [
7        "/vogon-poetry",
8        "/total-perspective-vortex"
9      ]
10    }
11  ],
12  "sitemap": "https://heartofgold.ship/sitemap.xml"
13}
Retrieve the robots.txt configuration for various user agents. <Warning title="Enterprise Only">This endpoint requires an Enterprise workspace.</Warning> Required scope: `site_config:read`
Was this page helpful?
Previous

Replace robots.txt

Next
Built with

Retrieve the robots.txt configuration for various user agents.

Enterprise Only
This endpoint requires an Enterprise workspace.

Required scope: site_config:read

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Path parameters

site_idstringRequiredformat: "objectid"
Unique identifier for a Site

Response

Request was successful
ruleslist of objects
List of rules for user agents.
sitemapstring
URL to the sitemap.

Errors

400
Bad Request Error
401
Unauthorized Error
404
Not Found Error
429
Too Many Requests Error
500
Internal Server Error