> This page is for Data API, version v2 Beta.
> For other versions, use one of these documentation indexes:
> - v2 (default): https://developers.webflow.com/data/v2.0.0/llms.txt
> - v2 Beta: https://developers.webflow.com/data/v2.0.0-beta/llms.txt
> - v1: https://developers.webflow.com/data/v1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://developers.webflow.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://developers.webflow.com/_mcp/server.

# Replace robots.txt

PUT https://api.webflow.com/beta/sites/{site_id}/robots_txt
Content-Type: application/json

Replace the `robots.txt` configuration for various user agents.This endpoint requires an Enterprise workspace.Required scope | `site_config:write`

Reference: https://developers.webflow.com/data/reference/enterprise/site-configuration/robots-txt/put

## Authentication

- `Authorization` header (bearer token, required) — Bearer authentication of the form `Bearer <token>`, where token is your auth token.

## Request

### Path parameters

- `site_id` (string, required) — Unique identifier for a Site

### Body (application/json)

This endpoint expects an object.

- `rules` (list of SitesSiteIdRobotsTxtPutRequestBodyContentApplicationJsonSchemaRulesItems, optional) — List of rules for user agents.
- `sitemap` (string, optional) — URL to the sitemap.

## Response

### 200

Request was successful

- `rules` (list of SitesSiteIdRobotsTxtPutResponsesContentApplicationJsonSchemaRulesItems, optional) — List of rules for user agents.
- `sitemap` (string, optional) — URL to the sitemap.

## Errors

### 400 Bad Request Error

Request body was incorrectly formatted.

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtPutResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 401 Unauthorized Error

Provided access token is invalid or does not have access to requested resource

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtPutResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 404 Not Found Error

Requested resource not found

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtPutResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 429 Too Many Requests Error

The rate limit of the provided access_token has been reached. Please have your application respect the X-RateLimit-Remaining header we include on API responses.

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtPutResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 500 Internal Server Error

We had a problem with our server. Try again later.

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtPutResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

## Types

### SitesSiteIdRobotsTxtPutRequestBodyContentApplicationJsonSchemaRulesItems

- `userAgent` (string, required) — The user agent the rules apply to.
- `allows` (list of string, optional) — List of paths allowed for this user agent.
- `disallows` (list of string, optional) — List of paths disallowed for this user agent.

### SitesSiteIdRobotsTxtPutResponsesContentApplicationJsonSchemaRulesItems

- `userAgent` (string, required) — The user agent the rules apply to.
- `allows` (list of string, optional) — List of paths allowed for this user agent.
- `disallows` (list of string, optional) — List of paths disallowed for this user agent.

### SitesSiteIdRobotsTxtPutResponsesContentApplicationJsonSchemaDetailsItems

## Examples

**Request**

```json
{
  "rules": [
    {
      "userAgent": "googlebot",
      "allows": [],
      "disallows": [
        "/vogon-poetry",
        "/total-perspective-vortex"
      ]
    }
  ],
  "sitemap": "https://heartofgold.ship/sitemap.xml"
}
```

**Response**

```json
{
  "rules": [
    {
      "userAgent": "googlebot",
      "allows": [],
      "disallows": [
        "/vogon-poetry",
        "/total-perspective-vortex"
      ]
    }
  ],
  "sitemap": "https://heartofgold.ship/sitemap.xml"
}
```

**SDK Code**

```python
import requests

url = "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt"

payload = {
    "rules": [
        {
            "userAgent": "googlebot",
            "allows": [],
            "disallows": ["/vogon-poetry", "/total-perspective-vortex"]
        }
    ],
    "sitemap": "https://heartofgold.ship/sitemap.xml"
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.put(url, json=payload, headers=headers)

print(response.json())
```

```javascript
const url = 'https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt';
const options = {
  method: 'PUT',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{"rules":[{"userAgent":"googlebot","allows":[],"disallows":["/vogon-poetry","/total-perspective-vortex"]}],"sitemap":"https://heartofgold.ship/sitemap.xml"}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go
package main

import (
	"fmt"
	"strings"
	"net/http"
	"io"
)

func main() {

	url := "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt"

	payload := strings.NewReader("{\n  \"rules\": [\n    {\n      \"userAgent\": \"googlebot\",\n      \"allows\": [],\n      \"disallows\": [\n        \"/vogon-poetry\",\n        \"/total-perspective-vortex\"\n      ]\n    }\n  ],\n  \"sitemap\": \"https://heartofgold.ship/sitemap.xml\"\n}")

	req, _ := http.NewRequest("PUT", url, payload)

	req.Header.Add("Authorization", "Bearer <token>")
	req.Header.Add("Content-Type", "application/json")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby
require 'uri'
require 'net/http'

url = URI("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Put.new(url)
request["Authorization"] = 'Bearer <token>'
request["Content-Type"] = 'application/json'
request.body = "{\n  \"rules\": [\n    {\n      \"userAgent\": \"googlebot\",\n      \"allows\": [],\n      \"disallows\": [\n        \"/vogon-poetry\",\n        \"/total-perspective-vortex\"\n      ]\n    }\n  ],\n  \"sitemap\": \"https://heartofgold.ship/sitemap.xml\"\n}"

response = http.request(request)
puts response.read_body
```

```java
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.put("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt")
  .header("Authorization", "Bearer <token>")
  .header("Content-Type", "application/json")
  .body("{\n  \"rules\": [\n    {\n      \"userAgent\": \"googlebot\",\n      \"allows\": [],\n      \"disallows\": [\n        \"/vogon-poetry\",\n        \"/total-perspective-vortex\"\n      ]\n    }\n  ],\n  \"sitemap\": \"https://heartofgold.ship/sitemap.xml\"\n}")
  .asString();
```

```php
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('PUT', 'https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt', [
  'body' => '{
  "rules": [
    {
      "userAgent": "googlebot",
      "allows": [],
      "disallows": [
        "/vogon-poetry",
        "/total-perspective-vortex"
      ]
    }
  ],
  "sitemap": "https://heartofgold.ship/sitemap.xml"
}',
  'headers' => [
    'Authorization' => 'Bearer <token>',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();
```

```csharp
using RestSharp;

var client = new RestClient("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt");
var request = new RestRequest(Method.PUT);
request.AddHeader("Authorization", "Bearer <token>");
request.AddHeader("Content-Type", "application/json");
request.AddParameter("application/json", "{\n  \"rules\": [\n    {\n      \"userAgent\": \"googlebot\",\n      \"allows\": [],\n      \"disallows\": [\n        \"/vogon-poetry\",\n        \"/total-perspective-vortex\"\n      ]\n    }\n  ],\n  \"sitemap\": \"https://heartofgold.ship/sitemap.xml\"\n}", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
```

```swift
import Foundation

let headers = [
  "Authorization": "Bearer <token>",
  "Content-Type": "application/json"
]
let parameters = [
  "rules": [
    [
      "userAgent": "googlebot",
      "allows": [],
      "disallows": ["/vogon-poetry", "/total-perspective-vortex"]
    ]
  ],
  "sitemap": "https://heartofgold.ship/sitemap.xml"
] as [String : Any]

let postData = JSONSerialization.data(withJSONObject: parameters, options: [])

let request = NSMutableURLRequest(url: NSURL(string: "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "PUT"
request.allHTTPHeaderFields = headers
request.httpBody = postData as Data

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```