> This page is for Data API, version v2 Beta.
> For other versions, use one of these documentation indexes:
> - v2 (default): https://developers.webflow.com/data/v2.0.0/llms.txt
> - v2 Beta: https://developers.webflow.com/data/v2.0.0-beta/llms.txt
> - v1: https://developers.webflow.com/data/v1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://developers.webflow.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://developers.webflow.com/_mcp/server.

# Delete robots.txt Content

DELETE https://api.webflow.com/beta/sites/{site_id}/robots_txt/content

Clears the **robots.txt** field under **Site settings > SEO > Indexing** and returns the empty value. Webflow ignores any request body.Clearing the field marks the site as having unpublished changes, unless the field is already empty. After the next site publish, the published robots.txt file contains only the lines that Webflow adds: the **Traffic controls** blocks and the `Sitemap:` line. When none of those lines apply, the published file is empty.When **Webflow subdomain indexing** is turned off, the robots.txt file on the webflow\.io subdomain blocks all crawlers, whatever the **robots.txt** field says.This endpoint requires an Enterprise workspace.Required scope | `site_config:write`

Reference: https://developers.webflow.com/data/v2.0.0-beta/reference/enterprise/site-configuration/robots-txt-content/delete

## Authentication

- `Authorization` header (bearer token, required) — Bearer authentication of the form `Bearer <token>`, where token is your auth token.

## Request

### Path parameters

- `site_id` (string, required) — Unique identifier for a Site

## Response

### 200

Request was successful

- `content` (string, required) — The robots.txt text.

## Errors

### 400 Bad Request Error

Validation failure

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentDeleteResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 401 Unauthorized Error

Provided access token is invalid or does not have access to requested resource

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentDeleteResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 403 Forbidden Error

The access token is missing the required scope, the site is not on an Enterprise plan, or the user who authorized the token doesn't have permission to perform this action on the site.

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentDeleteResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 404 Not Found Error

Requested resource not found

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentDeleteResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 429 Too Many Requests Error

The rate limit of the provided access_token has been reached. Please have your application respect the X-RateLimit-Remaining header we include on API responses.

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentDeleteResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 500 Internal Server Error

We had a problem with our server. Try again later.

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentDeleteResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

## Types

### SitesSiteIdRobotsTxtContentDeleteResponsesContentApplicationJsonSchemaDetailsItems

- string
- map from string to any

## Examples

**Response**

```json
{
  "content": ""
}
```

**SDK Code**

```python ClearedText
import requests

url = "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content"

headers = {"Authorization": "Bearer <token>"}

response = requests.delete(url, headers=headers)

print(response.json())
```

```javascript ClearedText
const url = 'https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content';
const options = {method: 'DELETE', headers: {Authorization: 'Bearer <token>'}};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go ClearedText
package main

import (
	"fmt"
	"net/http"
	"io"
)

func main() {

	url := "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content"

	req, _ := http.NewRequest("DELETE", url, nil)

	req.Header.Add("Authorization", "Bearer <token>")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby ClearedText
require 'uri'
require 'net/http'

url = URI("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Delete.new(url)
request["Authorization"] = 'Bearer <token>'

response = http.request(request)
puts response.read_body
```

```java ClearedText
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.delete("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content")
  .header("Authorization", "Bearer <token>")
  .asString();
```

```php ClearedText
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('DELETE', 'https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content', [
  'headers' => [
    'Authorization' => 'Bearer <token>',
  ],
]);

echo $response->getBody();
```

```csharp ClearedText
using RestSharp;

var client = new RestClient("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content");
var request = new RestRequest(Method.DELETE);
request.AddHeader("Authorization", "Bearer <token>");
IRestResponse response = client.Execute(request);
```

```swift ClearedText
import Foundation

let headers = ["Authorization": "Bearer <token>"]

let request = NSMutableURLRequest(url: NSURL(string: "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "DELETE"
request.allHTTPHeaderFields = headers

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```