> This page is for Data API, version v2 Beta.
> For other versions, use one of these documentation indexes:
> - v2 (default): https://developers.webflow.com/data/v2.0.0/llms.txt
> - v2 Beta: https://developers.webflow.com/data/v2.0.0-beta/llms.txt
> - v1: https://developers.webflow.com/data/v1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://developers.webflow.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://developers.webflow.com/_mcp/server.

# Get robots.txt Content

GET https://api.webflow.com/beta/sites/{site_id}/robots_txt/content

Returns the exact text value of the **robots.txt** field under **Site settings > SEO > Indexing**. When the field is empty, `content` is an empty string.The response doesn't include the lines that Webflow adds to the published robots.txt file when you publish the site:- A `Disallow: /` block for each known crawler when **Allow Search Engine Crawlers** or **Allow Artificial Intelligence Bots** is turned off under **Traffic controls**.
- A `Sitemap:` line when the site has a sitemap, unless **Remove sitemap.xml from robots.txt** is turned on.This endpoint requires an Enterprise workspace.Required scope | `site_config:read`

Reference: https://developers.webflow.com/data/v2.0.0-beta/reference/enterprise/site-configuration/robots-txt-content/get

## Authentication

- `Authorization` header (bearer token, required) — Bearer authentication of the form `Bearer <token>`, where token is your auth token.

## Request

### Path parameters

- `site_id` (string, required) — Unique identifier for a Site

## Response

### 200

Request was successful

- `content` (string, required) — The robots.txt text.

## Errors

### 400 Bad Request Error

Validation failure

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentGetResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 401 Unauthorized Error

Provided access token is invalid or does not have access to requested resource

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentGetResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 403 Forbidden Error

The access token is missing the required scope, the site is not on an Enterprise plan, or the user who authorized the token doesn't have permission to perform this action on the site.

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentGetResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 404 Not Found Error

Requested resource not found

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentGetResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 429 Too Many Requests Error

The rate limit of the provided access_token has been reached. Please have your application respect the X-RateLimit-Remaining header we include on API responses.

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentGetResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

### 500 Internal Server Error

We had a problem with our server. Try again later.

- `code` (enum, optional) — Error code
  - Allowed values: `aeo_input_validation`, `analyze_filter_conflict`, `analyze_input_validation`, `analyze_unsupported_filter`, `bad_request`, `batch_too_large`, `before_historical_floor`, `billing_provider_unavailable`, `branch_merge_conflict`, `branch_merge_in_progress`, `branch_not_found`, `branch_publish_failed`, `branch_sync_pending`, `branch_task_not_found`, `branch_update_in_progress`, `brand_not_configured`, `cannot_invite_self`, `cannot_remove_self`, `cannot_update_user_role`, `client_seats_unavailable`, `collection_not_found`, `competitors_not_configured`, `conflict`, `conflicting_scope_filters`, `default_site_role_required`, `duplicate_collection`, `duplicate_member_in_request`, `duplicate_site_in_request`, `duplicate_user_email`, `ecommerce_not_enabled`, `end_time_too_recent`, `enterprise_provisioning_cloud_app_not_supported`, `enterprise_provisioning_upstream_error`, `enterprise_self_provisioning_not_entitled`, `forbidden`, `forms_require_republish`, `incompatible_webhook_filter`, `insufficient_permissions`, `insufficient_seats`, `internal_error`, `invalid_auth_version`, `invalid_credentials`, `invalid_domain`, `invalid_resolution`, `invalid_site_role`, `invalid_time_range`, `invalid_user_email`, `invalid_workspace_role`, `invite_expired`, `invite_role_conflict`, `item_not_found`, `managed_by_scim`, `member_already_has_access`, `member_has_no_site_access`, `member_is_paying_client`, `member_not_found`, `member_not_removable`, `member_role_conflict`, `missing_scopes`, `no_domains`, `no_members_or_invites`, `not_authorized`, `not_enterprise_plan_site`, `not_enterprise_plan_workspace`, `optimize_before_historical_floor`, `optimize_input_validation`, `optimize_invalid_interval`, `optimize_unknown_dimension`, `optimize_unknown_goal`, `optimize_unknown_optimization`, `optimize_unsupported_group_by`, `order_not_found`, `page_cannot_branch`, `resource_not_found`, `service_unavailable`, `site_access_not_applicable_to_member`, `site_access_not_scopable_for_role`, `site_access_roles_must_match`, `site_not_in_workspace`, `site_role_id_invalid`, `site_role_id_not_allowed`, `site_role_id_required`, `site_specific_access_disabled`, `site_specific_access_not_entitled`, `time_range_too_wide`, `too_many_requests`, `unsupported_bot_provider`, `unsupported_version`, `unsupported_webhook_trigger_type`, `user_limit_reached`, `user_not_found`, `users_not_enabled`, `validation_error`, `workspace_invite_not_found`, `workspace_member_managed_by_scim`, `workspace_member_not_found`, `workspace_member_not_removable`, `workspace_role_not_grantable`
- `message` (string, optional) — Error message
- `externalReference` (string, optional) — Link to more information
- `details` (list of SitesSiteIdRobotsTxtContentGetResponsesContentApplicationJsonSchemaDetailsItems, optional) — Array of errors

## Types

### SitesSiteIdRobotsTxtContentGetResponsesContentApplicationJsonSchemaDetailsItems

- string
- map from string to any

## Examples

### RobotsTxtContent

**Response**

```json
{
  "content": "# Keep crawlers out of drafts\nUser-agent: *\nDisallow: /drafts\nCrawl-delay: 10\n"
}
```

**SDK Code**

```python RobotsTxtContent
import requests

url = "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content"

headers = {"Authorization": "Bearer <token>"}

response = requests.get(url, headers=headers)

print(response.json())
```

```javascript RobotsTxtContent
const url = 'https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content';
const options = {method: 'GET', headers: {Authorization: 'Bearer <token>'}};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go RobotsTxtContent
package main

import (
	"fmt"
	"net/http"
	"io"
)

func main() {

	url := "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content"

	req, _ := http.NewRequest("GET", url, nil)

	req.Header.Add("Authorization", "Bearer <token>")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby RobotsTxtContent
require 'uri'
require 'net/http'

url = URI("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Get.new(url)
request["Authorization"] = 'Bearer <token>'

response = http.request(request)
puts response.read_body
```

```java RobotsTxtContent
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.get("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content")
  .header("Authorization", "Bearer <token>")
  .asString();
```

```php RobotsTxtContent
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('GET', 'https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content', [
  'headers' => [
    'Authorization' => 'Bearer <token>',
  ],
]);

echo $response->getBody();
```

```csharp RobotsTxtContent
using RestSharp;

var client = new RestClient("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content");
var request = new RestRequest(Method.GET);
request.AddHeader("Authorization", "Bearer <token>");
IRestResponse response = client.Execute(request);
```

```swift RobotsTxtContent
import Foundation

let headers = ["Authorization": "Bearer <token>"]

let request = NSMutableURLRequest(url: NSURL(string: "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "GET"
request.allHTTPHeaderFields = headers

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```

### NoStoredText

**Response**

```json
{
  "content": ""
}
```

**SDK Code**

```python NoStoredText
import requests

url = "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content"

headers = {"Authorization": "Bearer <token>"}

response = requests.get(url, headers=headers)

print(response.json())
```

```javascript NoStoredText
const url = 'https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content';
const options = {method: 'GET', headers: {Authorization: 'Bearer <token>'}};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go NoStoredText
package main

import (
	"fmt"
	"net/http"
	"io"
)

func main() {

	url := "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content"

	req, _ := http.NewRequest("GET", url, nil)

	req.Header.Add("Authorization", "Bearer <token>")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby NoStoredText
require 'uri'
require 'net/http'

url = URI("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Get.new(url)
request["Authorization"] = 'Bearer <token>'

response = http.request(request)
puts response.read_body
```

```java NoStoredText
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.get("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content")
  .header("Authorization", "Bearer <token>")
  .asString();
```

```php NoStoredText
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('GET', 'https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content', [
  'headers' => [
    'Authorization' => 'Bearer <token>',
  ],
]);

echo $response->getBody();
```

```csharp NoStoredText
using RestSharp;

var client = new RestClient("https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content");
var request = new RestRequest(Method.GET);
request.AddHeader("Authorization", "Bearer <token>");
IRestResponse response = client.Execute(request);
```

```swift NoStoredText
import Foundation

let headers = ["Authorization": "Bearer <token>"]

let request = NSMutableURLRequest(url: NSURL(string: "https://api.webflow.com/beta/sites/580e63e98c9a982ac9b8b741/robots_txt/content")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "GET"
request.allHTTPHeaderFields = headers

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```