Logs Engine

Logs Engine gives you the ability to store your logs in R2 and query them directly.

​​ Store logs in R2

• Set up a Logpush to R2 job.
• Create an R2 access key with at least R2 read permissions.
• Ensure that you have Logshare read permissions.
• Alternatively, create a Cloudflare API token with the following permissions:
  • Account scope
  • Logs read permissions

​​ Query logs

You can use the API to query and download your logs by time range or RayID.

​​ Authentication

The following headers are required for all API calls:

• X-Auth-Email - the Cloudflare account email address associated with the domain
• X-Auth-Key - the Cloudflare API key

Alternatively, API tokens with Logs edit permissions can also be used for authentication:

• Authorization: Bearer <API_TOKEN>

​​ Required headers

In addition to the required authentication headers mentioned, the following headers are required for the API to access logs stored in your R2 bucket.

R2-access-key-id (required) - R2 Access Key Id R2-secret-access-key (required) - R2 Secret Access Key

​​ List files

List relevant R2 objects containing logs matching the provided query parameters, using the endpoint GET /accounts/{accountId}/logs/list.

​​ Query parameters

• start (required) string (TimestampRFC3339) - Start time in RFC 3339 format, for example start=2022-06-06T16:00:00Z.

• end (required) string (TimestampRFC3339) - End time in RFC 3339 format, for example end=2022-06-06T16:00:00Z.

• bucket (required) string (Bucket) - R2 bucket name, for example bucket=cloudflare-logs.

• prefix string (Prefix) - R2 bucket prefix logs are stored under, for example prefix=http_requests/example.com/{DATE}.

• limit number (Limit) - Maximum number of results to return, for example limit=100.

​​ Retrieve logs by time range

Stream logs stored in R2 that match the provided query parameters, using the endpoint GET /accounts/{accountId}/logs/retrieve.

​​ Query parameters

• start (required) string (TimestampRFC3339) - Start time in RFC 3339 format, for example start=2022-06-06T16:00:00Z

• end (required) string (TimestampRFC3339) - End time in RFC 3339 format, for example end=2022-06-06T16:00:00Z

• bucket (required) string (Bucket) - R2 bucket name, for example bucket=cloudflare-logs

• prefix string (Prefix) - R2 bucket prefix logs are stored under, for example prefix=http_requests/example.com/{DATE}

​​ Example API request

curl -s -g -X GET  'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/logs/retrieve?start=2022-06-01T16:00:00Z&end=2022-06-01T16:05:00Z&bucket=cloudflare-logs&prefix=http_requests/example.com/{DATE}' \
-H "X-Auth-Email: <EMAIL>" \
-H "X-Auth-Key: <API_KEY>" \ 
-H "R2-Access-Key-Id: R2_ACCESS_KEY_ID" \
-H "R2-Secret-Access-Key: R2_SECRET_ACCESS_KEY" | jq .

Results can be piped to a file using > logs.json.

Additionally, if you want to receive the raw GZIP bytes without them being transparently decompressed by your client, include the header -H 'Accept-Encoding: gzip'.

​​ ​Retrieve logs by RayID

Using your logs stored in R2 - the Logpull RayID Lookup feature allows you to query an indexed time range for the presence of an RayID and return the matching result. This feature is available to users with the Logpull RayID Lookup beta subscription.

The ability to look up a RayID is a two-step process. First, a time range needs to be indexed before being able to request a record by the RayID.

Indexes will automatically expire after seven days of no usage.

​​ Index a time range

Before executing your query, you can specify the time range you would like to index in order to narrow down the scope of the query. In the following example, we index one minute of logs stored in the R2 bucket "cloudflare-logs" under the prefix "http_requests/{DATE}".

​​ Example API request

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/logs/rayids/index' \
    -H 'Authorization: Bearer <API_TOKEN>' \
    -H 'R2-Access-Key-Id: <R2_ACCESS_KEY_ID>' \
    -H 'R2-Secret-Access-Key: <R2_SECRET_ACCESS_KEY>' \
    -H 'Content-Type: application/json' \
    --data-raw '{
    "start":"2022-08-16T20:30:00Z",
    "end":"2022-08-16T20:31:00",
    "bucket": "cloudflare-logs",
    "prefix": "http_requests/example.com/{DATE}"
}'

​​ Lookup a RayID

After indexing a time range, perform a GET request with the RayID. If a matching result is found in the indexed time range, the record will be returned. Note that the parameters have moved from the request body and into the URL. The -g flag is required to avoid the {DATE} parameter from being misinterpreted by cURL.

​​ Example API request

curl -g -X GET 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/logs/rayids/<RAY_ID>?bucket=cloudflare-logs&prefix=http_requests/example.com/{DATE}' \
    -H 'Authorization: Bearer <API_TOKEN>' \
    -H 'R2-Access-Key-Id: <R2_ACCESS_KEY_ID>' \
    -H 'R2-Secret-Access-Key: <R2_SECRET_ACCESS_KEY>'

​​ Troubleshooting