Enable Logpush to S3-compatible endpoints
Cloudflare Logpush supports pushing logs to S3-compatible destinations via the Cloudflare dashboard or via API, including:
- Alibaba Cloud OSS
- Backblaze B2
- DigitalOcean Spaces
- IBM Cloud Object Storage
- JD Cloud Object Storage Service
- Linode Object Storage
- Oracle Cloud Object Storage
- On-premise Ceph Object Gateway
For more information about Logpush and the current production APIs, refer to Cloudflare Logpush documentation.
Manage via the Cloudflare dashboard
Enable Logpush to an S3-compatible destination via the dashboard.
To enable the Cloudflare Logpush service:
Log in to the Cloudflare dashboard.
Select the Enterprise account or domain you want to use with Logpush.
Go to Analytics & Logs > Logs.
Click Add Logpush job. A modal window opens where you will need to complete several steps.
Select the dataset you want to push to a storage service.
Select the data fields to include in your logs. Add or remove fields later by modifying your settings in Logs > Logpush.
Select S3-Compatible.
Enter or select the following destination information:
- S3 Compatible Bucket Path
- Bucket region
- Access Key ID
- Secret Access Key
- Endpoint URL
Click Validate access.
Click Save and Start Pushing to finish enabling Logpush.
Once connected, Cloudflare lists the S3-compatible destination as a connected service under Logs > Logpush. Edit or remove connected services from here.
Manage via API
To set up S3-compatible endpoints:
- Create a job with the appropriate endpoint URL and authentication parameters.
- Enable the job to begin pushing logs.
1. Create a job
To create a job, make a POST
request to the Logpush jobs endpoint with the following fields:
- name (optional) - Use your domain name as the job name.
- destination_conf - A log destination consisting of an endpoint name, bucket name, bucket path, region, access-key-id, and secret-access-key in the following string format:
"s3://<BUCKET_NAME>/<BUCKET_PATH>?region=<REGION>&access-key-id=<ACCESS_KEY_ID>&secret-access-key=<SECRET_ACCESS_KEY>&endpoint=<ENDPOINT_URL>"
- dataset - The category of logs you want to receive. Refer to Log fields for the full list of supported datasets.
- logpull_options (optional) - To configure fields, sample rate, and timestamp format, refer to API configuration options.
Example request using cURL:
curl -s -X POST \https://api.cloudflare.com/client/v4/zones/<ZONE_ID>/logpush/jobs \
-d '{"name":"<DOMAIN_NAME>","destination_conf":"s3://<BUCKET_NAME>/<BUCKET_PATH>?region=<REGION>&access-key-id=<ACCESS_KEY_ID>&secret-access-key=<SECRET_ACCESS_KEY>&endpoint=<ENDPOINT_URL>", "logpull_options": "fields=ClientIP,ClientRequestHost,ClientRequestMethod,ClientRequestURI,EdgeEndTimestamp,EdgeResponseBytes,EdgeResponseStatus,EdgeStartTimestamp,RayID×tamps=rfc3339", "dataset": "http_requests"}' | jq .
Response:
{ "errors": [], "messages": [], "result": { "id": 100, "dataset": "http_requests", "enabled": false, "name": "<DOMAIN_NAME>", "logpull_options": "fields=ClientIP,ClientRequestHost,ClientRequestMethod,ClientRequestURI,EdgeEndTimestamp,EdgeResponseBytes,EdgeResponseStatus,EdgeStartTimestamp,RayID×tamps=rfc3339", "destination_conf": "s3://<BUCKET_NAME>/<BUCKET_PATH>?region=<REGION>&access-key-id=<ACCESS_KEY_ID>&secret-access-key=<SECRET_ACCESS_KEY>&endpoint=<ENDPOINT_URL>", "last_complete": null, "last_error": null, "error_message": null }, "success": true
}
2. Enable (update) a job
To enable a job, make a PUT
request to the Logpush jobs endpoint. You will use the job ID returned from the previous step in the URL, and send {"enabled": true}
in the request body.
Example request using cURL:
curl -s -X PUT \https://api.cloudflare.com/client/v4/zones/<ZONE_ID>/logpush/jobs/100 -d'{"enabled":true}' | jq .
Response:
{ "errors": [], "messages": [], "result": { "id": 100, "dataset": "http_requests", "enabled": true, "name": "<DOMAIN_NAME>", "logpull_options": "fields=ClientIP,ClientRequestHost,ClientRequestMethod,ClientRequestURI,EdgeEndTimestamp,EdgeResponseBytes,EdgeResponseStatus,EdgeStartTimestamp,RayID×tamps=rfc3339", "destination_conf": "s3://<BUCKET_NAME>/<BUCKET_PATH>?region=<REGION>&access-key-id=<ACCESS_KEY_ID>&secret-access-key=<SECRET_ACCESS_KEY>&endpoint=<ENDPOINT_URL>", "last_complete": null, "last_error": null, "error_message": null }, "success": true
}