Parse Cloudflare Logs JSON data
Overview
After downloading your Cloudflare Logs data, you can use different tools to parse and analyze your logs.
In this tutorial, you will learn how to parse your JSON log data using jq. To get started with jq, visit the jq official site.
Aggregating fields
To aggregate a field appearing in the log, such as by IP address, URI, or referrer, you can use several jq commands. This is useful to identify any patterns in traffic; for example, to identify your most popular pages or to block an attack.
The following examples match on a field name and provide a count of each field instance, sorted in ascending order by count.
$ jq -r .ClientRequestURI logs.json | sort -n | uniq -c | sort -n | tail2 /nginx-logo.png2 /poweredby.png2 /testagain3 /favicon.ico3 /testing3 /testing1236 /test7 /testing123410 /cdn-cgi/nexp/dok3v=1613a3a185/cloudflare/rocket.js54 /
$ jq -r .ClientRequestUserAgent logs.json | sort -n | uniq -c | sort -n | tail1 python-requests/2.9.12 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.174 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.365 curl/7.47.2-DEV36 Mozilla/5.0 (X11; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.051 curl/7.46.0-DEV
$ jq -r .ClientRequestReferer logs.json | sort -n | uniq -c | sort -n | tail2 http://example.com/testagain3 http://example.com/testing5 http://example.com/5 http://example.com/testing1237 http://example.com/testing123477 null
Filtering fields
Another common use case involves filtering data for a specific field value and then aggregating after that. This helps answer questions like Which URLs saw the most 502 errors? For example:
$ jq 'select(.OriginResponseStatus == 502) | .ClientRequestURI' logs.json | sort -n | uniq -c | sort -n | tail1 "/favicon.ico"1 "/testing"3 "/testing123"6 "/test"6 "/testing1234"18 "/"
To find out the top IP addresses blocked by the Cloudflare WAF, use the following query:
$ jq -r 'select(.WAFAction == "drop") | .ClientIP' logs.json | sort -n | uniq -c | sort -n1 127.0.0.1
Showing cached requests
To retrieve your cache ratios, try the following query:
$ jq -r '.CacheCacheStatus' logs.json | sort -n | uniq -c | sort -n3 hit3 null3 stale4 expired6 miss81 unknown
Showing TLS versions
To find out which TLS versions your visitors are using — for example, to decide if you can disable TLS versions that are older than 1.2 — use the following query:
$ jq -r '.ClientSSLProtocol' logs.json | sort -n | uniq -c | sort -n42 none58 TLSv1.2