Awk classic case – awk analysis of nginx access logs

Analyze access logs (Nginx as an example)

Nginx combined log format:

'$remote_addr - $remote_user  [$time_local]  '
' "$request"  $status  $body_bytes_sent  '
' "$http_referer"  "$http_user_agent" ';

1. Count the number of visits to the IP:

➜  ~ awk '{a[$1]++}END{for(i in a){print i, a[i]}}' www.linuxcommands.site_nginx.log

2. Count IPs with more than 5 visits:

In the following example, the awk if else conditional judgment statement is used to print out the IP address and the number of access times greater than 5 times.

➜  ~ awk '{a[$1]++}END{for(i in a){if(a[i] > 5) {print i, a[i]}}}' www.linuxcommands.site_nginx.log

3. Count the number of IP visits and sort the top 10:

In the following example, the awk command is used in conjunction with the sort command and the head command to obtain the IP with the top 10 access times.

➜  ~ awk '{a[$1]++}END{for(i in a){print i, a[i]}}' www.linuxcommands.site_nginx.log | sort -k2 -nr | head -10

4. Count access status is 404 ip and times:

➜  ~ awk '{if($9 == "404"){a[$1" "$9]++}}END{for(i in a){print i,a[i]}}' www.linuxcommands.site_nginx.log

5. Count the number of visits in the last minute

➜  ~ # mac 
➜  ~ date=$(date -v -1M +%d/%b/%Y:%H:%M:%S)     
➜  ~ awk -vdate=$date -F'[[ ]' '{if($5==date) c++}END{print c}' www.linuxcommands.site_nginx.log

6. Count the 10 most visited pages:

➜  ~ awk '{a[$7]++}END{for(v in a) print v,a[v]}' www.linuxcommands.site_nginx.log | sort -k2 -nr | head -10