To analyze a HTTP caching problem I needed to grab the network connections and take a look into the HTTP protocol and find possible problematic requests. I used tcpdump to capture the tcp streams and Wireshark to analyze the captured packets.
The following examples can come in handy.
Creating a pcap file is basically the same, as if you'd run tcpdump on the command line, just with an additional -w parameter for "writing an output file". Here's an example where all network traffic from or towards the host 192.168.0.1. After hitting [CTRL]+[C] the tcpdump process is stopped.
ckmint ~ $ sudo tcpdump -i any host 192.168.0.1 -w /tmp/tcpdump.pcap
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
^C44 packets captured
44 packets received by filter
0 packets dropped by kernel
The file (/tmp/tcpdump.pcap) can now be opened in Wireshark.
Note: There are also command line tools which can be used to analyze a pcap file.
After opening a pcap (tcpdump capture file) with Wireshark, a filter can be placed in the top bar. The following filter shows all HTTP responses which did NOT (!=) have a HTTP status 200:
Show connections which requested www.example.com/ (the main domain) on the webserver 192.168.168.5:
http.request.uri == "/" && http.host == "www.example.com" && ip.dst == 192.168.168.5
Show connections which contain a HTTP 200 response code but don't contain the "Content-Encoding" http header:
http.response.code == 200 && !http.content_encoding
Show http responses where the content was gzip compressed:
http.content_encoding == gzip
Don't show http content, only headers:
http.response !=0 || http.request.method != "TRACE"
To be able to create filters with custom http headers, I first needed to add them to Wiresharks preferences:
Edit -> Preferences -> Protocols -> HTTP -> Custom HTTP headers fields -> Edit
I added the following additional headers:
To use these custom http headers as a filter, you need to use the http.header prefix.
Show http responses which weren't cached, which don't contain the "Content-Encoding" header and which were treated by varnish-3 server:
http.header.X-Cache == MISS && !http.content_encoding && http.header.X-Varnish-Hostname == varnish-3
Show http responses which were cached (HIT) and treated by varnish-4 server:
http.header.X-Cache == HIT && http.header.X-Varnish-Hostname == varnish-4
Show http responses which pass through a varnish server (so the header X-Varnish-Hostname exists), have response code 200 and don't contain the "Content-Encoding" header:
http.header.X-Varnish-Hostname && http.response.code == 200 && !http.content_encoding
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder