Investigating HTTP server response (json) is cut/incomplete before the end

Written by Claudio Kuenzler - 1 comments

Published on March 8th 2019 - Listed in Nginx Linux Internet

And suddenly there was an alert on our Rancher 2 test environment:

Interestingly the warning popped up more or less at the same time, I updated the check_rancher2 monitoring plugin. Coincidence or did the new version introduce a bug? Turned out it was a coincidence *phew*, because the very same plugin worked correctly with the production environment.

So what was causing this alert? The output clearly showed that something with the received json data is not OK. I launched the plugin manually:

./check_rancher2.sh -H rancher2-stage.example.com -U token-xxxxx -P "secret" -S -t pod -p c-xxxxx:p-xxxxx
json read error: line 2 column 0: '}' expected near end of file
json read error: line 2 column 0: '}' expected near end of file
CHECK_RANCHER2 WARNING - No pods found in project c-xxxxx:p-xxxxx.

Indeed, something's off. I tried to see more using curl (what the plugin does in the backend):

$ curl -u token-xxxxx:secret rancher2-stage.example.com/v3/project/c-xxxxx:p-xxxxx/pods -v
* Trying 10.10.10.10...
* Connected to rancher2-stage.example.com (10.10.10.10) port 443 (#0)
[...]
< HTTP/1.1 200 OK
< Server: nginx/1.4.6 (Ubuntu)
< Date: Fri, 08 Mar 2019 11:50:37 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< Expires: Wed 24 Feb 1982 18:42:00 GMT
< X-Api-Schemas: https://rancher2-stage.example.com/v3/project/c-xxxxx:p-xxxxx/schemas
<
{"type":"collection","links":{"self":"https://rancher2-stage.example.com/v3/project [...]
,"imageID":"docker-pullable://application/logging@sha256:9bc* Connection #0 to host rancher2-stage.example.com left intact
6b69f025bae4fd54889da28d0833bddf9ebb0fce41babb41a215be4019643","lastState":{"type":"/v3/project/schemas/containerState"},"name":"logs-cron-internal-api","ready":false,"restartCount":0,"state":{"terminated":{"containerID":"docker://287286a02f121931ab4c11808c5c38b89aab6d88c07de84d269a40122662fbf2","exitCode":0,"finishedAt":"2019-03-08T11:50:05Z","finishedAtTS":1552045805000,"reason":"Completed","signal":0,"startedAt":"2019-03-08T11:50:05Z","startedAtTS":1552045805000,"type":"/v3/project/schemas/containerStateTerminate

The response's output clearly doesn't end properly! I verified on the Rancher 2 API in the browser and there everything seemed correct; the output ended in a correct json context.

I was already doing research and came across a couple of hints:

https://stackoverflow.com/questions/10557927/server-response-gets-cut-off-half-way-through
https://stackoverflow.com/questions/53922588/curl-json-output-cuts-off

But unfortunately they didn't help... However something hit me when I checked my mails and saw a critical disk space alert on the load balancer which is right in front of this Rancher 2 test environment!

And indeed, logged in on this LB:

# df -h /var
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg0-lvvar ext4 3.7G 3.4G 0 100% /var

Once I increased the disk space (or if I would have deleted some old logs in /var/log) the curl request came through correctly:

$ curl -u token-xxxxx:secret rancher2-stage.example.com/v3/project/c-xxxxx:p-xxxxx/pods -v
* Trying 10.10.10.10...
* Connected to rancher2-stage.example.com (10.10.10.10) port 443 (#0)
[...]
type":"/v3/project/schemas/volume"}],"workloadId":"daemonset:gamma:web"}]}
* Connection #0 to host rancher2-stage.example.com left intact

And the alert was gone:

$ ./check_rancher2.sh -H rancher2-stage.example.com -U token-xxxxx -P "secret" -S -t pod -p c-xxxxx:p-xxxxx
CHECK_RANCHER2 OK - All pods (37) in project c-xxxxx:p-xxxxx are running|'pods_total'=37;;;; 'pods_errors'=0;;;;

TL;DR: If you have a Reverse Proxy or Load Balancer using Nginx and your /var (or your partition where Nginx logging happens) is full, Nginx's server response might be cut off.

Add a comment

Show form to leave a comment

Comments (newest first)

Jim Johnson from wrote on Aug 23rd, 2019:

I wanted to mention that you can also get this error when your linux server doesn't trust the CA signer of whomever signed your HTTPS cert for rancher. To resolve you either need to add the correct signer certs to your linux server OR change the script to use curl -k for insecure / non-trusted CA Signers. Maybe you can somehow add this as an option? Like if you are using a self signed (or in our case a corporate self signed), add a --self-signed that will turn the curl command into curl -k ? Hope this helps others that may get this error ...

Investigating HTTP server response (json) is cut/incomplete before the end

Add a comment

Comments (newest first)

Blog Tags: