Investigating HTTP server response (json) is cut/incomplete before the end

Written by Claudio Kuenzler - 1 comments

Published on March 8th 2019 - Listed in Nginx Linux Internet

---

And suddenly there was an alert on our Rancher 2 test environment:

Interestingly the warning popped up more or less at the same time, I updated the check_rancher2 monitoring plugin. Coincidence or did the new version introduce a bug? Turned out it was a coincidence *phew*, because the very same plugin worked correctly with the production environment. 

So what was causing this alert? The output clearly showed that something with the received json data is not OK. I launched the plugin manually:

./check_rancher2.sh -H rancher2-stage.example.com -U token-xxxxx -P "secret" -S -t pod -p c-xxxxx:p-xxxxx
json read error: line 2 column 0: '}' expected near end of file
json read error: line 2 column 0: '}' expected near end of file
CHECK_RANCHER2 WARNING - No pods found in project c-xxxxx:p-xxxxx.

Indeed, something's off. I tried to see more using curl (what the plugin does in the backend):

$ curl -u token-xxxxx:secret rancher2-stage.example.com/v3/project/c-xxxxx:p-xxxxx/pods  -v
*   Trying 10.10.10.10...
* Connected to rancher2-stage.example.com (10.10.10.10) port 443 (#0)
[...]
< HTTP/1.1 200 OK
< Server: nginx/1.4.6 (Ubuntu)
< Date: Fri, 08 Mar 2019 11:50:37 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< Expires: Wed 24 Feb 1982 18:42:00 GMT
< X-Api-Schemas: https://rancher2-stage.example.com/v3/project/c-xxxxx:p-xxxxx/schemas
<
{"type":"collection","links":{"self":"https://rancher2-stage.example.com/v3/project  [...]
,"imageID":"docker-pullable://application/logging@sha256:9bc* Connection #0 to host rancher2-stage.example.com left intact
6b69f025bae4fd54889da28d0833bddf9ebb0fce41babb41a215be4019643","lastState":{"type":"/v3/project/schemas/containerState"},"name":"logs-cron-internal-api","ready":false,"restartCount":0,"state":{"terminated":{"containerID":"docker://287286a02f121931ab4c11808c5c38b89aab6d88c07de84d269a40122662fbf2","exitCode":0,"finishedAt":"2019-03-08T11:50:05Z","finishedAtTS":1552045805000,"reason":"Completed","signal":0,"startedAt":"2019-03-08T11:50:05Z","startedAtTS":1552045805000,"type":"/v3/project/schemas/containerStateTerminate

The response's output clearly doesn't end properly! I verified on the Rancher 2 API in the browser and there everything seemed correct; the output ended in a correct json context.

I was already doing research and came across a couple of hints:

• https://stackoverflow.com/questions/10557927/server-response-gets-cut-off-half-way-through
• https://stackoverflow.com/questions/53922588/curl-json-output-cuts-off

Add a comment

Show form to leave a comment

Comments (newest first)

Jim Johnson from wrote on Aug 23rd, 2019:

I wanted to mention that you can also get this error when your linux server doesn't trust the CA signer of whomever signed your HTTPS cert for rancher. To resolve you either need to add the correct signer certs to your linux server OR change the script to use curl -k for insecure / non-trusted CA Signers. Maybe you can somehow add this as an option? Like if you are using a self signed (or in our case a corporate self signed), add a --self-signed that will turn the curl command into curl -k ? Hope this helps others that may get this error ...

---