How to send HTTP requests in Python, handle authentication and JSON POST data

Written by Claudio Kuenzler - 0 comments

Published on February 15th 2022 - Listed in Coding Python

As I am currently working on a new script using Python, I had to find a simple but effective way to talking to a HTTP API with JSON responses. Turns out there are a couple of different possibilities, but the "requests" module seemed to be one of the most common and also easy to understand ways of sending HTTP requests.

Sending HTTP requests (GET and POST) and handle JSON data in Python.

Simple GET request

With curl this is the most basic request you can do:

$ curl https://www.claudiokuenzler.com

Using Python:

$ python
>>> import requests
>>> r=requests.get('https://www.claudiokuenzler.com')

The variable "r" now holds all data related to the requested URL. Including the HTTP response code:

>>> print(r.status_code)
200

This way you can easily check whether the request was successful or not. Let's check a page not found error.

With curl you would check the headers of an URL (quickest method):

$ curl https://www.claudiokuenzler.com/nothinghere -I
HTTP/2 404
server: nginx
date: Tue, 15 Feb 2022 08:10:49 GMT
content-type: text/html; charset=iso-8859-1
vary: Accept-Encoding

Using Python:

>>> r=requests.get('https://www.claudiokuenzler.com/nothinghere')
>>> if (r.status_code != 200):
... print("Sorry, something went wrong")
...
Sorry, something went wrong

Sending a POST request with JSON payload

Sending a POST uses the "post" instead of the "get" class.

With curl you would put the data payload into a parameter -d:

$ curl -X POST -d '{"id": 1, "method": "Switch.GetStatus", "params": {"id": 0}}' http://api.example.com/rpc

In Python requests, the data payload can be defined in various ways. The default is to simply define the payload in a data parameter:

>>> r=requests.post('http://api.example.com/rpc', data=somedata)

But as we are sending a JSON payload, there is a special parameter for JSON:

>>> r=requests.post('http://api.example.com/rpc', json={'key':'value','key2':'value2'})

Of course JSON payloads could become rather huge, so it make sense to put the payload into a separate variable before actually sending the POST request:

>>> postdata = { "id": 1, "method": "Switch.GetStatus", "params": {"id": 0} }
>>> r = requests.post('http://api.example.com/rpc', json=postdata)

And to make it even more dynamic, you can also use variables in the postdata payload:

>>> methodchoice="Switch.GetStatus"
>>> postdata = { "id": 1, "method": methodchoice, "params": {"id": 0} }
>>> print(postdata)
{'id': 1, 'method': 'Switch.GetStatus', 'params': {'id': 0}}
>>> r = requests.post('http://api.example.com/rpc', json=postdata)

HTTP authentication

Of course (REST) APIs are often using some kind of authentication. The requests module handles them with different classes/parameters, kind of similar to handling different data payload types. A request without a successful authentication returns a 401 response code.

For a "Basic Auth" authentication, a curl request looks like this:

$ curl https://www.example.com/hidden.txt -u "myuser:secret" -I
HTTP/2 200
server: nginx
date: Tue, 15 Feb 2022 08:36:16 GMT
content-type: text/plain
content-length: 187
[...]

In Python, the official parameter would be auth=BasicAuth, but as Basic Auth is the most common authentication method over HTTP, a simple auth= is enough:

>>> r=requests.get('https://www.example.com/hidden.txt', auth=('myuser', 'secret'))
>>> print(r.status_code)
200

To handle a different authentication method, for example Digest Auth, a different class needs to be loaded with the auth parameter:

>>> methodchoice="Switch.GetStatus"
>>> postdata = { "id": 1, "method": methodchoice, "params": {"id": 0} }
>>> r = requests.post('http://api.example.com/rpc', json=postdata, auth=HTTPDigestAuth('admin', 'secret'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'HTTPDigestAuth' is not defined

Whoa, an error shows up, that the HTTPDigestAuth authentication was not found/defined. To enable additional authentication methods, they must be loaded (imported) specifically:

>>> from requests.auth import HTTPDigestAuth
>>> methodchoice="Switch.GetStatus"
>>> postdata = { "id": 1, "method": methodchoice, "params": {"id": 0} }
>>> r = requests.post('http://api.example.com/rpc', json=postdata, auth=HTTPDigestAuth('admin', 'secret'))
>>> print(r.status_code)
200

A wrong password would lead to either 401 or, depending on the API, a 403 response code:

Reading and handling JSON response

As you can meanwhile guess from the article, my goal was to talk to an API which receives JSON data input and returns JSON output.

In curl you would combine the output with a json parser, such as jq, to have a "pretty view" for the human eye:

$ curl -X POST -d '{"id": 1, "method": "Switch.GetStatus", "params": {"id": 0}}' http://api.example.com/rpc --anyauth -u admin:secret
{"id":1,"src":"shellypro4pm-xxx","result":{"id":0, "source":"init", "output":true, "apower":445.1, "voltage":235.6, "current":2.164, "pf":-0.27, "aenergy":{"total":5297.006,"by_minute":[1139.782,7497.179,7637.339],"minute_ts":1644915308},"temperature":{"tC":49.8, "tF":121.6}}}

$ curl -s -X POST -d '{"id": 1, "method": "Switch.GetStatus", "params": {"id": 0}}' http://api.example.com/rpc --anyauth -u admin:secret | jq
{
"id": 1,
"src": "shellypro4pm-xxx",
"result": {
    "id": 0,
    "source": "init",
    "output": true,
    "apower": 460.7,
    "voltage": 235,
    "current": 2.237,
    "pf": -0.26,
    "aenergy": {
      "total": 5300.551,
      "by_minute": [
        4683.653,
        7497.179,
        7637.339
      ],
      "minute_ts": 1644915336
    },
    "temperature": {
      "tC": 49.9,
      "tF": 121.9
    }
}
}

To handle this in Python, the json module can be used. As we know, the HTTP response is saved in the "r" variable. By running "r" through the json function and saving it in another variable jsondata, the different key and values can then be used. To see what keys and values were parsed, use the json.dumps function on the jsondata variable:

>>> import json
>>> import requests
>>> from requests.auth import HTTPDigestAuth
>>> methodchoice="Switch.GetStatus"
>>> postdata = { "id": 1, "method": methodchoice, "params": {"id": 0} }
>>> r = requests.post('http://api.example.com/rpc', json=postdata, auth=HTTPDigestAuth('admin', 'secret'))
>>> jsondata=r.json()
>>> print(json.dumps(jsondata))
{"id": 1, "src": "shellypro4pm-xxx", "result": {"id": 0, "source": "init", "output": true, "apower": 454.5, "voltage": 235.2, "current": 2.218, "pf": -0.27, "aenergy": {"total": 5326.812, "by_minute": [385.795, 7667.42, 7639.309], "minute_ts": 1644915542}, "temperature": {"tC": 49.6, "tF": 121.4}}}

To retrieve a value from a specific key the array jsondata can now be used:

>>> print(jsondata['src'])
shellypro4pm-xxx

To go deeper into a nested json object, simply append the array with the nested object name:

>>> print(jsondata['result']['aenergy']['total'])
5326.812

>>> print(jsondata['result']['temperature']['tC'])
49.6

What is this all for?

You might wonder why I got through all this? Actually I'm currently working on a new monitoring plugin to monitor Shelly power switch and power meter devices. As I decided to use Python for the plugin's script language, I needed to dig deeper into HTTP requests (with and without authentication) and JSON handling in Python. The monitoring plugin check_shelly will be published on this blog when ready. Public repository can be found on GitHub.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.