How to send HTTP requests in Python, handle authentication and JSON POST data

Written by - 0 comments

Published on - Listed in Coding Python


As I am currently working on a new script using Python, I had to find a simple but effective way to talking to a HTTP API with JSON responses. Turns out there are a couple of different possibilities, but the "requests" module seemed to be one of the most common and also easy to understand ways of sending HTTP requests.

Sending HTTP requests (GET and POST) and handle JSON data in Python.

Simple GET request

With curl this is the most basic request you can do:

$ curl https://www.claudiokuenzler.com

Using Python:

$ python
>>> import requests
>>> r=requests.get('https://www.claudiokuenzler.com')

The variable "r" now holds all data related to the requested URL. Including the HTTP response code:

>>> print(r.status_code)
200

This way you can easily check whether the request was successful or not. Let's check a page not found error.

With curl you would check the headers of an URL (quickest method):

$ curl https://www.claudiokuenzler.com/nothinghere -I
HTTP/2 404
server: nginx
date: Tue, 15 Feb 2022 08:10:49 GMT
content-type: text/html; charset=iso-8859-1
vary: Accept-Encoding

Using Python:

>>> r=requests.get('https://www.claudiokuenzler.com/nothinghere')
>>> if (r.status_code != 200):
...   print("Sorry, something went wrong")
...
Sorry, something went wrong

Sending a POST request with JSON payload

Sending a POST uses the "post" instead of the "get" class.

With curl you would put the data payload into a parameter -d:

$ curl -X POST -d '{"id": 1, "method": "Switch.GetStatus", "params": {"id": 0}}' http://api.example.com/rpc

In Python requests, the data payload can be defined in various ways. The default is to simply define the payload in a data parameter:

>>> r=requests.post('http://api.example.com/rpc',  data=somedata)

But as we are sending a JSON payload, there is a special parameter for JSON:

>>> r=requests.post('http://api.example.com/rpc', json={'key':'value','key2':'value2'})

Of course JSON payloads could become rather huge, so it make sense to put the payload into a separate variable before actually sending the POST request:

>>> postdata = { "id": 1, "method": "Switch.GetStatus", "params": {"id": 0} }
>>> r = requests.post('http://api.example.com/rpc', json=postdata)

And to make it even more dynamic, you can also use variables in the postdata payload:

>>> methodchoice="Switch.GetStatus"
>>> postdata = { "id": 1, "method": methodchoice, "params": {"id": 0} }
>>> print(postdata)
{'id': 1, 'method': 'Switch.GetStatus', 'params': {'id': 0}}
>>> r = requests.post('http://api.example.com/rpc', json=postdata)

HTTP authentication

Of course (REST) APIs are often using some kind of authentication. The requests module handles them with different classes/parameters, kind of similar to handling different data payload types. A request without a successful authentication returns a 401 response code.

For a "Basic Auth" authentication, a curl request looks like this:

$ curl https://www.example.com/hidden.txt -u "myuser:secret" -I
HTTP/2 200
server: nginx
date: Tue, 15 Feb 2022 08:36:16 GMT
content-type: text/plain
content-length: 187
[...]

In Python, the official parameter would be auth=BasicAuth, but as Basic Auth is the most common authentication method over HTTP, a simple auth= is enough:

>>> r=requests.get('https://www.example.com/hidden.txt', auth=('myuser', 'secret'))
>>> print(r.status_code)
200

To handle a different authentication method, for example Digest Auth, a different class needs to be loaded with the auth parameter:

>>> methodchoice="Switch.GetStatus"
>>> postdata = { "id": 1, "method": methodchoice, "params": {"id": 0} }
>>> r = requests.post('http://api.example.com/rpc', json=postdata, auth=HTTPDigestAuth('admin', 'secret'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'HTTPDigestAuth' is not defined

Whoa, an error shows up, that the HTTPDigestAuth authentication was not found/defined. To enable additional authentication methods, they must be loaded (imported) specifically:

>>> from requests.auth import HTTPDigestAuth
>>> methodchoice="Switch.GetStatus"
>>> postdata = { "id": 1, "method": methodchoice, "params": {"id": 0} }
>>> r = requests.post('http://api.example.com/rpc', json=postdata, auth=HTTPDigestAuth('admin', 'secret'))
>>> print(r.status_code)
200

A wrong password would lead to either 401 or, depending on the API, a 403 response code:

>>> from requests.auth import HTTPDigestAuth
>>> methodchoice="Switch.GetStatus"
>>> postdata = { "id": 1, "method": methodchoice, "params": {"id": 0} }
>>> r = requests.post('http://api.example.com/rpc', json=postdata, auth=HTTPDigestAuth('admin', 'wrongpass'))
>>> print(r.status_code)
401

Reading and handling JSON response

As you can meanwhile guess from the article, my goal was to talk to an API which receives JSON data input and returns JSON output.

In curl you would combine the output with a json parser, such as jq, to have a "pretty view" for the human eye:

 $ curl -X POST -d '{"id": 1, "method": "Switch.GetStatus", "params": {"id": 0}}' http://api.example.com/rpc --anyauth -u admin:secret
{"id":1,"src":"shellypro4pm-xxx","result":{"id":0, "source":"init", "output":true, "apower":445.1, "voltage":235.6, "current":2.164, "pf":-0.27, "aenergy":{"total":5297.006,"by_minute":[1139.782,7497.179,7637.339],"minute_ts":1644915308},"temperature":{"tC":49.8, "tF":121.6}}}

$ curl -s -X POST -d '{"id": 1, "method": "Switch.GetStatus", "params": {"id": 0}}' http://api.example.com/rpc --anyauth -u admin:secret | jq
{
  "id": 1,
  "src": "shellypro4pm-xxx",
  "result": {
    "id": 0,
    "source": "init",
    "output": true,
    "apower": 460.7,
    "voltage": 235,
    "current": 2.237,
    "pf": -0.26,
    "aenergy": {
      "total": 5300.551,
      "by_minute": [
        4683.653,
        7497.179,
        7637.339
      ],
      "minute_ts": 1644915336
    },
    "temperature": {
      "tC": 49.9,
      "tF": 121.9
    }
  }
}

To handle this in Python, the json module can be used. As we know, the HTTP response is saved in the "r" variable. By running "r" through the json function and saving it in another variable jsondata, the different key and values can then be used. To see what keys and values were parsed, use the json.dumps function on the jsondata variable:

>>> import json
>>> import requests
>>> from requests.auth import HTTPDigestAuth
>>> methodchoice="Switch.GetStatus"
>>> postdata = { "id": 1, "method": methodchoice, "params": {"id": 0} }
>>> r = requests.post('http://api.example.com/rpc', json=postdata, auth=HTTPDigestAuth('admin', 'secret'))
>>> jsondata=r.json()
>>> print(json.dumps(jsondata))
{"id": 1, "src": "shellypro4pm-xxx", "result": {"id": 0, "source": "init", "output": true, "apower": 454.5, "voltage": 235.2, "current": 2.218, "pf": -0.27, "aenergy": {"total": 5326.812, "by_minute": [385.795, 7667.42, 7639.309], "minute_ts": 1644915542}, "temperature": {"tC": 49.6, "tF": 121.4}}}

To retrieve a value from a specific key the array jsondata can now be used:

>>> print(jsondata['src'])
shellypro4pm-xxx

To go deeper into a nested json object, simply append the array with the nested object name:

>>> print(jsondata['result']['aenergy']['total'])
5326.812

>>> print(jsondata['result']['temperature']['tC'])
49.6

What is this all for?

You might wonder why I got through all this? Actually I'm currently working on a new monitoring plugin to monitor Shelly power switch and power meter devices. As I decided to use Python for the plugin's script language, I needed to dig deeper into HTTP requests (with and without authentication) and JSON handling in Python. The monitoring plugin check_shelly will be published on this blog when ready. Public repository can be found on GitHub.



Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Icingaweb   Icingaweb2   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder   


Update cookies preferences