On our central systems monitoring, running Icinga 2, we use two kinds of alerts/notifications: E-Mail for most (normal) alerts and SMS (text messages) for high priority alerts.
For SMS we've been using Twilio as messaging provider since May 2022. Their API turned out to be easily understandable yet powerful and suited us perfectly to quickly drop off an alert message. However we recently noticed that no SMS were sent anymore.
When the https request is sent, the Twilio API responds with a JSON response. Until recently we never had to take a look at it as it just always worked. But either the API endpoint has changed or something else causes that SMS are not sent anymore.
By manually sending a POST request to the Twilio SMS API, we were able to confirm our guess: Sending SMS is not working anymore.
ck@icinga2:~# curl -X POST https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Messages.json --data-urlencode "Body=Hello from Infiniroot" --data-urlencode "From=$TWILIO_NUMBER" --data-urlencode "To=$TO_NUMBER" -u $TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN
{"code": 21408, "message": "Permission to send an SMS has not been enabled for the region indicated by the 'To' number: +4179XXXXXXX", "more_info": "https://www.twilio.com/docs/errors/21408", "status": 400}
The JSON response also contains a helpful more_info key, which shows a knowledge base entry for that particular error 21408. The page basically says that the destination countries of the phone numbers need to be enabled in the "Geo permissions" in the Twilio settings.
This turns out to be something new as we never had to enable any country before. In our case we enabled the countries of our advanced systems monitoring as a service customers, including Switzerland:
Once the countries were selected and saved, we sent another test message:
ck@icinga2:~# curl -X POST https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Messages.json --data-urlencode "Body=$message" --data-urlencode "From=INFINIROOT" --data-urlencode "To=+4179XXXXXXX" -u $TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN
{"body": "Hello from Infiniroot", "num_segments": "1", "direction": "outbound-api", "from": "INFINIROOT", "date_updated": "Thu, 30 Nov 2023 15:09:00 +0000", "price": null, "error_message": null, "uri": "/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Messages/SMXXXXXX0f5f1398ef49XXXXXXXXX643e.json", "account_sid": "$TWILIO_ACCOUNT_SID", "num_media": "0", "to": "+4179XXXXXXX", "date_created": "Thu, 30 Nov 2023 15:09:00 +0000", "status": "queued", "sid": "SMXXXXXX0f5f1398ef49XXXXXXXXX643e", "date_sent": null, "messaging_service_sid": null, "error_code": null, "price_unit": "USD", "api_version": "2010-04-01", "subresource_uris": {"media": "/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Messages/SMXXXXXX0f5f1398ef49XXXXXXXXX643e/Media.json"}}
The API confirmed and its JSON response now contains a "status":"queued" (compared to the "400" before). And 1-2 seconds later, the SMS arrived on the destination number.
What have we learned from this situation? We need to make sure we catch API errors. In our Icinga 2 notification scripts we have therefore added a condition which parses the "status" key from the JSON response. If it doesn't match "queued" an internal alert will be sent to us.
We would however prefer a HTTP status code to be part of the JSON response, too. Maybe Twilio will add this in the future.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb Icingaweb2 Influx Internet Java KVM Kibana Kodi Kubernetes LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder