An interesting error was seen today in our Ansible configuration management. A playbook, which uses the blockinfile module to write a "block" of content into a file, and was in use for many years, suddenly stopped working.
To reproduce the error, a very simple playbook, based on the original playbook was created:
ck@ansible:~$ cat /tmp/blockinfile.yaml
---
- name: ANSIBLE - Blockinfile test - Infiniroot LLC
hosts: '{{ target }}'
roles:
- yaegashi.blockinfile
tasks:
##########################
# Set facts
##########################
- name: Write a simple block into testfile
blockinfile:
dest: /tmp/blockinfile.test
create: yes
insertafter: "EOF"
marker: "# {mark} -- configured by Ansible"
block: |
Write some text
Write some more text
Write even more text
And again write some text
When the plugin was run against an Ubuntu 20.04 machine with Python 3.8, the blockinfile task failed with the following error:
ck@ansible:~$ ansible-playbook /tmp/blockinfile.yaml --extra-vars "target=target.example.com"
PLAY [ANSIBLE - Blockinfile test - Infiniroot LLC] **************************************************************************************************
TASK [Gathering Facts] **************************************************************************************************
ok: [target.example.com]
TASK [Write a simple block into testfile] **************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: startswith first arg must be bytes or a tuple of bytes, not str
fatal: [target.example.com]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 10.10.2.54 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/home/ansible/.ansible/tmp/ansible-tmp-1637765329.6180422-5446-104584190794430/AnsiballZ_blockinfile.py\", line 100, in <module>\r\n _ansiballz_main()\r\n File \"/home/ansible/.ansible/tmp/ansible-tmp-1637765329.6180422-5446-104584190794430/AnsiballZ_blockinfile.py\", line 92, in _ansiballz_main\r\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n File \"/home/ansible/.ansible/tmp/ansible-tmp-1637765329.6180422-5446-104584190794430/AnsiballZ_blockinfile.py\", line 40, in invoke_module\r\n runpy.run_module(mod_name='ansible.modules.blockinfile', init_globals=dict(_module_fqn='ansible.modules.blockinfile', _modlib_path=modlib_path),\r\n File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\r\n return _run_module_code(code, init_globals, run_name, mod_spec)\r\n File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\r\n _run_code(code, mod_globals, init_globals,\r\n File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\r\n exec(code, run_globals)\r\n File \"/tmp/ansible_blockinfile_payload_390rv7rl/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py\", line 268, in <module>\r\n File \"/tmp/ansible_blockinfile_payload_390rv7rl/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py\", line 215, in main\r\nTypeError: startswith first arg must be bytes or a tuple of bytes, not str\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
PLAY RECAP **************************************************************************************************
target.example.com : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
However when the target's Python version was changed to use Python 2.7 instead of 3.8 (by setting the ansible_python_interpreter in the Ansible inventory), the task was successfully executed:
ck@ansible:~$ grep target inventory/*
inventory/hosts:target.example.com ansible_ssh_host=10.10.2.54 ansible_python_interpreter=/usr/bin/python2.7
ck@ansible:~$ ansible-playbook /tmp/blockinfile.yaml --extra-vars "target=target.example.com"
PLAY [ANSIBLE - Blockinfile test - Infiniroot LLC] **************************************************************************************************
TASK [Gathering Facts] **************************************************************************************************
ok: [target.example.com]
TASK [Write a simple block into testfile] **************************************************************************************************
changed: [target.example.com]
PLAY RECAP **************************************************************************************************
target.example.com : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
The blockinfile task worked fine under Python 2.7, but not under the newer Python 3.8 on the target Ubuntu 20.04 machine. So it must be a Python bug, right?
That was our first thought but no! Although we created a new bug report #76358 in the Ansible repository, we continued the analysis. Another playbook was created in another customer environment with Debian 11 target systems (which have Python 3.9) and on these targets the playbook ran successfully!
ck@ansible2:~$ cat /tmp/simpleblockinfile.yaml
---
- name: ANSIBLE - minimal blockinfile test - Infiniroot LLC
hosts: '{{ target }}'
tasks:
# Write a block into testfile
- name: Write a block into testfile
blockinfile:
dest: /tmp/blockinfile.test
insertafter: "EOF"
marker: "# {mark} -- configured by Ansible"
block: |
Write some text
Write some more text
Write even more text
And again write some text
when: testfile.stat.exists == true
Maybe the problem only hits under a specific Python 3 version (before 3.9)? But the simplified playbook also ran on Ubuntu 18.04 targets with Python 3.6.
The simplified playbook was then copied to the original customer environment where the blockinfile problems occurred - and it worked!
Finally we realized what was causing the errors on Python 3: The original playbook was still using the yaegashi.blockinfile role. This was an old method to implement the blockinfile module before it became part of the "builtin" modules. That happened a while ago, but the playbook still contained the role include.
As long as the target hosts still linked to Python 2.7 under /usr/bin/python, the original blockinfile module continued to work. But recent distribution releases have now ditched Python 2.7 and /usr/bin/python points to a Python 3.x version. And bam! The playbooks stopped working on these targets.
By simply removing the yaegashi.blockinfile role in the playbook "headers", the blockinfile module is used from the Ansible builtin modules - which works correctly under both Python 2.7 and Python 3.x.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Container Containers CouchDB DB DNS Database Docker ELK Elasticsearch Filebeat FreeBSD Galera GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb2 InfluxDB Internet Java KVM Kibana Kodi Kubernetes LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Redis SSL Samba Seafile Security Shell SmartOS Solaris Surveillance SystemD Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder