A few weeks ago, an internal CouchDB 2.2 server, still running on Ubuntu 16.04 (Xenial) needed to be upgraded. Not only is Ubuntu 16.04 meanwhile EOL, also the CouchDB version 2.2 is outdated. This means: Both OS and CouchDB need to be upgraded.
Both ways of upgrading were tested and are described in this article:
And it turns out: The order of the upgrade is important!
Back when CouchDB 2.2 was installed, the apt repository was using apache.bintray.com. We found out quickly, that this repository was unavailable:
root@couchdb:~# apt-get update
[...]
W: The repository 'https://apache.bintray.com/couchdb-deb xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://apache.bintray.com/couchdb-deb/dists/xenial/main/binary-amd64/Packages 502 Bad Gateway
E: Some index files failed to download. They have been ignored, or old ones used instead.
Meanwhile the repositories have switched to a different URL (apache.jfrog.io). To add the new repository:
root@couchdb:~# apt-get update && apt-get install -y curl apt-transport-https gnupg
root@couchdb:~# curl https://couchdb.apache.org/repo/keys.asc | gpg --dearmor | sudo tee /usr/share/keyrings/couchdb-archive-keyring.gpg >/dev/null 2>&1
root@couchdb:~# source /etc/os-release
root@couchdb:~# echo "deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ ${VERSION_CODENAME} main" | sudo tee /etc/apt/sources.list.d/couchdb.list >/dev/null
This should add the repository in a dedicated file (/etc/apt/sources.list.d/couchdb.list). Depending on the current distribution release, the release name (here xenial) will be added:
root@couchdb:~# cat /etc/apt/sources.list.d/couchdb.list
deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ xenial main
The CouchDB documentation mentions that an upgrade from 2.x to 3.x should work without a problem. Even for clustered CouchDB setups, a "mixed" cluster (with cluster members running on CouchDB 2.x and others on CouchDB 3.x) are supported during the upgrade process:
CouchDB 2.x and 3.x are explicitly designed to allow “mixed clusters” during the upgrade process. This allows you to perform a rolling restart across a cluster, upgrading one node at a time, for a zero downtime upgrade.
But, there's just one thing: An administrative CouchDB user must exist. CouchDB 3.x removed the non-authenticated access (thank god!), which was a default in CouchDB 2.x. If you haven't created an admin user yet, follow this guide how to create a new admin user in a (clustered) CouchDB.
Also make a backup of the following files and folders related to your local CouchDB configuration. I created a backup directory and copied the relevant files and folders into it:
root@couchdb:~# mkdir /root/20220106-couchdb-upgrade
root@couchdb:~# cp /opt/couchdb/etc/default.ini /root/20220106-couchdb-upgrade/
root@couchdb:~# cp /opt/couchdb/etc/local.ini /root/20220106-couchdb-upgrade/
root@couchdb:~# cp -Rp /opt/couchdb/etc/local.d/ /root/20220106-couchdb-upgrade/
In this scenario, Ubuntu is upgraded. First from 16.04 (Xenial) to 18.04 (Bionic) and then finally to 20.04 (Focal). Prior to the upgrade, the CouchDB repositories are disabled:
root@couchdb:~# sed -i "s/^deb/#deb/g" /etc/apt/sources.list.d/couchdb.list
root@couchdb:~# cat /etc/apt/sources.list.d/couchdb.list
#deb [signed-by=/usr/share/keyrings/couchdb-archive-keyring.gpg] https://apache.jfrog.io/artifactory/couchdb-deb/ xenial main
root@couchdb:~# apt-get update
Then Ubuntu was upgraded to 18.04.
Note: There are a couple of ways, how to upgrade Ubuntu. I personally prefer to manually adjust the APT repositories in /etc/apt/sources.list and /etc/apt/sources.list.d/ and then run apt-get dist-upgrade. But use whatever floats your boat.
After the upgrade to Ubuntu 18.04 no problems could be detected. Next step: Upgrade to 20.04.
During the apt-get dist-upgrade process, a package dependency issue popped up:
The following packages have unmet dependencies:
couchdb : Depends: python
Depends: python-requests but it is not going to be installed
Recommends: python-progressbar but it is not installable
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
According to this error, it seems that the old CouchDB (still at version 2.2) requires old python packages, which were removed in Ubuntu 20.04. The rest of the dist-upgrade seemed to run through just fine, but this error keeps on hanging over the apt command.
This is the moment when CouchDB definitely needs to be upgraded. The CouchDB is enabled again, then the CouchDB package is installed/upgraded:
root@couchdb:~# sed -i "s/^#deb/deb/g" /etc/apt/sources.list.d/couchdb.list
root@couchdb:~# apt-get update
root@couchdb:~# apt-get install couchdb
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
couch-libmozjs185-1.0 libcurl3 libicu55 libnspr4 python-asn1crypto python-certifi python-cffi-backend python-chardet python-cryptography python-enum34 python-idna python-ipaddress python-openssl python-pkg-resources python-requests python-six python-urllib3
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
libmozjs-68-0
The following NEW packages will be installed:
libmozjs-68-0
The following held packages will be changed:
couchdb
The following packages will be upgraded:
couchdb
1 upgraded, 1 newly installed, 0 to remove and 14 not upgraded.
Need to get 28.4 MB of archives.
After this operation, 10.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
The output also shows that the packages, previously mentioned in the upgrade error, will be removed after the CouchDB upgrade.
The installation of the new CouchDB version worked without any problem. Another apt-get dist-upgrade was executed to properly finish the OS upgrade. Finally no package dependency issues anymore!
The OS is upgraded to Ubuntu 20.04 and CouchDB is running:
root@couchdb:~# ps auxf | grep couch
root 18962 0.0 0.0 8904 668 pts/4 S+ 15:20 0:00 \_ grep couch
couchdb 17756 0.0 0.2 3718552 49168 ? Ssl 15:19 0:02 /opt/couchdb/bin/../erts-9.3.3.15/bin/beam.smp -K true -A 16 -Bd -- -root /opt/couchdb/bin/.. -progname couchdb -- -home /opt/couchdb -- -boot /opt/couchdb/bin/../releases/3.2.1/couchdb -name couchdb@127.0.0.1 -setcookie monster -kernel error_logger silent -sasl sasl_error_logger false -noshell -noinput -smp enable -ssl session_lifetime 300 -config /opt/couchdb/bin/../releases/3.2.1/sys.config
couchdb 17787 0.0 0.0 2496 588 ? Ss 15:19 0:00 \_ erl_child_setup 65536
couchdb 17768 0.0 0.0 3884 92 ? S 15:19 0:00 /opt/couchdb/bin/../erts-9.3.3.15/bin/epmd -daemon
This scenario (upgrading the OS first) worked, but had a glitch with the package dependencies. Let's see if scenario 2 works better and smoother.
In this scenario, CouchDB is upgraded first. As all the Ubuntu packages were up to date, only the couchdb package is shown as upgrade-able:
root@couchdb:~# apt-get update
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 https://esm.ubuntu.com/infra/ubuntu xenial-infra-security InRelease
Hit:5 https://esm.ubuntu.com/infra/ubuntu xenial-infra-updates InRelease
Get:6 https://apache.jfrog.io/artifactory/couchdb-deb xenial InRelease [3,433 B]
Get:7 https://apache.jfrog.io/artifactory/couchdb-deb xenial/main amd64 Packages [3,664 B]
Fetched 7,097 B in 1s (5,227 B/s)
Reading package lists... Done
root@couchdb:~# apt-show-versions -u
couchdb:amd64/xenial 2.2.0~xenial upgradeable to 3.2.1~xenial
Prior to the upgrade, CouchDB is stopped:
root@couchdb:~# /etc/init.d/couchdb stop
[ ok ] Stopping couchdb (via systemctl): couchdb.service.
And then the CouchDB package is upgraded:
root@couchdb:~# apt-get install couchdb
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libcurl3 python-cffi-backend python-chardet python-cryptography python-enum34 python-idna python-ipaddress python-ndg-httpsclient python-openssl python-pkg-resources python-progressbar python-pyasn1 python-requests python-six
python-urllib3
Use 'apt autoremove' to remove them.
The following held packages will be changed:
couchdb
The following packages will be upgraded:
couchdb
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 24.8 MB of archives.
After this operation, 2,467 kB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 https://apache.jfrog.io/artifactory/couchdb-deb xenial/main amd64 couchdb amd64 3.2.1~xenial [24.8 MB]
Fetched 24.8 MB in 5s (4,533 kB/s)
Preconfiguring packages ...
(Reading database ... 21507 files and directories currently installed.)
Preparing to unpack .../couchdb_3.2.1~xenial_amd64.deb ...
Unpacking couchdb (3.2.1~xenial) over (2.2.0~xenial) ...
Processing triggers for systemd (229-4ubuntu21.31) ...
Processing triggers for ureadahead (0.100.0-19.1) ...
Setting up couchdb (3.2.1~xenial) ...
Configuration file '/opt/couchdb/etc/local.ini'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** local.ini (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /opt/couchdb/etc/local.ini ...
Configuration file '/opt/couchdb/etc/vm.args'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** vm.args (Y/I/N/O/D/Z) [default=N] ? Y
Note: It is recommended to apply the new configuration files from the package installation, by answering with Y or I in the prompt. You created a backup of the CouchDB configs first anyway (you did, right?!), so you can compare the configs after the installation.
The CouchDB upgrade ran through without any error. Now Ubuntu was upgraded to 18.04 (no errors) and then to 20.04 (again, no errors).
Awesome! This scenario was way smoother!
After CouchDB was upgraded to 3.2, only one configuration parameter needed to be manually adjusted in /opt/couchdb/etc/local.ini. In my case in CouchDB 2.2 I needed to increase the max_http_request_size to 4 GB in the [httpd] section. As of CouchDB 3.2 this setting is now located in [chttpd]:
[chttpd]
...
# 20180425 added by Claudio Kuenzler
max_http_request_size = 4294967296 ; 4 GB
...
One more thing I needed to do for legacy applications was to re-enable the auth_plugins in /opt/couchdb/etc/default.ini:
; List of replicator client authentication plugins to try. Plugins will be
; tried in order. The first to initialize successfully will be used for that
; particular endpoint (source or target). Normally couch_replicator_auth_noop
; would be used at the end of the list as a "catch-all". It doesn't do anything
; and effectively implements the previous behavior of using basic auth.
; There are currently two plugins available:
; couch_replicator_auth_session - use _session cookie authentication
; couch_replicator_auth_noop - use basic authentication (previous default)
; Currently, the new _session cookie authentication is tried first, before
; falling back to the old basic authenticaion default:
;auth_plugins = couch_replicator_auth_session,couch_replicator_auth_noop
; To restore the old behaviour, use the following value:
;auth_plugins = couch_replicator_auth_noop
auth_plugins = couch_replicator_auth_noop
To summarize this article, here are my suggested CouchDB + OS upgrade steps:
This should help reduce the downtime during the CouchDB upgrade to a minimum.
You don't want to worry about privileges, backups, file system growth, memory allocation etc and just want to focus using CouchDB? Check out the dedicated managed CouchDB server server at Infiniroot!
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb Icingaweb2 Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder