After (and especially before) every Gitlab update, it is advised to keep an eye on the update/migration jobs, which run in the background. These so-called Background migrations are launched automatically after every update. They make sure that data (for example database) is adjusted to the new version.
Before continuing with the Gitlab upgrade path (if you need to do multiple version updates in a row), you should wait until these Background migrations are finished.
After our Gitlab CE instance was updated (in this situation from 18.8 to 18.9), the background migration tasks can be listed with the following command:
ck@gitlab:~# sudo gitlab-rake gitlab:background_migrations:list
id | table_name | job_class_name | status | progress
---------|-------------------|-----------------------------------------|-----------|-----------
main_1 | integrations | BackfillIntegrationsTypeNew | finished | 100.00%
main_2 | namespaces | BackfillUserNamespace | finished | 100.00%
main_3 | issues | BackfillIssueSearchData | finished | 100.00%
main_4 | routes | BackfillNamespaceIdForNamespaceRoute | finished | 100.00%
[...]
The output shows a lot of entries. Relevant for your eye are the status and progress. To retrieve a list of still active jobs, you could filter for the "active" filter or don't show the 100% entries:
ck@gitlab:~# sudo gitlab-rake gitlab:background_migrations:list | grep -v "100.00%"
id | table_name | job_class_name | status | progress
---------|-------------------------|---------------------------------------------|-----------|----------
main_450 | web_hook_logs | BackfillPartitionedWebHookLogsDaily | active |
main_703 | oauth_applications | BackfillOrganizationIdOnOauthApplications | active | 50000.00%
main_704 | projects | BackfillMissingProjectPushRules | active | 0.00%
main_705 | project_authorizations | MigrateProjectAuthorizations | active |
main_708 | web_hook_logs_daily | BackfillWebHookLogsDailyShardingKeys | active |
This means after the update of the Omnibus package was completed, five background migrations were active.
A couple of minutes later only one remained:
ck@gitlab:~# sudo gitlab-rake gitlab:background_migrations:list | grep -v "100.00%"
id | table_name | job_class_name | status | progress
---------|-------------------------|---------------------------------------------|-----------|----------
main_450 | web_hook_logs | BackfillPartitionedWebHookLogsDaily | active |
I waited and checked the same command again every couple of minutes. But this migration task still remained. And it didn't seem to make any progress either.
In the Admin UI, the background migration also shows up as running but with 0% progress:
After some (traditional non AI) research, I came across a Gitlab issue, that mentioned the same problem, with the same background migration being stuck.
According to one of the Gitlab maintainers, this background migration can be safely deleted, as the relevant table does no longer exist in the database.
We can mark this migration as completed as the web_hook_logs table was removed in %18.0.
In order to delete the task, we need to run a "Gitlab-internal" rails command. This can either be done by using the Gitlab console (sudo gitlab-rails console) or by using gitlab-rails as wrapper.
First let's verify we're looking at the correct background migration. According to the output above the id is 450. This can also be verified in the UI (the ID shows up in the link of this particular background migration task).
ck@gitlab:~# sudo gitlab-rails runner "pp Gitlab::Database::BackgroundMigration::BatchedMigration.find(450)"
#<Gitlab::Database::BackgroundMigration::BatchedMigration:0x00007f79a6324250
id: 450,
created_at: "2025-10-28 06:57:31.718787000 +0000",
updated_at: "2025-10-28 06:57:31.718787000 +0000",
min_value: 1,
max_value: nil,
batch_size: 1000,
sub_batch_size: 100,
interval: 120,
status: 1,
job_class_name: "BackfillPartitionedWebHookLogsDaily",
batch_class_name: "PrimaryKeyBatchingStrategy",
table_name: "web_hook_logs",
column_name: "id",
job_arguments: [],
total_tuple_count: nil,
pause_ms: 100,
max_batch_size: nil,
started_at: "2025-10-28 06:57:31.717179000 +0000",
on_hold_until: nil,
gitlab_schema: "gitlab_main",
finished_at: nil,
queued_migration_version: nil,
min_cursor: [0, "2025-09-28 06:57:31 UTC"],
max_cursor: [0, "2025-10-28 06:57:31 UTC"]>
Yes, this is indeed our stuck background migration task "BackfillPartitionedWebHookLogsDaily".
Now we can delete this task:
ck@gitlab:~# sudo gitlab-rails runner "Gitlab::Database::BackgroundMigration::BatchedMigration.find(450).delete"
And the active background migrations no longer shows this stuck task:
ck@gitlab:~# sudo gitlab-rake gitlab:background_migrations:list | grep -v "100.00%"
id | table_name | job_class_name | status | progress
---------|---------------------------|------------------------------------------|-----------|---------
The Admin UI also confirms that there are no pending or active background migrations anymore:
Now I can continue with the Gitlab updates :-).
AI AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Observability Office OpenSearch PHP Perl Personal PostgreSQL PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Znuny Zoneminder