How to clean up stuck background migration task BackfillPartitionedWebHookLogsDaily in Gitlab CE

Written by - 0 comments

Published on - Listed in Git Linux


After (and especially before) every Gitlab update, it is advised to keep an eye on the update/migration jobs, which run in the background. These so-called Background migrations are launched automatically after every update. They make sure that data (for example database) is adjusted to the new version. 

Before continuing with the Gitlab upgrade path (if you need to do multiple version updates in a row), you should wait until these Background migrations are finished.

Multiple background migration tasks after update

After our Gitlab CE instance was updated (in this situation from 18.8 to 18.9), the background migration tasks can be listed with the following command:

ck@gitlab:~# sudo gitlab-rake gitlab:background_migrations:list
id       | table_name        | job_class_name                          | status    | progress  
---------|-------------------|-----------------------------------------|-----------|-----------
main_1   | integrations      | BackfillIntegrationsTypeNew             | finished  | 100.00%   
main_2   | namespaces        | BackfillUserNamespace                   | finished  | 100.00%   
main_3   | issues            | BackfillIssueSearchData                 | finished  | 100.00%   
main_4   | routes            | BackfillNamespaceIdForNamespaceRoute    | finished  | 100.00% 
[...]

The output shows a lot of entries. Relevant for your eye are the status and progress. To retrieve a list of still active jobs, you could filter for the "active" filter or don't show the 100% entries:

ck@gitlab:~# sudo gitlab-rake gitlab:background_migrations:list | grep -v "100.00%"

id       | table_name              | job_class_name                              | status    | progress 
---------|-------------------------|---------------------------------------------|-----------|----------
main_450 | web_hook_logs           | BackfillPartitionedWebHookLogsDaily         | active    |          
main_703 | oauth_applications      | BackfillOrganizationIdOnOauthApplications   | active    | 50000.00%
main_704 | projects                | BackfillMissingProjectPushRules             | active    | 0.00%    
main_705 | project_authorizations  | MigrateProjectAuthorizations                | active    |          
main_708 | web_hook_logs_daily     | BackfillWebHookLogsDailyShardingKeys        | active    |         

This means after the update of the Omnibus package was completed, five background migrations were active.

A couple of minutes later only one remained:

ck@gitlab:~# sudo gitlab-rake gitlab:background_migrations:list | grep -v "100.00%"
id       | table_name              | job_class_name                              | status    | progress 
---------|-------------------------|---------------------------------------------|-----------|----------
main_450 | web_hook_logs           | BackfillPartitionedWebHookLogsDaily         | active    |

I waited and checked the same command again every couple of minutes. But this migration task still remained. And it didn't seem to make any progress either.

In the Admin UI, the background migration also shows up as running but with 0% progress:

Removing the background migration task

After some (traditional non AI) research, I came across a Gitlab issue, that mentioned the same problem, with the same background migration being stuck.

According to one of the Gitlab maintainers, this background migration can be safely deleted, as the relevant table does no longer exist in the database.

We can mark this migration as completed as the web_hook_logs table was removed in %18.0.

In order to delete the task, we need to run a "Gitlab-internal" rails command. This can either be done by using the Gitlab console (sudo gitlab-rails console) or by using gitlab-rails as wrapper.

First let's verify we're looking at the correct background migration. According to the output above the id is 450. This can also be verified in the UI (the ID shows up in the link of this particular background migration task).

ck@gitlab:~# sudo gitlab-rails runner "pp Gitlab::Database::BackgroundMigration::BatchedMigration.find(450)"
#<Gitlab::Database::BackgroundMigration::BatchedMigration:0x00007f79a6324250
 id: 450,
 created_at: "2025-10-28 06:57:31.718787000 +0000",
 updated_at: "2025-10-28 06:57:31.718787000 +0000",
 min_value: 1,
 max_value: nil,
 batch_size: 1000,
 sub_batch_size: 100,
 interval: 120,
 status: 1,
 job_class_name: "BackfillPartitionedWebHookLogsDaily",
 batch_class_name: "PrimaryKeyBatchingStrategy",
 table_name: "web_hook_logs",
 column_name: "id",
 job_arguments: [],
 total_tuple_count: nil,
 pause_ms: 100,
 max_batch_size: nil,
 started_at: "2025-10-28 06:57:31.717179000 +0000",
 on_hold_until: nil,
 gitlab_schema: "gitlab_main",
 finished_at: nil,
 queued_migration_version: nil,
 min_cursor: [0, "2025-09-28 06:57:31 UTC"],
 max_cursor: [0, "2025-10-28 06:57:31 UTC"]>

Yes, this is indeed our stuck background migration task "BackfillPartitionedWebHookLogsDaily".

Now we can delete this task:

ck@gitlab:~# sudo gitlab-rails runner "Gitlab::Database::BackgroundMigration::BatchedMigration.find(450).delete"

And the active background migrations no longer shows this stuck task:

ck@gitlab:~# sudo gitlab-rake gitlab:background_migrations:list | grep -v "100.00%"

id       | table_name                | job_class_name                           | status    | progress
---------|---------------------------|------------------------------------------|-----------|---------

The Admin UI also confirms that there are no pending or active background migrations anymore:

Now I can continue with the Gitlab updates :-).


More recent articles:

RSS feed

Blog Tags:

  AI   AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Observability   Office   OpenSearch   PHP   Perl   Personal   PostgreSQL   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Znuny   Zoneminder