Gitlab access tokens not working anymore after upgrade to 16.x

Written by - 2 comments

Published on - last updated on July 31st 2023 - Listed in Linux Git


After upgrading a local Gitlab CE installation from 14.x to 16.x I noticed that access tokens didn't work anymore for pulling a private repository:

ck@ansible:/pub/ansible$ cd /pub/ansible && /usr/bin/git pull https://ansible:my-super-secret-token@gitlab.example.com/infrastructure/ansible.git
fatal: unable to access 'https://gitlab.example.com/infrastructure/ansible.git/': The requested URL returned error: 500

During the Gitlab upgrade path, which involved multiple version upgrades, I tested git pull both with login credentials and access tokens:

  • 14.6.1 (initial version): both working
  • 14.9.5: both working
  • 14.10.5: both working
  • 15.0.5: both working
  • 15.1.6: both working
  • 15.4.6: both working
  • 15.11.13: both working
  • 16.2.1: login working, access tokens not working

The access token (ansible) is still showing up in the repository settings in the now upgraded Gitlab:

But "something" seems to have changed concerning the token handling and using this token now results in an internal server error (HTTP 500). 

Creating a new repository access token

I created a new access token (ansible2023) with the same permissions (role: developer, scope: read_repository) as the old token (ansible):

Both tokens now show up in the list:

Access tokens of a Gitlab repository

Note: I noticed the "Expires" column has changed for the original ansible token from "never" to "in 11 months"... ?

By using the new token (ansible2023) and its secret, the repository pull was working again:

ck@ansible:/pub/ansible$ cd /pub/ansible && /usr/bin/git pull https://ansible2023:my-new-fancy-secret@gitlab.example.com/infrastructure/ansible.git
From https://gitlab.example.com/infrastructure/ansible
 * branch            HEAD       -> FETCH_HEAD
Already up to date.

By the way, if you assign the "guest" role (instead of "developer"), this won't be enough to pull the repository and you get the following error:

ck@ansible:/pub/ansible$ cd /pub/ansible && /usr/bin/git pull https://ansible2023:my-new-fancy-secret@gitlab.example.com/infrastructure/ansible.git
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://gitlab.example.com/infrastructure/ansible.git/': The requested URL returned error: 403

In the Gitlab access logs, the pull can be seen with a HTTP 200:

xxx.xxx.xxx.xxx - - [28/Jul/2023:11:15:01 +0200] "GET /infrastructure/ansible.git/info/refs?service=git-upload-pack HTTP/1.0" 401 279 "" "git/2.30.2" -
xxx.xxx.xxx.xxx - ansible2023 [28/Jul/2023:11:15:01 +0200] "GET /infrastructure/ansible.git/info/refs?service=git-upload-pack HTTP/1.0" 200 164 "" "git/2.30.2" -
xxx.xxx.xxx.xxx - ansible2023 [28/Jul/2023:11:15:01 +0200] "POST /infrastructure/ansible.git/git-upload-pack HTTP/1.0" 200 145 "" "git/2.30.2" -

Where did the breaking change come from?

Of course the big question is: Why did it work before and since the upgrade to 16.x not anymore? The cause seems to be a change mentioned in the deprecation and removals documentation:

Non-expiring access tokens

    Announced in GitLab 15.4 (Sep 2022)
    Removal in GitLab 16.0 (May 2023) (breaking change)

Access tokens that have no expiration date are valid indefinitely, which presents a security risk if the access token is divulged. Because access tokens that have an exipiration date are better, from GitLab 15.3 we populate a default expiration date.
In GitLab 16.0, any personal, project, or group access token that does not have an expiration date will automatically have an expiration date set at one year.

It doesn't say whether or not the previously active tokens will become immediately invalid, but this seems to be the case given what I just experienced.

Update: Old token working again

Now a few days after the Gitlab upgrade and a weekend with food poisoning (ugh!), I tested the old access token once more. And interestingly it worked:

ck@ansible:/pub/ansible$ cd /pub/ansible && /usr/bin/git pull https://ansible:my-super-secret-token@gitlab.example.com/infrastructure/ansible.git
From https://gitlab.example.com/infrastructure/ansible
 * branch            HEAD       -> FETCH_HEAD
Already up to date.

So to me it looks that the upgrade to Gitlab 16.x triggered the following internally once Gitlab was started:

  • Detect old access tokens with a expiry date set to "never"
  • Migrate/adjust these tokens to an expiry date of ~1 year (in 11 months, as seen in the screenshot)
  • During that migration the token doesn't work and an internal server error (500) is shown to the git client


Add a comment

Show form to leave a comment

Comments (newest first)

ck from Switzerland wrote on Jul 31st, 2023:

Manuel, thanks for the comment. Yes, I saw that too. But as you can see from my upgrade experience, the previously used ansible token stopped working after the upgrade to Gitlab 16.x. I noticed that the expiry date was adjusted from never to in 11 months however this existing token did not work anymore after the upgrade. Only new tokens would work again. According to a Gitlab employee, I talked to in the meantime, this could be a bug.


Manuel Kraft from wrote on Jul 31st, 2023:

Hey Claudio,

Thank you for posting your upgrade documentation.

Just want to share with you and your readers, that the token DO NOT get enforced to stop working after upgrading to 16.0 / 16.x .

The personal access tokens will receive an expiration of 1 year in the future after you upgrade and batched background migrations are run.

As GitLab is open source, and if you need to see code to be confident, the value is set to 365.days.from_now here

Ref:
https://gitlab.com/gitlab-org/gitlab/-/blob/eb0f0465ff1f55f1347b981f2502a8e4d4fe6fe4/lib/gitlab/background_migration/cleanup_personal_access_tokens_with_nil_expires_at.rb#L10

Regarding project / group tokens: Under the hood Project and Group access tokens are also Personal Access tokens; the user they belong to is a "bot" user rather than a real human. So this new expiration applies to all 3 types of tokens.

Ref: https://docs.gitlab.com/ee/development/internal_users.html


RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder