How to renew expired certificates in MinIO S3 Object Storage Server

Written by Claudio Kuenzler - 0 comments

Published on October 26th 2023 - Listed in Linux Minio Cloud

---

MinIO is an open source alternative to the famous S3 object storage service from AWS, offering so-called S3 Buckets for static files. As AWS is very costly, the MinIO alternative comes in very handy. And best of it: It's not that complicated.

I've worked with MinIO for quite a while now, on and off since 2018. At first I needed to learn the hard way that MinIO, offering a stateful (persistent) object storage, should better not be put into Kubernetes (or you'll likely hit HA cluster issues). But this time I was called in for a different problem from a customer, running MinIO on-premise.

Oh no, certificates have expired

The customer contacted me because the backup process, which writes the backups into a S3 bucket on MinIO, failed with the following error:

RequestError: send request failed caused by: Get "https://minio.example.com:9000/backup?delimiter=%2F&encoding-type=url&max-k[..]": x509: certificate has expired or is not yet valid: current time 2023-08-30T02:08:04+02:00 is after 2023-07-21T12:43:49Z

Luckily the error message is quite clear about the problem: The TLS certificate has expired. Let's verify this on the MinIO server:

root@monitoring:~# /usr/lib/nagios/plugins/check_http -H minio.example.com -p 9000 -C 14
CRITICAL - Certificate 'minio.example.com' expired on Fri Jul 21 12:43:49 2023 +0000.

The certificate expiry check (here with check_http) confirms the error.

Where are the MinIO certificates?

It might not be obvious, but the MinIO certificates are located inside the HOME directory of the MinIO user (minio by default). To find the home directory of the minio user one way is to check out /etc/passwd:

root@minio:~# grep minio /etc/passwd
minio:x:997:992:MinIO:/var/local/minio:/usr/sbin/nologin

The home directory is therefore /var/local/minio, at least on this server and this installation. Located inside this path we can find a "hidden" directory .minio and located within .minio another directory called certs:

root@minio:~# ls -la /var/local/minio/.minio
total 12
drwx------ 3 minio minio 4096 Aug 26  2020 .
drwxr-xr-x 3 minio minio 4096 Aug 26  2020 ..
drwx------ 3 minio minio 4096 Aug 26  2020 certs

Inside the certs directory we can find the private key (private.key) and the certificate (public.crt) which are, in this case, symbolic links:

root@minio:~# ls -la /var/local/minio/.minio/certs
total 12
drwx------ 3 minio minio 4096 Aug 26  2020 .
drwx------ 3 minio minio 4096 Aug 26  2020 ..
drwx------ 2 minio minio 4096 Aug 26  2020 CAs
lrwxrwxrwx 1 root  root    34 Aug 26  2020 private.key -> /etc/pki/realms/domain/default.key
lrwxrwxrwx 1 root  root    38 Aug 26  2020 public.crt -> /etc/pki/realms/domain/public/full.pem

With openssl the certificate can be verified:

root@minio:~# openssl x509 -text -in /var/local/minio/.minio/certs/public.crtCertificate
    Data:
        Version: 3 (0x2)
        Serial Number:
            7e:86:b6:84:75:30:f4:26:e4:17:00:04:d3:f6:69:d1
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Example, OU = MinIO
        Validity
            Not Before: Jul 21 12:43:49 2020 GMT
            Not After : Jul 21 12:43:49 2023 GMT
        Subject: CN = minio.example.com
[...]

We can see the initial certificate was created with a 3 year validity - and has now expired.

How to create a new MinIO certificate

There are a couple of howto's and alternative ways how to use MinIO with TLS certificates. When using internal certificates, directly handled by the MinIO server, the best working way to create a new certificate was the certtool method.

First create a backup of the current private.key and public.crt. In this situation they are symbolic links, so the symlinks can just be deleted:

root@minio:/var/local/minio/.minio/certs# rm private.key public.crt

Then create a new private key using certtool:

root@minio:/var/local/minio/.minio/certs# certtool --generate-privkey --outfile private.key
Generating a 3072 bit RSA private key...

Note: Of course you can set a password on the private key, but then you'll also need to adjust the MinIO configuration so the key is automatically decrypted during server start.

Now create a new (self-signed) certificate using the new private key. This will ask a lot of questions (much more than you might be used to when creating certificates with openssl):

root@minio:/var/local/minio/.minio/certs# certtool --generate-self-signed --load-privkey private.key --outfile public.crt
Generating a self signed certificate...
Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.
Common name: minio.example.com
UID:
Organizational unit name: MinIO
Organization name: Example
Locality name: Zurich
State or province name: Zurich
Country name (2 chars): CH
Enter the subject's domain component (DC):
This field should not be used in new certificates.
E-mail:
Enter the certificate's serial number in decimal (123) or hex (0xabcd)
(default is 0x3f0d3a886bada6d95de2e2e47f51a99082f7c8c6)
value:


Activation/Expiration time.
The certificate will expire in (days): 3000


Extensions.
Does the certificate belong to an authority? (y/N): N
Is this a TLS web client certificate? (y/N): N
Will the certificate be used for IPsec IKE operations? (y/N): N
Is this a TLS web server certificate? (y/N): y
Enter a dnsName of the subject of the certificate: minio.example.com
Enter an additional dnsName of the subject of the certificate: example.com
Enter an additional dnsName of the subject of the certificate:
Enter a URI of the subject of the certificate:
Enter the IP address of the subject of the certificate:
Will the certificate be used for signing (DHE ciphersuites)? (Y/n): n
Will the certificate be used for encryption (RSA ciphersuites)? (Y/n): n
Will the certificate be used for data encryption? (y/N):
Will the certificate be used to sign OCSP requests? (y/N):
Will the certificate be used to sign code? (y/N):
Will the certificate be used for time stamping? (y/N):
Will the certificate be used for email protection? (y/N):
Enter the URI of the CRL distribution point:
X.509 Certificate Information:
    Version: 3
[...]

Is the above information ok? (y/N): y

Signing certificate...

Don't forget to change the ownership of the newly created files to minio:

root@minio:/var/local/minio/.minio/certs# chown minio:minio private.key public.crt

At the end this shows us a new private.key and public.crt in the certs directory:

root@minio:~# ls -la /var/local/minio/.minio/certs
total 24
drwx------ 3 minio minio 4096 Okt 17 17:26 .
drwx------ 3 minio minio 4096 Aug 26  2020 ..
drwx------ 2 minio minio 4096 Aug 26  2020 CAs
-rw------- 1 minio minio 8177 Okt 17 17:25 private.key
-rw-r--r-- 1 minio minio 1708 Okt 17 17:28 public.crt

Now restart the MinIO service and the new key and certificate are automatically read and activated:

root@minio:~# systemctl restart minio

What about external (public) access?

In the example above, MinIO is only used for internal backup purposes. But if you use MinIO as a public S3 Object Storage, I advise to use real certificates (either bought or from Let's Encrypt) on a reverse proxy and from there proxy-pass to MinIO.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.