Contabo re-enables root SSH login on every boot (Ubuntu 24.04)

If you rent a Contabo VPS running Ubuntu 24.04, harden SSH, and reboot, root login over SSH comes back. Not because your configuration was wrong — because cloud-init rewrites it at every single boot.

We found this while locking down a fresh Contabo box. The hardening applied cleanly, sshd -T confirmed it, and then the reboot quietly undid part of the work. Here is exactly what happens, and how to make it stick.

What cloud-init actually runs on your server#

Contabo ships provisioning data with the image. You can read it on any of their Ubuntu instances:

sudo cat /var/lib/cloud/instance/cloud-config.txt

Inside it there is a bootcmd block:

bootcmd:
- sed 's/#PermitRootLogin .*/PermitRootLogin yes/' -i /etc/ssh/sshd_config
- grep PermitRootLogin /etc/ssh/sshd_config || echo PermitRootLogin yes >> /etc/ssh/sshd_config
- pkill -HUP sshd

The critical detail is the module it lives in. Most cloud-init modules run once per instance — they set a semaphore file in /var/lib/cloud/instance/sem/ and never run again. bootcmd is different. It is per_always. It runs on every boot, forever.

So every restart, your sshd_config gets rewritten to permit root login.

The failure that hides it#

You will probably notice a failed service before you notice the SSH change:

systemctl --failed
● cloud-init.service   loaded failed failed Cloud-init: Network Stage

It looks alarming and it is almost always ignored, because networking works fine. Here is why it fails:

Failed to run bootcmd module bootcmd
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Exit code: 1

The third command, pkill -HUP sshd, returns non-zero. Ubuntu 24.04 uses socket activation for SSH: ssh.socket listens, and systemd spawns an sshd process per connection. There is no long-lived daemon for pkill to signal, so it exits 1 and cloud-init reports the module as failed.

This is the dangerous part. The failure is cosmetic; the damage is already done. The sed ran first and succeeded. Your config was modified. Only the pointless reload failed.

Why your hardening file loses#

The usual advice is to drop your settings into /etc/ssh/sshd_config.d/ with a high number, like 99-hardening.conf. On this image that file does nothing.

Two rules combine against you:

1. sshd uses the first value it finds for any keyword, not the last. This is the opposite of how most config systems behave, and it is where nearly everyone goes wrong.
2. The Include sits near the top of sshd_config, and files inside load in alphabetical order.

So the resolution order is:

/etc/ssh/sshd_config.d/50-cloud-init.conf        PasswordAuthentication yes   ← wins
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf PasswordAuthentication no    ← ignored
/etc/ssh/sshd_config.d/99-hardening.conf         PasswordAuthentication no    ← ignored
/etc/ssh/sshd_config                             PermitRootLogin yes          ← ignored

Ubuntu’s own image ships a file saying no that is beaten by cloud-init’s file saying yes. Your 99- file never had a chance.

This is not specific to Contabo. It was reported upstream against cloud-init and closed as “not planned” — the behaviour is considered intentional. It is not going to be fixed for you.

The fix#

1. Name the file so it sorts first#

Not 99-. Use 01-, so it is read before anything cloud-init writes:

sudo tee /etc/ssh/sshd_config.d/01-hardening.conf > /dev/null <<'EOF'
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
MaxAuthTries 3
AllowUsers deploy
EOF

Because first-match wins, this now outranks both 50-cloud-init.conf and the bootcmd rewrite of the main file. The sed still runs every boot — it just no longer matters.

2. Confirm a key works before you commit#

Do this before restarting anything, in a second terminal, keeping your current session open:

ssh -o BatchMode=yes deploy@your-server 'echo ok'

BatchMode=yes disables the password prompt, so this only succeeds if key authentication genuinely works. If it fails, fix that first — with PasswordAuthentication no and no working key, you are locked out permanently.

3. Make the file hard to remove by accident#

Six months from now, someone tidying sshd_config.d deletes the odd 01- file and root SSH silently returns at the next reboot. Prevent it:

sudo chattr +i /etc/ssh/sshd_config.d/01-hardening.conf

Even root cannot delete it without clearing the flag first. To edit it later:

sudo chattr -i /etc/ssh/sshd_config.d/01-hardening.conf
# edit
sudo chattr +i /etc/ssh/sshd_config.d/01-hardening.conf

Put a comment at the top of the file explaining why it is named 01-. The next person will not know, and the failure is silent.

Verify it after a reboot, not before#

Never trust the file you wrote. Ask the daemon what it actually resolved:

sudo sshd -T | grep -E 'permitrootlogin|passwordauthentication|allowusers'

You want:

permitrootlogin no
passwordauthentication no
allowusers deploy

Then reboot and run it again. A hardening job that has not survived a restart is not finished — on this image, the restart is precisely when it comes undone.

If you would rather remove cloud-init entirely#

On a long-lived server that is already provisioned, cloud-init has no remaining job:

sudo touch /etc/cloud/cloud-init.disabled

This stops the boot-time rewrite at its source and clears the failed unit. Be aware it also stops cloud-init managing hostname and /etc/hosts, so check those are set statically first. If your provider ever rebuilds the instance from the image, the provisioning runs again from scratch anyway.

Worth checking on your own servers#

If you run Ubuntu 24.04 anywhere, this takes ten seconds:

sudo sshd -T | grep permitrootlogin
sudo grep -r PasswordAuthentication /etc/ssh/sshd_config.d/

If the first says yes when you believed you had disabled it, you have this problem. We found 75 failed root login attempts in 24 hours on a box that had been online less than a day — those attempts are constant, automated, and they only need the door to be open once.

---