Ubuntu's unattended-upgrades breaking the internet

Today many Ubuntu hosted websites breaks or experience high error rates.

Ubuntu Security Notices (USNs) are announced, https://www.ubuntu.com/usn/usn-3239-2/.

The culprit was a library called libc6. Specifically of this version 2.23-0ubuntu6 introduced an internal ABI change within the resolver library.

Which means you cannot resolve DNS and make calls to remote database e.g. RDS or external API requests.

If you are affected and running PHP, you will probably see this in error log:

... php_network_getaddresses: getaddrinfo failed: Name or service not known ...

Run this command to see if the faulty version of libc is installed via unattended upgrades.

$ sudo cat /var/log/unattended-upgrades/unattended-upgrades-dpkg.log | grep libc6

Preparing to unpack .../libc6_2.23-0ubuntu6_amd64.deb ...
Unpacking libc6:amd64 (2.23-0ubuntu6) over (2.23-0ubuntu5) ...
Setting up libc6:amd64 (2.23-0ubuntu6) ...

One of my colleague suggested a solution to downgrade the libc version.

$ apt-get -y --allow-downgrades install libc6=2.23-0ubuntu5

At the time of writing this blog, Ubuntu has released a fixed, glibc (2.23-0ubuntu7). Read more here https://launchpad.net/ubuntu/+source/glibc/2.23-0ubuntu7.

Just apply new updates with:

$ sudo apt-get update
$ sudo apt-get dist-upgrade

Lastly, you can turn off unattended upgrades. To disable it, run the following command.

$ sudo dpkg-reconfigure unattended-upgrades

And confirm the changes.

$ cat /etc/apt/apt.conf.d/20auto-upgrades

APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Unattended-Upgrade "0";

There you go. Have a nice day!