My pager went off at 2:14 AM last Thursday. Our three-node staging cluster running AlmaLinux 9 had completely dropped off the network. No database connections. No API responses. Just a wall of connection timeouts.

Well, that’s not entirely accurate — I spent two hours pulling my hair out before I found the culprit. A routine, automated dnf update had pulled down the latest packages for the 9.7 release. And right there in the logs was the poison pill: OpenSSL had been bumped from 3.2 to 3.5.

In a minor release. Let that sink in for a second. We’ve spent decades operating under the assumption that enterprise Linux distributions don’t break Application Binary Interface (ABI) compatibility during minor updates. But Red Hat decided to ship a major OpenSSL upgrade in RHEL 10.1 and 9.7. And because AlmaLinux is faithfully bug-for-bug compatible with upstream, they inherited this exact behavior.

The collateral damage

The immediate result? Anything dynamically linked against the older OpenSSL 3.2 libraries panicked. My logs were absolutely flooded with this garbage:

ImportError: /lib64/libssl.so.3: version 'OPENSSL_3.2.0' not found (required by /usr/pgsql-16/lib/libpq.so.5)

PostgreSQL was dead in the water. Custom Python extensions we compiled locally? Segfaulting. Our older PgBouncer instances just refused to start. I had to scramble to roll 22 VMs back to their pre-update snapshots just to get our internal billing API back online.

The frustrating part is that this wasn’t a mistake in the code. It was a deliberate policy choice upstream that broke trust. When you swap out fundamental cryptographic libraries, you break the ecosystem. Period.

How the PostgreSQL community fixed it

And while I was busy drafting an angry internal post-mortem, I noticed something actually helpful happening in the database world. The maintainers of the PostgreSQL RPM repository clearly saw this trainwreck coming, and they quietly rolled out support for multiple RHEL minor versions in their YUM repositories.

Pinning the release version

First, you need to stop AlmaLinux from blindly upgrading to 9.7. I forced the release version in DNF:

echo "9.6" > /etc/dnf/vars/releasever
dnf clean all

Then, I updated our PostgreSQL repository configuration to use the new minor-version-specific paths. If you look at your /etc/yum.repos.d/pgdg-redhat-all.repo file, you need to ensure the baseurl is respecting that specific minor version variable instead of defaulting to a generic path.

A nasty Python edge case

I want to share a specific headache I ran into while testing the actual 9.7 upgrade path on a sandbox machine. If you’re running Python 3.12 and using the pre-compiled psycopg2-binary package from PyPI to connect to Postgres, you are going to have a bad time with OpenSSL 3.5. I ran a load test against a dummy database using the new OS libraries.

Under normal conditions on 9.6 (OpenSSL 3.2), our test API handles about 850 requests per second. But when I ran the exact same container on 9.7, the throughput randomly dropped to zero for 5-10 seconds at a time. The Python worker processes were silently dying and restarting.

The only way I fixed it was ditching the binary wheel and forcing a local compilation against the new OpenSSL 3.5 headers:

# Uninstall the binary wheel
pip uninstall psycopg2-binary

# Install the build dependencies for AlmaLinux 9.7
dnf install postgresql16-devel gcc python3.12-devel openssl-devel

# Build from source
pip install psycopg2 --no-binary psycopg2

Don’t trust pre-compiled database drivers right now if you’re making the jump to 9.7 or 10.1. Build them yourself.

I’m glad the PostgreSQL team adapted their repo structure to handle this mess. But honestly, we shouldn’t have to defensively configure our package managers just to survive a minor OS update. Until upstream decides to respect ABI boundaries again, pin your minor versions. Always.