The library was never the variable: four layers between OpenSSL 3.5 and a post-quantum database session
PostgreSQL could not name the hybrid group, the distribution withheld the signature algorithms, the authentication protocol failed closed, and the CA's runtime chose nothing.
The objective was to establish whether the connections to a certificate authority's database could be post-quantum, and to attribute precisely what each component in the path contributed to the answer. The experiment was planned as a comparison of two OpenSSL versions. It became a study of four layers, none of which was OpenSSL.
The database cannot name the algorithm. PostgreSQL 17 configures key exchange through ssl_ecdh_curve, which resolves names in OpenSSL's elliptic-curve table. The hybrid group is not a curve:
FATAL: ECDH: unrecognized curve name: X25519MLKEM768So every session on both PostgreSQL 17 images, one linked against OpenSSL 3.0 and one against 3.5.7, was P-256. The library comparison was moot at the application layer. PostgreSQL 18 replaces the parameter with ssl_groups, which hands the list to SSL_CTX_set1_groups_list; the stack was migrated by pg_dump and pg_restore onto a new volume, and a hybrid-only offer completed.
The distribution decides what a client offers. With an ML-DSA-65 certificate on the database, every client, including one linked against OpenSSL 3.5.7 with ML-DSA in its algorithm list, was refused: no suitable signature algorithm. The cause was a single include line in the system openssl.cnf that pulls in the distribution's crypto policy, which fixes a classical list of signature algorithms for every OpenSSL process on the host. With -sigalgs mldsa65 the handshake completed, and with -groups X25519MLKEM768 as well it was a fully post-quantum session to a database: hybrid key exchange, ML-DSA-65 authentication, ML-DSA-87 root. For psql, which has no such flag, a private OPENSSL_CONF did the same.
The authentication protocol fails closed. libpq's SCRAM channel binding hashes the server certificate with the digest of its signature algorithm. ML-DSA has no digest:
could not find digest for NID UNDEFThat is correct behavior on both sides, and the only remedy today is channel_binding=disable, a trade-off someone must sign.
The runtime chose nothing. The certificate authority's own JDBC session, on the OpenJDK 17 its image ships, negotiated a classical group beside a psql that negotiated the hybrid; against the ML-DSA certificate it could not connect at all, thirty-three handshake failures and a dead command line. The same database, the same evening, the same certificate: a post-quantum session for the client, none for the authority that issued it. A later experiment with Java 25 changed nothing on the wire.
An operator who upgrades the library and measures nothing has raised the floor and left every ceiling where it was. The method that found each ceiling was the same: a single-group, single-scheme offer from s_client, read against the server's log, which on PostgreSQL names the refusal precisely enough to tell the layers apart.
In the book: Chapter 19 carries the full transcript, the figures and the runbook. In the apparatus: the fix is in the repository, pinned to the versions that were measured. In your estate: if this is the problem you are looking at, this is what an engagement does about it.