I'm not sure why the latest round of tests failed - The only change in this last commit was a doc file change, so I'm guessing it's a transient CI failure. Is there a way to trigger it to retry?

Is there a way to trigger it to retry?

You can make an empty commit or ask a triager to rerun the CI (but there is none apart from me that is watching the PR I think)

This looks great. I was planning to add some PQ support to Python now that ML-KEM/DEM were standardized but this is a good start as well. I'll have a look at the tests when I'm on my laptop again (Friday) but you can consider this PR to be approved unless I find something by then.

Thanks very much! There seems to be less urgency in providing support for ML-DSA and SLH-DSA for authentication than there was for ML-KEM for key agreement mainly because of the "capture now, decrypt later" concerns, but I'd be happy to contribute to an effort around supporting those as well. I'd actually like to learn more about what OpenSSL APIs actually need to change here.

I am currently modernizing the use of OpenSSL HMAC in hashlib and I actually wondered whether we used APIs that were deprecated in 3.0. If you want, you can help me here, at least for the SSL module (I'd prefer that we work on separate modules to avoid merge conflicts), namely look at the calls we make to OpenSSL and check if there are deprecated calls in 3.x. If there are, open an issue and a PR that modernize such calls (for HMAC, we moved from using the old HMAC_* interface to the more generic EVP_MAC interface)

If you want, you can help me here, at least for the SSL module (I'd prefer that we work on separate modules to avoid merge conflicts), namely look at the calls we make to OpenSSL and check if there are deprecated calls in 3.x.

I don't know if I'll have the cycles to actually take on fixing all the issues that we find, but I gave this a quick look to get a sense for the scope of the problem. There aren't as many changes as I was expecting to find, but it's still fairly complicated if we want to continue to support all the way back to OpenSSL 1.1.1 while avoiding functions deprecated in OpenSSL 3.x.

In many cases, the old API calls still exist in 3.x but the new API will only work on 3.x. I'm thinking leaving the old calls in place may be our best bet for now, to avoid conditional compilation. An example is replacing all references to BIO_new() with BIO_new_ex() where we pass in an explicit NULL argument for the library context. It doesn't seem worth a #if everywhere that appears as long as 3.x continues to provide a wrapper for the old BIO_new() call. A similar issue shows up around one use of BIO_new_mem_buf().

Here are some of the items I've found:

• It looks like OpenSSL has split the cipher setting function into two separate functions for TLS 1.2 and below vs. TLS 1.3. The code currently calls SSL_CTX_set_cipher_list() which is TLS 1.2-only and there's another function SSL_CTX_set_ciphersuites() for TLS 1. Since we don't call the latter function right now, I think that means there's no way currently to set a non-default TLS 1.3 cipher suite list. I'm thinking this is a good candidate for treating as a bug.
• There are some calls (d2i_X509_bio and i2d_X509) for reading certs that should be changed to use the OSSL_DECODER and OSSL_ENCODER APIs.
• There's code reading DH params with PEM_read_DHparams() which returns a low-level DH object. I think should change to use PEM_read_bio_Parameters(), which returns an EVP_PKEY. There's also a call to DH_free that would change to EVP_PKEY_free().
• Another place is setting DH params with SSL_CTX_set_tmp_dh(), and that should change to SSL_CTX_set0_tmp_dh_pkey(), which operates on EVP_PKEYs.
• There's code using SSL_CTX_set_psk_client_callback() and SSL_CTX_set_psk_server_callback() which could be changed to use the OpenSSL 3.x SSL_CTX_set_psk_use_session_callback() and SSL_CTX_set_psk_find_session_callback() APIs. However, I think those new APIs only work on TLS 1.3, so this may be another case where we need to keep both around, or just continue to use the deprecated API for now, since I think it is possible to use that for both TLS 1.2 and TLS 1.3 connections if you're careful. There were some notes on this in the OpenSSL migration guide.

Following up on my last comment: In the SSL module docs, there's a "TLS 1.3" section which actually mentions that set_ciphers() can't be used to control which TLS 1.3 ciphers are enabled, even though both TLS 1.3 and older ciphers are returned by the get_ciphers() call. So, I guess this isn't a bug. There are a number of other documented limitations here, though, so I wonder if correcting some of those issues might be more useful than any deprecation issues in the OpenSSL APIs.

Helping fixing those issues or modernizing API calls is appreciated so choose what you want!

Thanks!

I've actually got another change ready which adds support for setting TLS 1.3 ciphers using a new set_ciphersuites() function on SSLContext. This mirrors the OpenSSL API which has SSL_CTX_set_cipher_list() and SSL_CTX_set_ciphersuites() functions. I thought about trying to do it all from within the existing set_ciphers() call, but that actually turns out to be really messy, especially given that the SSL_CTX_set_cipher_list() supports punctuation and partial matching on the various parts of the cipher suite name, whereas SSL_CTX_set_ciphersuites() just takes a simple colon-delimited list of exact TLS 1.3 cipher names. Keeping them separate is also important to let each of the two have its own independent default value. So, if you only call one of these two functions, the other one remains set to its default. It also does a better job reporting errors when the calls are separate, as you can report separately whether it failed to match any of the supported ciphers of the associated version in each of the two calls.

It'll probably be easier if I wait until this PR is merged before I create a new issue and PR to avoid conflicts in the "what's new" entry. Alternately, if you want me to expand the scope of this issue/PR to cover both setting TLS 1.3 cipher and TLS 1.3 groups and check in the additional changes hree, I could do that. While they're independent, both of them are about providing more complete TLS 1.3 support, so I'm good with either option.

I would prefer having separate issues and PRs even though they are related. It makes reverting stuff easier. I will be available tomorrow (maybe this evening, Paris time)

No problem - I'll hold onto the cipher suite change for now. Also, I have a third change planned to add support for getting and setting sigalgs. It should be able to follow the same pattern as ciphers and groups.

Once those changes are in, I'll revisit use of deprecated APIs, but for now I'd recommend only rewriting such code if the preferred replacement is supported in 1.1.1 or earlier. That will avoid needing to keep two versions of the code around depending on the OpenSSL version. As an example, I think replacing code which uses DH APIs with EVP APIs should be doable without any compile-time conditionals, but for now I'd avoid replacing calls to BIO_new, as the replacement function BIO_new_ex() doesn't appear until OpenSSL 3.0.

I just noticed that I left off the "gh-136306" off of my last commit. Should I amend and force push to fix that, or is it ok to proceed without that?

Is there anything else you'd like me to change?

Ah, you need to regenerate the files as there was a conflict created by something.

Should I amend and force push to fix that, or is it ok to proceed without that?

In general, we don't force push commits because we squash everything at the end (and usually I rewrite the commit message as well to remove some commits messages that are not necessary). It's also better not to force-push as it makes incremental review harder. I've left a comment somewhere saying when it's fine to force-push but I never find it again when I need to...

Do you need anything else from me on this change? I'm looking forward to getting this in so I can move on to the cipher suite and signature algs updates!

Scratch that - I just saw your other message about the conflict. I'll look into it.