test_pkey.rb: Add PQC algorithm SLH-DSA (FIPS 205) test

[junaruga]

I want to add SLH-DSA (FIPS 205) test to ruby/openssl to make sure SLH-DSA works in Ruby OpenSSL.

Commit message

Add PQC algorithm SLH-DSA (FIPS 205) test as well as existing ML-KEM (FIPS 203) and ML-DSA (FIPS 204) in test/openssl/test_pkey.rb.

https://csrc.nist.gov/projects/post-quantum-cryptography

OpenSSL 3.5.0 supports PQC algorithm SLH-DSA. It seems LibreSSL and AWS-LC don't support SLH-DSA yet.

https://openssl-library.org/post/2025-04-08-openssl-35-final-release/

Note we don't add SLH-DSA tests in test/openssl/test_ssl.rb. SLH-DSA is used for the signature case like ML-DSA. But SLH-DSA's signature size is larger than ML-DSA's. So, I am not sure that SLH-DSA is used in SSL/TLS signature case.

The used SLH-DSA parameter set in test_slh_dsa is SLH-DSA-SHA2-128s which is one of the total 12 parameter sets. This is the lowest security strength (security category 1) with small key and signature sizes.

This aligns with ML-DSA-44 used in test_ml_dsa. ML-DSA-44 also has the lowest security strength (security category 2) in the 3 ML-DSA parameter sets, ML-DSA-44, ML-DSA-65, ML-DSA-87.

References:

• SLH-DSA parameter sets: FIPS 205 Section 11 Parameter Sets - https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.205.pdf
• ML-DSA parameter sets: FIPS 204 Section 4 Parameter Sets - https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.204.pdf

Assisted-by: Claude:Opus 4.6

[rhenium]

Does SLH-DSA require any different handling from the other signature algorithms in the OpenSSL C API? If SLH-DSA works through the same generic EVP_PKEY paths, I feel these tests would be mostly duplicate.

It also seems rather slow: https://github.com/ruby/openssl/actions/runs/35600244714/job/106334376765?pr=1113#step:10:61

ML-DSA was interesting because it was one of the first algorithms to not have an NID and required special treatment (we changed #inspect and .new_raw_*_key).