Silent salt_len truncation in signature_algorithm_identifier() causes signature verification failures

[MinghuaWang]

Summary

SigningKey::signature_algorithm_identifier() truncates salt_len from usize to u8 via an as u8 cast. When salt_len >= 256, the encoded AlgorithmIdentifier contains an incorrect salt length (e.g., 256 becomes 0). Any compliant verifier that reads this AlgorithmIdentifier will use the wrong salt length, causing valid signatures to fail verification.

This is a functional correctness bug caused by numeric truncation.

Details

In src/pss/signing_key.rs:223-224, the DynSignatureAlgorithmIdentifier implementation for SigningKey<D>:

fn signature_algorithm_identifier(&self) -> spki::Result<AlgorithmIdentifierOwned> {
    get_pss_signature_algo_id::<D>(self.salt_len as u8)  //truncation here
}

self.salt_len is usize (set by the user via SigningKey::new_with_salt_len(key, salt_len)), but it is cast to u8 before being passed to get_pss_signature_algo_id. This discards all bits above the lowest 8, so any value >= 256 wraps modulo 256.

The same bug exists in src/pss/blinded_signing_key.rs:187:

fn signature_algorithm_identifier(&self) -> spki::Result<AlgorithmIdentifierOwned> {
    get_pss_signature_algo_id::<D>(self.salt_len as u8)  //same truncation
}

get_pss_signature_algo_id (src/pss.rs:280) encodes the truncated value into RsaPssParams, which is then serialized as DER into the AlgorithmIdentifier:

fn get_pss_signature_algo_id<D>(salt_len: u8) -> spki::Result<AlgorithmIdentifierOwned> {
    let pss_params = RsaPssParams::new::<D>(salt_len);
    Ok(AlgorithmIdentifierOwned {
        oid: ID_RSASSA_PSS,
        parameters: Some(Any::encode_from(&pss_params)?),
    })
}

There is no validation that salt_len fits in u8 anywhere in the call chain.

PoC

The following PoC demonstrates that when salt_len = 256, verification fails because the AlgorithmIdentifier contains a truncated salt length. For comparison, it also includes two valid cases, salt_len = 255 and salt_len = 128, where the salt length fits in u8 and verification succeeds.

use rsa::pss::{Signature, SigningKey, VerifyingKey};
use rsa::RsaPrivateKey;
use sha2::Sha256;
use signature::{Keypair, RandomizedSigner, Verifier};
use spki::DynSignatureAlgorithmIdentifier;

fn sign_verify_test(
    rng: &mut impl signature::rand_core::CryptoRng,
    private_key: RsaPrivateKey,
    message: &[u8],
    salt_len: usize,
) -> bool {
    let signing_key = SigningKey::<Sha256>::new_with_salt_len(private_key, salt_len);
    let signature: Signature = signing_key.sign_with_rng(rng, message);

    let algo_id = signing_key
        .signature_algorithm_identifier()
        .expect("encoding AlgorithmIdentifier");

    use spki::der::{Decode, Encode};
    let params_bytes = algo_id.parameters.unwrap().to_der().unwrap();
    let pss_params = pkcs1::RsaPssParams::from_der(&params_bytes).unwrap();
    let decoded_salt_len = pss_params.salt_len;
    println!("salt_len: {salt_len} truncated as u8: {decoded_salt_len}"); // BUG: 256 as u8 == 0!

    let pub_key = signing_key.verifying_key().as_ref().clone();
    let verifying_key =
        VerifyingKey::<Sha256>::new_with_salt_len(pub_key, decoded_salt_len as usize);
    let verifies = verifying_key.verify(message, &signature).is_ok();

    verifies
}

fn main() {
    let mut rng = rand::rng();
    let private_key = RsaPrivateKey::new(&mut rng, 3072).expect("key generation");

    let message = b"hello, world";

    // success case
    let salt_len = 255;
    let verifies_255 = sign_verify_test(&mut rng, private_key.clone(), message.as_slice(), salt_len);
    println!("salt_len = {salt_len}, verification {verifies_255}");
    println!();

    // success case
    let salt_len = 128;
    let verifies_128 = sign_verify_test(&mut rng, private_key.clone(), message.as_slice(), salt_len);
    println!("salt_len = {salt_len}, verification {verifies_128}");
    println!();

    // failed case
    let salt_len = 256;
    let verifies_256 = sign_verify_test(&mut rng, private_key.clone(), message.as_slice(), salt_len);
    println!("salt_len = {salt_len}, verification {verifies_256}");
}

Reproduce:

cargo run --example poc_salt_len_truncation --features encoding

Output:

salt_len: 255 truncated as u8: 255
salt_len = 255, verification true

salt_len: 128 truncated as u8: 128
salt_len = 128, verification true

salt_len: 256 truncated as u8: 0
salt_len = 256, verification false

Explanation

The PoC output shows that verification fails for salt_len = 256 because the AlgorithmIdentifier encodes the salt length as 0 instead of 256. In contrast, verification succeeds for salt_len = 255 and salt_len = 128 because both values fit in u8 and are encoded correctly.

When salt_len = 256, the values used during the signing and verification are as follows:

In the signing process:

SigningKey::sign_with_rng(rng, msg)
  -> sign_digest::<_, Sha256>(rng, false, key, &hash, 256)
    -> fills 256-byte random salt
    -> sign_pss_with_salt_digest::<_, Sha256>(rng, key, hash, &salt)
      -> emsa_pss_encode_digest::<Sha256>(hash, em_bits=3071, &salt)
        -> db[em_len - s_len - h_len - 2] = 0x01   // db[94] = 0x01
        -> db[em_len - s_len - h_len - 1..].copy_from_slice(salt)  // salt at db[95..351]

db layout: [0x00 × 94] [0x01] [256 bytes salt]

In the verifying process (truncated salt_len=0 from AlgorithmIdentifier):

VerifyingKey::verify(msg, &sig)   // self.salt_len = Some(0)
  -> verify_digest::<Sha256>(pub_key, &hash, &sig.inner, Some(0))
    -> emsa_pss_verify_digest::<Sha256>(hash, em, Some(0), key_bits=3072)
      -> emsa_pss_verify_pre(hash, em, 3071, Some(0), 32)
        // guard passes: em_len(384) >= h_len(32) + s_len(0) + 2
      -> mgf1 unmasks db (recovers original db correctly)
      -> emsa_pss_verify_salt(db, em_len=384, s_len=0, h_len=32)
        -> split_at(em_len - h_len - s_len - 2)     // split_at(350)
        -> checks: are db[0..350] all zero?           // NO — db[94]=0x01 -> fails
        -> checks: is db[350] == 0x01?                // NO — db[350] is salt data
        -> returns salt_valid = FALSE
      -> Err(Error::Verification)

The verifier expects db layout for salt_len=0: [0x00 × 350] [0x01], but the actual db has 0x01 at position 94 and random salt bytes from 95 to 350. Both checks fail, so salt_valid = FALSE and verification returns Err.

Suggested fix

Validate at encoding time and return an error:

fn signature_algorithm_identifier(&self) -> spki::Result<AlgorithmIdentifierOwned> {
     fn signature_algorithm_identifier(&self) -> spki::Result<AlgorithmIdentifierOwned> {
-        get_pss_signature_algo_id::<D>(self.salt_len as u8)
+        use spki::der::{ErrorKind, Tag::Integer};
+        let salt_len: u8 = self
+            .salt_len
+            .try_into()
+            .map_err(|_| spki::der::Error::from(ErrorKind::Length { tag: Integer }))?;
+        get_pss_signature_algo_id::<D>(salt_len)
    }