Add composite signature interface for RSA and ECDSA signatures

[mamckee]

OpenSSL 3.5 introduced hardcoded composite signature + digest interfaces for RSA and ECDSA. This PR adds the following composite signature interfaces to the SymCrypt provider, implementing the sign/verify message functions in place of sign/verify digest functions. The original RSA and ECDSA signature interfaces are still available.

[Copilot]

Pull request overview

This PR adds OpenSSL 3.5 “composite” signature algorithm entries (e.g., RSA-SHA256 / ECDSA-SHA256) to the SymCrypt provider and implements message-based sign/verify flows for those composite algorithms while keeping the existing RSA/ECDSA interfaces intact.

Changes:

• Introduces shared RSA/ECDSA signature context headers and refactors existing implementations to expose reusable internal sign/verify helpers.
• Adds new RSA/ECDSA composite (“sigalg”) signature implementations that use SIGN_MESSAGE / VERIFY_MESSAGE entry points.
• Registers the new composite algorithm names in the provider’s signature algorithm table and adds the new sources to the build.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
SymCryptProvider/src/signature/p_scossl_rsa_signature.h	New shared RSA signature context + internal helper declarations.
SymCryptProvider/src/signature/p_scossl_rsa_signature.c	Refactor/expose RSA internal sign/verify; add state tracking for message APIs.
SymCryptProvider/src/signature/p_scossl_rsa_sigalg_signature.c	New RSA composite signature implementations (RSA-SHA* / RSA-SHA3-*).
SymCryptProvider/src/signature/p_scossl_ecdsa_signature.h	New shared ECDSA signature context + internal helper declarations.
SymCryptProvider/src/signature/p_scossl_ecdsa_signature.c	Refactor/expose ECDSA internal sign/verify; add state tracking for message APIs.
SymCryptProvider/src/signature/p_scossl_ecdsa_sigalg_signature.c	New ECDSA composite signature implementations (ECDSA-SHA* / ECDSA-SHA3-*).
SymCryptProvider/src/p_scossl_names.h	Adds composite algorithm name strings for RSA/ECDSA.
SymCryptProvider/src/p_scossl_base.c	Registers composite signature algorithms in the provider dispatch table.
SymCryptProvider/CMakeLists.txt	Adds new sigalg signature source files to the build.
ScosslCommon/src/scossl_rsa.c	Removes version guards around RSA_PSS_SALTLEN_AUTO_DIGEST_MAX usage (OpenSSL 3.5 baseline).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

SymCryptEngine/src/e_scossl_ecc.c:640

• The function comment now states it returns 1 (valid) / 0 (invalid) / -1 (error), but several error paths in this function still return SCOSSL_FAILURE (0). That makes internal errors indistinguishable from an invalid signature for OpenSSL callers. Either update the error returns here to -1, or adjust the comment to reflect the actual behavior.

// Return
// 1 (SCOSSL_SUCCESS) for valid signature
// 0 (SCOSSL_FAILURE) for invalid signature
// -1 for error
int e_scossl_eckey_verify(int type, _In_reads_bytes_(dgst_len) const unsigned char* dgst, int dgst_len,
                          _In_reads_bytes_(sig_len) const unsigned char* sigbuf, int sig_len, _In_ EC_KEY* eckey)
{
    const EC_KEY_METHOD* ossl_eckey_method = NULL;
    SCOSSL_ECC_KEY_CONTEXT *keyCtx = NULL;

    switch( e_scossl_get_ecc_context(eckey, &keyCtx) )
    {
    case SCOSSL_FAILURE:
        SCOSSL_LOG_ERROR(SCOSSL_ERR_F_ENG_ECKEY_VERIFY, ERR_R_OPERATION_FAIL,
            "e_scossl_get_ecc_context failed.");
        return SCOSSL_FAILURE;
    case SCOSSL_FALLBACK:
        ossl_eckey_method = EC_KEY_OpenSSL();
        PFN_eckey_verify pfn_eckey_verify = NULL;
        EC_KEY_METHOD_get_verify(ossl_eckey_method, &pfn_eckey_verify, NULL);
        if (!pfn_eckey_verify)
        {
            return SCOSSL_FAILURE;
        }
        return pfn_eckey_verify(type, dgst, dgst_len, sigbuf, sig_len, eckey);
    case SCOSSL_SUCCESS:
        break;
    default:
        SCOSSL_LOG_ERROR(SCOSSL_ERR_F_ENG_ECKEY_VERIFY, ERR_R_INTERNAL_ERROR,
            "Unexpected e_scossl_get_ecc_context value");
        return SCOSSL_FAILURE;
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

In VERIFYMSG mode, this frees ctx->pbSignature but doesn't set it to NULL before early returns (e.g., invalid signature size / memdup failure). That leaves a dangling pointer and can lead to double-free in p_scossl_rsa_freectx or later reinitialization. Set ctx->pbSignature = NULL immediately after OPENSSL_free(ctx->pbSignature) (matching the ECDSA sigalg implementation).

Suggested change

	OPENSSL_free(ctx->pbSignature);
	OPENSSL_free(ctx->pbSignature);
	ctx->pbSignature = NULL;

[Copilot]

p_scossl_rsa_sigalg_verify_message_final calls p_scossl_rsa_verify_internal with ctx->pbSignature/ctx->cbSignature without verifying that a signature was actually provided via OSSL_SIGNATURE_PARAM_SIGNATURE. If pbSignature is NULL (or length 0), scossl_rsa_pkcs1_verify will pass the NULL pointer into SymCrypt, which can crash. Add an explicit check that ctx->pbSignature != NULL && ctx->cbSignature > 0 (and raise an appropriate provider error) before verifying.

[Copilot]

There is a stray standalone \ line here. While it will splice the newline during preprocessing, it's very likely accidental and hurts readability (and may trigger compiler warnings). Remove the backslash line.

Suggested change

\

[Copilot]

p_scossl_ecdsa_sigalg_verify_message_final verifies using ctx->pbSignature/ctx->cbSignature without checking that a signature was provided via OSSL_SIGNATURE_PARAM_SIGNATURE. If pbSignature is NULL (or length 0), scossl_ecdsa_verify will dereference it while parsing DER and can crash. Add a guard that ctx->pbSignature != NULL && ctx->cbSignature > 0 (and raise a provider error) before calling p_scossl_ecdsa_verify_internal.

Suggested change

	if (ctx->pbSignature == NULL || ctx->cbSignature == 0)
	{
	ERR_raise(ERR_LIB_PROV, PROV_R_MISSING_SIGNATURE);
	return 0;
	}

[Copilot]

This header now defines ECDSA signature-size constants, and ScosslCommon/src/scossl_ecc.c changes scossl_ecdsa_verify to return a tri-state int (1 valid / 0 invalid / -1 error). However, the prototype below still returns SCOSSL_STATUS (annotated as success only when return==1), which makes -1 look like a successful status to static analysis/callers. Update the declaration to return int (and update any documentation accordingly) to match the implementation and new contract.

[Copilot]

This declaration was changed to return int and the comment above says it returns 1/0/-1. Ensure the implementation returns -1 on internal errors (not SCOSSL_FAILURE == 0), or update the comment to match the actual return behavior so callers interpret the result correctly.