maskGenAlgorithm for RSA signature with PKCS1-PSS padding - rsa

Question

I am generating RSA signature using RSA_PKCS1_PSS_PADDING. I am setting digest algorithm as SHA256 using EVP_get_digestbyname() and EVP_DigestSignInit(). And salt length parameter as -1 using EVP_PKEY_CTX_set_rsa_pss_saltlen().
I have EVP_MD_CTX, EVP_MD and EVP_PKEY_CTX structures used for signature generation.
How can I get the name of Mask generation algorithm name used by OpenSSL by default? Is there any API provided for getting it?
Edit: OpenSSL version used: 1.1.0g.

Answer 1

RSASSA-PSS is in practice always used with MGF1 as the Mask Generation Function. The only variation is which Message Digest is used internally by MGF1.
Sometime that's the same Message Digest as the one used for hashing the message and building the tag in PSS, because that makes the most sense. Other times it is SHA-1 because that used to be the default MD for early RSASSA-PSS APIs, thus for the associated MGF1.
In an ideal world, some attribute (in the signature, or/and in the public key certificate used to check the signature) would tell MGF1-with-such-MD, perhaps by way of some Object IDentifier like we have to specify PSS. But crypto APIs are hell.
In order to control what Message Digest is used by MGF1, we want something on the tune of what -sigopt rsa_mgf1_md:sha256 does in the openssl dgst command.
My best guess is to set the MGF1 digest using
assert(EVP_PKEY_CTX_set_rsa_mgf1_md(ctx, EVP_sha256)>=0);
or get it using EVP_PKEY_CTX_get_rsa_mgf1_md() as documented:
The EVP_PKEY_CTX_get_rsa_mgf1_md() macro gets the MGF1 digest for ctx. If not explicitly set the signing digest is used. The padding mode must have been set to RSA_PKCS1_OAEP_PADDING or RSA_PKCS1_PSS_PADDING.