Sign sha256 hash with RSA using pkcs#11 api? - rsa

Question

I'm currently using pyPkcs11 to sign files.
The following call works for signing common files with RSA and sha256,
session.sign(privKey, toSign, Mechanism(CKM_SHA256_RSA_PKCS, None)
But some of my files are already hashed (sha256), and the signature needs to give the same output that would be given by this openSSL command:
openssl pkeyutl -sign -pkeyopt digest:sha256 -in <inFilePath> -inkey <keyPath> -out <outFilePath>
I have tried the following call, which does not generate a hash of the file before signature,
session.sign(privKey, toSign, Mechanism(CKM_RSA_PKCS, None)
But the result is not the one i expected, and according to the first answer of this post CKM_RSA_PKCS vs CKM_RSA_X_509 mechanisms in PKCS#11,
CKM_RSA_PKCS on the other hand also performs the padding as defined in the PKCS#1 standards. This padding is defined within EMSA-PKCS1-v1_5, steps 3, 4 and 5. This means that this mechanism should only accept messages that are 11 bytes shorter than the size of the modulus. To create a valid RSASSA-PKCS1-v1_5 signature, you need to perform steps 1 and 2 of EMSA-PKCS1-v1_5 yourself.
After some research it appears that my file contains the first step of the signature described by the RFC 3447, so the missing part is the second one, where the ASN.1 value is generated.
Can I force this operation with pkcs11 and how ?
The PKCS#11 documentation doesn't seem to contain any information about it.

Answer 1

I see two ways to do this; the appropriate one depends on the token (not all tokens/wrappers do all machanisms).
As explained in this other answer, you could decorate the 32-octet SHA-256 that you start from into a full padded message representative. Basically, as explained in PKCS#1, if the RSA key is k octets (with I assume k≥51+11=62 octets, that is a public modulus at least 8⋅62-7=489 bits, which is a must for security), you
Append on the left the 19-octet string
30 31 30 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20
yielding a 51-octet string, which really is the ASN.1 DER encoding for the hash type and value per
DigestInfo ::= SEQUENCE {
digestAlgorithm DigestAlgorithm,
digest OCTET STRING
}
Further append on the left the string of k-51 octets (of which k-51-3 are FF)
00 01 FF FF FF FF..FF FF FF FF 00
yielding a k-octet string
Sign with mechanism CKM_RSA_X_509 (length k octets).
Or, alternatively: perform as in [1.]; skip [2.]; and in [3.] use mechanism CKM_RSA_PKCS (length 51 octets).
Disclaimer: I did not check, and have not used a PKCS#11 device lately.
Note: While there is no known attack against proper implementations of it, use of PKCS#1 v1.5 signature padding is increasingly frowned at; e.g. French authorities recommend
RecomSignAsym-1. Il est recommandé d’employer des mécanismes de signature asymétrique disposant d’une preuve de sécurité.
Or, in English:
It is recommended to use asymmetric signature mechanisms featuring a security proof
They mention RSA-SSA-PSS (sic). As a bonus, the PKCS#11 implementation of that is mechanism CKM_RSA_PKCS_PSS which accepts a hash, rather than the data to sign, making what's asked trivial.

Answer 2

I'm afraid, there is no PKCS#11 function, which can do what you want.
The only solution that I am aware of would be, to apply the PKCS#1 v1.5 padding manually to your hash and then sign the block using the CKM_RSA_X_509 (raw or textbook RSA) mechanism.

Answer 3

Rather than issuing the command,
openssl pkeyutl -sign -pkeyopt digest:sha256 -in <inFilePath> -inkey <keyPath> -out <outFilePath>
try
openssl dgst --sha256 -sign <keyPath> -sha256 -out <signature file> <inFilePath>
This produces 256 bytes signature that would probably be identical to
one generated by Pkcs11 C_Sign