return value issues

[JohnLegrandRichards]

PKCS11 RSA signature is returning a bit string or something. Is there a way to specify the format of the signature that gets returned?

[larskanis]

A RSA signature is just a big number represented as a byte string. It should have the size of the RSA modulus, which is 256 bytes for a 2048 bit RSA key for instance.

There is CKM_RSA_PKCS which is a raw RSA signature without any digest computation. In contrast CKM_SHA256_RSA_PKCS does some precalculation like so before building the RSA signature:

          # Build the digest info TLV as PKCS#1
          ary1 = [
            OpenSSL::ASN1::ObjectId.new('SHA256'),
            OpenSSL::ASN1::Null.new(nil),
          ]
          ary2 = [
            OpenSSL::ASN1::Sequence.new(ary1),
            OpenSSL::ASN1::OctetString.new(OpenSSL::Digest::SHA256.new(document_to_be_signed).digest),
          ]
          raw_value_to_be_signed = OpenSSL::ASN1::Sequence.new(ary2).to_der

Similarly for CKM_SHA256_RSA_PKCS_PSS, which has a bit more complicated precalculation and padding.

Simple raw RSA signature is also in our tests:

pkcs11/test/test_pkcs11_crypt.rb

Lines 72 to 89 in eeda7cb

	def test_sign_verify
	plaintext = "important text"
	signature = session.sign( :SHA1_RSA_PKCS, rsa_priv_key, plaintext)
	assert signature.length>10, 'The signature should contain some data'
	signature2 = session.sign( :SHA1_RSA_PKCS, rsa_priv_key){|c|
	c.update(plaintext[0..3])
	c.update(plaintext[4..-1])
	}
	assert_equal signature, signature2, 'results of one-step and two-step signatures should be equal'
	valid = session.verify( :SHA1_RSA_PKCS, rsa_pub_key, signature, plaintext)
	assert valid, 'The signature should be correct'
	assert_raises(CKR_SIGNATURE_INVALID, 'The signature should be invalid on different text') do
	session.verify( :SHA1_RSA_PKCS, rsa_pub_key, signature, "modified text")
	end
	end