RSA Public Key problem c++ client to java server



I have a c++ client that generates a keypair and then sends the public key to
the server(java).
Server receives key and converts it to "java format" (reverses endian, etc),
all this works fine. Then server
uses its converted public key to encrypt a message that it sends back to the
client.
Problem is that when the client decrypts the message, it gets a NTE_BAD_DATA
(0x80090005) error and the decryption
fails. Encryption/Decryption works fine locally on the client.

My guess is that this has something to do with
padding/algorithm/ecb/cipher/CRYPT_OAEP...

I been working on this for a couple of days now so please, any guru in
heaven who can help with this?


/********** C++ SIDE ************/

/* Initiate Provider and Key */

#define ENCRYPT_ALGORITHM
// The high order WORD 0x0200 (decimal 512) determines the key length in
bits.
#define KEYLENGTH 0x02000000


// Attempt to acquire a context and a key container.
// The context will use Microsoft Enhanced Cryptographic
// Provider for the RSA_FULL provider type.
CryptAcquireContext(&m_hCryptProv, wszContainerName, MS_ENHANCED_PROV,
PROV_RSA_FULL, CRYPT_MACHINE_KEYSET );

// Call the CryptGenKey method to get a handle to a new exportable key-pair.
CryptGenKey(m_hCryptProv, CALG_RSA_KEYX , KEYLENGTH | CRYPT_EXPORTABLE,
&m_hCryptKey);


/* C++ side configuration now looks like this when prompted with
CryptGetKeyParam(...) function */

//ALGORITHM: CALG_RSA_KEYX
//BLOCKSIZE: 512
//KEYLEN: 512
//PADDING: PKCS5_PADDING
//CIPHERMODE: none


/* To export the public key as MSPublicKeyBlob, following is done: */

// This call determines the length of the key blob.
CryptExportKey(m_hCryptKey, NULL, PUBLICKEYBLOB, 0, NULL, &dwBlobLen);

// Do the actual exporting into the key BLOB.
CryptExportKey(m_hCryptKey, NULL, PUBLICKEYBLOB, 0, pbKeyBlob, &dwBlobLen);


/* To test encrypt locally, the function is called as follows */

CryptDecrypt(m_hCryptKey, NULL, TRUE, 0, pbData, pdwDataLen);


/* To decrypt with local or remote data, the Encrypt function is called as
follows */

CryptEncrypt(m_hCryptKey, NULL, TRUE, 0, pbData, pdwDataLen, dwBufLen);





/********** JAVA SIDE ************/

/* Convert MSPublicKeyBlob to Java PublicKey */

private static final byte PUBLICKEYBLOB = 0x06;
private static final byte CUR_BLOB_VERSION = 0x02;
private static final short RESERVED = 0x0000;
private static final int CALG_RSA_KEYX = 0x0000a400;
private static final int CALG_RSA_SIGN = 0x00002400;
private static final String MAGIC = "RSA1" ; // 0x31415352

public static PublicKey msPkeyToPkey(byte[] mspublickeyblob)
{
DataInputStream dis = null;

int jint = 0 ; // int to build Java int from little-endian ordered byte data
int bitlen = 0;
int pubexp = 0;


//------ Read the "BLOBHEADER" fields -------------

ByteArrayInputStream bis = new ByteArrayInputStream(mspublickeyblob);
dis = new DataInputStream(bis);

if( dis.readByte() != PUBLICKEYBLOB || dis.readByte() !=
CUR_BLOB_VERSION || dis.readShort() != RESERVED)
{
System.out.println("Fail # 1");
return null;
}

jint = 0;
for (int i=0; i<4; i++)
jint += dis.readUnsignedByte() *(int)Math.pow(256,i);

if(jint != CALG_RSA_KEYX && jint != CALG_RSA_SIGN)
{
System.out.println("Fail # 2");
return null;
}

//------ Read the RSAPUBKEY struct members ---------
StringBuffer magic = new StringBuffer(4);

for (int i=1; i<=4; i++)
magic.append((char)dis.readByte());

if(!magic.toString().equals(MAGIC))
{
System.out.println("Fail # 3");
return null;
}

for (int i=0; i<4; i++)
bitlen += dis.readUnsignedByte() *(int)Math.pow(256,i);

for (int i=0; i<4; i++)
pubexp += dis.readUnsignedByte() *(int)Math.pow(256,i);

int keysize = bitlen;

//----- Finally, get the modulus data, and reverse bytes to get big-endian
value --------
byte[] modulus = new byte[bitlen/8] ; //should be this many bytes left
int modbytes = dis.read(modulus);

if(modbytes != (bitlen/8))
{
System.out.println("Fail # 4");
return null;
}

Utils.reverseEndian(modulus) ; //reverse bytes to put in big-endian order


//----- Create the PublicKey from modulus and public exponent --------
RSAPublicKeySpec pubKeySpec = new RSAPublicKeySpec(new BigInteger(1,
modulus), BigInteger.valueOf(pubexp));
KeyFactory keyFactory = KeyFactory.getInstance("RSA");
PublicKey pubKey = keyFactory.generatePublic(pubKeySpec);

return pubKey;
}


/* Instantiate a Cipher used for encryption */

cipher = Cipher.getInstance("RSA"); // or possibly "RSA/ECB/PKCS1Padding"?
cipher.init(Cipher.ENCRYPT_MODE, key);

/* Encrypt data */

cipher.doFinal(bytedata, 0, bytedata.len);



The problem is probably to correctly set MODE and PADDING on both sides but
it seems everybody has different opinions to which configuration
that actually works, and how it can be set up. Is it even possible using the
CryptoAPI together with JCE or do I need BouncyCastle, Crypto++ etc..
to find an equal setting in both languages.

Any suggestion using some other assymetic encryption system would do just
fine if some two-side example code is accessible.
.



Relevant Pages

  • Re: RSA Public Key problem c++ client to java server
    ... // This call determines the length of the key blob. ... CryptExportKey(m_hCryptKey, NULL, PUBLICKEYBLOB, 0, NULL, &dwBlobLen); ... /* To decrypt with local or remote data, the Encrypt function is called as ... int bitlen = 0; ...
    (microsoft.public.platformsdk.security)
  • CryptImportKey
    ... To secure the transmission i use public key encrypting. ... The privatekeyblob goes with the server and the publickeyblob exists in the client application. ...
    (microsoft.public.platformsdk.security)
  • RE: CryptExportKey() NTE_BAD_KEY (80090003) error
    ... goto Exit_MyEncryptFile; ... // Encrypt the file with a random session key, ... PUBLICKEYBLOB, ... TEXT("Memory is allocated for the key BLOB. ...
    (microsoft.public.platformsdk.security)
  • RE: Having problem with Encryption using CryptoAPI
    ... the PUBLICKEYBLOB format. ... After that you are extracting exponent and modulus ... > When I run this program and try to encrypt the file using the "Encrypt" ...
    (microsoft.public.dotnet.security)
  • Having problem with Encryption using CryptoAPI
    ... When I run this program and try to encrypt the file using the "Encrypt" ... EncryptDecrypt.Wincrypt.MyStore); // MyStore ... and data encryption ... PUBLICKEYBLOB. ...
    (microsoft.public.dotnet.security)