Re: Crypto API using RC2 instead of RC4?

From: Valery Pryamikov (valery_at_harper.no)
Date: 11/23/05


Date: Wed, 23 Nov 2005 21:47:50 +0100

and where did you plug your random string generator? if you try to decrypt
randomly generated bytes with default padding mode - you'll get error at
least on 14 inputs out of each 16. you should only decrypt what was
encrypted with the same key and same cipher mode and padding mode.
as i already told you - block cipher operating on whole blocks. rc2 is 64
bit block cipher (8 bytes block). if you encrypt only 4 bytes with block
cipher, you need to pad 4 bytes to whole block (8 bytes). padding modes may
instruct cipher to add padding for you automatically, but that also means
that cipher should should automatically remove padding bytes when
decrypting. Default PKCS padding is to add as many byte as necessary with
each byte being numeric presentation of length of padding. in case of 4
bytes of text - padding it will be {0x04, 0x04, 0x04, 0x04}. if you use
standard padding mode and lenth of text is multiple of 8 bytes, whole block
will be added filled with 8 0x08 bytes.

-Valery.
http://www.harper.no/valery

"Matt" <mtaylor@gladstonemrm.com> wrote in message
news:1132769805.428158.167520@g14g2000cwa.googlegroups.com...
> Looks like I jumped the gun a bit.
> I've run it through a random string generator to really test it and it
> still generates an error every so often.
> Below is the Decrypt function if anybody is able to figure out what is
> wrong.
> I'm tearing my hair out.
> Its the call to the CryptDecrypt API that is causing the problem.
>
> I'm not entirely sure what you mean by the padding mode Valery?
> I'm sure I haven't changed anything like that.
>
> *** Code Starts ***
>
> Private Function CryptoDecrypt(intHashType As Integer, _
> intCipherType As Integer) As Boolean
>
> Dim lngHashHnd As Long ' Hash handle
> Dim lngkey As Long
> Dim lngRetCode As Long ' return value from an API call
> Dim lngHashType As Long
> Dim lngLen As Long
> Dim lngAlgoType As Long
> Dim lngHExchgKey As Long
> Dim lngEncDataLength As Long
> Dim lngEnctBuffLen As Long
> Dim strEncBuffer As String
> Dim strOutData As String
> Dim strPassword As String
>
> '--------------------------- Initialize
> variables-------------------------------
> CryptoDecrypt = False ' preset to FALSE
> Erase g_abytOutData()
> strOutData = ""
> strEncBuffer = ""
> strPassword = ""
> lngHashType = CALG_SHA1
> lngAlgoType = CALG_RC2
>
> '----------------------- Aquire the provider
> handle---------------------------
> If g_lngCryptoContext = 0 Then
> If Not GetProvider Then
> Call Class_Terminate ' Failed. Time to leave.
> Exit Function
> End If
> End If
>
> On Error GoTo CryptoDecrypt_Error
>
> '---------------------- convert password to
> string-----------------------------
> If UBound(g_abytPassword) > 0 Then
> strPassword = ByteArrayToString(g_abytPassword())
> End If
>
> '----------------------- Create a hash
> object----------------------------------
> If Not CBool(CryptCreateHash(g_lngCryptoContext, lngHashType, ByVal
> 0&, _
> ByVal 0&, lngHashHnd)) Then
>
> Err.Raise vbObjectError + 6, "clsCryptoAPIWrapper.CryptoDecrypt",
> "Error: " & CStr(GetLastError) & " during CryptCreateHash!"
> End If
>
> '------------------------- Hash in the password
> text----------------------------
> If Not CBool(CryptHashData(lngHashHnd, strPassword,
> Len(strPassword), ByVal 0&)) Then
> Err.Raise vbObjectError + 7, "clsCryptoAPIWrapper.CryptoDecrypt",
> "Error: " & CStr(GetLastError) & " during CryptHashData!"
> End If
>
> '-------------- Create a session key from the hash
> object--------------------
> If Not CBool(CryptDeriveKey(g_lngCryptoContext, lngAlgoType, _
> lngHashHnd, ByVal CRYPT_NO_SALT, lngkey)) Then
>
> Err.Raise vbObjectError + 8, "clsCryptoAPIWrapper.CryptoDecrypt",
> "Error: " & CStr(GetLastError) & " during CryptDeriveKey!"
> End If
>
> '----------------------- Destroy hash
> object--------------------------------
> If lngHashHnd <> 0 Then
> lngRetCode = CryptDestroyHash(lngHashHnd)
> End If
> lngHashHnd = 0
>
> '------------------------- Prepare data for
> decryption.-----------------------
> lngEncDataLength = Len(g_strInData)
> lngEnctBuffLen = lngEncDataLength * 2
> strEncBuffer = String$(lngEnctBuffLen, vbNullChar)
> LSet strEncBuffer = g_strInData
>
> '--------------------- Decrypt the text
> data---------------------------------
> If Not CBool(CryptDecrypt(lngkey, ByVal 0&, ByVal 1&, ByVal 0&, _
> strEncBuffer, lngEncDataLength)) Then
>
> Err.Raise vbObjectError + 9, "clsCryptoAPIWrapper.CryptoDecrypt",
> "Bytes required:" & CStr(lngEnctBuffLen) & vbCrLf & vbCrLf & _
> "Error: " & CStr(GetLastError) & " during CryptEncrypt!"
> End If
>
> '--------------- Return the decrypted data string in a byte
> array-------------
> strOutData = Mid$(strEncBuffer, 1, lngEncDataLength)
> g_abytOutData = StringToByteArray(strOutData)
> CryptoDecrypt = True ' Successful finish
>
> CleanUp:
>
> ' -------------------------Destroy session
> key.-----------------------------
> If lngkey <> 0 Then
> lngRetCode = CryptDestroyKey(lngkey)
> End If
>
> '-------------------------- Destroy key exchange key
> handle---------------------
> If lngHExchgKey <> 0 Then
> lngRetCode = CryptDestroyKey(lngHExchgKey)
> End If
>
> '---------------------------- Destroy hash
> object--------------------------------
> If lngHashHnd <> 0 Then
> lngRetCode = CryptDestroyHash(lngHashHnd)
> End If
>
> '----------------------------- Empty
> variables-----------------------------------
> lngHashHnd = 0
> strPassword = String$(250, 0)
> Exit Function
>
> CryptoDecrypt_Error:
>
> ' -------------------------Destroy session
> key.-----------------------------
> If lngkey <> 0 Then
> lngRetCode = CryptDestroyKey(lngkey)
> End If
>
> '-------------------------- Destroy key exchange key
> handle---------------------
> If lngHExchgKey <> 0 Then
> lngRetCode = CryptDestroyKey(lngHExchgKey)
> End If
>
> '---------------------------- Destroy hash
> object--------------------------------
> If lngHashHnd <> 0 Then
> lngRetCode = CryptDestroyHash(lngHashHnd)
> End If
>
> '----------------------------- Empty
> variables-----------------------------------
> lngHashHnd = 0
> strPassword = String$(250, 0)
>
> Err.Raise vbObjectError + 10, "clsCryptoAPIWrapper.CryptoDecrypt",
> "Error: " & CStr(Err.Number) & " " & Err.Description & vbCrLf & _
> vbCrLf & "A critical error ocurred during the decryption
> process."
>
> End Function
>
> *** Code Ends ***
>