Re: 2.5.7 gnutls_x509_privkey_generate() returns GNUTLS_E_INVALID_REQUEST

[Sam Varshavchik]

Nikos Mavrogiannopoulos writes:

> Sam Varshavchik wrote:
>
> > Unfortunately, there appears to be more problems with private key
> > functions in 2.5.7. I'm now getting segfaults in
> > gnutls_x509_privkey_import(). My debugging results are as follows:
> [...]
> > ==3339== 5,886 (168 direct, 5,718 indirect) bytes in 3 blocks are
> > definitely lost in loss record 8 of 9
> > ==3339==    at 0x4A05174: calloc (vg_replace_malloc.c:397)
> > ==3339==    by 0x3158C0A211: (within /usr/lib64/libtasn1.so.3.0.14)
> > ==3339==    by 0x3158C0A3F2: (within /usr/lib64/libtasn1.so.3.0.14)
> > ==3339==    by 0x3158C0A7BD: asn1_create_element (in
> > /usr/lib64/libtasn1.so.3.0.14)
> > ==3339==    by 0x4C829B2: _gnutls_asn1_encode_rsa (privkey.c:1075)
> > ==3339==    by 0x4C83E0B: gnutls_x509_privkey_export (privkey.c:739)
>
> Hello Sam,
>  I've commited fixes for these issues, so that latest code in git should
> not have these issues. All except the last one (quoted). It would be
> very helpful to give me some pointers on how to reproduce it.
>
> As far as I understand it should happening when
> gnutls_rsa_params_export_raw() is called?

The following minimal program triggers the leak:

#include <gcrypt.h>
#include <gnutls/gnutls.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <pthread.h>

GCRY_THREAD_OPTION_PTHREAD_IMPL;

int main()
{
        int i;
        gnutls_rsa_params_t rsa;
        size_t buflen;

        gcry_control(GCRYCTL_SET_THREAD_CBS, &gcry_threads_pthread);
        gcry_control(GCRYCTL_ENABLE_QUICK_RANDOM);
        gnutls_global_init();

        gnutls_rsa_params_init(&rsa);

        printf("vers %s %s\n", LIBGNUTLS_VERSION, gnutls_check_version (NULL));

        gnutls_rsa_params_generate2(rsa, 1024);

        gnutls_rsa_params_export_pkcs1(rsa, GNUTLS_X509_FMT_DER, NULL, &buflen);

        char b[buflen];

        gnutls_rsa_params_export_pkcs1(rsa, GNUTLS_X509_FMT_DER, b, &buflen);

        gnutls_rsa_params_deinit(rsa);

        return(0);
}


The leak gets triggered only when gnutls_rsa_params_export_pkcs1() gets 
invoked twice, once with a NULL buffer pointer, to compute the buffer size, 
and a second time with the real buffer pointer. Invoking it just once is not 
sufficient.

What looks like is happening is that in gnutls_x509_privkey_export():

       case GNUTLS_PK_RSA:
         ret = _gnutls_asn1_encode_rsa (&key->key, key->params);

First time through, key->key is NULL, and _gnutls_asn1_encode_rsa() 
allocates it using asn1_create_element(). On the second call, key->key holds 
the previously allocated buffer, but _gnutls_asn1_encode_rsa() allocates a 
new one, without freeing the first one, and leaks it.

I also see similar logic for the GNUTLS_PK_DSA case, and 
_gnutls_asn1_encode_dsa(), so there may also be a similar leak there.

pgpERb0VYShZA.pgp
Description: PGP signature