Re: 2.5.7 gnutls_x509_privkey_generate() returns GNUTLS_E_INVALID_REQUEST

[Sam Varshavchik]

Simon Josefsson writes:

> Sam Varshavchik <address@hidden> writes:
>
> > The following short test program runs when compiled against
> > 2.4.0. Compiling it against 2.5.7 causes it to report a
> > GNUTLS_E_INVALID_REQUEST from the second call to
> > gnutls_x509_privkey_generate().
>
> I can't reproduce this, adding this somewhere:
>
>   printf ("vers %s %s\n", LIBGNUTLS_VERSION, gnutls_check_version (NULL));
>
> Does print 2.5.7 for both, confirming that I really use 2.5.7.  So it
> seems something else is required to reproduce this.  Can you try to
> debug gnutls_x509_privkey_generate and see what happens?
>
> Does 'certtool -p' trigger the same problem for you?

The bug seems to be easy to spot. I think this is it:

int
gnutls_x509_privkey_generate (gnutls_x509_privkey_t key,
                             gnutls_pk_algorithm_t algo, unsigned int bits,
                             unsigned int flags)
{
 int ret;
 unsigned int params_len;

// . . .

   ret = _gnutls_rsa_generate_params (key→params, &params_len, bits);

This goes into:

static int
_generate_params (int algo, bigint_t *resarr, unsigned int *resarr_len,
                 int bits)

// . . .

 if (resarr && resarr_len && *resarr_len > params.params_nr)
                             ===========

Looks like *resarr_len points to uninitialized memory at this point. 
gnutls_x509_privkey_generate() never initialized params_len, as far as I can 
tell.

pgp9TdEwbYzTu.pgp
Description: PGP signature