bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

Here's a emacs -Q recipe:

    $ emacs -Q
    M-x eval-expression RET (setq gnus-select-method '(nnimap 
"imaps.mycompany.com")) RET
    M-x gnus

An error pops up:

    nnimap (imaps.mycompany.com) open error: ''gnutls-cli' is not recognized as 
an internal or external command,

In the *Message* buffer, we can see at at least one tls command succeeded

    Opening connection to imaps.mycompany.com via tls...
    gnutls.c: [1] (Emacs) GnuTLS library not found
    Opening TLS connection to `imaps.mycompany.com'...
    Opening TLS connection with `gnutls-cli --insecure -p 993 
imaps.mycompany.com'...failed
    Opening TLS connection with `gnutls-cli --insecure -p 993 
imaps.mycompany.com --protocols ssl3'...failed
    Opening TLS connection with `openssl s_client -connect 
imaps.mycompany.com:993 -no_ssl2 -ign_eof'...done
    Opening TLS connection to `imaps.mycompany.com'...done
    nnimap (imaps.mycompany.com) open error: ''gnutls-cli' is not recognized as 
an internal or external command,
    '.  Continue? (y or n)  n

This works fine, on the same server with linux.

My analysis and possible fix is as follows: The problem is that
`network-stream-open-tls' returns invalid information in its final form:

    (list stream
          (network-stream-get-response stream start eoc)
          (network-stream-command stream capability-command eoc)
          'tls)

This is turn is because code immediately preceding it

    ;; If we're using tls.el, we have to delete the output from
    ;; openssl/gnutls-cli.
    (when (and (null use-builtin-gnutls)
               eoc)
      (network-stream-get-response stream start eoc)
      (goto-char (point-min))
      (when (re-search-forward eoc nil t)
        (goto-char (match-beginning 0))
        (delete-region (point-min) (line-beginning-position))))

is not really doing what it says it's doing: it's not deleting enough
input, and the search for capabilities actually returns the IMAP
greeting.

This in turn is related to the CRLF lineendings that the function
`open-tls-stream' running under w32 only (in linux it doesn't happen)
inserts in the process buffer before the end of the tls handshake. The
code in `network-stream-open-tls' for getting rid of useless starting
output deletes up to the first CRLF (and apparently it should do so just
once, since other CRLF's are legitimate). But if fails and catches a
very early CRLF which is still part of the now useless TLS
handshake. The search performed by

    (network-stream-command stream capability-command eoc)

doesn't work. That said, a reasonable way to fix this might be adding a
line

    (delete-region (point-min) (point))

as the first form in the body of the `when'. This is because
`open-tls-stream' apparently guarantees that point will be left where
the tls handshake ended, as per its code snippet and comment:

    (if start-of-data
        ;; move point to start of client data
        (goto-char start-of-data))

Alternalively, make `open-tls-stream' do that itself in the end. I'm
doing just that with.

    (defadvice open-tls-stream (after dont-forget-to-erase-buffer-up-to-point 
activate)
      (delete-region (point-min) (point)))

Or maybe improve searches by `network-stream-get-response' or be smarter
when deleting output.

Here's the contents of the nnimap process buffer. Notice the CRLFs.


'gnutls-cli' is not recognized as an internal or external command,
operable program or batch file.

Process *nnimap* exited abnormally with code 1
'gnutls-cli' is not recognized as an internal or external command,
operable program or batch file.

Process *nnimap* exited abnormally with code 1
depth=0 C = PT, L = Lisboa, O = SISCOG.COM, OU = IMAP server, CN = 
*.siscog.com, emailAddress = suporte@siscog.pt
verify error:num=18:self signed certificate
verify return:1
depth=0 C = PT, L = Lisboa, O = SISCOG.COM, OU = IMAP server, CN = 
*.siscog.com, emailAddress = suporte@siscog.pt
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP 
server/CN=*.siscog.com/emailAddress=suporte@siscog.pt
   i:/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP 
server/CN=*.siscog.com/emailAddress=suporte@siscog.pt
---
Server certificate
-----BEGIN CERTIFICATE-----
[elided]
-----END CERTIFICATE-----
subject=/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP 
server/CN=*.siscog.com/emailAddress=suporte@siscog.pt
issuer=/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP 
server/CN=*.siscog.com/emailAddress=suporte@siscog.pt
---
No client certificate CA names sent
---
SSL handshake has read 1240 bytes and written 518 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 9334E31E9FD9A90C9D301227A2799EB8210AAC9FFD583B6597251744CB6EBFB3
    Session-ID-ctx:
    Master-Key: 
7A5A40FEFC3A3493C9AFC4B80BCB044FAC48C94E0B4DE6E35035866F90EC78FFDE18D1B40E93B125F4F3F73468A9D5F0
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Compression: 1 (zlib compression)
    Start Time: 1368189548
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---
* OK Dovecot ready.

In GNU Emacs 24.3.1 (i386-mingw-nt5.1.2600)
 of 2013-03-17 on MARVIN
Windowing system distributor `Microsoft Corp.', version 5.1.2600
Configured using:
 `configure --with-gcc (4.7) --cflags
 -ID:/devel/emacs/libs/libXpm-3.5.8/include
 -ID:/devel/emacs/libs/libXpm-3.5.8/src
 -ID:/devel/emacs/libs/libpng-dev_1.4.3-1/include
 -ID:/devel/emacs/libs/zlib-dev_1.2.5-2/include
 -ID:/devel/emacs/libs/giflib-4.1.4-1/include
 -ID:/devel/emacs/libs/jpeg-6b-4/include
 -ID:/devel/emacs/libs/tiff-3.8.2-1/include
 -ID:/devel/emacs/libs/gnutls-3.0.9/include
 -ID:/devel/emacs/libs/libiconv-1.13.1-1-dev/include
 -ID:/devel/emacs/libs/libxml2-2.7.8/include/libxml2'

Important settings:
  value of $LANG: C.UTF-8
  locale-coding-system: cp1252
  default enable-multibyte-characters: t