bug#35969: 26.2, Excorporate

[tenspd137 .]

Thomas -

For the unauthenticated site, the buffer " *http www.gnu.org:443*" returns

HTTP/1.1 200 OK
Date: Wed, 29 May 2019 17:31:43 GMT
Server: Apache/2.4.7
Content-Location: home.html
.....
[successfuol output]
.....

and

URL-DEBUG

http -> Contacting host: www.gnu.org:443
http -> Marking connection as busy: www.gnu.org:443 #<process www.gnu.org>
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http www.gnu.org:443*>)
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http www.gnu.org:443*>)
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process www.gnu.org>'
http -> url-http-parse-response called in ( *http www.gnu.org:443*)
http -> Request is:
GET https://www.gnu.org/ HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: www.gnu.org
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)


retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http www.gnu.org:443*>)
http -> Calling after change function
`url-http-wait-for-headers-change-function' for `#<process
www.gnu.org>'
http -> url-http-wait-for-headers-change-function ( *http www.gnu.org:443*)
http -> Saw end of headers... ( *http www.gnu.org:443*)
http -> url-http-parse-response called in ( *http www.gnu.org:443*)
http -> Got a content-length, being smart about document end.
http -> Calling initial content-length for extra data at end of headers
http -> Spinning waiting for headers...
http -> Calling after change function
`url-http-content-length-after-change-function' for `#<process
www.gnu.org>'
http -> Calling after change function
`url-http-content-length-after-change-function' for `#<process
www.gnu.org>'
http -> Calling after change function
`url-http-content-length-after-change-function' for `#<process
www.gnu.org>'
http -> Calling after change function
`url-http-content-length-after-change-function' for `#<process
www.gnu.org>'
http -> Calling after change function
`url-http-content-length-after-change-function' for `#<process
www.gnu.org>'
http -> Marking connection as free: www.gnu.org:443 #<process www.gnu.org>
http -> url-http-parse-headers called in ( *http www.gnu.org:443*)
http -> url-http-parse-response called in ( *http www.gnu.org:443*)
http -> Parsed HTTP headers: class=2 status=200
http -> Finished parsing HTTP headers: t
http -> Marking connection as free: www.gnu.org:443 #<process www.gnu.org>
http -> Activating callback in buffer ( *http www.gnu.org:443*): #[128
"\302\303\304p#\210\300\305\240\210\301p\240\207" [(nil) (#<buffer
*http www.gnu.org:443*>) url-debug retrieval "Synchronous fetching
done (%S)" t] 5 "

(fn &rest IGNORED)"] ((:peer (:certificate (:version 3 :serial-number
"03:da:73:db:ff:c6:23:a4:16:f3:45:6f:fe:05:8e:04:b8:c2" :issuer
"C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3" :valid-from
"2019-04-02" :valid-to "2019-07-01" :subject "CN=emacs.org"
:public-key-algorithm "RSA" :certificate-security-level "Medium"
:signature-algorithm "RSA-SHA256" :public-key-id
"sha1:89:0c:f7:08:4e:65:16:0b:43:08:43:eb:e3:33:77:fa:fb:1c:a5:70"
:certificate-id
"sha1:0d:61:97:26:9a:58:d7:04:de:52:1f:29:ca:45:55:ec:67:a9:b1:60")
:key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-128-GCM"
:mac "AEAD")))
retrieval -> Synchronous fetching done (#<buffer  *http www.gnu.org:443*>)

The authenticated site buffer " *http httpbin.org:443" returns
HTTP/1.1 401 UNAUTHORIZED
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Date: Wed, 29 May 2019 17:38:27 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
WWW-Authenticate: Basic realm="Fake Realm"
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 0
Connection: Close

and the URL-DEBUG buffer gives
http -> Contacting host: httpbin.org:443
http -> Marking connection as busy: httpbin.org:443 #<process httpbin.org>
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http httpbin.org:443*>)
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process httpbin.org>'
http -> url-http-parse-response called in ( *http httpbin.org:443*)
http -> Request is:
GET https://httpbin.org/basic-auth/user/passwd HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: httpbin.org
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)


retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http httpbin.org:443*>)
http -> Calling after change function
`url-http-wait-for-headers-change-function' for `#<process
httpbin.org>'
http -> url-http-wait-for-headers-change-function ( *http httpbin.org:443*)
http -> Saw end of headers... ( *http httpbin.org:443*)
http -> url-http-parse-response called in ( *http httpbin.org:443*)
http -> Got a content-length, being smart about document end.
http -> Got 0-length content-length, activating callback immediately.
http -> Marking connection as free: httpbin.org:443 #<process httpbin.org>
http -> url-http-parse-headers called in ( *http httpbin.org:443*)
http -> url-http-parse-response called in ( *http httpbin.org:443*)
http -> Parsed HTTP headers: class=4 status=401
http -> Handling normal authentication
http -> Contacting host: httpbin.org:443
http -> Marking connection as busy: httpbin.org:443 #<process httpbin.org>
http -> Finished parsing HTTP headers: nil
http -> Spinning waiting for headers...
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer
*http httpbin.org:443*>)
http -> Calling after change function
`url-https-proxy-after-change-function' for `#<process httpbin.org>'
http -> url-http-parse-response called in ( *http httpbin.org:443*-54346)
http -> Request is:
GET https://httpbin.org/basic-auth/user/passwd HTTP/1.1
MIME-Version: 1.0
Connection: close
Extension: Security/Digest Security/SSL
Host: httpbin.org
Accept-encoding: gzip
Accept: */*
User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
Authorization: Basic dXNlcjpwYXNzd2Q=


http -> Calling after change function
`url-http-wait-for-headers-change-function' for `#<process
httpbin.org>'
http -> url-http-wait-for-headers-change-function ( *http
httpbin.org:443*-54346)
http -> Saw end of headers... ( *http httpbin.org:443*-54346)
http -> url-http-parse-response called in ( *http httpbin.org:443*-54346)
http -> Got a content-length, being smart about document end.
http -> Calling initial content-length for extra data at end of headers
http -> Marking connection as free: httpbin.org:443 #<process httpbin.org>
http -> url-http-parse-headers called in ( *http httpbin.org:443*-54346)
http -> url-http-parse-response called in ( *http httpbin.org:443*-54346)
http -> Parsed HTTP headers: class=2 status=200
http -> Finished parsing HTTP headers: t
http -> Marking connection as free: httpbin.org:443 #<process httpbin.org>
http -> Activating callback in buffer ( *http httpbin.org:443*-54346):
#[128 "\302\303\304p#\210\300\305\240\210\301p\240\207" [(t) (#<buffer
 *http httpbin.org:443*>) url-debug retrieval "Synchronous fetching
done (%S)" t] 5 "

(fn &rest IGNORED)"] ((:peer (:certificate (:version 3 :serial-number
"06:0c:5d:3e:f4:d0:2e:f5:c1:ac:09:8e:b3:0e:39:ab" :issuer
"C=US,O=Amazon,OU=Server CA 1B,CN=Amazon" :valid-from "2019-02-17"
:valid-to "2020-03-17" :subject "CN=httpbin.org" :public-key-algorithm
"RSA" :certificate-security-level "Medium" :signature-algorithm
"RSA-SHA256" :public-key-id
"sha1:1b:e3:b6:57:70:0a:15:df:dc:05:20:f2:66:da:8e:2f:74:4b:e8:ed"
:certificate-id
"sha1:cd:9a:4d:21:af:57:65:e6:d3:be:3a:d0:1c:dc:88:26:b8:96:bb:f7")
:key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-128-GCM"
:mac "AEAD")))
retrieval -> Synchronous fetching done (#<buffer  *http httpbin.org:443*-54346>)
http -> Spinning waiting for headers...

Just as a check:
echo $http_proxy
http://web-proxy:8080

echo $https_proxy
http://web-proxy:8080

using wget, I get a file with the following contents:
 wget --user user --password passwd https://httpbin.org/basic-auth/user/passwd
--2019-05-29 11:46:25--  https://httpbin.org/basic-auth/user/passwd
Resolving web-proxy... xx.xx.xxx.xxx
Connecting to web-proxy|xx.xx.xxx.xxx]:8080... connected.
Proxy request sent, awaiting response... 401 UNAUTHORIZED
Authentication selected: Basic realm="Fake Realm"
Reusing existing connection to httpbin.org:443.
Proxy request sent, awaiting response... 200 OK
Length: 47 [application/json]
Saving to: ‘passwd’

passwd
100%[======================================================================================================================>]
     47  --.-KB/s    in 0s

2019-05-29 11:46:26 (3.59 MB/s) - ‘passwd’ saved [47/47]


cat passwd
{
  "authenticated": true,
  "user": "user"
}

Thanks!

-C

}


On Wed, May 29, 2019 at 11:02 AM Thomas Fitzsimmons <fitzsim@fitzsim.org> wrote:
>
> Hi,
>
> "tenspd137 ." <dcday137@gmail.com> writes:
>
> > Hi - thanks for the reply.  Here are the steps I took (I had to
> > configure the proxy as well....):
> >
> > M-: (setq url-proxy-services '(("http" . "web-proxy:8080") ("https" . 
> > "web-proxy:8080")))
> > M-: (setq url-debug t)
> > M-: (url-retrieve-synchronously "https://server/EWS/Exchange.asmx";)
> >
> > -Asks for name and password-, I fill them in.....
>
> OK, I think the buffer contents you pasted are fine/expected; it's the
> web site saying you're unauthorized, which then causes "url" to prompt
> for credentials.  That part seems to have worked through the proxies.
>
> Then the url library created a second buffer:
>
> " *http outlook.office365.com:443*-229815"
>
> which I'm inferring from the logs contained "400 Bad Request" headers.
> I'm not sure why that might be; I haven't tried Emacs's url proxy
> support before.  That said, I'd like to keep trying to fix this since
> I'd like Excorporate to work via proxies.
>
> You could try an unauthenticated site to see if that works through the
> proxies, e.g.:
>
> [same initial steps]
> M-: (url-retrieve-synchronously "https://www.gnu.org/";)
>
> and then a different authenticated site, like this test page:
>
> M-: (url-retrieve-synchronously "https://httpbin.org/basic-auth/user/passwd";)
>
> Username: user
> Password: passwd
>
> If that all works, then we'll have to figure out why url proxy support
> is not working with the Exchange server specifically.
>
> [...]
>
> > But - just for reference - here is what is in the file I receive when
> > I use wget:
> >
> >  wget --user me@work --password xxxxxxxxxxx 
> > https://outlook.office365.com/EWS/Exchange.asmx
> >
> > [successful output]
>
> Are you configuring wget to use the same proxies as in the Emacs
> attempt, via the http_proxy and https_proxy environment variables?
>
> Thanks,
> Thomas