Re: [Help-smalltalk] NetClients.HTTP

[Stephen]

Hi

Thanks for looking into this, Paolo, and for showing how the code could 
best be tested.

I have been away with work for a week with work and hence the delay 
responding.

I've applied your patch to latest GST from Git today (3.2.90-3cd3a73), 
however there is still an error.

~/gst-devel $ ./sitemonitor.st
Object: MimeScanner new "<0x4039f568>" error: Invalid Field (Missing colon)
Error(Exception)>>signal (ExcHandling.st:254)
Error(Exception)>>signal: (ExcHandling.st:264)
NetClients.MIME.MimeScanner(Object)>>error: (SysExcept.st:1415)
NetClients.MIME.MimeScanner(NetClients.MIME.SimpleScanner)>>notify: 
(NetClients.star#VFS.ZipFile/MIME.st:356)
NetClients.MIME.MimeScanner(NetClients.MIME.SimpleScanner)>>mustMatch:notify: 
(NetClients.star#VFS.ZipFile/MIME.st:386)
NetClients.MIME.HeaderField class>>readFieldNameFrom: 
(NetClients.star#VFS.ZipFile/MIME.st:1443)
NetClients.MIME.HeaderField class>>readFrom: 
(NetClients.star#VFS.ZipFile/MIME.st:1451)
NetClients.MIME.MimeEntity>>parseFieldFrom: 
(NetClients.star#VFS.ZipFile/MIME.st:1141)
NetClients.MIME.MimeEntity>>parseFieldsFrom: 
(NetClients.star#VFS.ZipFile/MIME.st:1153)
NetClients.HTTP.HTTPResponse>>parseResponse: 
(NetClients.star#VFS.ZipFile/HTTP.st:520)
NetClients.HTTP.HTTPResponse class(NetClients.NetResponse 
class)>>fromClient: (NetClients.star#VFS.ZipFile/Base.st:951)
NetClients.HTTP.HTTPProtocolInterpreter(NetClients.NetProtocolInterpreter)>>getResponse 
(NetClients.star#VFS.ZipFile/Base.st:792)
NetClients.HTTP.HTTPProtocolInterpreter>>readResponseInto: 
(NetClients.star#VFS.ZipFile/HTTP.st:207)
NetClients.HTTP.HTTPProtocolInterpreter>>get:requestHeaders:into: 
(NetClients.star#VFS.ZipFile/HTTP.st:150)
NetClients.HTTP.HTTPClient>>get:requestHeaders:into: 
(NetClients.star#VFS.ZipFile/HTTP.st:76)
[] in NetClients.HTTP.HTTPClient class>>exampleURL:host:port: 
(NetClients.star#VFS.ZipFile/HTTP.st:63)
BlockClosure>>ensure: (BlkClosure.st:269)
NetClients.HTTP.HTTPClient class>>exampleURL:host:port: 
(NetClients.star#VFS.ZipFile/HTTP.st:68)
UndefinedObject>>executeStatements (sitemonitor.st:9)



And as a check, here is a diff on MIME.st after I had applied your patch...

diff --git a/packages/net/MIME.st b/packages/net/MIME.st
index 0460bc2..7a4c199 100644
--- a/packages/net/MIME.st
+++ b/packages/net/MIME.st
@@ -588,6 +588,7 @@ Object subclass: SimpleScanner [
        <category: 'stream interface -- reading'>
        lookahead notNil ifTrue: [^lookahead].
        self atEnd ifTrue: [^nil].
+        hereChar := nil.
        lookahead := source next.
        ^lookahead
     ]
@@ -1144,9 +1145,12 @@ MessageElement subclass: MimeEntity [
        <category: 'parsing'>
        | cr nl |
        
-       [(cr := rfc822Stream peekFor: Character cr)
-           | (nl := rfc822Stream peekFor: Character nl)]
-               whileFalse: [self parseFieldFrom: rfc822Stream]
+       [(cr := rfc822Stream hereChar == Character cr)
+                    ifTrue: [rfc822Stream step].
+        (nl := rfc822Stream hereChar == Character nl)
+                    ifTrue: [rfc822Stream step].
+         cr or: [nl]]
+                whileFalse: [self parseFieldFrom: rfc822Stream]
     ]

     parseMultipartBodyFrom: rfc822Stream [
@@ -2863,14 +2867,13 @@ MailScanner subclass: RFC822Scanner [
        <category: 'private'>
        | char |
        self atEnd ifTrue: [^false].
-       char := source next.
+       char := self peek.
        ^((self classificationMaskFor: char) anyMask: WhiteSpaceMask)
            ifFalse:
-               [lookahead := char.
-               self resetToken.
+               [self resetToken.
                false]
            ifTrue:
-               [self sourceTrailNextPut: char.
+               [self next. self sourceTrailNextPut: char.
                true]
     ]

Thanks
Stephen

On 29/03/11 9:24 PM, Paolo Bonzini wrote:
> On 03/28/2011 09:52 PM, Stephen wrote:
> >     request := NetClients.URIResolver openStreamOn: siteurl ifFail: [
> > Transcript show: 'Request failed'].
> >     Transcript show: request contents.
>
> The problem is in the failure.  The error happens simply because your
> ifFail block is returning the transcript itself.
>
> The first step in analyzing the failure is to lift the abstraction
> provided by URIResolver: this
>
> NetClients.HTTP.HTTPClient exampleURL: 'http://moodle.org/' host: 'moodle.org' 
> port: 80
>
> provides a better error:
>
> Object: MimeScanner new "<0x2b3b619fb4e0>" error: cannot step back twice
> Error(Exception)>>signal (ExcHandling.st:254)
> Error(Exception)>>signal: (ExcHandling.st:264)
> NetClients.MIME.MimeScanner(Object)>>error: (SysExcept.st:1415)
> NetClients.MIME.MimeScanner(NetClients.MIME.SimpleScanner)>>stepBack 
> (NetClients.star#VFS.ZipFile/MIME.st:432)
> NetClients.MIME.MimeScanner(NetClients.MIME.SimpleScanner)>>scanWhile: 
> (NetClients.star#VFS.ZipFile/MIME.st:421)
> NetClients.MIME.MimeScanner(NetClients.MIME.RFC822Scanner)>>skipWhiteSpace 
> (NetClients.star#VFS.ZipFile/MIME.st:2790)
> NetClients.MIME.HeaderField class>>readFieldNameFrom: 
> (NetClients.star#VFS.ZipFile/MIME.st:1442)
> NetClients.MIME.HeaderField class>>readFrom: 
> (NetClients.star#VFS.ZipFile/MIME.st:1449)
> NetClients.MIME.MimeEntity>>parseFieldFrom: 
> (NetClients.star#VFS.ZipFile/MIME.st:1142)
> NetClients.MIME.MimeEntity>>parseFieldsFrom: 
> (NetClients.star#VFS.ZipFile/MIME.st:1151)
> NetClients.HTTP.HTTPResponse>>parseResponse: 
> (NetClients.star#VFS.ZipFile/HTTP.st:520)
>
> So the next step is grabbing the response with wireshark.  You can put
> it into a file and "serve it" with
>
> nc -l localhost 8080<  g
>
> if you change the gst script to
>
> NetClients.HTTP.HTTPClient exampleURL: 'http://moodle.org/' host: 'localhost' 
> port: 8080
>
> After some attempts it looks like it's choking on a header "Expires: ".
> I think this fixes it, but it needs more testing:
>
> diff --git a/packages/net/MIME.st b/packages/net/MIME.st
> index 0460bc2..3bbc8da 100644
> --- a/packages/net/MIME.st
> +++ b/packages/net/MIME.st
> @@ -588,6 +588,7 @@ Object subclass: SimpleScanner [
>         <category: 'stream interface -- reading'>
>         lookahead notNil ifTrue: [^lookahead].
>         self atEnd ifTrue: [^nil].
> +       hereChar := nil.
>         lookahead := source next.
>         ^lookahead
>       ]
> @@ -1144,8 +1145,11 @@ MessageElement subclass: MimeEntity [
>         <category: 'parsing'>
>         | cr nl |
>         
> -       [(cr := rfc822Stream peekFor: Character cr)
> -           | (nl := rfc822Stream peekFor: Character nl)]
> +       [(cr := rfc822Stream hereChar == Character cr)
> +             ifTrue: [rfc822Stream step].
> +       (nl := rfc822Stream hereChar == Character nl)
> +             ifTrue: [rfc822Stream step].
> +        cr or: [nl]]
>                 whileFalse: [self parseFieldFrom: rfc822Stream]
>       ]
>
> @@ -2863,14 +2867,13 @@ MailScanner subclass: RFC822Scanner [
>         <category: 'private'>
>         | char |
>         self atEnd ifTrue: [^false].
> -       char := source next.
> +       char := self peek.
>         ^((self classificationMaskFor: char) anyMask: WhiteSpaceMask)
>             ifFalse:
> -               [lookahead := char.
> -               self resetToken.
> +               [self resetToken.
>                 false]
>             ifTrue:
> -               [self sourceTrailNextPut: char.
> +               [self next. self sourceTrailNextPut: char.
>                 true]
>       ]
>
>
> Travis, do you know who wrote the MIME/RFC822 packages for VW?
> Cincom released them as open source, so GNU Smalltalk's packages
> are based on those.
>
> Paolo