bug#13857: Unhandled case in module/web/response.scm

[Jason Earl]

response.scm does not seem to handle the case where the server does not
specify a content length.  Here's a minimal example that should work,
but doesn't:

--8<---------------cut here---------------start------------->8---
#!/usr/local/bin/guile -s
!#

(use-modules (srfi srfi-8)
             ((web uri)    #:select (string->uri))
             ((web client) #:select (http-get)))

(receive (res-headers res-body)
    (http-get (string->uri 
"http://www.blogger.com/feeds/4777343509834060826/posts/default";))
  (display res-body)
  (newline))
--8<---------------cut here---------------end--------------->8---

Now the reason that I started experimenting with guile in the first
place was that I wanted to learn more about scheme, and fixing this
seemed like a good opportunity at a practical application of my basic
scheme skills.

So I did a little debugging and created this little patch that fixes
this issue.

>From 74af03051fec22e01376f3f4597eb8de2dab87b5 Mon Sep 17 00:00:00 2001
From: Jason Douglas Earl <address@hidden>
Date: Fri, 1 Mar 2013 23:58:41 -0700
Subject: [PATCH] Handle case where server does not specify a Content-Length.

* module/web/response.scm (response-body-port,
  make-undelimited-input-port): The server does not have to return a
  content length.  Handle that case.
---
 module/web/response.scm |   28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/module/web/response.scm b/module/web/response.scm
index 7e14f4d..33dcf1e 100644
--- a/module/web/response.scm
+++ b/module/web/response.scm
@@ -264,6 +264,26 @@ closes PORT, unless KEEP-ALIVE? is true."
 
   (make-custom-binary-input-port "delimited input port" read! #f #f close))
 
+(define (make-undelimited-input-port port keep-alive?)
+  "Return an input port that reads from PORT, until EOF.  Closing the
+returned port closes PORT, unless KEEP-ALIVE? is true."
+  (define bytes-read 0)
+
+  (define (read! bv start count)
+    (let ((ret (get-bytevector-n! port bv start count)))
+      (if (eof-object? ret)
+         0
+          (begin
+            (set! bytes-read (+ bytes-read ret))
+           ret))))
+
+  (define close
+    (and (not keep-alive?)
+         (lambda ()
+           (close port))))
+
+  (make-custom-binary-input-port "undelimited input port" read! #f #f close))
+
 (define* (response-body-port r #:key (decode? #t) (keep-alive? #t))
   "Return an input port from which the body of R can be read.  The
 encoding of the returned port is set according to R's ‘content-type’
@@ -277,9 +297,11 @@ response port."
         (make-chunked-input-port (response-port r)
                                  #:keep-alive? keep-alive?)
         (let ((len (response-content-length r)))
-          (and len
-               (make-delimited-input-port (response-port r)
-                                          len keep-alive?)))))
+          (if len
+             (make-delimited-input-port (response-port r)
+                                        len keep-alive?)
+             (make-undelimited-input-port (response-port r)
+                                          keep-alive?)))))
 
   (when (and decode? port)
     (match (response-content-type r)
-- 
1.7.10.4

With that patch my little test program works.

Now, please forgive my ignorance.  I probably misunderstand what
"delimited" means in this context, and I am probably using it
incorrectly.  I would be shocked if this even slightly resembles how
this should be fixed.  I have been lurking on the guile mailing lists
for a few months and I don't understand half of what you guys are
talking about (which is actually why I am so keen to play with guile).
Sharing this patch seemed to be the easiest way to explain what is
happening.

I am not even going to pretend that I have spent a great deal of time
reading the HTTP 1.1 protocol specs, but it does appear that the server
does not have to return a Content-Length.  I certainly have run across
servers that don't.

Poking at this issue has been quite a bit of fun for me.  So, thanks for
all of your hard work on the system.  Now I must admit that I am
interested in seeing how (and if) this gets fixed.

Jason