David> It seems the the current implementation of HTTPURLConnection.connect()
David> buffers the entire response before returning.
David> Is that a correct analysis?
Yes.
I'd also like this to be fixed. It has shown up in a few ways; I
think it turns out that java programs expect to be able to make an
http connection and read data in a streamy way, not pause for a
humongous read and then just get bytes from a buffer.
E.g., netx (JNLP implementation) puts up a dialog showing percent
completion as it reads; with our implementation is just says 0% and
then (after a long pause) it is done. Not very friendly IMO.
David> Does anyone know the rational behind doing it this way?
I've talked to Chris a couple of times about this. He is the real
expert. Meanwhile I'll give you my less than perfect opinion.
I think the biggest reason for this is handling multiplexing multiple
http requests onto a single connection. In this situation you want to
read whatever data is available, since (e.g.) data might be available
on one of the connections, but that connection might not actively be
read atm; meanwhile another thread could try to read from its
connection but block indefinitely. (I hope that abbreviated example
makes sense, I can try to expand on it. Assuming it is even close to
reality, Chris?)
Delving into this code has been on my to-do list for a while but it
keeps getting pushed down. My vague plan was to see if we could limit
connection reuse to situations where either the underlying connection
was quiescent, or perhaps when the data was known to be small and
buffering would be reasonable.