Re: Problem with url-retrieve when HTTP status is 304

[Mark Plaksin]

Stefan Monnier <address@hidden> writes:

>> With CVS Emacs, url-retrieve triggers an error when it gets an HTTP status
>> of 304 *and* the requested page is not in the cache.  To reproduce, try
>> this:
>
>> (setq url-request-extra-headers
>> '(("If-Modified-Since" . "Sun, 18 Sep 2005 17:34:28 GMT")
>>   ("If-None-Match" . "\"91ad777b5809a56e2b2b9d7c4833fceb\"")))
>> (url-retrieve-synchronously "http://tumbleblindly.org/feed/";)
>
> Just so I understand the problem better: where did you get info such as
> "Sun, 18 Sep 2005 17:34:28 GMT" ?

>From the HTTP headers of the URL in question.  I fetched the headers
outside of the URL library.

>> It's not clear which should change--the section of url-http-parse-headers
>> which handles 304 or url-cache-extract.  One of them should check to see
>> whether the cache file exists before trying to insert its contents.
>
> It seems neither can do anything: when we receive a 304 and there's no cache
> file, we simply don't have the body, so we can't complete the operation.
>
> I.e. I'm tempted to say the problem is that the ("If-Modified-Since"
> . "Sun, 18 Sep 2005 17:34:28 GMT") was used even though there's no cache
> file for it.

Good point.

> Would you want URL to handle such a situation in a similar way to
> a redirect (i.e. it removes the If-Modified-Since header and tries again)?
> Or should it return an empty body?

Good question!

Perhaps all this means that any application that chooses to bypass the URL
library's caching mechanism should use condition-case to handle errors in
cases like this.

Then again the HTTP 1.1 RFC (2616) says exactly what you said--remove the
If-Modified-Since header and If-None-Match headers and retry:
  If a 304 response indicates an entity not currently cached, then the
  cache MUST disregard the response and repeat the request without the
  conditional."

The RFC defines a conditional GET as:
  The semantics of the GET method change to a "conditional GET" if the
  request message includes an If-Modified-Since, If-Unmodified-Since,
  If-Match, If-None-Match, or If-Range header field. A conditional GET
  method requests that the entity be transferred only under the
  circumstances described by the conditional header field(s). The
  conditional GET method is intended to reduce unnecessary network usage by
  allowing cached entities to be refreshed without requiring multiple
  requests or transferring data already held by the client.
(From: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html)

Here's the entire section about 304s:
  10.3.5 304 Not Modified
  
  If the client has performed a conditional GET request and access is
  allowed, but the document has not been modified, the server SHOULD respond
  with this status code. The 304 response MUST NOT contain a message-body,
  and thus is always terminated by the first empty line after the header
  fields.
  
  The response MUST include the following header fields:
  
  - Date, unless its omission is required by section 14.18.1
  
  If a clockless origin server obeys these rules, and proxies and clients add
  their own Date to any response received without one (as already specified
  by [RFC 2068], section 14.19), caches will operate correctly.
  
  - ETag and/or Content-Location, if the header would have been sent
  in a 200 response to the same request
  
  - Expires, Cache-Control, and/or Vary, if the field-value might
  differ from that sent in any previous response for the same
  variant
  
  If the conditional GET used a strong cache validator (see section 13.3.3),
  the response SHOULD NOT include other entity-headers. Otherwise (i.e., the
  conditional GET used a weak validator), the response MUST NOT include other
  entity-headers; this prevents inconsistencies between cached entity-bodies
  and updated headers.
  
  If a 304 response indicates an entity not currently cached, then the cache
  MUST disregard the response and repeat the request without the conditional.
  
  If a cache uses a received 304 response to update a cache entry, the cache
  MUST update the entry to reflect any new field values given in the
  response.
(From: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html)