From MAILER-DAEMON Tue Apr 30 17:00:33 2013 Received: from list by lists.gnu.org with archive (Exim 4.71) id 1UXHf7-0004wS-BD for mharc-websec-users@gnu.org; Tue, 30 Apr 2013 17:00:33 -0400 Received: from eggs.gnu.org ([208.118.235.92]:60725) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UXHf1-0004wL-MJ for websec-users@nongnu.org; Tue, 30 Apr 2013 17:00:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UXHex-0000m2-22 for websec-users@nongnu.org; Tue, 30 Apr 2013 17:00:27 -0400 Received: from smtprelay-b31.telenor.se ([213.150.131.20]:32961) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UXHew-0000l0-R4 for websec-users@nongnu.org; Tue, 30 Apr 2013 17:00:22 -0400 Received: from ipb2.telenor.se (ipb2.telenor.se [195.54.127.165]) by smtprelay-b31.telenor.se (Postfix) with ESMTP id E4BFEE9C8C for ; Tue, 30 Apr 2013 23:00:18 +0200 (CEST) X-SENDER-IP: [83.227.168.100] X-LISTENER: [smtp.bredband.net] X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AktcAMgvgFFT46hkPGdsb2JhbABSgwcBNbdrh3YXAwEBAQE4NYJNExyBMQwKiD2fdJEWjmiPNoM6A5cmkl6BXjo X-IronPort-AV: E=Sophos;i="4.87,584,1363129200"; d="scan'208";a="541189566" Received: from ua-83-227-168-100.cust.bredbandsbolaget.se (HELO mimmi.uddeborg.se) ([83.227.168.100]) by ipb2.telenor.se with ESMTP; 30 Apr 2013 23:00:18 +0200 Received: from freddi.uddeborg (freddi.uddeborg [172.17.0.3]) by mimmi.uddeborg.se (8.14.5/8.14.5) with ESMTP id r3UL0FMU020704 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 30 Apr 2013 23:00:16 +0200 Received: from freddi.uddeborg (localhost [127.0.0.1]) by freddi.uddeborg (8.14.5/8.14.5) with ESMTP id r3UL0EH0011041 for ; Tue, 30 Apr 2013 23:00:14 +0200 Received: (from goeran@localhost) by freddi.uddeborg (8.14.5/8.14.5/Submit) id r3UL0EB8011040; Tue, 30 Apr 2013 23:00:14 +0200 From: =?utf-8?Q?G=C3=B6ran_Uddeborg?= MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20864.12510.66803.514996@freddi.uddeborg> Date: Tue, 30 Apr 2013 23:00:14 +0200 To: websec-users@nongnu.org Subject: Gzip doesn't seem to play well with HTTPS X-Mailer: VM 8.1.2 under 24.1.1 (x86_64-redhat-linux-gnu) X-Face: "m&!q8; elk)j/GjdH0A1-5euH[VSc"z|=FSJFsZMltGCVh^{K :^Y;hs=IW^!,^]<.Do*c6+Eg2{+(^IjaW<2X,!<6=S\`8s3$0p'*sQ"tm\<(tRW?,CmQ3 X-URL: http://www.uddeborg.se/g%C3%B6ran/ X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 213.150.131.20 X-BeenThere: websec-users@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Users discussions regarding Web Secretary List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Apr 2013 21:00:32 -0000 I instructed websec to monitor a page which is only available via HTTPS. But I never got any reasonable results, always only garbled text. I stripped down the code piece by piece, and realised this somehow conflicts with the gzip compression. If I disable that, by setting $feature_compress to 0 rather than 1 on the first line, things seems to work much better. LWP::UserAgent also seems to have built-in support for doing the unzipping. If I use the convenience function "get" augmented with support for compression like this: my $enc = HTTP::Message::decodable; my $response = $ua->get(, "Accept-Encoding" => $enc); if ($response->is_success) { print $response->decoded_content; I do get a correct response. For some reason, decoded_content does not seem to be available when building the request "by hand" like in websec. I haven't figured out why, but I'm not very proficient with perl.