Re: Wget fails to download some URLs from www.investing.com

[Tim Rühsen]

On 2/14/24 11:21, Chris Smith via Primary discussion list for GNU Wget 
wrote:
> Hi guys,
> This is not so much a bug as requests being blocked by the cloudflare server.
>
> Checkout:
>      https://www.investing.com/crypto/bitcoin/btc-usd-historical-data

The server just gives a 403 Forbidden.

> The URL works in Firefox, but fails to download using cURL or Wget.
> I have tried various user-agent strings, so that is not the problem.

Maybe the page requires a Referrer: header? In theory it could be 
anything, e.g. the page is HTTP/3 only or whatnot :|

You can start with F12 (developer console) in Firefox, load the page and 
check the request headers. Or better: there is a copy-as-curl when 
right-clicking on the request line that puts a full curl command line 
into your clipboard.

> The following URL works for Firefox, cURL and Wget:
>      https://www.investing.com/equities/lloyds-banking-grp-historical-data
>
> Kind regards,
> Chris Smith <chris.smith@eclipso.eu>
>
> _________________________________________________________________
> ________________________________________________________
> Your E-Mail. Your Cloud. Your Office. eclipso Mail Europe. 
> https://www.eclipso.de

OpenPGP_signature.asc
Description: OpenPGP digital signature