Wget - Sync remote site with dynamic URLs that contain tokens?

[Casey Jensen]

Hello,

First of all, apologies, this doesn't fit the category of a bug, because I
can't figure out whether it's possible or an ever-intended solution for
wget. I reviewed the man page and did some searching, but didn't find a
solution for the problem.

I was interested in leveraging wget to keep a remote list of files in sync.
I construct a file list on the fly that I'm feeding to wget, and a location
I'm downloading files to.
wget -nc -a $logfile -i $filePath/filesynclist.txt -P $folderSyncPath

The issue is that these are coming from a CDN and are signed URLs. So they
come with a temporary token value appended onto the file name, for example:
installer.pkg?token=088817451a9c490093t5ob22eyaorncifukx

However, the next time those URLs are looked up, that token value may
change. Thus, that file list will also change because the old token value
is not present/valid any longer.

Ultimately, wget needs the token value to access the file, verify its
presence, see if there has been a change, and download it, but because the
token value itself is dynamic, it results in the content being downloaded
over and over again.

Is it possible to tell wget to ignore the token for the purposes of the
comparison in the existing file list (eg wget looks up the full url, but
then removes the token string and confirms if that matches something
locally downloaded), and only download if there is not a "base url string"
that matches?

This is quite a complex problem. Thank you for the help.




Casey Jensen

Associate Experience Engineering

Macintosh Engineering

casey.jensen@capitalone.com

Click to engage our team
<https://intake.cloud.capitalone.com/mac-engineering>

______________________________________________________________________



The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.