Re: [Discuss-gnuradio] PyBOMBS-Mirror: Scripts that help build a mirror for PyBOMBS repos

[Wang Kang]

Hi Martin,

I think it's better that PyBOMBS not deal with mirror fetching jobs. In
fact, it's more bash-friendly. And it's the job for mirror site
maintainers rather than PyBOMBS users. In theory, mirror maintainers
even need not to have any knowledge of PyBOMBS. (Maybe PyBOMBS providing
an official rsync:// upstream is a more friendly way for those mirror
maintainers.)

The real problem I'm encounting with is that when adding a new recipe, I
can only add "git+{http,https,git}://" URLs, which means as a mirror
site maintainer, I have to do a lot of git merge jobs so that users have
a smooth git tree and are able to do `pybombs recipes update`
continously.

However, if PyBOMBS can read URL replacement lists distributed by mirror
site, things will be much easier.
Mirror site maintainers can simply serve a plain txt file with http or
https. (Maybe with gpg signatures.)

I think the following demo may describe my thoughts better. 
(Commandline scheme may not be the best, only for the purpose of
description)

PyBOMBS mirror demo
====================

$ pybombs recipes mirror-add
http://example.com/pybombs/gr-recipes.mirror.list
Mirror list: gr-recipes.mirror.list added.
Refreshing receipes mirror list from
http://example.com/pybombs/gr-recipes.mirror.list
Done.

$ pybombs recipes mirror-add
file:///tmp/gr-recipes.user-custom-upstream.list

$ pybombs recipes mirror-update
Refreshing receipes mirror list from
http://example.com/pybombs/gr-recipes.mirror.list ......
Done.

$ pybombs recipes mirror-remove
http://example.com/pybombs/gr-recipes.mirror.list

$ pybombs recipes mirror-remove gr-recipes.mirror.list

$ pybombs install gnuradio
Looking for gnuradio URLs in mirror list...
Found! Using git+http://example.com/gnuradio.git instead of
git+https://blahblah/gnuradio.git

$ pybombs install --disable-mirror gnuradio 

$ cat gr-recipes.mirror.list
git+git://git.osmocom.org/osmo-tetra.git
git+https://example.com/pybombs/osmo-tetra.git
wget+https://blahblahblah/somepackage.tgz
wget+https://example.com/somepackage.tgz
.....


Other thoughts
==============

Checksum
--------------

I think there is need for adding a optional checksum validation in
recipes. Just as what homebrew is like.
In case of broken or trojan packages.

Local cache
---------------

I think local cache is not as useful as  network mirror that can be fast
accessed. 


BTW: I am also a maintainer of https://mirrors.tuna.tsinghua.edu.cn/, I
am working on setting up an stable PyBOMBS running on it. Most problems
of GNU Radio users in China are bad connectivity to gnuradio repos. 

I am happy to contribute Python codes for PyBOMBS when needed.  

Thanks.

--
Wang Kang

On Mon, 13 Jun 2016, Martin Braun wrote:

> Wang,
>
> that's pretty cool, I'll take a closer look at the scripts in a bit.
> (Also, thanks for posting bug reports etc. on PyBOMBS recently).
>
> I have it on the roadmap to integrate something like this into the
> default PyBOMBS feature set, would you be interested in porting this to
> Python?
>
> Cheers,
> Martin
>
> On 06/13/2016 10:11 AM, Wang Kang wrote:
>> Hi, all
>>
>> I just hacked a bunch of scripts that help build a mirror site for
>> PyBOMBS repos.
>>
>>      https://github.com/scateu/pybombs-mirror
>>
>> It seems to work fine on my laptop now. So it will be very fast to
>> deploy on new computers using a local PyBOMBS mirror.
>>
>> Hope it will help for anyone who also want to build a local mirror site.
>>
>> The whole mirror is about 1.2GB.
>> Ettus X300 stuff not included. It's about ~1GB extra.
>>
>> You can add repos that you don't want to fetch into `ignore.urls`.
>> It will be passed through upstream.
>>
>> Besides, you can define custom upstreams according to your network
>> condition into `pre-replace-upstream.urls` to gain better speed.
>>
>>
>> Any comments or pull requests are appreciated.
>>
>>
>
>
> _______________________________________________
> Discuss-gnuradio mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>