Re: [Mingw-cross-env-list] Build automation and binary distribution (was Qt5 Shared / Travis-CI)

[Jonathan Greig]

Tony,

Using GitHub to counteract Travis-CI limitations and vice versa does seem like a dead end and could have other technical issues. But many people use Travis-CI to send data or deploy to a remote server. You wouldn't necessarily have to deploy to GitHub(The only reason we do is because our website is hosted on GitHub pages and it will be braindead easy to link to the .zip buttons for dev builds and our scenario is much different than the one MXE faces). I think using S3 for caching is any better an idea than using GitHub. They are both sub-optimal workarounds.

The enemy here is time, so Travis-CI just needs to figure out what to send and where to send it as quick as possible. In fact, Travis-CI probably just needs to signal the remote server to tell it to clone your GitHub repo and do the build and once the build is complete, notify developers via email/mailing list/etc... Digital Ocean ( https://www.digitalocean.com ) has cheap VPS hosting that could be used for this purpose.

Another option would be to contact Travis-CI people directly and explain the situation and point them to the Issues and the discussions on the mailing list. Since they provide the service for open source projects, they may make an exception to the time limit. Maybe not.

Another option could be to ask Ryan Gordon about this. I previously was considering BuildBot and later discovered Travis-CI. Ryan is familiar with BuildBot and if hosting the MXE CI system is a problem, it may be possible that he'll set you up with an account on icculus.org considering the nature of the project. I copied him on this email in case he wishes to chime in on the matter.

Cheers,

Jonathan

On Fri, Mar 7, 2014 at 10:37 AM, Tony Theodore <address@hidden> wrote:

On 7 Mar 2014, at 20:51, Jonathan Greig <address@hidden> wrote:

[…]

(like pulling a known previously built binary or source package right from an MXE github repo similar to what I mentioned above concerning the Travis-CI issue Tim explained above).

I noticed Tim's pull request was closed, but I hope that MXE does incorporate Travis-CI into the workflow and that it is a high priority. Not only does that immediately notify the commiter that they have broken the build, it notifies anyone making a pull request that they will break the build, and others such as myself can actually look at the build history and immediately know that the build is failing before even cloning it just to find out it doesn't build.

Travis-CI would be a very welcome addition to MXE, the benefits are very clear and it’s the “nicest” CI system I’ve seen. However, using github as a binary cache to workaround the time limits doesn’t seem like a reasonable use of either service to me. I’ve seen some blog posts about using S3 as a cache, but those are aimed at reducing build times, not breaking a build into restartable stages.

I’d rather use some distributed buildbot system that can take up idle time on various user-contributed hardware, possibly using our S3 bucket as cache. That can’t work as no normal user could possibly upload a full build anywhere. Maybe some hybrid of testing and collecting results from distributed machines and then building known working versions on something with a backbone connection to upload the build artefacts?. I know that is sub-optimal, but a full build of MXE is many hours and many gigabytes - not really suited to services designed to encourage unit testing and code sharing (in my mind).

A clean solution to this would be great, there are related issues[1] and recurring themes. An apt repository would probably work and remove the need for caching, but how would we test and populate that in the first place?

Suggestions welcome, it could be that I’m too conservative in my estimation of how Travis-CI and GitHub should be used.

Cheers,

Tony

[1]

https://github.com/mxe/mxe/issues/140

https://github.com/mxe/mxe/issues/129

https://github.com/mxe/mxe/issues/145