Enhacement: Try to compile online

[nyancat18]

You should try to use an online compriles: Travis CI

that lets the users without monster hardware help your project.

i am a long user or your browser

[nyancat18]

other method would be opensuse build service

[Eloston]

@Lourens-Rich Why are you downvoting this? This is a good suggestion since a continuous integration service would allow for anyone to get the latest binaries for the latest changes, and it simplifies testing.

[Lourens-Rich]

@Eloston, sorry, but on this moment i think we need more developers than hamsters, who will be only test this builds.
If someone helps @gcarq , it would be great!

[DragoonAethis]

The point is that @gcarq and any other contributor wouldn't have to waste time checking out any contributions that won't pass the Chromium test suite. And instead of rebuilding the project locally CI would build and test it automatically, so that next to no time is wasted testing buggy patches.

[Eloston]

@DragoonAethis Actually some of these patches break the Chromium test suite. For example, fix-building-with-safebrowsing.patch, which I wrote for ungoogled-chromium, will definitely break the test suite.

We would probably need to patch the tests too if we want to use them. Not just because of the patches breaking them, but also because we're tweaking GN flags to abnormal values (which breaks assumptions made in the tests)

[DragoonAethis]

Some breakage is expected, but tests eg for layout rendering shouldn't ever fail if you're tweaking network-related features, for instance.

[Eloston]

@DragoonAethis Good point, but there's not much value in the tests if they're testing features mostly unaffected by our changes.

EDIT: Until we get more people working on patches, I don't think the tests are going to help us very much. So the only real value in CI systems would be for its computing resources, in that users can get builds and the developer can see if the code compiles with some changes to the code. It's not a very "idealistic" workflow, but it's more efficient to let the compiler/linker tells us what we forgot to patch or screwed up than reading tons of source code.

[gcarq]

I had this idea in the past, but never invested much time in this.
Does anyone know whether travis-ci allows to allocate that much resources to build chromium?, as it seems easier to setup than opensuse build service.

@Eloston As you mentioned the direct benefit would be the computing power and maybe a more automated release cycle. Also I think the Iridium guys are doing a pretty good job with adapting tests, so it should be not that much work once we know where to look and adapt.

[Eloston]

@gcarq I suppose. But the fix-building-with-safebrowsing.patch is pretty agressive since I remove some functions entirely to make the compiler catch any overlooked safebrowsing references. I think these changes will also break a number of unittests. I can't say for sure until I've looked more into them.

EDIT: This isn't much of a problem for Inox, but more unittests would break for ungoogled-chromium due to source cleaning, unless I exclude a number of files from being deleted.

[gcarq]

I've tried travis-ci but it has a maximum job runtime of 50 minutes, so this won't be an option.
https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts

[Eloston]

@gcarq What constitutes a job? Is a job defined to be some process to be launched by Travis CI as described in the configuration file? Are subprocesses considered part of the same job? If so, I don't believe any single leaf-node target dependency in ninja takes more than 50 minutes, so it may be possible to create a huge list of jobs that builds each target in the order that they need to be in (i.e. doing the work that ninja does for us at runtime).

Unless there's something lenient about the definition of a job and some convenient feature in GN or Ninja that will allow us to conform to the definition of a job, this is probably much more work than it's worth.

Maybe OBS can work?

[DragoonAethis]

A job would be something like the whole project built with compiler X under OS Y for CPU archtecture Z. A single build under Travis consists of all combinations of these (which you have to define).

[Eloston]

@DragoonAethis How is the compiler invoked? I'm trying to understand the relationship between this abstract concept of a "job" with the more concrete (and many) ways that software projects do things.

[DragoonAethis]

If I remember correctly (haven't used Travis in a while), results of each job couldn't "see" each other. Essentially each job was run in a separate VM, then tested in that VM, then deployed on success if you wanted to, and once done, discarded. You'd write a config file which defined how you want your build environment to be configured (OS, compilers, deps), then a set of commands to install remaining deps (not available from what Travis predefines) and the build itself. Then more commands to test the thing, what to do after success/failure, etc.

If you really wanted to push this system to its limits, you could write a wrapper which would periodically check whenever there's enough time left for another component (eg what ninja builds) and if not, let the job continue as successful, upload the results somewhere, then in another job you'd download the results of the previous build and continue building. Now, that's a mess.

If you really want CI, you'll probably want https://jenkins.io/ and a dedicated server. Or a shared server and a wiiiiiide smile to its administrator so that you could evade a ban after a build or two due to resource exhaustion ;) (Eg this is how many Android ROMs are built - despite being open, Travis just can't handle something this big.)

[Eloston]

Alright, thanks for the info. Not doable as I suspected. Fun thought investigation though.

[gcarq]

I'm currently experimenting with www.wercker.com. They provide a way to overwrite the maximum build time: http://devcenter.wercker.com/docs/faq/how-to-bypass-timeouts#how-do-i-increase-the-timeout-for-individual-steps-. Right now it looks very promising, I will post an update if the build goes through. For anyone interested here is the current build status: https://app.wercker.com/gcarq/inox-patchset/runs

[Eloston]

@gcarq Something looks really wrong with the current build. It's at 5.5 hours and it seems to be stuck at 12335/22334. The output of the current step isn't changing either. The output for the previous build that timed out at 25 minutes went farther than this one.

[nyancat18]

you should increase even more the time

360-600

[gcarq]

@Eloston yes this is very strange, I think they have some kind of memory or cpu limit despite the unlimited job time.

[nyancat18]

could you make this
compile a test program (10 minutes at i5 + inox instructions)

and test it at wrecker (example, it needs 30)

after you put the right timeout (9*60)

[nyancat18]

https://app.wercker.com/caimandelta/inox-patchset/runs/build/58751ce5bcacfe01008dd20a?step=58751cf6f9ed0b0001f6e4eb

[nyancat18]

i0ve gone to unsanity

https://app.wercker.com/caimandelta/inox-patchset/runs

[nyancat18]

forked your repo

hardore compiling

[gcarq]

@triceratops1 are you making some progress? looks like your linked build also timed out about 12000-ish

[nyancat18]

i've got 19k
ut fails

[nyancat18]

i'm trying opensuse bild (arch extra)

[nyancat18]

NEEW TEST
https://build.opensuse.org/package/show/home:meh924f/inox-binn

[nyancat18]

@gcarq you could test it at opensuse build service
the only issue its

YOU MUST upload the chromium-xxxxx.tar.xz (source code)

i've limited bandwith, but if you've a good internet speed

you can test opensuse build service (please take as inspiration my attempt)
https://build.opensuse.org/package/show/home:meh924f/inox-binn

[gcarq]

@triceratops1 thanks, I will check it out ASAP (probably this weekend).

[nyancat18]

@gcarq please let me make you a small warning

YOU NEED upload the chromium source code (chromium-xxx.tar.xz) the 490mb file
i would test if it helps, but my internet its bad :D

[nyancat18]

@gcarq i'seen that you created and removed a travis setup, and i've two questions

1 had travis worked, or failed (too)
2 obs (the opensuse build service), hard to config, but UNLIMITED build time/resources, hard because you need download, and UPLOAD the source

[gcarq]

@triceratops1

1. travis was my first attempt but without success
2. I will try opensuse build service next, but this has not a very high priority at the moment. Why someone has to upload the source, isn't it possible to fetch them via HTTP in the build job?

[nyancat18]

no :(

but you've an ajax loader (after you upload ALL files), with right names
included the chormum code

[human-bean]

At least in the webinterface of obs it is sufficient to provide the source url, not sure about osc. However according to the documentation they do not allow certain software:
https://en.opensuse.org/openSUSE:Build_Service_application_blacklist

[nyancat18]

@human-bean

the patented parts (chrome) needs serials (api key)

inox doesnt has api key

with it only free code works