Rebuild only unique dependencies

[infinity0]

Continuing on from #317, sometimes when people do a "rebuild" what they mean is not precisely make -B which rebuilds all dependencies, but a narrower form that only rebuilds dependencies that are not reverse-dependencies of other targets. This keeps the effect of the rebuild localised.

Example:

y.c -> y.o -> y
           \
            v
x.c -> x.o -> x

I'd like "rebuild x" to only clean x.o and x, and not y.o - nor y later. For example, if I updated the tool that generates x.o in a way that I know doesn't affect anything else. It would also be useful for these other scenarios that people hinted upon in #317

• re-run tests without rebuilding all objects
• rebuilding docs if you updated doxygen but not the compiler

Slightly more complex example, to demonstrate how the concept generalises into code:

z.c -> z.o -> z
           \
            v
y.c -> y.o -> y
           \
            v
x.c -> x.o -> x

"rebuild x y" would look at the ancestors of x or y and exclude ancestors of other (top-level) targets i.e. z, the result being to clean y.c, y.o, x.c, x.o. In general terms you'd only clean/rebuild:

  ancestors(requested top-level targets)
\ ancestors(unrequested top-level targets)
\ source code

where \ means set subtraction. (There is a corner case to figure out what should happen if a non-top-level target is requested.)

---

Then there is a separate question of how this functionality should be exposed, unresolved from #317, where the functionality can be conceptually decomposed into a clean and build steps. However high-level human usage probably more commonly wants to do the composed functionality, rather than its decompositions separately. (citation = myself)

[jhasse]

For example, if I updated the tool that generates x.o in a way that I know doesn't affect anything else.

Why not simply delete x.o and call ninja x in that case?

[infinity0]

For example, if I updated the tool that generates x.o in a way that I know doesn't affect anything else.

Why not simply delete x.o and call ninja x in that case?

This isn't convenient in the general case, in a big project with lots of object files, I have no convenient way of determining that I should only delete x.o but not y.o. This PRissue is to add a feature to ninja to easily tell me this information.

[jhasse]

Okay so you have a big project where you "don't know things". But apparently you know that "the tool that generates x.o in a way that doesn't affect anything else". Isn't that information also something you would want to automatically inferred? For example rebuild everything which uses a build rule (e.g. in your example everything that the doc tool touches)?

[infinity0]

Isn't that information also something you would want to automatically inferred?

If you are suggesting that ninja could detect when build tools change, in effect inferring the undeclared part of the full real actual dependency graph of build inputs (including build tools as well as source code), that would be pretty cool. I don't believe it does this currently, and I thought it would be quite hard to achieve, e.g. if the build tool links to other system libraries that could change but the actual build tool executable doesn't change.

What I suggested in this issue seems much easier to implement, since it's entirely in terms of the declared dependency graph.

[infinity0]

rebuild everything which uses a build rule

Doing this manually, would be an alternative idea that achieves something similar to what this issue describes. But I also think this would be hard, since there could be many different ways to refer to the same build tool in a build rule. A very complex build rule could even dynamically choose the build tool, so that it's not obvious that it's running doxygen but it is.

[eli-schwartz]

A very complex build rule meaning what? The first thing that springs to mind here is Makefile metaprogramming (where the build tool can be selected differently depending on environment variables, the result of running a command on each make run to define the value of the Makefile variable containing the build tool name, etc.)

Ninja doesn't support that sort of thing -- the command line that gets executed by ninja should be the same command line as statically defined when build.ninja is created.

[infinity0]

Ninja supports arbitrary shell commands. Arbitrary shell commands can do things like eval $(echo ZG95eGdlbgo= | base64 -d).

But this is getting sidetracked and has nothing to do with the actual issue description.

[eli-schwartz]

Ninja supports arbitrary shell commands. Arbitrary shell commands can do things like eval $(echo ZG95eGdlbgo= | base64 -d).

But this is getting sidetracked and has nothing to do with the actual issue description.

This is not getting sidetracked at all, it's a direct follow-up for:

rebuild everything which uses a build rule

Doing this manually, would be an alternative idea that achieves something similar to what this issue describes. But I also think this would be hard, since there could be many different ways to refer to the same build tool in a build rule. A very complex build rule could even dynamically choose the build tool, so that it's not obvious that it's running doxygen but it is.

However I don't think that the "eval to choose the build tool without telling ninja" case is a reasonable thing for ninja to be concerned enough about that it implements new features. The fact that it's technically possible doesn't mean it's supported.

[infinity0]

reasonable thing

If you don't think that is a reasonable thing, then you don't think arbitrary shell commands is a reasonable thing. But ninja already supports arbitrary shell commands, as it should, and as any other general multi-language build system does and should.

it's technically possible doesn't mean it's supported.

I'm not sure what you mean by "technically possible", and arbitrary shell commands are certainly already fully supported. What is the difference between cc and eval $(echo ZG95eGdlbgo= | base64 -d) that you think the former is "possible" but the latter is only "technically possible"?

[jhasse]

That we design ninja's features around the former.

[digit-google]

Note that a command like ninja -t inputs x y will return the list of all ancestors of x or y, so can be used to implement the feature with something simple like:

touch $(ninja -t inputs x y)
ninja x y

The touch will invalidate timestamps on the filesystem, which may be slightly undesirable though for future build queries.

The alternative would be to modify Ninja, but the changes are non-trivial, since which commands to run depend on internal state of the build graph that is touched from multiple classes and locations in the source code. Doable, but I don't know if this is worth it (or maybe an opportunity for some light refactor / cleanup?)

[infinity0]

That we design ninja's features around the former.

This will break the latter in unforeseeable ways and downgrade it from "technically possible" to "impossible", upon which I and others will no longer have a use for ninja.

The alternative would be to modify Ninja, but the changes are non-trivial, [..]

As I explained in the OP, the changes are equivalent to calculating set subtraction ninja -t inputs x y - ninja -t inputs (top-level targets except x y). This would be straightforward.

[infinity0]

can be used to implement the feature with something simple like:

touch $(ninja -t inputs x y)
ninja x y

This doesn't work for the reasons explained in the OP and #317. What the feature requires is an easy way to calculate the expression ninja -t inputs x y - ninja -t inputs (top-level targets except x y). This is infeasible to do in shell script but could be done inside ninja much more easily.