Design a unified target graph for projects and solutions

[Nirmal4G]

Goals

• Basic targets for Project dev loop support.
• Basic targets for Solution dev loop support.
• Integration targets for Project<->Project.
• Integration targets for Project<->Solution.
• Graph Build support for both project and solution.
• TBA

Targets List:

• Restore
• Build
• Clean
• Rebuild (wrapper around Clean + Build)
• Test
• Run
• Publish

NOTE: Design of the Target Graph is not final!

[jzabroski]

This is pretty neat, but I think it's less clear what the unified target graph means across various SDKs.

I always thought the Maven way of thinking about this was a lot cleaner and better followed the principles of dependency inversion by declaring an interface in the form of a goal that the plug-in targets had to achieve. I think that also aligns closely with what the original "The Build Master" textbook on MSBuild advocated for. If you think broadly about "what do we really want that other ecosystems have" and "what we have today (via binlog replays)" you can see that the main gap is still in traceability of inputs, mostly around goal-directed values passed into black boxes for these targets vs the targets themselves. There's no way to "observe" what the black box does in generating outputs/task items.

[Nirmal4G]

I think it's less clear what the unified target graph means across various SDKs.

May be I should come up with better naming. That said, with the current MSBuild target graph, although extensible, they are not what I call a concept based or more accurately—action based graph.

What I mean is, if you take a ProjectReference or any ~Reference item which can be a link/reference to other things in the project/solution or even outside of the project/solution cone, typically you would have certain behavior control on how the references are consumed by the project/solution. You would also have a way of "collecting" those item, do some "preparation" like transforms, de-duplication, etc... and then you would "resolve" those references into assets that the project/solution can consume.

This is just one example of an action based target graph design, I'm experimenting with. Likewise, you'd have a similar pattern of thought for other concepts that exist in the Build System (Restore, Build, Test, Publish, etc...). Remember that MSBuild is an orchestrator and not a build system. It was called XMake for a reason! The common tasks/targets that MSBuild ships with, is the build system or build definition but only designed for .NET only. That's why C++ team, when they migrated from the old system, they had to have their own Common tasks/targets and not use the ones that ship with MSBuild.

I always thought the Maven way of thinking about this was a lot cleaner and better followed the principles of dependency inversion by declaring an interface in the form of a goal that the plug-in targets had to achieve.

I personally don't like certain things about Maven, first is Java but that's subjective 😅. Then again, I have to say, when I was working on some java projects, Maven felt a lot cleaner, even though it was verbose, it had defined the language better. Once you know the language, the concepts, you can easily understand a POM file on its own. Maven and other systems like CMake, Eclipse IDE is where I saw specialization done best. But nowhere I saw generalization except in MSBuild and Visual Studio. The VS project system is one of the most extensible open platform I've ever come across. Nothing comes even close. What I mean is the design choices and philosophy of the older system. Not the garbage implementation we have today.

The main gap is still in traceability of inputs, mostly around goal-directed values passed into black boxes for these targets vs the targets themselves. There's no way to "observe" what the black box does in generating outputs/task items.

TRUE THAT. As you said binlog works but it's still a text log encoded in binary format. It depends on the caller to log the message as every other logging mechanisms work. But there's no evaluation level tracing in MSBuild orchestration process. There's also no scope or encapsulation concept applicable to properties, items or things that are either read-only or once-defined. But I for one, need a step-though debugger and tracer for MSBuild eval engine, badly! With that my life would be complete. 😇

TLDR; XMake design and philosophy was better.

You could literally have any kind of action to be done with MSBuild (at least in the XMake days). You still can but a lot of workarounds entered into the XMake and turned it into MSBuild, we all know and love today and made it bloat. Though the recent improvements to the Schema made it simpler, it suffers from things that every other "All-in-One" Swiss army tools suffer: Generalization and Specialization. Without the right balance of those, you would have all sorts of issues. These could stem from historical baggage that plagues the .NET Community or even issues that are a consequence of certain decisions that allowed specialization into an otherwise mostly a generalized product.

Even Miguel-de-Icaza who ported .NET Framework over to Mono has said some beautiful, yet sad things about XBuild (MSBuild on Mono). This is not the first instance. M$FT could get hands on some of the futuristic tech/concepts/ideas and they butcher it with "business" decisions. I have seen too many to list. All the products that ever produced by M$FT could have the potential to be better than anything at that time. In fact, every single thing was ahead of their time. But alas, it's fate.

AnyBuild is there too but I won't go too much into it. One thing I'll say is that AnyBuild could have been MSBuild Extended!? It was created for whole different set of requirements/use cases and even now it does work well in those scenarios. But what I'm hoping to have, is to learn from Build Systems, and to build a generalized orchestrator and a specialized declaration system to achieve what MSBuild couldn't. Everybody who loved and hated MSBuild, was hoping to see a XAML/POCO based system.

When I say XAML build system, I don't mean TFS XAML, NO, that was a horrible application of XAML. With XAML, you could have the most verbose doc or the most simplest doc. It's up to the writer. That's also why I'm experimenting with a XAML syntax based MSBuild fork privately. I'm still working up the design and MVP to check feasibility but it's just an experiment to learn about build systems.

Finally,

what do we really want that other ecosystems have?

I'm not even going there. What I want, not even a single existing build system offers. It's either too much generalization and too much specialization. Nothing that says, this is a declarative task, orchestrator, manager and runner. NOTHING! Believe me, I have looked all over.

When you are designing a solution for a problem through software, what you need is not just the right set of tools to create a solution at that time. NO! NAH! That only works when you're physically solving the problem. But when you are solving it via software, one of its core philosophy is Don't Repeat Yourself. Then you have Create once, innovate forever. So, What I need is a single solution that is created once for this kind of problem and can be innovated forever. It's like building the DNA of the software world. Playing God. That's what made me interested in software and friends.