Recommended way to ensure a set of projects use the same baseline?

[sketch34]

We have a large production workspace with dozens of Makefile projects, many that depend upon each other. I'm augmenting the make build system so that the Makefile projects can also depend upon:

1. New internal cmake+vcpkg (manifest mode) projects
2. Public vcpkg ports

It is important in this situation that we do not have different versions of the same dependency being introduced in a build. For example, we don't want the following potential version conflict to occur:

A (Makefile) --> B (cmake+vcpkg) --> C v1.0 (vcpkg port)
D (Makefile) --> E (cmake+vcpkg) --> C v2.0 (vcpkg port)

*** Version conflict, potential issues with both C v2.0 and C v1.0 in dependency chain ***
F (Makefile) --> A
             --> D

I've done some research and there doesn't seem to be a way to achieve this that doesn't involve manually checking that all vcpkg.json / vcpkg-configuration.json in every project set a consistent baseline, and needing to update them all manually whenever we want to do a dependency update across all projects.

If all internal vcpkg.json do not specify port versions in their dependencies (i.e. no "version>=" unless required) then the vcpkg versioning algorithm will resolve the above issue provided all baselines are the same.

It would be great if there was a way to set a global baseline. It appears this can be done in Classic Mode by adding a vcpkg-configuration.json to VCPKG_ROOT. But I don't see any way to do this with manifest mode.

[dg0yt]

There is nothing more global than the top-level manifest.

[dg0yt]

If there is no top-level project, then there is no single vcpkg-root once you leave a single computer.

[sketch34]

I don't see why that is a problem. Each dev machine, CI build, whatever would merely need to use the same "global baseline" commit SHA within whatever VCPKG_ROOT it happens to be using. The minimum versioning algorithm would then guarantee they are all consistent.

[sketch34]

I thought one solution could be to fork vcpkg and add a vcpkg-configuration.json to the root of our fork. The fork is then used by all projects. So that single vcpkg-configuration.json would set the "global baseline" for all projects. But this doesn't work in manifest mode.

[dg0yt]

So with regard to canonical manifest mode, the problems are here:

A (Makefile) --> B (cmake+vcpkg) --> C v1.0 (vcpkg port)
D (Makefile) --> E (cmake+vcpkg) --> C v2.0 (vcpkg port)
^                    ^
|                    --- Should be internal vcpkg ports
 ---- Should be internal vcpkg ports


F (Makefile) --> A
             --> D
^
 ---- Should be top-level project (or internal vcpkg port, used by another top-level project)

where the internal ports would be published through an internal registry.

[sketch34]

I can't convert all the projects from Makefile to be internal vcpkg ports. That would be a huge undertaking to do all at once. We need an interim solution.

B and E are in fact internal vcpkg ports provided by an internal vcpkg registry. However we want the ability to consume them as "cmake + vcpkg" projects rather than directly as ports for workflow reasons.

[AugP]

This is an interesting scenario and one I think we need to consider further. I added it to the agenda for the next meeting between the maintainers of vcpkg. We will get back to you hopefully later this week with our thoughts.

[sketch34]

@AugP any thoughts yet?

[BillyONeal]

Everyone picking 1 baseline is insufficient to resolve this situation, because C[featurea] and C[featureb] are not necessarily link compatible. As far as I know you have to combine all dependencies you want into one graph.

[sketch34]

Ahh right, good point thanks. Looking forward to what @AugP comes back with when you guys meet next.

[AugP]

+1 to what @BillyONeal said earlier.

What you originally proposed won't achieve what you want. The fundamental you want to get to is one install tree with one install command.

It sounds like you are trying to address 2 constraints:

1. You need to avoid version conflicts throughout your dependency graph, even though not all dependencies are consumed as vcpkg ports.
2. Your internal vcpkg ports (B & E in the example) need to also be available as project submodules or something similar rather than always being consumed as ports. Let me know if I misunderstood something here.

Our proposed solution allows you to configure your projects such that you can call "vcpkg install" on project F to resolve your dependencies, before building it with the Makefile, while respecting the constraints above.

This requires several steps:

1. Create overlay ports for B and E, each with stub portfile.cmake files and vcpkg.json files (that reference C and any other upstream dependencies). When these are used, B & E will not actually be built through vcpkg directly, but vcpkg will still install transitive dependencies like C. The portfile.cmake files would be mostly empty, but should still be configured with appropriate policies, similar to: https://devblogs.microsoft.com/cppblog/using-system-package-manager-dependencies-with-vcpkg/.
2. The overlay vcpkg.json files should be a symlink to the original vcpkg.json files for B & E. Put the symlink and portfile.cmake files in separate sub-directories of B/E without any other files at that location. This is to avoid a potential performance issue where "vcpkg install" may otherwise waste time unnecessarily.
3. The vcpkg.json files for Projects A and D can reference the overlay ports when you want to build B & E separately or reference the original ports directly when you just want to consume them as vcpkg ports.
4. Configure some CMake variables for B & E: Set VCPKG_MANIFEST_DIR to the directory F is located in, turn off VCPKG_MANIFEST_INSTALL, and set VCPKG_INSTALLED_DIR to a common directory like F/vcpkg_installed. Also see the docs: https://learn.microsoft.com/en-us/vcpkg/users/buildsystems/cmake-integration#vcpkg_manifest_dir
5. Your Makefile for F should run "vcpkg install --overlay-ports:/overlays" early on, before the step where it runs cmake configuration on B & E.

You would still want to have a vcpkg.json file for project F as well as some vcpkg.json files for A and D. The manifest files for A and D would determine if you are using the overlay ports or the original ones. When you run "vcpkg install" it should be run at the project F level

Below is an example directory structure:

/Makefile                      -- F
/A/Makefile                    -- uses 'B'
/D/Makefile                    -- uses 'E'
/B                             -- submodule for 'B'
/B/CMakeLists.txt
/B/vcpkg.json                  -- definition & deps for 'B', such as 'C'
/E                             -- submodule for 'E'
/E/CMakeLists.txt
/E/vcpkg.json                  -- definition & deps for 'E', such as 'C'
/overlays                      -- intra-workspace overlays
/overlays/B                    -- stub overlay for B
/overlays/B/vcpkg.json         -- symlink to /B/vcpkg.json
/overlays/B/portfile.cmake     -- Empty portfile (with appropriate policies; see [1])
/overlays/E                    -- stub overlay for E
/overlays/E/vcpkg.json         -- symlink to /E/vcpkg.json
/overlays/E/portfile.cmake     -- Empty portfile (with appropriate policies; see [1])

[sketch34]

Thank you very much, I will try this out.

[sketch34]

@AugP working on this now. So far so good, will update with details of implementation when I have it working end-to-end.

[sketch34]

Hi @AugP,
I have it working as you describe above, thank you.

A further goal: I would like the ability for B to depend on E.

It would be nice to simply use the overlay port for this. But I'm not sure how to fit it into the scheme above.

First attempt:

/B/vcpkg.json

{
  "name": "B",
  "builtin-baseline": "a34c873a9717a888f58dc05268dea15592c2f0ff",
  "dependencies": [
    "E"
  ],
  "vcpkg-configuration": {
    "overlay-ports": [
      "../overlays"
    ]
  }
}

Cmake configure B

[main] Configuring project: B
[proc] Executing command: /home/linuxbrew/.linuxbrew/bin/cmake -DCMAKE_LINKER=/usr/local/bin/g++ -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/home/jarrod/engines/B/out/install/linux-debug -S/home/jarrod/engines/B -B/home/jarrod/engines/B/out/build/linux-debug -G Ninja
[cmake] -- VCPKG_ROOT: /home/jarrod/vcpkg
[cmake] -- Running vcpkg install
[cmake] warning: Embedding `vcpkg-configuration` in a manifest file is an EXPERIMENTAL feature.
[cmake] Detecting compiler hash for triplet x64-linux...
[cmake] Compiler found: /usr/local/bin/g++

[cmake] The following packages will be built and installed:
[cmake]     E[core,table-rdf-submodule]:[email protected] -- /home/jarrod/engines/B/../overlays/E
[cmake] Restored 1 package(s) from /home/jarrod/.cache/vcpkg/archives in 2.99 ms. Use --debug to see more details.
[cmake] Installing 1/1 E[core,table-rdf-submodule]:[email protected]...
[cmake] Elapsed time to handle E:x64-linux: 15.9 ms
[cmake] E:x64-linux package ABI: 4854c921efd0952d705239628177a201d02f7d2034f1c5f65edcf850e93ee063
[cmake] Total install time: 16 ms

...

[cmake] CMake Error at /home/jarrod/vcpkg/scripts/buildsystems/vcpkg.cmake:859 (_find_package):
[cmake]   Could not find a package configuration file provided by "E" with any
[cmake]   of the following names:
[cmake] 
[cmake]     EConfig.cmake
[cmake]     E-config.cmake

[sketch34]

I have the above "first attempt" working, more or less with the following modified /overlays/E/portfile.cmake:

set(VCPKG_POLICY_EMPTY_PACKAGE enabled)

vcpkg_check_features(OUT_FEATURE_OPTIONS FEATURE_OPTIONS
    FEATURES
        internal-overlays E_VCPKG_INTERNAL_OVERLAYS
)

if(E_VCPKG_INTERNAL_OVERLAYS)
  message(STATUS "Internal overlays feature is enabled.")

  # Configure using the source of E located at /E (see dir structure in AugP's post).
  vcpkg_cmake_configure(
    SOURCE_PATH ${CURRENT_PORT_DIR}/../../E
    OPTIONS
        -DVCPKG_MANIFEST_INSTALL=ON   # I needed to explicitly set these to on.
        -DVCPKG_MANIFEST_MODE=ON
  )

  vcpkg_cmake_install()
  vcpkg_cmake_config_fixup(CONFIG_PATH lib/cmake/E)
  vcpkg_fixup_pkgconfig()
endif()

The internal-overlays feature of E simply adds a host dependency to vcpkg-cmake and vcpkg-cmake-config so vcpkg_cmake_configure() are available in the above portfile.

I'm feeling like the complexity of all this is just barely manageable at this point. I'm willing to persevere a bit more with it. Perhaps for B (cmake+vcpkg) --> E (cmake+vcpkg) dependencies it should just always use our internal vcpkg registry, and there is no other option. But it would be a better workflow if I could avoid the need to update the registry every time a source change is made in E that I want available in B, for example.