Relates to #2750: Fix Substrate Docker

[ltfschoen]

The goal is to provide a fix so that if you run ./substrate/docker/build.sh from the project root directory of the polkadot-sdk, it will build a Docker container that has all relevant substrate-related binaries available.

That is the command that I've been running whilst using Docker Desktop for Mac 4.22.0 (117440), using macOS Sonoma 14.1.1.

I believe it should allow the user to have access to the binary of the following polkadot-sdk packages names:

• polkadot
• polkadot-parachain-bin
• staging-node-cli
• staging-chain-spec-builder
• subkey
• solochain-template-node
• parachain-template-node
• minimal-template-node
• test-parachain-adder-collator

where those packages generate the following binaries:

• polkadot
• polkadot-parachain
• substrate-node
• chain-spec-builder
• subkey
• solochain-template-node
• parachain-template-node
• minimal-template-node
• adder-collator

But NOT polkadot-parachain just yet, since it generates an error #4448.

So far I am trying to add support for all of those binaries, even though later some of those could go into a different Dockerfile, by specifically building those packages here.

So rather than building everything with cargo build --workspace --locked --release in the workspace, we allow them to choose specific packages that are relevant (or only those that don't generate an error).

The progress made in trying fix #2750 include this error:

 => ERROR [builder 4/4] RUN cargo build --locked --release                                                                                                                                                0.3s
------
 > [builder 4/4] RUN cargo build --locked --release:
0.311 error: could not find `Cargo.toml` in `/substrate` or any parent directory
------
substrate_builder.Dockerfile:6
--------------------
   4 |     WORKDIR /substrate
   5 |     COPY . /substrate
   6 | >>> RUN cargo build --locked --release
   7 |
   8 |     # This is the 2nd stage: a very small image where we copy the Substrate binary."
--------------------
ERROR: failed to solve: process "/bin/sh -c cargo build --locked --release" did not complete successfully: exit code: 101

Where the fix has just been to modify this line so it finds the Cargo.toml in the project root https://github.com/paritytech/polkadot-sdk/compare/master...ltfschoen:polkadot-sdk:fix-2750-substrate-docker-file?expand=1#diff-42b72709c8ec68097a5de02440c9263d5c0c3acbd13076b482183991597b8e49R6

The next error was:

150.5 error: failed to select a version for `env_logger`.
150.5     ... required by package `sc-executor v0.32.0 (/substrate/substrate/client/executor)`
150.5     ... which satisfies path dependency `sc-executor` (locked to 0.32.0) of package `sc-client-api v28.0.0 (/substrate/substrate/client/api)`
150.5     ... which satisfies path dependency `sc-client-api` (locked to 28.0.0) of package `substrate-test-runtime-client v2.0.0 (/substrate/substrate/test-utils/runtime/client)`
150.5     ... which satisfies path dependency `substrate-test-runtime-client` (locked to 2.0.0) of package `sc-block-builder v0.33.0 (/substrate/substrate/client/block-builder)`
150.5 versions that meet the requirements `^0.11` (locked to 0.11.3) are: 0.11.3
150.5
150.5 the package `sc-executor` depends on `env_logger`, with features: `anstream` but `env_logger` does not have these features.
150.5  It has an optional dependency with that name, but that dependency uses the "dep:" syntax in the features table, so it does not have an implicit feature with that name.
150.5
150.5
150.5 failed to select a version for `env_logger` which could resolve this conflict
------
substrate_builder.Dockerfile:6
--------------------
   4 |     WORKDIR /substrate
   5 |     COPY . /substrate
   6 | >>> RUN cargo build --locked --release
   7 |
   8 |     # This is the 2nd stage: a very small image where we copy the Substrate binary."
--------------------
ERROR: failed to solve: process "/bin/sh -c cargo build --locked --release" did not complete successfully: exit code: 101

Where the solution has been to go through each of the packages and change it like this https://github.com/paritytech/polkadot-sdk/compare/master...ltfschoen:polkadot-sdk:fix-2750-substrate-docker-file?expand=1#diff-19bdfdcfbc7d4526e1df401f04e4fa041ed5cc9ce9b44ec234dedf9301c7585cR46

Then the next issue was this error:

357.8 error: package `jsonrpsee-types v0.22.5` cannot be built because it requires rustc 1.74.1 or newer, while the currently active rustc version is 1.70.0
357.8 Either upgrade to rustc 1.74.1 or newer, or use
357.8 cargo update -p [email protected] --precise ver
357.8 where `ver` is the latest version of `jsonrpsee-types` supporting rustc 1.70.0
------
substrate_builder.Dockerfile:6
--------------------
   4 |     WORKDIR /substrate
   5 |     COPY . /substrate
   6 | >>> RUN cargo build --locked --release
   7 |
   8 |     # This is the 2nd stage: a very small image where we copy the Substrate binary."
--------------------
ERROR: failed to solve: process "/bin/sh -c cargo build --locked --release" did not complete successfully: exit code: 101

That error is because at the top of substrate/docker/substrate_builder.Dockerfile here it depends on docker.io/paritytech/ci-linux:production that is using an old version RUST_NIGHTLY=2023-05-23 that breaks the build.

So to fix that I could either create a PR to modify https://github.com/paritytech/scripts/blob/master/dockerfiles/ci-linux/Dockerfile to use the latest Rust nightly version that is compatible with the latest polkadot-sdk, and if the PR gets merged then a new release of docker.io/paritytech/ci-linux:production would get created and that would hopefully resolve this issue.

But instead I decided to just add the proposed modifications here to resolve that error.

Then there was a warning

941.6 warning: /substrate/cumulus/client/relay-chain-rpc-interface/Cargo.toml: `default_features` is deprecated in favor of `default-features` and will not work in the 2024 edition
941.6 (in the `smoldot` dependency)
941.6 warning: /substrate/cumulus/client/relay-chain-rpc-interface/Cargo.toml: `default_features` is deprecated in favor of `default-features` and will not work in the 2024 edition
941.6 (in the `smoldot-light` dependency)

That I fixed here

Then there was another error that I resolved somehow (can't remember how I solved it)

941.8     Updating crates.io index
941.8     Updating git repository `https://github.com/w3f/ring-vrf`
949.8     Updating git repository `https://github.com/paritytech/litep2p`
953.6     Updating git repository `https://github.com/w3f/fflonk`
956.0     Updating git repository `https://github.com/w3f/ring-proof`
957.4     Updating git repository `https://github.com/paritytech/arkworks-substrate`
959.2     Updating git repository `https://github.com/paritytech/polkadot-sdk`
1036.7 error: failed to get `sp-crypto-ec-utils` as a dependency of package `sp-ark-bls12-381 v0.4.2 (https://github.com/paritytech/arkworks-substrate#caa2eed7)`
1036.7     ... which satisfies git dependency `sp-ark-bls12-381` (locked to 0.4.2) of package `bandersnatch_vrfs v0.0.4 (https://github.com/w3f/ring-vrf?rev=e9782f9#e9782f93)`
1036.7     ... which satisfies git dependency `bandersnatch_vrfs` (locked to 0.0.4) of package `sp-core v28.0.0 (/substrate/substrate/primitives/core)`
1036.7     ... which satisfies path dependency `sp-core` (locked to 28.0.0) of package `asset-hub-westend-integration-tests v1.0.0 (/substrate/cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend)`
1036.7
1036.7 Caused by:
1036.7   failed to load source for dependency `sp-crypto-ec-utils`
1036.7
1036.7 Caused by:
1036.7   Unable to update https://github.com/paritytech/polkadot-sdk#82912acb

Then i got this error:

907.7 From https://github.com/paritytech/arkworks-substrate
907.7  * [new ref]         caa2eed74beb885dd07c7db5f916f2281dad818f -> refs/commit/caa2eed74beb885dd07c7db5f916f2281dad818f
908.4     Updating git repository `https://github.com/paritytech/polkadot-sdk`
3876.0 error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet.
3876.0 error: 900 bytes of body are still expected
3876.0 fetch-pack: unexpected disconnect while reading sideband packet
3876.0 fatal: early EOF
3876.0 fatal: index-pack failed
3876.1 error: failed to get `sp-crypto-ec-utils` as a dependency of package `sp-ark-bls12-381 v0.4.2 (https://github.com/paritytech/arkworks-substrate#caa2eed7)`
3876.1     ... which satisfies git dependency `sp-ark-bls12-381` (locked to 0.4.2) of package `bandersnatch_vrfs v0.0.4 (https://github.com/w3f/ring-vrf?rev=e9782f9#e9782f93)`
3876.1     ... which satisfies git dependency `bandersnatch_vrfs` (locked to 0.0.4) of package `sp-core v28.0.0 (/substrate/substrate/primitives/core)`
3876.1     ... which satisfies path dependency `sp-core` (locked to 28.0.0) of package `asset-hub-westend-integration-tests v1.0.0 (/substrate/cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend)`
3876.1

Which I solved here by adding the lines

	export GIT_TRACE_PACKET=1 && \
	export GIT_TRACE=1 && \
	export GIT_CURL_VERBOSE=1 && \
	git config --global pack.window 1 && \
	git config --global http.postBuffer 52428800 && \
	git config --global core.compression 0 && \
	git config --global http.version HTTP/1.1 && \

Then I got error:

 > [stage-1 5/6] COPY --from=builder /substrate/target/release/chain-spec-builder /usr/local/bin:
------
substrate_builder.Dockerfile:64
--------------------
  62 |     COPY --from=builder /substrate/target/release/substrate /usr/local/bin
  63 |     COPY --from=builder /substrate/target/release/subkey /usr/local/bin
  64 | >>> COPY --from=builder /substrate/target/release/node-template /usr/local/bin
  65 |     COPY --from=builder /substrate/target/release/chain-spec-builder /usr/local/bin
  66 |
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 4b02f120-ba24-4d54-b2be-a38dc6dce883::iqsvu4z4zrpu3jq9hstthsbsv: "/substrate/target/release/node-template": not found

Which is because the node-template binary has been replaced with other templates in the ./templates folder of the polkadot-sdk that i mentioned earlier, such as solochain-template-node and minimal-template-node, so i've replaced that with the latest binaries here

Now that latest error that I've gotten has been the one that i've reported in this issue #4448, and i still haven't resolved it yet.

This PR now compiles.

[ltfschoen]

@bkchr why isn't this PR needed anymore?

[bkchr]

Yeah sorry, forgot to put a message!

Basically we still need the fix, but this pr is doing modifications in too many places. I assume that the real fix of the issue is much smaller than what this pr proposed. It was also stale, which in the end lead me to close this.