installer: Download and install bitcoind from GUI installer

[jp1ac4]

This is a follow-up PR to #592 as part of #570 to download and install bitcoind.

I'm creating this draft PR now to facilitate discussion. Once #592 has been merged, I'll rebase on master.

[jp1ac4]

Please note that all commits shown so far are from #592.

[jp1ac4]

@darosior For the download, do you have a preferred HTTP client to use?

[edouardparis]

I would personally suggest that you follow the example of https://github.com/iced-rs/iced/tree/0.9/examples/download_progress
using reqwest and that provide a progress bar example. (you may add a subscription method to the step trait)

[jp1ac4]

Closed this accidentally when rebasing on master. Will reopen again when I have some commits 🥲

[jp1ac4]

As part of this PR, would it be worth moving all the bitcoind-related code (both internal & external) from src/installer/step/mod.rs to src/installer/step/bitcoind.rs?

[jp1ac4]

This is now working and ready to try.

The presentation probably still needs some work and there may be further refactoring needed.

Most of the changes are in 3882f8a and 1e3272e.

I've added DownloadState and InstallState for use in InternalBitcoindStep, but there's not yet an overall state for the step.

I've kept all changes to dependencies in their own commits for now in case we want to tweak versions or make things OS-dependent.

[darosior]

I'm going to test this ASAP. How about splitting off the first, large but trivial, commit into its own PR already?

[darosior]

Although i've got an installed bitcoind i'm getting an error about the binary.

This begs the question though. It's probably easier to reason about this if it's always the case that we both manage a bitcoind binary and the Bitcoin datadir. So i think what you did makes sense.

However if we are going down this route this should be a red error to not have the bitcoind binary yet, as it's the expected state for anybody installing the wallet.

Instead of having the user both first select "Let Liana manage my node", then be shown an error and have to click on "Download bitcoind", how about we simply explain that we'll download the bitcoind binary if they choose "Let Liana manage my node"? Less clicks, no error shown.

[darosior]

The progress bar for the download is great. However there is a lapse of time after the download is finished where we can't do anything, how about displaying the status here (like "Unpacking..")?

Also, we could drop the progress bar once the download is complete:

[darosior]

Ok, i had to click "Start bitcoind" again to be able to get to the next step. But there is no other way than starting bitcoind at this stage, so it should be automatic IMO.

[darosior]

I'm testing this under various scenarii, it's working great. Exciting!

[jp1ac4]

Thanks for your initial feedback.

How about splitting off the first, large but trivial, commit into its own PR already?

I think that makes sense. I'll create a separate PR for it.

Instead of having the user both first select "Let Liana manage my node", then be shown an error and have to click on "Download bitcoind", how about we simply explain that we'll download the bitcoind binary if they choose "Let Liana manage my node"? Less clicks, no error shown.

Yep, I think that's better and will avoid any error messages appearing. Will change it.

Also, we could drop the progress bar once the download is complete

Agree, will change that.

Ok, i had to click "Start bitcoind" again to be able to get to the next step. But there is no other way than starting bitcoind at this stage, so it should be automatic IMO.

Yep, I agree, will change that.

[darosior]

It looks like the latest version does not download a bitcoind binary if one is present on the system already, contrary to the previous binary (which i found less error-prone as in #630 (comment)). It's not a big deal but wonder if that was on purpose?

[pythcoiner]

i'm trying build this PR on a fresh install Debian12 and got this issue:

pythcoiner@debian12:~/liana/gui$ git fetch origin pull/630/head:pr630 && git checkout pr630
remote: Enumerating objects: 75, done.
remote: Counting objects: 100% (75/75), done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 61 (delta 45), reused 51 (delta 35), pack-reused 0
Unpacking objects: 100% (61/61), 23.53 KiB | 1.38 MiB/s, done.
From https://github.com/wizardsardine/liana
 * [new ref]         refs/pull/630/head -> pr630
Switched to branch 'pr630'
pythcoiner@debian12:~/liana/gui$ cargo build --release
    Updating crates.io index
  Downloaded fnv v1.0.7
  Downloaded http-body v0.4.5
  Downloaded filetime v0.2.22
  Downloaded want v0.3.1
  Downloaded native-tls v0.2.11
  Downloaded openssl-macros v0.1.1
  Downloaded hyper-tls v0.5.0
  Downloaded url v2.3.1
  Downloaded tower-service v0.3.2
  Downloaded serde_urlencoded v0.7.1
  Downloaded httpdate v1.0.3
  Downloaded try-lock v0.2.4
  Downloaded form_urlencoded v1.1.0
  Downloaded ipnet v2.8.0
  Downloaded openssl-probe v0.1.5
  Downloaded mime v0.3.17
  Downloaded httparse v1.8.0
  Downloaded xattr v1.0.0
  Downloaded tokio-native-tls v0.3.1
  Downloaded bzip2 v0.4.4
  Downloaded zip v0.5.13
  Downloaded serde_derive v1.0.186
  Downloaded quote v1.0.30
  Downloaded openssl-sys v0.9.91
  Downloaded tar v0.4.40
  Downloaded serde v1.0.186
  Downloaded http v0.2.9
  Downloaded reqwest v0.11.20
  Downloaded tokio-util v0.7.8
  Downloaded hyper v0.14.27
  Downloaded h2 v0.3.21
  Downloaded syn v2.0.29
  Downloaded idna v0.3.0
  Downloaded openssl v0.10.56
  Downloaded bzip2-sys v0.1.11+1.0.8
  Downloaded encoding_rs v0.8.33
  Downloaded 36 crates (4.2 MB) in 0.82s (largest was `encoding_rs` at 1.4 MB)
   Compiling quote v1.0.30
   Compiling futures-core v0.3.28
   Compiling futures-sink v0.3.28
   Compiling futures-channel v0.3.28
   Compiling serde v1.0.186
   Compiling either v1.8.1
   Compiling rayon v1.7.0
   Compiling syn v2.0.29
   Compiling syn v1.0.109
   Compiling wayland-scanner v0.29.5
   Compiling wayland-client v0.29.5
   Compiling bytemuck_derive v1.4.1
   Compiling thiserror-impl v1.0.40
   Compiling bytemuck v1.13.1
   Compiling futures-macro v0.3.28
   Compiling serde_derive v1.0.186
   Compiling futures-util v0.3.28
   Compiling thiserror v1.0.40
   Compiling toml v0.5.11
   Compiling wayland-protocols v0.29.5
   Compiling phf_macros v0.11.1
   Compiling futures-executor v0.3.28
   Compiling find-crate v0.6.3
   Compiling calloop v0.10.5
   Compiling phf v0.11.1
   Compiling wayland-cursor v0.29.5
   Compiling palette_derive v0.6.1
   Compiling futures v0.3.28
   Compiling pin-project-internal v1.0.12
   Compiling foreign-types-macros v0.2.3
   Compiling palette v0.6.1
   Compiling jpeg-decoder v0.3.0
   Compiling pin-project v1.0.12
   Compiling openssl-sys v0.9.91
   Compiling rustybuzz v0.7.0
   Compiling iced_core v0.9.0
   Compiling iced_style v0.8.0
   Compiling foreign-types v0.5.0
   Compiling flume v0.10.14
error: failed to run custom build command for `openssl-sys v0.9.91`

Caused by:
  process didn't exit successfully: `/home/pythcoiner/liana/gui/target/release/build/openssl-sys-8467c151f1b50e67/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_DIR
  OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=OPENSSL_STATIC
  cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  run pkg_config fail: `PKG_CONFIG_ALLOW_SYSTEM_CFLAGS="1" "pkg-config" "--libs" "--cflags" "openssl"` did not exit successfully: exit status: 1
  error: could not find system library 'openssl' required by the 'openssl-sys' crate

  --- stderr
  Package openssl was not found in the pkg-config search path.
  Perhaps you should add the directory containing `openssl.pc'
  to the PKG_CONFIG_PATH environment variable
  Package 'openssl', required by 'virtual:world', not found


  --- stderr
  thread 'main' panicked at '

  Could not find directory of OpenSSL installation, and this `-sys` crate cannot
  proceed without this knowledge. If OpenSSL is installed and this crate had
  trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
  compilation process.

  Make sure you also have the development packages of openssl installed.
  For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

  If you're in a situation where you think the directory *should* be found
  automatically, please open a bug at https://github.com/sfackler/rust-openssl
  and include information about your system as well as this message.

  $HOST = x86_64-unknown-linux-gnu
  $TARGET = x86_64-unknown-linux-gnu
  openssl-sys = 0.9.91

  ', /home/pythcoiner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-sys-0.9.91/build/find_normal.rs:190:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

[pythcoiner]

it seems it's just about libssl-dev is introduced as BUILD dependencies, will dig more on this later-on today, but i think we should update the docs

[jp1ac4]

It looks like the latest version does not download a bitcoind binary if one is present on the system already, contrary to the previous binary (which i found less error-prone as in #630 (comment)). It's not a big deal but wonder if that was on purpose?

It should still download a binary if there isn't already one in the expected location (<liana_datadir>/bitcoin-25.0/bin/bitcoind), but if it finds one there, it will say that bitcoind is already installed. I checked just now and it seems to be working this way still. Could you please check again. Thanks.

[jp1ac4]

it seems it's just about libssl-dev is introduced as BUILD dependencies, will dig more on this later-on today, but i think we should update the docs

Thanks for spotting that. Perhaps we could avoid adding this dependency by using a different HTTP client or one of Reqwest's features that uses rustls-tls.

[darosior]

Thanks for spotting that. Perhaps we could avoid adding this dependency by using a different HTTP client or one of Reqwest's features that uses rustls-tls.

I'll look into this.

[darosior]

I checked just now and it seems to be working this way still. Could you please check again. Thanks.

In my testing it does not.

$ cargo run -- --datadir new_dir_doesnt_exist
   Finished dev [unoptimized + debuginfo] target(s) in 0.89s
     Running `target/debug/liana-gui --datadir new_dir_doesnt_exist`
  2023-08-29T10:10:31.689014Z  INFO liana_gui:120: Created a fresh data directory at new_dir_doesnt_exist

2023-08-29T10:10:31.689068Z  INFO liana_gui: Created a fresh data directory at new_dir_doesnt_exist
  2023-08-29T10:10:47.744328Z  INFO liana_gui::installer::step::bitcoind:181: Writing to file new_dir_doesnt_exist/bitcoind_datadir/bitcoin.conf

And then:

It did not download and unpack it.

[darosior]

Actually, if the path to the bitcoind binary is fixed it does not need to be part of the config anymore? Nor does the bitcoind datadir actually?

Removing this would probably fix my bug from above as a side-effect.

[jp1ac4]

Ah, I'm not sure what was causing that, but there's a more recent version now where I've even removed the "Start bitcoind" button as it now starts automatically. I'll finish some final changes and then mark the PR as ready for review (with possible change to HTTP client pending) and you can check again.

Actually, if the path to the bitcoind binary is fixed it does not need to be part of the config anymore? Nor does the bitcoind datadir actually?

That's true, except currently the location depends on the bitcoind version (<liana_datadir>/bitcoin-25.0/bin/bitcoind), which might be useful for managing upgrades of the Liana-managed node over time or if later on we want to use the same managed node with different Liana data directories (to avoid repeating IBD). I think the bug you encountered should be resolved once my changes here are finalised. Not storing the config would be a bigger change so might be better to save for a follow-up PR.

[darosior]

Same here it looks like we don't need xattr, so maybe it should work to specify no-default-features for tar too?

[darosior]

I've done the move to reqwest with rustls along with addressing my comments above here: https://github.com/darosior/liana/tree/2308_jp_download_bitcoind. I'll perform a release build on this branch to make sure we haven't introduced new system dependencies.

[jp1ac4]

Thanks for the suggested changes to dependencies. I've just committed those and dropped the minreq commits so that we're back to using reqwest.

I did those changes in two commits as reqwest relates to 8caecf3 while the other dependencies relate to 1427ace.

EDIT: At the end, I can squash these commits further.

[jp1ac4]

I've added e7985ff to log the version and hash of the download. Perhaps these log statements could be moved elsewhere, but this seemed the quickest way to add them for now.

Also, some wording and alignment changes on the node management step: 6799889.

[darosior]

This looks good to me from testing. I've also successfully built this using our reproducible build system for releases. No new dependency introduced. @kloaec is currently testing a reproducibly-built binary on Windows.

[darosior]

From @kloaec's testing on Windows:

[pythcoiner]

it seems it's just about libssl-dev is introduced as BUILD dependencies, will dig more on this later-on today, but i think we should update the docs

Thanks for spotting that. Perhaps we could avoid adding this dependency by using a different HTTP client or one of Reqwest's features that uses rustls-tls.

Installing libssl-dev solves the build issue, I don't think we need to change the http client, maybe any client handling https will require it, but we should update the doc, #555 still open and follow the same purpose, I can add here if needed
edit: i commented before reading your today's messages

[pythcoiner]

ACK 6799889 => build on Debian12 + download/install core + IBD, everything run smoothly (don't revew the code), great job @jp1ac4 !

[jp1ac4]

Have added two new commits:

• 7d8d683 - thanks @darosior & @kloaec for this 🙏
• 9314e7b - hopefully fixes a bug on Windows where the exe path was wrong

[edouardparis]

Generally, I prefer to not have circular dependencies. Can you cut the view in multiple functions and call in the step method view the one according to the current step state ? Or would you prefer we do it in other PRs ?

[jp1ac4]

Ah yes, that's a good point. I'll have a look now at how best to split it and if I can do it quickly, but otherwise it might be better in a follow-up PR in order not to delay the release.

[jp1ac4]

Would it be OK to define parts of the view within the step as an Element and pass this to the view function? That way I could match on different values of DownloadState and InstallState within the step and avoid the need to define different functions for each case.

[edouardparis]

In a followed up PR I would prefer finally

[darosior]

Note to self: make sure the wait for the cookie file is enough for mainnet. (Most likely not)

EDIT: we should probably use try_wait as a less brittle way of detecting if it errored.

[darosior]

ACK bf67f94 -- modulo a few changes we'll address in follow-ups. I've not reviewed the code but significantly tested it on both Windows and Linux.

[darosior]

Congrats @jp1ac4 on keeping up with the review (totaling above 200 review comments on boths PRs) and all the challenges.

[darosior]

@edouardparis i'll let you have the final call on this

[edouardparis]

ACK bf67f94