-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automatically startup and download bitcoind
when starting up the GUI
#570
Comments
@jp1ac4 will take care of this. Anyone feel free to chime in about the approach. |
Meta point: the development could be split in two PRs: 1) find and startup bitcoind 2) download bitcoind if we can't find it. |
Related: #101. |
do we have the ability to check if the space available is on SSD rather than HDD? (I guess users will not wait several weeks for IBD) |
At least ask the user to confirm that we can use data from this directory? |
under Linux distro using systems, should check if there is some edit: still under Linux check under |
under Linux I guess in |
I don't think so, what if the user want to uninstall Liana and remove (unwantedly) Bitcoin data? |
under linux I guess in |
do we have the ability to check if the space available is on SSD rather than HDD?
I haven't looked into it but i assume we'd look for the space available on the partition on which the path we'd store the datadir at resides. So the physical type of the drive shouldn't matter.
(I guess users will not wait several weeks for IBD)
For now they have to anyways. AssumeUtxo (linked above) could help with that.
…-------- Original Message --------
On Jul 13, 2023, 9:22 PM, pythcoiner wrote:
> - What warning should appear about those in the GUI (like "around 20G of space will be used")? Any other check (like checking available storage)?
do we have the ability to check if the space available is on SSD rather than HDD? (I guess users will not wait several weeks for IBD)
—
Reply to this email directly, [view it on GitHub](#570 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AFLK3F3GXC33BDTXCL4GYPLXQBDJFANCNFSM6AAAAAA2JDUXAA).
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
my mean, it will be useful to at least hint to users that datadir location on HDD will be the cause of several weeks IBD, so, the user can decide to move to another machine or wait |
I think both options should be "editable" maybe locked by default, for users who know what they are doing. Regarding the pruning, I think 6GB (1 for the blocks, 5 for UTXOs) is already a lot for basic users.
10GB seems already a lot, unless they are very comfortable with more. Regarding the HDD vs SSD, I don't think it matters so much as to TELLING the user that it will be long anyways. 2 (full) days with SSD, a lot more with HDD. We could tell the user to move their Liana to ssd entirely if they have both, instead of doing a specific thing for core? not sure. Should we keep Core running for initial IBD even when Liana is closed? |
don't bitcoind do a full IBD before pruning? in that case, the user needs 550G during IDB and then 20/10G after pruning. |
That's risky. It's like what, 4 days of full blocks? To synchronize with the chain Liana loads a watchonly wallet on Bitcoin Core. Upon loading it will rescan the blocks since it was last loaded. To do this it needs the block data. This means a user needs not keep bitcoind running without the Liana watchonly wallet loaded for more than 4 days and that they must load the Liana watchonly wallet at startup to sync it before the historical data gets deleted. Otherwise they would have to reindex the whole chain with the wallet loaded (which'd take almost as long as an IBD). I was thinking of something like 20G, which would be around 2 months and a half of full blocks. And i would err more on the side of not being too aggressive: disk space is cheap nowadays, being stuck and having to manually reindex through the command line is not.
Why? This is huge and i don't think it is necessary. Rather the contrary, i would try to limit resources consumption (but then it's a tradeoff with writing more to disk, which is the bottleneck for IBD i think).
If you are going to start tweaking with bitcoind's parameter you might as well run your own.. I don't think we should have any option for this: we are just trying to find reasonable defaults for most of our non-technical users.
Good question. Without thinking more about it i'd lean more on the side of cleanly shutting it down when Liana is closed, as otherwise a user could just stop their machine without cleanly shutting down bitcoind, which could result in a partially written cache and therefore an even longer IBD.
Historical data is deleted as we go. Note this also means that a lower prune target makes IBD marginally slower. |
under Linux (systemd) I guess we can handle that using the poweroff.target |
Currently, when installing Liana for the first time on a network, the bitcoind connection check comes towards the end after the descriptors have been defined. On the other hand, if Liana was previously installed on a network, an error will occur when trying to start the Liana daemon if bitcoind is not running. I'm thinking that in both cases, there could be a common first step to check the bitcoind connection and then provide options if it's not running/installed etc. This could be an extension of the current Bitcoin Core settings page or perhaps a standalone set of steps to run/install bitcoind. |
I'm on the fence because while i agree we could have this step in common (if it's reworked a bit), we probably always want to show the bitcoind connection configuration in the installer. Let's see what it could look like if we don't think it's necessary. When arriving to this step:
The reworked "bitcoind configuration" step would have the following look and behaviour. First of all it would look for the What do you think? Does that make sense to jump-through during the installer if a bitcoind's running? Does that make sense to try starting up bitcoind automatically? |
That sounds fine. I think the main thing is to reuse the same screens (where possible) to configure bitcoind in both cases, even if they are shown at different steps depending on installation vs "regular startup". On a related note, I wonder if at some point it would be worth separating the wallet creation with its descriptors from the bitcoind connection, as you might want to define the wallet without having any connection.
I think it would still be worth asking for confirmation from the user that the pre-populated configuration is the required one.
I've been thinking about the following:
Perhaps here we should focus on running a minimal "throwaway" node required for running Liana just in the present moment. This would be mainly for those users who don't intend to run their own node or are not able to connect to their usual node at the present moment and just want to quickly use Liana. We could define a |
Yeah i was initially kinda opposed to this. But i've been recently considering for multisig scenarii where cosigners shouldn't be required to run a bitcoind.
In my opinion if there is already a configuration, we should use it. If there is none and no datadir we should create our own. But i don't feel too strongly about it.
Yes and we're compatible with all of them? It's not an issue if the user want to tweak their config, as long as they know what they are doing. Could you expand on why it could come to be an issue?
This is pretty compelling. One issue i can see in doing this is if they ever want to use a bitcoind for something else, it could clash for ports and it would duplicate disk space. Ports issues are fixable and using a couple dozens GB of disk space shouldn't be treated as an issue IMO. So i'd say go for this approach. If there is no existing datadir at the default location, create a new |
Also @kloaec mentioned we could create watchonly wallets with |
I'm happy to go with this approach (see also my comments below about technical users and choosing the right configuration options).
My main concern was that for more technical users, if we try to find their existing config and start bitcoind for them, we may choose options they don't want at that particular moment, e.g. if we choose the wrong I suppose it boils down to identifying different user personas and understanding their requirements. If we focus here on less technical users who don't want to manage bitcoind themselves, it may simplify the design and implementation.
That sounds good to me.
I'll take a look at that. |
Yeah my thinking was that we'd use their own bitcoin.conf if a bitcoind datadir already exist, and would bail out if we can't find it. But really the "segregated minimal bitcoind" approach seems best to me.
Sure, it's a bit orthogonal but should also be trivial (basically setting an argument to |
I'll take care of this, it's starting to be pressing as i'd like to have it in a 1.1 bugfix release. |
OK thanks. Regarding the other changes, I'm working in this branch: https://github.com/jp1ac4/liana/tree/start-bitcoind-from-liana-gui. I've updated the installer so that it creates I'm using the toml crate for writing to I was planning to use a single |
… true` 8ae597f bitcoind: create watchonly wallet with load_on_startup = true (Antoine Poinsot) Pull request description: Upon startup bitcoind will update loaded wallets before pruning historical chain data. This is a trivial fix for people using an aggressively pruned `bitcoind` to not be forced to `-reindex` if they haven't started their Liana wallet for longer than their prune target allows for. See #570 (comment) for a discussion about this. ACKs for top commit: darosior: self-ACK 8ae597f Tree-SHA512: 022ab1224d182372963a6ceb9baaf13aecc568b9d0d546d490ac1fb15450a8c70611b4b25a3a8a2f6cf613e28e8b54c8f5e4c55a4da723927ffc4c6e179f2a63
Yeah
Good question. I'd say a single |
Yep, I think this will be the easiest approach.
I agree. I'll continue with a single file ( |
Yeah, absolutely, perform a random 4k read test on the data directory, really chainstate directory's device. If it's a HDD you will get 100-200 max IOPS, and if it is a SSD or even good USB flash drive you will get thousands. The random read performance of storage often becomes the bottle neck when the machine doesn't have enough RAM to keep the chainstate database entirely in memory. You can sync very fast on a HDD if you have 16 or 24GB of memory and a high -dbcache. But with 4 or 8 it will become glacial once the disk chainstate needs to be read to connect most blocks. |
@darosior For parsing an existing config file, it might be convenient to use https://github.com/zonyitoo/rust-ini, or would you prefer not to add any dependencies for this? |
Note that if we use a separate |
From a quick skim |
@jp1ac4 what's your ETA for the PR? I'd like to proceed with a feature freeze for v2 around the 15th of August and what you're working on is a must have. :) |
I think I'll be ready to create a draft PR within a few days with the main logic in place for configuring and starting an already-installed bitcoind. I have an updated Liana installer currently "working" in https://github.com/jp1ac4/liana/tree/start-bitcoind-from-liana-gui. The main things missing are error handling and tests, plus general refactoring, but you should be able to get an idea of how it's working and if any major changes in approach are required. |
We might be interested in using |
I am leaving my comments about the current state of the PR, as wording and maybe order of things needs to change a bit. "Liana requires a Bitcoin node running. Do you prefer to: Info: I would also prefer to have a detection step of Core present on the machine or not BEFORE this step, so we can avoid overwhelming newbies with technical questions. |
Ok, the current one is pretty good too. Up you you guys to choose. For the extra info, I am not sure how to display it. Maybe instead of "buttons" we can have 2 big ass rectangles, with all the info in it (kinda like when you choose a pricing on a website, which "tier" do you want to use?) |
My initial idea was to have a radio button / checkbox on the left-hand side, with the corresponding description on the right-hand side. So there would be two rows, one for each option, which should give more space for longer descriptions. But I'm also happy to go with your suggestion. |
698eff7 doc: Add step to choose bitcoind type (jp1ac4) 36cf85d gui: add option to use internal bitcoind (jp1ac4) 765c68b installer: allow for different previous messages (jp1ac4) Pull request description: This is to resolve part of #570 (configure and start an already-installed bitcoind). I'm opening this draft PR so that you can provide feedback and check if any changes in approach are required. I've added optional steps to the installer for the user to configure and start an "internal" bitcoind that uses `~/.liana/bitcoind_datadir` as its data directory. The main things missing are: - [x] Make it work on Windows. - [x] Stop internal bitcoind when Liana is not running. - [x] Start this internal bitcoind, if applicable, when returning to Liana after installation. One option for this would be to check in the Liana config file whether the bitcoind `.cookie` file is within the `~/.liana/bitcoind_datadir` folder, which would indicate that the internal bitcoind should be used, and then to start it (we might need to store the executable path if it's not in PATH). - [x] Tests! ACKs for top commit: edouardparis: ACK 698eff7 Tree-SHA512: ce561cd74944b9a80e73bf0f45eafc613a033b115276c208cba95a00920409d3ec56b81cf50fd29eb60c82b96c1d9295a51b8df7e7ca62a4474dd77461564dd0
The first part (
You'll need a way of working with Zip and Tar archives. You'll need a way to SHA256 a file, you can probably read the file content and reuse Anything i could be missing here? cc @edouardparis @kloaec. |
That sounds good. Would you suggest in
Yep, I think I can use https://github.com/rust-bitcoin/rust-bitcoin/tree/master/hashes. |
I'd say one level up. |
I'll create a separate issue about "updates" of the software. The option of hardcoding the SHA256 prevents us to deal with this, without releasing a new Liana version. If the binaries are removed from Bitcoincore.org, our software would break. An alternative would be to check against a bunch of signatures from coredevs, and we grab the Shasum file + latest release on bitcoincore.org . We can hardcode the pubkeys of some coredevs in our release, similar to hardcoding the shasum in terms of security. The issue with this alternative option is that we may break other things, if a release of Core isn't compatible with Liana for some reason. Maybe we could restrict this to minor releases of Core (i.e., looking for 25.x only, not 26+) for each Liana release. |
If the binaries were removed from bitcoincore.org, this functionality would be unavailable (which is for the better: we wouldn't want people to download a vulnerable binary). I don't think it's worth trying to be smart by guessing what the latest release is (how would we even do that, Github API?) nor checking GPG sigs. Just checking a hash is lower tech and the probability of a binary being removed is IMO far lower than the risk of installing an incompatible new version of Bitcoin Core. However there may be a point to nudge users to update Liana if the download fails. But then again presumably at this step they just downloaded the software. So to run into the error they would have to either:
This seems quite unrealistic to me. |
It could also be that it's bad timing, latest version of Liana but with a Core version that doesn't exist anymore. That would cause problems for however long we need to put the fix, probably less than 24h? If looking for a latest version of Core is out of question, we could also have a "fall back" Core version, assuming the bug is only with the latest release. Maybe not useful though, i'm just trying to think about edge cases.
I assume Bitcoincore.org being unreachable is covered already? Should we look at mirrors or other sources before throwing an error? |
The fallback to a lower version could make sense, although it sounds a bit overthought to me. (Again, this is super not probable.) Mirrors definitely make sense, although it's not necessary for now. We should create a dedicated issue for this to be addressed in a later version. |
Following up on the text edits from @kloaec.
Both are fine by me. I feel like Kevin's idea would be a bit fancier, if that's not too hard to implement. But the current screen is also fine. Let's keep it this way for v2 unless we really need to change it. |
All: anything else here that should be addressed for v2 after #630? |
bf67f94 installer: download and install bitcoind (jp1ac4) d507d57 gui: add subscription to Step trait (jp1ac4) ae23fbc gui: add download module (jp1ac4) b3bc943 gui: add dependencies to download and install bitcoind (jp1ac4) Pull request description: This is a follow-up PR to #592 as part of #570 to download and install bitcoind. I'm creating this draft PR now to facilitate discussion. Once #592 has been merged, I'll rebase on master. ACKs for top commit: darosior: ACK bf67f94 -- modulo a few changes we'll address in follow-ups. I've not reviewed the code but significantly tested it on both Windows and Linux. edouardparis: ACK bf67f94 Tree-SHA512: bda4b8bfbb3a59917d9ea60c074c2b0021229213240bebc4bd176f9909c62ab323d0a8d4becffbac192f29fad92adb81b2da0bc9b37c8eea1654c26ec6699077
If
bitcoind
is not already running when the GUI is started, we should run it. If we can't find anybitcoind
binary, we should download it from https://bitcoincore.org/bin/bitcoin-core-25.0/.There is a number of things to be taken into account. Here is a non exhaustive list:
bitcoind
with? Pruned? By how much? Lower cache?bitcoin.conf
, and not just startupbitcoind
with those. This is to avoid any surprising behaviour to the user, and also in case we update those options.bitcoind
binary?bitcoind
binary, but not a bitcoin.conf?-prune
.)bitcoind
binary we download?The text was updated successfully, but these errors were encountered: