Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] main from linera-io:main #9

Open
wants to merge 1,151 commits into
base: main
Choose a base branch
from
Open

Conversation

pull[bot]
Copy link

@pull pull bot commented Sep 8, 2023

See Commits and Changes for more details.


Created by pull[bot]

Can you help keep this open source service alive? 💖 Please sponsor : )

andresilva91 and others added 27 commits August 1, 2024 14:24
## Motivation

As we replace different existing functionality with blobs, we'll need different blob types.

## Proposal

Make `BlobId` into an enum

## Test Plan

CI
## Motivation

`HashedBlob` isn't really a hashed blob anymore, as the `BlobId` now has more than just the hash.

## Proposal

Change `Blob` to be `BlobContent`, and change `HashedBlob` to `Blob`. Hopefully this makes the code a bit less confusing.

## Test Plan

CI
* Remove Python HTTP server from LLM example README

It's no longer needed, now that the service can fetch it directly from
the internet.

* Fix the directory used in the commands

It doesn't seem like the `llm` directory is entered at any time before
that command.
…gStore`. (#2339)

* Replace async_lock with sync_lock for the hashable_wrapper.
* Replace the async_lock by sync lock for the LRU cache.
* Remove one store and restructure the test code for value-splitting.
* Rename the "create_memory_context/store" as "create_test_memory_context/store".
* Put the functions "create_test_memory_*" under "with_testing" control.
* `linera-version`: use a committed cache of the API hashes

* `scripts/test_publish.sh`: allow publishing to an existing registry

* `linera-version`: only check API hashes

* `linera-version`: correct command in test failure output

* Only enable the `tokio` feature for `fs-err` in `linera-service`
* Introduce the `MEMORY_STORES` for storing the memory stores.
* Switch the memory stores from async `Mutex/RwLock` to sync ones for memory.
* The `now_or_never` is removed from memory code.
* The `MemoryStore` now satisfy the AdminKeyValueStore` and the testing framework for this trait has been improved.
* The random namespaces are now used for the `MemoryStore`.
* The `MemoryStorage` has been changed accordingly.
* For testing cases, the namespace is killed on drop.
## Motivation

We renamed this in a previous PR, but some names were still incorrect

## Proposal

Use blob content wherever is appropriate 

## Test Plan

CI
## Motivation

Right now we just have the NFT's image "payload" (the bytes of the image) loaded in the example app's state.

## Proposal

Let's use blobs instead for that

## Test Plan

e2e test was modified to work with blobs
* Simplify linera-sdk testing.

* Fix typos, avoid unwrap.

* Remove with_rejection.
* `linera-client`: put metrics and filesystem support behind a feature flag

* `linera-client`: feature-flag the storage service

* `linera-client`: build `persistent::LocalStorage` and conditionally use it on the Web

* `linera-client`: add dummy memory-based `Persist` implementation to help linting

* `.github/workflows`: test building `linera-client` for the browser instead of `linera-core`

* Rename missed `storage_service` features

* `linera-client`: add `web-default` feature set for ease of compilation

---------

Co-authored-by: Andreas Fackler <[email protected]>
## Motivation

UI is now outdated since we're not using Blobs on the NFT example

## Proposal

Update UI to also use blobs

## Test Plan



<div class='graphite__hidden'>
          <div>🎥 Video uploaded on Graphite:</div>
            <a href="https://app.graphite.dev/media/video/HlciHFAoHZW62zn13apJ/86647f85-e5e0-4c14-b9a5-4bbd39e16443.mov">
              <img src="https://app.graphite.dev/api/v1/graphite/video/thumbnail/HlciHFAoHZW62zn13apJ/86647f85-e5e0-4c14-b9a5-4bbd39e16443.mov">
            </a>
          </div>
<video src="https://graphite-user-uploaded-assets-prod.s3.amazonaws.com/HlciHFAoHZW62zn13apJ/86647f85-e5e0-4c14-b9a5-4bbd39e16443.mov">Screen Recording 2024-08-06 at 12.08.07.mov</video>
* feat: use "serde_json" for "BlobType" string conversion

* chore: rebuild CLI.md

* fix: use "serde_json" inside original string conversion code

* fix: improve FromStr impl

* fix: roll back "clap-markdown" to "0.1.3"

* fix: roll back "clap" to "4.4.11"

* fix: roll back "Cargo.lock"

* fix: use "match" to avoid "unwrap"

* chore: clippy
## Motivation

This is a follow up suggested on #2354

## Proposal

Check if the `Blob` exists before actually minting the NFT

## Test Plan

CI
* build: update the "Cargo.lock"

* doc: rebuild the "CLI.md"
* feat: add "user_data" arg in "transfer" cli

* doc: rebuild the CLI.md

* fix: update the "UserData::from_option_string" impl

* fix: correct the "UserData::from_option_string"

* doc: add more documentation to "user_data" option

* fix: handle the error case of parsing UserData from string

* Update linera-service/src/linera/main.rs

Co-authored-by: Andreas Fackler <[email protected]>
Signed-off-by: duguorong009 <[email protected]>

---------

Signed-off-by: duguorong009 <[email protected]>
Co-authored-by: Andreas Fackler <[email protected]>
…antCollectionView`. (#2366)

* Move the KeyValueStoreView to sync.

* Replace the async_lock::Mutex by std::sync::Mutex.

* Formatting.

* Corrections from clippy.
## Motivation

We don't want `BlobId` exposed to the SDK. Users will only be aware of `Data` blobs, the other types will be only internally used.

## Proposal

Only expose `Data` blobs to the SDK, and assume when dealing with blobs that it's a `Data` blob. Because of that we don't need to use `BlobId` everywhere, and can just use the hash of the `Data` blob in most places.

## Test Plan

CI + also retested the NFT UI
…omically. (#2343)

* Atomic message bundles

* Add a TransactionTracker; have one oracle responses list per txn.

* Deduplicate bundle and operation handling.

* Extract message execution from execute_block.

* Move next_message_index to TransactionTracker.

* Add to_posted test helper function.

* Fix message_id_for_operation.

* Return iterator from make_message_bundles_for.

* Address review comments.

* Address review comments; don't track messages twice.
* `linera-client`: remove `anyhow`

* `linera-service`: move error type out of `linera_client::storage`

* `linera-client`: return concrete types from `ClientOptions::wallet`
jvff and others added 30 commits October 23, 2024 12:57
* Create a `DeliveryNotifier` helper type

Prepare to only allow the chain worker to manipulate it.

* Use the `DeliveryNotifier` in `WorkerState`

Use the new helper type as a first step to moving its control to the
chain worker.

* Pass a `DeliveryNotifier` to the chain worker

Start moving it into the chain worker, so that the `WorkerState` only
manages caching them, instead of actually using them.

* Garbage collect delivery notifiers

Avoid leaking delivery notifiers as the chain workers in the cache are
evicted.

* Move delivery notification to chain worker

This is the first part to prevent a race between adding a listener to
the `DeliveryNotifier` and notifying that messages have been delivered.

* Move notifier registration to chain worker

Finish preventing a race between adding a listener to the
`DeliveryNotifier` and notifying that messages have been delivered. Now
both steps are performed in the chain worker, which serializes them.

* Protect registration and notification methods

Ensure that only the chain worker is able to perform those operations.

* Improve documentation links

Don't link to `true` and use a slightly shorter link to external type.

Co-authored-by: Andreas Fackler <[email protected]>
Signed-off-by: Janito Vaqueiro Ferreira Filho <[email protected]>

---------

Signed-off-by: Janito Vaqueiro Ferreira Filho <[email protected]>
Co-authored-by: Andreas Fackler <[email protected]>
## Motivation

CI was failing on PR #2646 with a JSON serialization failure. That was fixed by PR #2668 but we realized none of our tests covered this case.

## Proposal

Create a test that tries to save a wallet with pending blobs in it. Verified that before #2668 this test fails, and now it succeeds.

## Test Plan

CI

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
## Motivation

#2668 was not necessarily the best way we can fix this issue.

## Proposal

Always serialize `BlobId` as string if `is_human_readable`. Fixes #2671

## Test Plan

Reverted #2668, made sure the `test_save_wallet_with_pending_blobs` test failed, and made sure it went back to passing with the new serialization implementation.

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
## Motivation

This step sometimes fails with API rate limit exceeded, but we don't actually need to run this from scratch everytime. Example failure: https://github.com/linera-io/linera-protocol/runs/31950311713

## Proposal

Cache this step

## Test Plan

CI

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
## Motivation

Currently if a call to `blob_last_used_by` returns an `Err`, we'll print a `ERROR` level log. That is not very accurate because validators could still be not fully in sync and some of them could be missing some blobs.

## Proposal

Change log level to `WARN`

## Test Plan

CI

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
…nic ≥ 0.12 (#2700)

* `linera-rpc`: manually configure TLS for the client as required by Tonic ≥ 0.12

* `linera-rpc`: add CA roots from `webpki-roots`
## Motivation

While getting #2683 across the finish line, I noticed some speed ups we could do to our Docker github workflow

## Proposal

We already build the docker image in `compose.sh`, so no need to do it again before running it.

## Test Plan

#2683 made Docker CI last 20+ minutes. This brings it back to around 10 minutes. Docker CI should be sufficient testing here

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
* Ensure received certificates' messages are delivered.

* Remove unnecessary client mutex locking.

* Add a comment: Why are we retrying?
* Scylla under docker-compose doesn't run in developer mode

* Test changes to docker/docker-compose.yml.

* Added upterm session

* Running on gcp-large with updated aio-max-nr

* Revert change of runner

The custom runner isn't currently working.

* Comment out filesystem tweak

It doesn't work on GitHub's runners.

* Patch `docker-compose.yml` file in CI

Run it in developer mode in CI.

* Comment out upterm session configuration

Should only be needed in our custom runner.

* Add TODOs for the steps to do for custom runners

Ensure that the quick patches are easily remembered to be removed.

---------

Co-authored-by: Christos Hadjiaslanis <[email protected]>
Co-authored-by: Christos Hadjiaslanis <[email protected]>
…le (#2711)

* Don't read environment variable unless needed

Return sooner if the `faucet_url` was provided as an argument.

* Avoid allocating a `String` unless needed

Only allocate the fallback `String` if the environment variable was not
configured.

* Replace `.to_string()` with `.to_owned()`

Make it explicit that we are moving the string to newly allocated heap
memory.

* Document the constructor

Provide a quick description of how the `RemoteNetTestingConfig` is
configured.

* Refactor to reduce one nesting level

Use only one final `unwrap` instead of nested `unwrap`s.

Co-authored-by: Andreas Fackler <[email protected]>
Signed-off-by: Janito Vaqueiro Ferreira Filho <[email protected]>

* Tweak review suggestion

Fix small typos to make it compile.

---------

Signed-off-by: Janito Vaqueiro Ferreira Filho <[email protected]>
Co-authored-by: Andreas Fackler <[email protected]>
The URL is currently incorrect, and changes often. It makes more sense
to require the user to specify it when running the tests.
…2715)

## Motivation

When using Docker Compose, we're not currently properly waiting for the service dependencies.

## Proposal

Add a health check, and remove the `init_done` file

## Test Plan

CI: we can see from the Docker CI logs that both the `proxy` and `shard`s services only get created after `shard-init` exits:
```
Network docker_default  Creating
 Network docker_default  Created
 Volume "docker_grafana-storage"  Creating
 Volume "docker_grafana-storage"  Created
 Volume "docker_linera-scylla-data"  Creating
 Volume "docker_linera-scylla-data"  Created
 Container scylla  Creating
 Container prometheus  Creating
 Container grafana  Creating
 Container prometheus  Created
 Container scylla  Created
 Container grafana  Created
 Container shard-init  Creating
 Container shard-init  Created
 Container proxy  Creating
 Container docker-shard-4  Creating
 Container docker-shard-2  Creating
 Container docker-shard-1  Creating
 Container docker-shard-3  Creating
 Container docker-shard-1  Created               <----  Creations
 Container proxy  Created                               <----
 Container docker-shard-4  Created              <----
 Container docker-shard-3  Created              <----
 Container docker-shard-2  Created              <----
 Container grafana  Starting
 Container prometheus  Starting
 Container scylla  Starting
 Container scylla  Started
 Container grafana  Started
 Container shard-init  Starting
 Container prometheus  Started
 Container shard-init  Started
 Container shard-init  Waiting
 Container shard-init  Waiting
 Container shard-init  Exited                           <----    Exits
 Container proxy  Starting                               <----   Starts
 Container shard-init  Exited
 Container docker-shard-4  Starting              <----
 Container proxy  Started
 Container docker-shard-4  Started
 Container docker-shard-1  Starting               <----
 Container docker-shard-1  Started
 Container docker-shard-3  Starting               <----
 Container docker-shard-3  Started
 Container docker-shard-2  Starting                <----
 Container docker-shard-2  Started
 Container prometheus  Waiting
 Container grafana  Waiting
 Container scylla  Waiting
 Container proxy  Waiting
 Container docker-shard-4  Waiting
 Container docker-shard-1  Waiting
 Container docker-shard-3  Waiting
 Container docker-shard-2  Waiting
 Container shard-init  Waiting
 Container docker-shard-4  Healthy
 Container docker-shard-1  Healthy
 Container docker-shard-3  Healthy
 Container docker-shard-2  Healthy
 Container proxy  Healthy
 Container shard-init  Exited
 Container scylla  Healthy
 Container grafana  Healthy
 Container prometheus  Healthy
```

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
* Add a blob size limit.

* Add a bytecode size limit.

* Add unit tests for limits.

* Don't enforce the limit for already published bytecode.

* Simplify LimitedWriter; add unit test.

* Add decompressed_size_at_most.

* Update and copy comment about #2710.
* Add `forget_chain` method to wallet CLI wrapper

Allow removing chains from the wallet in tests.

* Forget irrelevant newly created chains in test

The chains only serve to increase the Faucet chain's length in a way
that simulates the Faucet from being executed for a long time, so they
can be forgotten after they have been created.
Use its `Display` implementation when serializing to human-readable
formats.
* Don't cache blobs that are too large.

* Check size first, then decompress.
* updated project template, includes tests

* updating test case instantiation_argument

* suggested changes
* Shard editing command

* reduce version printing

* add unit test

* fix

* fixes after review

* remove write_json
## Motivation

This README was just showing how to build the Block Explorer code

## Proposal

Include also instructions on how to serve the web application and open it in your browser

## Test Plan

Went through the steps myself

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
* increase logging for subscriptions

* Update linera-rpc/src/grpc/client.rs

Co-authored-by: Janito Vaqueiro Ferreira Filho <[email protected]>
Signed-off-by: Mathieu Baudet <[email protected]>
## Motivation

There's a lot of time in our release process that is used building docker images, so we should do something about that to speed up our process.

## Proposal

Create a public registry with linera-io/linera-infra#97
And use a CI job to build and push docker images to this public registry

## Test Plan

Modified the CI workflow to trigger for this PR, changed it to publish to a test image. Made sure CI was green, and the docker image was visible and pullable from the public registry:


![Screenshot 2024-10-29 at 19.06.12.png](https://graphite-user-uploaded-assets-prod.s3.amazonaws.com/HlciHFAoHZW62zn13apJ/3ab377b6-ce75-44b3-82c1-63e6524a07e3.png)

Example successful run with the test code: https://github.com/linera-io/linera-protocol/actions/runs/11583023872/job/32247304661

Test code: 78c30d5

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.