merge from upstream

Signed-off-by: Anthony Petrov <[email protected]>

.github/workflows/config/node-release.yaml

@@ -1,11 +1,11 @@
 release:
   branching:
     execution:
-      time: "19:00:00"
+      time: "20:00:00"
     schedule:
-      - on: "2023-12-01"
-        name: release/0.45
+      - on: "2023-02-27"
+        name: release/0.48
         initial-tag:
-          create: true
+          create: false
           name: v0.45.0-alpha.2
 

hedera-node/docs/design/services/smart-contract-service/atomic-crypto-transfer.md

@@ -0,0 +1,76 @@
+# Atomic Crypto Transfer
+
+## Purpose
+
+[HIP-206](https://hips.hedera.com/hip/hip-206) defines functionality to perform atomic crypto transfers of value - hbar and tokens - between accounts using the IHederaTokenService interface. This document will define the architecture and implementation of this functionality.
+
+## Goals
+
+Describe the implementation of the atomic crypto transfer functionality in the Hedera Smart Contract Service.  
+
+## Non Goals
+
+- The implementation of the atomic crypto transfer functionality in the mono service codebase will not be discussed.
+- The implementation details of crypto `allowance` and `approvals` for hbar from within smart contracts is outside the scope of this document and will be described in a separate document.
+
+## Architecture
+
+The architecture for atomic crypto transfer follows the existing framework defined for handling all calls to the HederaTokenService system contract in the modularization services and is described in more detail in the `Implementation` section below.
+
+## Implementation
+Handling token related functionality is generally cooperatively performed by the smart contracts service module and the token service module.  The following classes are used to handle the call to the atomic crypto transfer function:
+
+### Smart Contract Service Module
+1.  The smart contract service implements a `CustomMessageCallProcessor` class which overrides the Besu `MessageCallProcessor` class in order to potentially intercept calls to system contract addresses.
+If the contract address of the current call is determined to be the contract address for the HederaTokenService (0x167), the call is redirected to the `HtsSystemContract` class for processing. 
+2.  The `HtsSystemContracts` creates an instance of the `HtsCallAttempt` class from the input bytes and the current message frame to encapsulate the call.
+3.  The `HtsCallAttempt` class iterates through a list of `Translator` classes (provided by Dagger) in order to determine which translator will be responsible for processing the call by attempting to match the call's 
+function signature to a signature known by the translator.  In the case of atomic crypto transfer calls, the `ClassicTransferTranslator` class will be responsible for processing the calls which has the following function signature: \
+```cryptoTransfer(((address,int64,bool)[]),(address,(address,int64,bool)[],(address,address,int64,bool)[])[])```
+4.  The `ClassicTransferTranslator` class will call the `ClassicTransferDecoder` class to decode the parameters of the call and translating the encoded parameter into a `TransactionBody` object.
+5.  A class called  `ClassicTransfersCall` will then take the created `TransactionBody` object and dispatches a new transaction to the Token Service Module for processing.  
+- Before dispatching the relevant feature flag `contracts.precompile.atomicCryptoTransfer.enabled=true` will be checked.
+- A `VerificationStrategy` class will be provided to the token service during dispatch in order to ensure that the security model is adhered to.
+- It is also responsible for other miscellaneous tasks such as checking for sufficient gas and encoding the response.
+
+![image info](./class_diagram.drawio.png)
+
+### Token Service Module
+
+Once the smart contract service dispatches the transaction to the Hedera Token Service the following steps are performed:
+
+1. Validate the semantic correctness of the transaction.
+2. Handle any aliases found and potentially create hollow accounts as necessary.
+3. Handle auto associations
+4. Add transfers to the transfer list
+5. Handle custom fees and add the resulting transfers to the transfer list
+6. Perform the transfers between accounts using the transfer list
+7. Create the resulting record and return to the caller. 
+
+The implementation can be found starting in the `CryptoTransferHandler` class.
+
+## Acceptance Tests
+
+As outlined above most of the implementation of the atomic crypto transfer functionality has already been completed.  What remains is to write acceptance tests
+to validate the functionality with a particular emphasis on security and edge cases and fixing issues as they arise. Below is a diagram that enumerates various transfer scenarios that need to be tested.
+
+![image info](./transfer_scenarios.drawio.png)
+
+
+### BDD Tests
+
+BDD tests are required to cover security concerns which require complex signing scenarios.  Many of these tests
+are already implemented and need not be repeated as XTests.
+
+#### Positive Tests
+- Successful transfer of hbars only and HTS tokens only between accounts from sender account via contract.
+- Successful transfer of hbars and HTS tokens with a custom fees (including fallback fee scenarios).
+- Successful transfer of hbars and HTS tokens with available auto token association slots on the receiver.
+- Successful transfer of hbars and HTS tokens from EOA account given approval to another EOA.
+- Successful transfer of hbars and HTS tokens from EOA account given approval to a caller contract.
+- Successful transfer of hbars and HTS tokens from owner contract via transfer contract.
+
+#### Negative Tests
+
+- Failure when attempting to transfer from special system accounts.
+- Failure when receiver signature is required and not provided.

hedera-node/hedera-app/src/xtest/java/README.md

@@ -0,0 +1,47 @@
+# XTests
+
+## Introduction
+XTests are intended to fill the gap between unit tests and integration tests. 
+They are designed to test the functionality of a services module in its entirety, as well as 
+interactions between module.
+
+## Goals and Benefits
+- Easier to write, read and understand, and to debug into than the HAPI suite BDDs because they just exercise the service (or services directly) through its API instead of being an end-to-end test where a HAPI transaction goes in and a record stream comes out, with all kinds of concurrency and asynchronicity happening in between, and the measurement of the effect being rather indirect.
+- At the same time making it easier to test edge cases because you can hand-craft test data using the actual data structures taken by the API, and can thus drive the system into states it can't normally get to by using only valid HAPI transactions.
+- A more complete test of the service-under-test than (our current) unit tests because mocking is extremely discouraged. So you're actually testing the real code instead of hard-wiring in to the test a bunch of assumptions of how the real code is supposed to work.
+- And yet they should have the flavor of a unit test: small, tightly focused to one specific issue and in fact one specific edge case (and thus compartmentalized from other tests testing in the same area)
+- xTests are ideally suited for testing and understanding inter module interactions such as dispatching transactions from smart contracts service to token service
+- xTests executes more quickly and thus allows the developer to iterate between test and functionality more efficiently than integration tests
+
+## When xTests Are Not Appropriate
+
+While xTests offer benefits as described above, they are not appropriate for all testing scenarios.
+Some specific scenarios where xTests are not appropriate include:
+
+- Any test that require multiple nodes to be running
+- Tests that require complex signing/signature verification
+- Any test where the existing scaffolding infrastructure needs to be significantly modified (although this will likely become more flexible over time)
+
+## Mocking
+
+xTests support mocking of objects just as in any unit test however in general mocking is discouraged unless the mocked object has no bearing on the functionality under test.  One objective to xTests is to understand potential side effects of tested functionality which may be hidden by excessive mocking. 
+
+## xTest Structure and Examples
+
+Just as during regular execution of nodes, xTests depend on dagger to provide the necessary dependencies such as state, context, fees and configuration.
+The classes that provide these basic scaffolding dependencies are 
+[BaseScaffoldingComponent](https://github.com/hashgraph/hedera-services/blob/develop/hedera-node/hedera-app/src/xtest/java/common/BaseScaffoldingComponent.java) and [BaseScaffoldingModule](https://github.com/hashgraph/hedera-services/blob/develop/hedera-node/hedera-app/src/xtest/java/common/BaseScaffoldingModule.java).
+
+There is a base class for all xTests called `AbstractXTest` which provide basic, common and useful functionality.  In particular the [scenarioPasses()](https://github.com/hashgraph/hedera-services/blob/develop/hedera-node/hedera-app/src/xtest/java/common/AbstractXTest.java#L119)
+method provides the structure for a test execution - from setup to scenario execution to assertions.
+
+Currently, xTests have been written for the smart contract service (primarily centered around system contracts) and the token service.
+All xTests for the smart contract service should descend from [AbstractContractXTest](https://github.com/hashgraph/hedera-services/blob/develop/hedera-node/hedera-app/src/xtest/java/contract/AbstractContractXTest.java)
+which provides common functionality such as calling smart contract handlers with synthetic transactions and calling system contract functions.
+Several prototypical examples to illustrate common xTest patterns are 
+[AssociationsXTest](https://github.com/hashgraph/hedera-services/blob/develop/hedera-node/hedera-app/src/xtest/java/contract/AssociationsXTest.java), 
+[ClassicViewsXTest](https://github.com/hashgraph/hedera-services/blob/develop/hedera-node/hedera-app/src/xtest/java/contract/ClassicViewsXTest.java)
+and [HtsErc20TransfersXTest](https://github.com/hashgraph/hedera-services/blob/develop/hedera-node/hedera-app/src/xtest/java/contract/HtsErc20TransfersXTest.java)
+
+Generally, xTests follow the pattern of setting up initial structures (accounts, tokens, aliases, tokenRels etc.), executing a scenario and then asserting the results. 
+The scenarios are defined by overriding the `doScenarioOperations()` method in the xTest class.  

hedera-node/test-clients/scripts/diff-testing/00-steps.md

@@ -0,0 +1,127 @@
+<!--
+# Copyright (C) 2024 Hedera Hashgraph, LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+-->
+
+## GCP setup
+
+* `gcloud components update`
+* `gcloud init` -- if needed
+  * pick zone 'us-west1-c` -- arbitrarily (?)
+* `gcloud auth application-default login` -- and sign in w/ account
+
+## Python setup
+* have `venv` available, install `3.12.0` (or something else recent)
+* `pip install -r requirements.txt`
+  * `pip install google-cloud-storage`
+* `open /Applications/Python\ 3.12/Install\ Certificates.command`
+
+## Get hostnames from `swirlds/infrastructure` repo
+* see `mainnet/ansible/host_vars_mainnet/node*.yml`
+* `grep source_hostname * | sed 's/[.]yml:source_hostname://g'`
+* use file `mainnet_hostnames-2024-02-04.txt`
+
+## To run:
+
+First you use the script to create a file that contains the names of all the 
+stream files in the interval that are up there in Google Cloud Storage.  
+(And this file contains other date, such as the size of the file, and 
+whether or not it has been downloaded already.)
+
+Then you use that file to download all the data files (which Google Cloud 
+Storage calls "blobs").
+
+Easy.  Except due to cloud and network problems the smooth processing of 
+calls to Google Cloud Storage may get interrupted from time to time.  So the 
+script allows you to _rerun_ both phases until they're finally complete 
+(keeping track, via that file created in the first phase, of stuff that's 
+already been done, so it isn't done again.)
+
+1. First create the list of blobs (record/event files) you want to download from gcp:
+   ```bash
+   python3 main.py get_blob_names -root <dir-for-files> \
+                         -b <bloblist-filename>         \
+                         -s <start-time> -e <end-time>  \
+                         -node <node#>
+   ```
+   where the start and end of the interval are specified like `2024-02-01T00:00:00`,
+   and the node number is the node you want to pull files for.
+
+   This will tell you how many files it found in that interval.
+
+   But it may happen the node you picked was down for some time during
+   that interval.  So run the script again using the command `reget_blob_names`
+   (instead of `get_blob_names`) and specify a different node number.  It
+   will _merge_ additional files found into the bloblist you already
+   have.  Repeat until it finds no new files.
+
+2. Download the blobs with the command
+   ```bash
+   python3 main.py download_blobs -root <dir-for-files> \
+                        -b <bloblist-filename> 
+   ```
+   It will fetch files in batches - and give you a progress report on
+   how many batches it is doing.
+
+   Files can fail to download.  Keep repeating this command until you see,
+   by the metrics reported, that all the files are downloaded.
+
+   You can "tune" the performance by changing the batch size with the
+   `-batch nnn` argument, and by changing the level of concurrency with the
+   `-concurrency nnn` argument.
+
+## More explanation
+
+- What are all these "blobs" in the names of things in the source code?  
+"Blob" is what Google Cloud Storage calls a "file".
+
+- All nodes put the streams out on GCP, even though they're _identical_ (except for the 
+  signatures, of course).  So you can pick any node number (1..29) to pull 
+from, your choice.  But sometimes nodes go down so the data is missing out 
+there (without any indication).  So this script lets you _rerun_ getting the 
+names of all the files, using a different node number ... you do that until 
+no new files are discovered.  Then, presumably, you have a full set of 
+stream file names.
+
+  But the node you pull from doesn't matter.  And so the default chosen by 
+the script, if you don't specify one, is arbitrary.
+
+- Speaking of nodes, nodes are always referred to by their _number_ (1..
+29/30).  And that's true whether the script calls it a "number" or an "id" or 
+whatever.
+
+- This script - and I still think of it as a script - has grown a bit.  It
+might be a bit larger than a "script" now ... but there it is.  And so, even
+though it is not good software engineering to use _global variables_ this 
+script does use global variables.  And since those global variables are 
+referred to everywhere, even though it is not good software engineering to 
+use _single character variable names **especially** for global variables_ 
+this script _does_ use single character variable names to refer to global 
+variables.  Because I wanted the minimum syntax overhead to specify the 
+context.  Because it's necessary to have the context for readability, but 
+too much of it is just visual cruft.
+
+  - `a` is the global holder of command line arguments.  All commands line 
+arguments are in there, with default values for those that weren't actually 
+on the command line.  It is of the class `Args`, and both are declared in 
+`cli_arguments.py` and imported elsewhere.
+  - `g` is the global holder of global variables.  It is of the class 
+`Globals`, and both are declared in `globals.py` and imported elsewhere.
+
+  And unfortunately importing a variable doesn't work the way I expected in 
+Python.  I'm not sure exactly what happens, but I think a copy is made. 
+Shallow copy probably.  Anyway, it hasn't impacted things so far but it's 
+something to be aware of and I'll probably have to fix it if the script has 
+further developments.
+

hedera-node/test-clients/scripts/diff-testing/bloblist.py

@@ -0,0 +1,89 @@
+# Copyright (C) 2024 Hedera Hashgraph, LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import annotations
+
+import copy
+from dataclasses import dataclass, replace as clone
+from datetime import datetime
+from typing import List, Set, Optional
+
+import jsons
+
+from utils import *
+from versioned_output_file import VersionedOutputFile
+
+
+@dataclass
+class Blob:
+    name: str
+    type: str
+    have: bool
+    node: int
+    crc32c: str  # base64, big-endian
+    md5hash: str  # base64, big-endian (presumably)
+    size: int
+
+    def __copy__(self):
+        return clone(self)
+
+    def __hash__(self):
+        """For these (mutable) objects, the 'primary key" is the type+name"""
+        return hash((self.name, self.type))
+
+
+@dataclass
+class BlobList:
+    interval_start_time: datetime
+    interval_end_time: datetime
+    nodes: Set[int]
+    blobs: List[Blob]
+
+    def __len__(self):
+        return len(self.blobs)
+
+    def __copy__(self):
+        return clone(self)
+
+    def __deepcopy__(self, memo):
+        bl = clone(self)
+        bl.nodes = bl.nodes.copy()
+        bl.blobs = [copy.copy(b) for b in bl.blobs]
+        return bl
+
+    @staticmethod
+    def save(path: str, bloblist: BlobList) -> None:
+        contents = jsons.dumps(bloblist, ensure_ascii=False, jdkwargs={"indent": 4})
+        with VersionedOutputFile(path) as f:
+            f.write(contents)
+
+    @staticmethod
+    def load(filename: str) -> Optional[BlobList]:
+        try:
+            with open(filename, 'r', encoding='utf-8') as file:
+                contents = file.read()
+        except IOError as ex:
+            print(f"*** BlobList.load({filename}): IOError: {ex}")
+            contents = None
+        bloblist = jsons.loads(contents, BlobList) if contents is not None else None
+
+        # Our use of datetime.fromisoformat needs naive datetimes (without timezone) that json package puts there
+        bloblist.interval_start_time = naiveify(bloblist.interval_start_time)
+        bloblist.interval_end_time = naiveify(bloblist.interval_end_time)
+        return bloblist
+
+
+# For various reasons related to formatting blob names we use naive datetime - without timezones - so we must tell
+# jsons library not to complain about that
+jsons.suppress_warning('datetime-without-tz')