Nonce too low error when trying to deploy safe singleton factory

[mshakeg]

Description

I'm trying to deploy the safe-global/safe-singleton-factory contract as described here in order to then deterministically deploy a canonical version of safe-global/safe-contracts

The safe-global/safe-singleton-factory is only needed if Hedera implements EIP-155, not sure if this is the case, but if not the Arachnid/deterministic-deployment-proxy can be used instead.

In any case it should be possible to deploy the canonical version of this contract, however attempting to deploy it fails as previously described in safe-global/safe-singleton-factory/issues/206

Steps to reproduce

1. Construct a curl for the transaction for either the Hedera testnet or mainnet using the transaction data.

curl -X POST --data '{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["0xf8a88086019c1c38a40083088b808080b853604580600e600039806000f350fe7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe03601600081602082378035828234f58015156039578182fd5b8082525050506014600cf3820271a0b0291bc53df97aceabf620b5e1dd22e97dfd26e5a4df68acdf3a91d8738eec4ba0193cf7b2c1246e376ac3af37183c8cc31a2ef96f41a60216442260efac2dba7c"],"id":1}' -H "Content-Type: application/json" https://mainnet.hashio.io/api

2. Submit the transaction via the hashio relay
3. Observe a response like:

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": 32001,
    "name": "Nonce too low",
    "message": "[Request ID: 673c81b3-fe0b-4584-94df-3500d06214a8] Nonce too low. Provided nonce: 0, current nonce: 0"
  }
}

4. On hashscan observe that the transaction signer/deploy i.e. 0xE1CB04A0fA36DdD16a06ea828007E35e1a3cBC37 has not actually made any transactions and so the transaction should not revert with a nonce error especially a non-sensical one such as Provided nonce: 0, current nonce: 0

Additional context

No response

Hedera network

mainnet

Version

relay/0.32.0

Operating system

None

[mshakeg]

Seems related to #1791

[Nana-EC]

Hey @mshakeg ,

We'll take a look. I think it is related as you noted, we'll get to the bottom of this

[konstantinabl]

@mshakeg Hi can you provide the transaction data which you used to construct the transaction, so I can replicate your actions?

[mshakeg]

Hey @konstantinabl the transaction data for the testnet and mainnet transactions are in the linked json files in step 1. As you can see it's the same data used in the curl at least for mainnet. It's been a while since the transactions was signed and the gas price has changed since then so trying to execute that transaction will result in:

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32009,
    "name": "Gas price too low",
    "message": "[Request ID: 4c4b7b64-094c-4c26-8267-18171fa3ca75] Gas price '1770000000000' is below configured minimum gas price '1870000000000'"
  }
}

[konstantinabl]

@mshakeg So, can they sign it again in order for us to be able to reproduce the issue?

[mshakeg]

@konstantinabl I asked, though in the meantime I signed a testnet transaction using an account I control:

{
	"gasPrice": 2200000000000,
	"gasLimit": 100000,
	"signerAddress": "0x13D65d7fA66A2970eE8862ba8633D064B43Bf091",
	"transaction": "0xf8a8808602003a37f000830186a08080b853604580600e600039806000f350fe7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe03601600081602082378035828234f58015156039578182fd5b8082525050506014600cf3820273a02c2dc95853f2d9809a846eb31ce58e2061c20f3699ebbf1f641c4e2a818f6940a078feaec02b382bb4e0b610dba69f8643a6f6b1f1dec60e0d7ff079b568438472",
	"address": "0xAfb3D5C0cd6a610F87365ce1BF8Eb6A0AA985988"
}

Interestingly my transaction seems to have succeeded without this nonce issue and the contract deployed to the expected address

curl -X POST --data '{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["0xf8a8808602003a37f000830186a08080b853604580600e600039806000f350fe7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe03601600081602082378035828234f58015156039578182fd5b8082525050506014600cf3820273a02c2dc95853f2d9809a846eb31ce58e2061c20f3699ebbf1f641c4e2a818f6940a078feaec02b382bb4e0b610dba69f8643a6f6b1f1dec60e0d7ff079b568438472"],"id":1}' -H "Content-Type: application/json" https://testnet.hashio.io/api

And if you attempt to replay my above transaction then it correctly fails with a nonce issue:

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": 32001,
    "name": "Nonce too low",
    "message": "[Request ID: 33547925-df5a-42d3-af06-b43309c3b78b] Nonce too low. Provided nonce: 0, current nonce: 1"
  }
}

[konstantinabl]

@mshakeg Sorry I got confused, can you explicitly tell me which transaction succeeded and which one failed?
Did the one you signed on your own succeed and then when you replayed it failed?

[mshakeg]

@konstantinabl so the transaction I signed just an hour ago to deploy a non-canonical safe-singleton-factory contract succeeded, however the transaction signed by the safe deployer account to deploy the canonical safe-singleton-factory now fails but due to a gas price issue, since my transaction succeeded it is entirely possible that if the gas price wasn't an issue that the canonical safe transaction would be processed without failing with "Nonce too low error" which was the case when I originally created this issue, this would suggest that the issue is intermittent.

[mshakeg]

It would seem that Hedera implements EIP-155 as attempting to deploy the Arachnid/deterministic-deployment-proxy contract results in the following error:

curl -X POST --data '{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["0xf8a58085174876e800830186a08080b853604580600e600039806000f350fe7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe03601600081602082378035828234f58015156039578182fd5b8082525050506014600cf31ba02222222222222222222222222222222222222222222222222222222222222222a02222222222222222222222222222222222222222222222222222222222222222"],"id":1}' -H "Content-Type: application/json" https://testnet.hashio.io/api

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32000,
    "name": "ChainId not supported",
    "message": "[Request ID: cac363ab-1905-4231-a10e-eadf5ec7c95c] ChainId (0x0) not supported. The correct chainId is 0x128"
  }
}

This makes the safe-global/safe-singleton-factory contract a requirement for a canonical deployment of safe-contracts.

[konstantinabl]

@mshakeg Thank you for your thorough explanation :) So the issue with the nonce is resolved and we can close this?

[mshakeg]

@konstantinabl it's still an issue as confirmed by one of the Safe maintainers. They won't share the new transaction data with a higher gas price, you could try convince them to share it if it's blocking you from debugging, but I don't think it is as you can just disable the gas price check when debugging.

[mshakeg]

@konstantinabl have you managed to make progress on this issue?

[konstantinabl]

@mshakeg Hi, I managed to reproduce the issue and I am investigating it :) Will get back to you

[konstantinabl]

@mshakeg The issue is in our services when using a lazy-created account, such as the one used by the person from safe.
hashgraph/hedera-services#9310 You can follow the issue here

[mshakeg]

@konstantinabl thanks for the investigation. What gas price are you using for the transaction? IMU if you use a gas price that is lower than the latest gas price returned by eth_gasPrice then that'll result in the relay returning an error response like the one mentioned here, and not original the "Nonce too low" error?

If we have to specify a higher gas price than the one returned by eth_gasPrice for the transaction to be processed successfully then that would indicate an issue with the eth_gasPrice method implementation.

[konstantinabl]

Hi @mshakeg sorry that it took me so long to answer.
Regarding the gasPrice, you can use the value returned from the eth_gasPrice to sign a transaction and it should be successful.
Otherwise this issue is supposed to be fixed in the next release of services, version 0.43
I will close this now but if you have any problems after the release, feel free to reopne

[mshakeg]

Hi @stoqnkpL @Nana-EC I'm still getting this issue. Thought it would've been resolved by now.

{
    "error": {
        "code": 32001,
        "name": "Nonce too low",
        "message": "[Request ID: eaebfea4-6089-4d79-bdac-5f9d02900a8b] Nonce too low. Provided nonce: 76, current nonce: 76"
    },
    "jsonrpc": "2.0",
    "id": 56
}

[Nana-EC]

Hi @stoqnkpL @Nana-EC I'm still getting this issue. Thought it would've been resolved by now.

{

    "error": {

        "code": 32001,

        "name": "Nonce too low",

        "message": "[Request ID: eaebfea4-6089-4d79-bdac-5f9d02900a8b] Nonce too low. Provided nonce: 76, current nonce: 76"

    },

    "jsonrpc": "2.0",

    "id": 56

}

@mshakeg the deployment of all the related fixes may not have gone out yet.
We needed some updates on mirror node and 0.47 services.
We planned to reach out when everything was in the all clear.
Please allow for the release cycles to complete.

FYI @georgi-l95 and @stoqnkpL

[mshakeg]

@Nana-EC thanks, btw don't you think the relay should not forward txs to consensus nodes if the gas price for the tx is below the latest gas price? I don't want to have to pay for txs that failed due to the gas price being too low, but even if the relay had such logic that probably wouldn't help as at least for me the gas price that I'm specifying for the tx is the value returned by the relay which indicates that the relay doesn't have/calculate the latest gas price correctly and so the relay would submit the tx(with a too low gas price) in any case.

[Nana-EC]

@Nana-EC thanks, btw don't you think the relay should not forward txs to consensus nodes if the gas price for the tx is below the latest gas price? I don't want to have to pay for txs that failed due to the gas price being too low, but even if the relay had such logic that probably wouldn't help as at least for me the gas price that I'm specifying for the tx is the value returned by the relay which indicates that the relay doesn't have/calculate the latest gas price correctly and so the relay would submit the tx(with a too low gas price) in any case.

That's a good suggestion.
Can you open a separate ticket with details for us to triage.
Gas price should not change often and usually only by a few tiny bar when the exchange rate changes.

[mshakeg]

Hi @Nana-EC I'm still getting this error despite the testnet upgrade to v0.47.1 a week ago.

[Nana-EC]

Hi @Nana-EC I'm still getting this error despite the testnet upgrade to v0.47.1 a week ago.

Thanks @mshakeg can you share details around your issue.
RequestID or transaction submitted to the relay.
@georgi-l95 and I would love to peer deeper into it as it should self heal with with 0.47.1 and Mirror Node 0.99.0 in testnet

[mshakeg]

@Nana-EC here is the tx submitted to the relay:

curl -X POST https://testnet.hashio.io/api \
     -H "Content-Type: application/json; charset=utf-8" \
     -d '{"method":"eth_sendRawTransaction","params":["0x02f902b48201284c85a0a734740085a0a73474008307bef3940b10e483aac4340256772754d23131b6e0dc31ea80b902441749e1e30000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000120000000000000000000000000897bab966eb0619ad1513f87c3fbfa43b5478781000000000000000000000000000000000000000000000000000000000003b7ff000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000444585e33b000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000066b124805d63ff0d8059f4e80e1deb459b754f6700000000000000000000000000000000000000000000000000000000000343a4000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000444585e33b0000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c080a0b3807be8f9655e1d37d91c23da4859a90ee839bbb76c7040a4f43af88bf3466c9f8bd867b33ea68b7db6b9b1df43c2166bf56c773c18a620cbd3b541a47eddb3"],"id":55,"jsonrpc":"2.0"}'

Here is the response:

{
    "error": {
        "code": 32001,
        "name": "Nonce too low",
        "message": "[Request ID: 480e89d6-1dc3-4479-bcfd-dddffcd22e83] Nonce too low. Provided nonce: 76, current nonce: 76"
    },
    "jsonrpc": "2.0",
    "id": 55
}

[natanasow]

@mshakeg Your transaction falls into an edge case which is fixed in v0.48. As a workaround, you can manually set a transaction's nonce to 77 and submit it to the relay again.

[Nana-EC]

@mshakeg did the above unblock you?
There's an issue around self healing which we hope to have resolved in all cases in 0.48.
However, in 0.47 the issue shouldn't occur for accounts with consensus and mirror node aligned nonces.

[mshakeg]

@Nana-EC it did, will the safe team then have to action something? This isn't urgent at all as I've deployed my own instance of safe.

[Nana-EC]

Awesome, great news.
Yes, the Safe team would probably just need to take some steps since they require their nonce to be 0 at deployment, but they should wait until after 0.48 is deployed.
They will need to

1. Delete their account using CryptoDelete
2. Create a new account with the same evm address using CryptoCreate
3. Submit the transaction to deploy the

@georgi-l95 can help out with a new ticket for us to track and a script that anyone can run in this 0 nonce unique scenario

[georgi-l95]

Hey @mshakeg, you can try using this short workflow. Just npm i and npm run deploy, should execute the steps that Nana described.
This will be executed against local-node, but can be easily modified to do it against other networks. Ping me, if I can help with anything
https://github.com/georgi-l95/hedera-approve-example/tree/alias-deletion-test

[mshakeg]

Hey @georgi-l95 great, the safe team will have to action this. What should they set the HEDERA_NETWORK var to? Seems like the logic that uses this env var is commented out?

[georgi-l95]

@mshakeg Okay, I've reworked the script. You can follow the .env.example to make .env. For operator id and key you can pass ecdsa account and key. This should delete and release the alias of this account, so this alias can be used again.

[Nana-EC]

Hey @georgi-l95 great, the safe team will have to action this. What should they set the HEDERA_NETWORK var to? Seems like the logic that uses this env var is commented out?

Hey @mshakeg not sure if you missed the part where I said we should probably wait till 0.48 is deployed (next month) - conservative guess but we're working to reduce the time.

Can you ask them to hold off on running it so we coordinate properly.
I've update the issue to ask them to hold off whiles we coordinate properly
I wanted to avoid the team having to go back and forth and giving up again.

My understanding is they probably also don't have lots of context on Hedera details.