ReadableKVStateBase.readCache is not properly synchronized

[OlegMazurov]

Description

The following exception is thrown by a node when under heavy load:

2023-11-27 20:33:31.143 ERROR 133  PreHandleWorkflowImpl - Unexpected error while pre handling a transaction!
java.lang.ClassCastException: class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
        at java.base/java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1994) ~[?:?]
        at java.base/java.util.HashMap$TreeNode.treeify(HashMap.java:2110) ~[?:?]
        at java.base/java.util.HashMap.treeifyBin(HashMap.java:778) ~[?:?]
        at java.base/java.util.HashMap.putVal(HashMap.java:650) ~[?:?]
        at java.base/java.util.HashMap.put(HashMap.java:618) ~[?:?]
        at com.hedera.node.app.spi.state.ReadableKVStateBase.markRead(ReadableKVStateBase.java:124) ~[app-spi-0.45.0-SNAPSHOT.jar:?]
        at com.hedera.node.app.spi.state.ReadableKVStateBase.get(ReadableKVStateBase.java:70) ~[app-spi-0.45.0-SNAPSHOT.jar:?]
        at com.hedera.node.app.service.token.impl.ReadableAccountStoreImpl.getAccountLeaf(ReadableAccountStoreImpl.java:132) ~[app-service-token-impl-0.45.0-SNAPSHOT.jar:?]
        at com.hedera.node.app.service.token.impl.ReadableAccountStoreImpl.getAccountById(ReadableAccountStoreImpl.java:104) ~[app-service-token-impl-0.45.0-SNAPSHOT.jar:?]
        at com.hedera.node.app.workflows.prehandle.PreHandleWorkflowImpl.preHandleTransaction(PreHandleWorkflowImpl.java:174) ~[HederaNode.jar:?]
        at com.hedera.node.app.workflows.prehandle.PreHandleWorkflowImpl.lambda$preHandle$0(PreHandleWorkflowImpl.java:128) ~[HederaNode.jar:?]
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) [?:?]
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) [?:?]
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) [?:?]
        at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) [?:?]
        at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754) [?:?]
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) [?:?]
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312) [?:?]
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843) [?:?]
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808) [?:?]
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188) [?:?]

The reason is unsynchronized calls to markRead() -> readCache.put() from multiple threads.

Steps to reproduce

The exception was observed when running the NftTransferLoadTest benchmark (during the token creation phase).

Additional context

No response

Hedera network

other

Version

develop

Operating system

Linux

[OlegMazurov]

Functionality is fixed by #10965 with a heavy price on performance: ConcurrentHashMap should have been used instead of Collections.synchronizedMap(), which introduced a heavily contended lock.