Open issues

RocksDB performance problems
PAN-3245
Release - Documentation
PAN-1745
Title within Configure High Availability of APIs documentation tutorial has a spelling mistake
PAN-3276
EVM Evolution EIP (EIP-2348)
PAN-3274
Modify Besu to accept TLS connections on ws JSON RPC interface
PAN-3271
Modify Besu to accept TLS connections on graphql interface
PAN-3269
Bootnode did not report all connected Peers
PAN-3260
metrics-category details incorrect
PAN-3250
Low gas limit causes root mismatch with private transactions
PAN-3243
Default database location for homebrew is dangerous
PAN-3236
Remove build from source docs from user doc
PAN-3225
Support web3js contract API
PAN-3224
Add GraphQL support for logs
PAN-3223
Besu behaviour when Orion not available to process a private transaction
PAN-3210
Migration to new private state storage
PAN-3200
Detailed design for onchain privacy group management
PAN-3199
Doc that you can't use throwaway keys with account permsissioning
PAN-3196
Segment private state storage
PAN-3190
Besu Wiki Article Explaining Release Philosophy
PAN-3176
Include privateFrom in RPC parameters rather than inject from CLI key
PAN-3167
Make the Getting Started documentation more accessible
PAN-3164
Write acceptance tests for db migration
PAN-3162
Measure Pantheon TPS For Token Transfers
PAN-3132
refactor privacynode.getTransactionSigningKey
PAN-3113
Add a setPrivacyGroupState endpoint on Pantheon
PAN-3106
Doc deployment docker image
PAN-3101
Test failure on Master Build for p2p
PAN-3099
Privacy mainnet
PAN-3096
Rework "convertEnclaveInvalidReason" used in EeaSendRawTransaction
PAN-3086
Doc how to install and build plugins
PAN-3078
Doc pruning
PAN-3049
1.3 Monitoring
PAN-3041
1.3 Doc retesteth
PAN-3036
Doc using Pantheon Docker image on Windows
PAN-3034
Add priv_getCode
PAN-3029
1.2 Tech Debt
PAN-3018
Move Privacy solution away from Base-64 Strings
PAN-3014
Add priv_call
PAN-3008
Fast Sync Meta Epic
PAN-3002
Monitor Stratum Bounty
PAN-2993
Modify priv_getTransactionReceipt result fields
PAN-2984
Create docs repos for each project
PAN-2939
Reorganise docs navigation
PAN-2938
Prevent PrivacyGroup data from being modified by a non-member
PAN-2913
Sign jars
PAN-2895
Permissioning doc 1.2
PAN-2880
Basic RBAC for Permissioning to support the management of nodes and accounts
PAN-2862
1.3 Doc privacy updates
PAN-2858
Limit what types of transactions and what each account can do
PAN-2834
Permissioning check to only connect to nodes running a specified client/version
PAN-2833
issue 1 of 121

RocksDB performance problems

Description

We have been experiencing performance issues since pantheon 1.1.3 due the rocksdb read performance.

Our context is the following:

  • We use Eventeum as the event listener platform for BESU. We are core committers of the project.

  • Eventeum internally stores a kind of index, to store which was the latestblock read, to get all the events happening on the chain

  • When eventeum stops for a while, either by a planned maintenance, configuration change, or any read problem at the node level, being the node able to get new blocks from the chain, but not able to serve any query to eventeum, eventeum is not synced with the head block of the chain

  • When eventeum starts it creates logs per any event filte via eth_newFilter, syncs the pending information via eth_getFilterLogs, and then it uses eth_getFilterChanges to get updates when on sync

We are seeing that when for whatever reason, the amount of unsynced blocks si higher than 10000, the performance drops, 100k of blocks, makes BESU hanf at query level locking rocks db with traces like "Thread Thread[vert.x-worker-thread-18,5,main] has been blocked for 76419 ms, time limit is 60000 ms"

Based on this scenario we have done some performance checks compared to goquorum

  • 10k blocks: BESU = 5 secs, QUORUM = 0,9 secs

  • 50k blocks: BEDU = 10 secs , Quorum = 4 sec
    -100k blocks: BESU = 20 sec, Quorum = 10secs

> 100k BESU rocksdb threads blocks, quorum takes some time but answers

Its strange based on a comparison between leveldb and rocksdb.

With this scenario, we really need and urgent solution on it, to boost the performance.

We have several things, menawhile , on mind:

  • Monitor rocks db with the following metrics : "Latency for read from RocksDB.", "Latency of remove requests from RocksDB.", "Latency for write to RocksDB.", "Latency for commits to RocksDB."), raising alerts when it is underperforming. The problem is that we cannot see that metrics on the metrics endpoint neither at 1.2.2 or 1.3.1. Can you review the following bug? Any suggestion to monitor rocksdb

  • At eventeum level, split the unsynced blocks, in a number of chunks based on a kinf of window. any suggestion on the mas window to include?

Kind regards

Environment

None

Status

Assignee

Danno Ferrin

Reporter

Fernando Paris

Labels

None

Scrum Team

Chupacabra

Refinement State

Not Started

Components

Sprint

Fix versions

Priority

P2
Configure