org.rocksdb.RocksDBException: block checksum mismatch

Description

Our Rinkeby node stopped importing blocks because of the RocksDB errors below. A copy of the corrupt database is in ec2-18-221-72-217.us-east-2.compute.amazonaws.com:/data/broken-db

{{2019-08-31 10:49:43.020+00:00 | EthScheduler-Workers-0 | WARN | BlockPropagationManager | Failed to import announced block 5007328 (0x8dcd3b365e92ea6e1f979535409dd5aa1301f389aaf4ea95b6565a91ed3c570f).
2019-08-31 10:49:58.031+00:00 | nioEventLoopGroup-3-1 | INFO | BlockPropagationManager | Saving announced block 5007329 (0x745b1e0f565d4a147e6bda126def67c02f7af75eae0347db47196f22fa9131bd) for future import
2019-08-31 10:50:00.474+00:00 | nioEventLoopGroup-3-1 | INFO | SyncTargetManager | Found common ancestor with peer 0x15256f9aaee5eb7478... at block 5007327
2019-08-31 10:50:00.928+00:00 | EthScheduler-Services-96 (importBlock) | ERROR | PipelineChainDownloader | Chain download failed. Restarting after short delay.
java.util.concurrent.CompletionException: tech.pegasys.pantheon.services.kvstore.KeyValueStorage$StorageException: org.rocksdb.RocksDBException: block checksum mismatch: expected 1366259193, got 309731942 in /data/database/027588.sst offset 8262489 size 1655
at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?]
at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376) ~[?:?]
at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1019) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
at tech.pegasys.pantheon.services.pipeline.Pipeline.abort(Pipeline.java:150) ~[pantheon-pipeline-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:132) ~[pantheon-pipeline-1.2.2.jar:1.2.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: tech.pegasys.pantheon.services.kvstore.KeyValueStorage$StorageException: org.rocksdb.RocksDBException: block checksum mismatch: expected 1366259193, got 309731942 in /data/database/027588.sst offset 8262489 size 1655
at tech.pegasys.pantheon.services.kvstore.RocksDbKeyValueStorage$RocksDbTransaction.doCommit(RocksDbKeyValueStorage.java:189) ~[pantheon-kvstore-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.services.kvstore.KeyValueStorage$AbstractTransaction.commit(KeyValueStorage.java:108) ~[pantheon-kvstore-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.storage.keyvalue.WorldStateKeyValueStorage$Updater.commit(WorldStateKeyValueStorage.java:156) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.worldstate.DefaultMutableWorldState.persist(DefaultMutableWorldState.java:192) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.mainnet.MainnetBlockProcessor.processBlock(MainnetBlockProcessor.java:154) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.mainnet.MainnetBlockProcessor.processBlock(MainnetBlockProcessor.java:35) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:57) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:86) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.mainnet.MainnetBlockImporter.importBlock(MainnetBlockImporter.java:45) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.core.BlockImporter.importBlock(BlockImporter.java:42) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:43) ~[pantheon-eth-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:27) ~[pantheon-eth-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.services.pipeline.CompleterStage.run(CompleterStage.java:35) ~[pantheon-pipeline-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:128) ~[pantheon-pipeline-1.2.2.jar:1.2.2]
... 5 more
Caused by: org.rocksdb.RocksDBException: block checksum mismatch: expected 1366259193, got 309731942 in /data/database/027588.sst offset 8262489 size 1655
at org.rocksdb.Transaction.commit(Native Method) ~[rocksdbjni-5.15.10.jar:?]
at org.rocksdb.Transaction.commit(Transaction.java:206) ~[rocksdbjni-5.15.10.jar:?]
at tech.pegasys.pantheon.services.kvstore.RocksDbKeyValueStorage$RocksDbTransaction.doCommit(RocksDbKeyValueStorage.java:187) ~[pantheon-kvstore-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.services.kvstore.KeyValueStorage$AbstractTransaction.commit(KeyValueStorage.java:108) ~[pantheon-kvstore-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.storage.keyvalue.WorldStateKeyValueStorage$Updater.commit(WorldStateKeyValueStorage.java:156) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.worldstate.DefaultMutableWorldState.persist(DefaultMutableWorldState.java:192) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.mainnet.MainnetBlockProcessor.processBlock(MainnetBlockProcessor.java:154) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.mainnet.MainnetBlockProcessor.processBlock(MainnetBlockProcessor.java:35) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:57) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:86) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.mainnet.MainnetBlockImporter.importBlock(MainnetBlockImporter.java:45) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.core.BlockImporter.importBlock(BlockImporter.java:42) ~[pantheon-core-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:43) ~[pantheon-eth-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:27) ~[pantheon-eth-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.services.pipeline.CompleterStage.run(CompleterStage.java:35) ~[pantheon-pipeline-1.2.2.jar:1.2.2]
at tech.pegasys.pantheon.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:128) ~[pantheon-pipeline-1.2.2.jar:1.2.2]
... 5 more}}

Environment

None

Status

Assignee

Unassigned

Reporter

Adrian Sutton

Labels

None

Scrum Team

Chupacabra

Refinement State

Not Started

Fix versions

Priority

P3