Shiden node stopped syncing

[gflow33]

Description

Tell us what happened. In particular, tell us how and why you are using this project, and describe the bug that you encountered. Please note that we are not able to support all conceivable use cases, but the more information you are able to provide the more equipped we will be to help.

I run a shiden node as an rpc node and it stopped syncing and finalizing blocks

Steps to Reproduce

Replace the example steps below with actual steps to reproduce the bug you're reporting.

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Service-arguments used to run the node

./astar-collator --relay-chain-rpc-urls ws://<IP>:9933 ws://<IP>:9933 \
--node-key-file /home/polkadot/node-key \
--name "🛡 DWELLIR SHIDEN RPC 🛡" \
--chain shiden \
--pruning archive \
--pool-limit 65536 \
--rpc-max-connections 3000 \
--enable-evm-rpc \
--rpc-external \
--rpc-port 9933 \
--rpc-cors all \
--rpc-methods unsafe \
--prometheus-external \
--telemetry-url "wss://telemetry.polkadot.io/submit 0" \
--runtime-cache-size 8 \
--max-runtime-instances 32 \
-- \
--sync warp'

Expected vs. Actual Behavior

What did you expect to happen after you followed the steps you described in the last section? What actually happened?

The node is supposed to be syncing vs stopped syncing

Environment

Describe the environment in which you encountered this bug. Use the list below as a starting point and add additional information if you think it's relevant.

• Operating system: Ubuntu 24.04 LTS (Noble Numbat)
• Project version/tag: v5.45.0
• Rust version (run rustup show): Rust not installed

Logs, Errors or Screenshots

Please provide the text of any logs or errors that you experienced; if applicable, provide screenshots to help illustrate the
problem.

Jan 20 12:50:24 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:24 [Parachain] 💔 Error importing block 0x8ae714cbc7326fab079a1323a8018a5e3ccaa819da15f032f9576255bb3d6b27: consensus error: Import failed: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:25 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:25 [Parachain] Block import error: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:25 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:25 [Parachain] 💔 Error importing block 0x8ae714cbc7326fab079a1323a8018a5e3ccaa819da15f032f9576255bb3d6b27: consensus error: Import failed: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:25 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:25 [Parachain] 💤 Idle (1 peers), best: #8644461 (0x0aee…deef), finalized #8642092 (0x4526…22e9), ⬇ 775.6kiB/s ⬆ 0.9kiB/s
Jan 20 12:50:26 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:26 [Relaychain] Received finalized block via RPC: #26729074 (0x3910…dca8 -> 0xe75b…23c8)
Jan 20 12:50:27 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:27 [Parachain] Block import error: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:27 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:27 [Parachain] 💔 Error importing block 0x8ae714cbc7326fab079a1323a8018a5e3ccaa819da15f032f9576255bb3d6b27: consensus error: Import failed: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:30 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:30 [Parachain] ⚙️  Syncing  0.0 bps, target=#8697507 (3 peers), best: #8644461 (0x0aee…deef), finalized #8642092 (0x4526…22e9), ⬇ 1.1MiB/s ⬆ 0.8kiB/s
Jan 20 12:50:30 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:30 [Relaychain] Received finalized block via RPC: #26729075 (0xe75b…23c8 -> 0x2016…3af9)
Jan 20 12:50:30 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:30 [Relaychain] Received imported block via RPC: #26729079 (0xd4c9…2e3f -> 0xb219…703a)
Jan 20 12:50:31 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:31 [Parachain] Block import error: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:31 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:31 [Parachain] 💔 Error importing block 0x8ae714cbc7326fab079a1323a8018a5e3ccaa819da15f032f9576255bb3d6b27: consensus error: Import failed: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:34 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:34 [Relaychain] Received finalized block via RPC: #26729076 (0x2016…3af9 -> 0xe90a…0705)
Jan 20 12:50:35 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:35 [Parachain] 💤 Idle (1 peers), best: #8644461 (0x0aee…deef), finalized #8642092 (0x4526…22e9), ⬇ 685.9kiB/s ⬆ 0.9kiB/s
Jan 20 12:50:36 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:36 [Relaychain] Received imported block via RPC: #26729080 (0xb219…703a -> 0xeaf2…7fab)
Jan 20 12:50:40 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:40 [Parachain] 💤 Idle (1 peers), best: #8644461 (0x0aee…deef), finalized #8642092 (0x4526…22e9), ⬇ 0.9kiB/s ⬆ 0.8kiB/s
Jan 20 12:50:40 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:40 [Parachain] Block import error: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:40 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:40 [Parachain] 💔 Error importing block 0x8ae714cbc7326fab079a1323a8018a5e3ccaa819da15f032f9576255bb3d6b27: consensus error: Import failed: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:41 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:41 [Parachain] Block import error: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:41 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:41 [Parachain] 💔 Error importing block 0x8ae714cbc7326fab079a1323a8018a5e3ccaa819da15f032f9576255bb3d6b27: consensus error: Import failed: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:42 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:42 [Relaychain] Received imported block via RPC: #26729081 (0xeaf2…7fab -> 0x779f…aed5)
Jan 20 12:50:42 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:42 [Relaychain] Received finalized block via RPC: #26729077 (0xe90a…0705 -> 0xeb3f…d586)
Jan 20 12:50:44 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:44 [Parachain] Block import error: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error
Jan 20 12:50:44 juju-8f236b-0 polkadot[816]: 2025-01-20 12:50:44 [Parachain] 💔 Error importing block 0x8ae714cbc7326fab079a1323a8018a5e3ccaa819da15f032f9576255bb3d6b27: consensus error: Import failed: Database error: IO error: While pread offset 60553504 len 25591: /home/polkadot/.local/share/polkadot/chains/shiden/db/full/324753.sst: Input/output error

Additional Information

Please add any other details that you think may help us solve your problem.

[Dinonard]

From the logs, it seems your DB got corrupted.

Try to restore it from some snapshot, or try syncing the node from scratch.

[gflow33]

If I may ask what could be the cause of the db getting corrupted? And how to avoid it happening again.

[Dinonard]

Not sure how to avoid it since it can happen due to various reasons.

Issues with polkadot-sdk (e.g. a bug, some concurrency issue, etc.), might be intermittent issue on the particular setup the collator is being run on (related to hardware, OS, etc.).