[Historical testnet] Fresh v0.1.0 nodes cannot sync: HistoryStep matrix whitelist slot 0 drift #5

Closed
opened 2026-08-12 20:49:44 +00:00 by Arttemkaaa · 2 comments
Arttemkaaa commented 2026-08-12 20:49:44 +00:00 (Migrated from github.com)

Fresh v0.1.0 mainnet nodes are unable to sync from height 0.

The issue reproduces on multiple independent x86_64 machines.

P2P connectivity works correctly. The nodes connect to multiple peers and receive state manifests from peers with current chain tips, but the manifests are rejected during HistoryStep verification.

Example error:

BadHistoryStepTerminal(
"HistoryStep terminal rejected: HistoryStep bank:
HistoryStep matrix whitelist slot 0 drift"
)

The node then reports:

state manifest round produced no usable candidate — re-requesting

I have also observed:

HistoryStep proof: Zerocheck(SumcheckFinalFailed)

One affected machine:

  • OS: Ubuntu 24.04 x86_64
  • CPU: AMD EPYC 9654
  • 192 threads
  • Backend: avx512bw+vpclmul
  • parano1d: v0.1.0
  • Hardware check: NODE READY

The same sync failure also reproduces on another x86_64 machine with 32 threads.

On the EPYC node, peers are discovered successfully and manifests are received, for example:

received state manifest ... tip=133 segments=1

but then:

ignoring manifest from a peer that supplied an invalid recursive terminal

The node remains at:

Height 0

while connected peers advertise chain heights well above 100.

Steps to reproduce:

  1. Download the official v0.1.0 Linux x86_64 release.
  2. Run:

./parano1d --check-hardware

Result:

BACKEND avx512bw+vpclmul
NODE READY

  1. Start a fresh mainnet node:

./parano1d

  1. Wait for P2P bootstrap.

Peers connect successfully, but the node remains at height 0 and cannot accept a state snapshot.

Running with:

./parano1d --log debug

shows the HistoryStep terminal verification failure described above.

This appears reproducible across different hardware, so it does not seem specific to the EPYC system.

Fresh v0.1.0 mainnet nodes are unable to sync from height 0. The issue reproduces on multiple independent x86_64 machines. P2P connectivity works correctly. The nodes connect to multiple peers and receive state manifests from peers with current chain tips, but the manifests are rejected during HistoryStep verification. Example error: BadHistoryStepTerminal( "HistoryStep terminal rejected: HistoryStep bank: HistoryStep matrix whitelist slot 0 drift" ) The node then reports: state manifest round produced no usable candidate — re-requesting I have also observed: HistoryStep proof: Zerocheck(SumcheckFinalFailed) One affected machine: - OS: Ubuntu 24.04 x86_64 - CPU: AMD EPYC 9654 - 192 threads - Backend: avx512bw+vpclmul - parano1d: v0.1.0 - Hardware check: NODE READY The same sync failure also reproduces on another x86_64 machine with 32 threads. On the EPYC node, peers are discovered successfully and manifests are received, for example: received state manifest ... tip=133 segments=1 but then: ignoring manifest from a peer that supplied an invalid recursive terminal The node remains at: Height 0 while connected peers advertise chain heights well above 100. Steps to reproduce: 1. Download the official v0.1.0 Linux x86_64 release. 2. Run: ./parano1d --check-hardware Result: BACKEND avx512bw+vpclmul NODE READY 3. Start a fresh mainnet node: ./parano1d 4. Wait for P2P bootstrap. Peers connect successfully, but the node remains at height 0 and cannot accept a state snapshot. Running with: ./parano1d --log debug shows the HistoryStep terminal verification failure described above. This appears reproducible across different hardware, so it does not seem specific to the EPYC system.
ignotusnemo commented 2026-08-12 22:30:31 +00:00 (Migrated from github.com)

Confirmed. Thank you for the detailed report and logs.

This is not hardware-specific and is not caused by the AVX-512 backend. The public v0.1.0 binaries were built by CI with the pre-mainnet test HistoryStep pack, while the final mainnet pack was used in the local build tested before release. That mismatch causes both errors reported here:

  • HistoryStep matrix whitelist slot 0 drift
  • HistoryStep proof: Zerocheck(SumcheckFinalFailed)

The source code, genesis and mainnet parameters are correct. The published v0.1.0 binaries are not and must not be used.

I have hidden v0.1.0 and stopped all seed nodes. v0.1.1 is currently being built for all supported platforms with the final mainnet HistoryStep pack.

Once v0.1.1 is published, affected users must close v0.1.0, delete the complete Parano1d data directory, install v0.1.1 and start again from a clean directory:

  • Linux: ~/.parano1d/data
  • macOS: /Users/YourName/.parano1d/data
  • Windows: %USERPROFILE%\.parano1d\data

I will keep this issue open until v0.1.1 is published, all seed nodes are restarted, and clean public synchronization is verified.

Confirmed. Thank you for the detailed report and logs. This is not hardware-specific and is not caused by the AVX-512 backend. The public v0.1.0 binaries were built by CI with the pre-mainnet test HistoryStep pack, while the final mainnet pack was used in the local build tested before release. That mismatch causes both errors reported here: - `HistoryStep matrix whitelist slot 0 drift` - `HistoryStep proof: Zerocheck(SumcheckFinalFailed)` The source code, genesis and mainnet parameters are correct. The published v0.1.0 binaries are not and must not be used. I have hidden v0.1.0 and stopped all seed nodes. v0.1.1 is currently being built for all supported platforms with the final mainnet HistoryStep pack. Once v0.1.1 is published, affected users must close v0.1.0, delete the complete Parano1d data directory, install v0.1.1 and start again from a clean directory: - Linux: `~/.parano1d/data` - macOS: `/Users/YourName/.parano1d/data` - Windows: `%USERPROFILE%\.parano1d\data` I will keep this issue open until v0.1.1 is published, all seed nodes are restarted, and clean public synchronization is verified.
ignotusnemo commented 2026-08-16 00:18:54 +00:00 (Migrated from github.com)

Historical testnet issue.

This report concerns the retired pre-mainnet network and its early v0.1.0/v2.x binaries. It does not apply to the current Parano1d mainnet, which launched from a new genesis on August 21, 2026.

The original failure was caused by incompatible HistoryStep packs in early CI and local builds. Current mainnet binaries use the canonical authenticated pack and bind its identity before P2P synchronization, preventing incompatible packs from joining the same network.

The networking redesign is documented in PR #6.

Current mainnet release: https://github.com/ignotusnemo/parano1d/releases/latest

Historical testnet issue. This report concerns the retired pre-mainnet network and its early v0.1.0/v2.x binaries. It does not apply to the current Parano1d mainnet, which launched from a new genesis on August 21, 2026. The original failure was caused by incompatible HistoryStep packs in early CI and local builds. Current mainnet binaries use the canonical authenticated pack and bind its identity before P2P synchronization, preventing incompatible packs from joining the same network. The networking redesign is documented in [PR #6](https://github.com/ignotusnemo/parano1d/pull/6). Current mainnet release: [https://github.com/ignotusnemo/parano1d/releases/latest](https://github.com/ignotusnemo/parano1d/releases/latest)
Sign in to join this conversation.
No description provided.