[Node] Correct DA stream timeout error.

[musitdev]

Summary

• RFCs: Link to RFC, Link to RFC, or $\emptyset$.
• Categories: any of protocol-units, networks, scripts, util, cicd, or misc.
  Add a Heartbeat on the DA stream from the leader to the follower.

Changelog

• Integrate DA changes from PR Fixes Issues w/ Stream Crashing Over Large Replay #996.
• Add heartbeat on the the DA stream.

Testing

To test run the command:

CELESTIA_LOG_LEVEL=FATAL nix develop --extra-experimental-features nix-command --extra-experimental-features flakes --command bash  -c "just movement-full-node native build.setup.eth-local.celestia-local.test-followers --keep-tui"

You can update the test-followers overlay so that the test never end and you'll see on each follower the heartbeat notification.

Outstanding issues

[0xmovses]

On movement-full-follower-1

I get the error:

service_type="table_info_service" step="2"
2025-02-12T17:59:50.428023Z  INFO movement_full_node::node::tasks::execute_settle: Receive heartbeat blob
2025-02-12T17:59:51.743095Z  INFO movement_full_node::node::manager: Receive Terminate Signal
thread 'tokio-runtime-worker' panicked at /Users/movses/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/blocking/shutdown.rs:51:21:
Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And the same for the other follower nodes and movement-full-node

[musitdev]

It's normal when the test end process compose send a SYGKILL signal that stop the node abruptly. So depending on how the Tokio runtime is, you can get this error. If you remove the exit_on_end: true in the follower process compose script at the end of the test, the node never stop and you get no error.

[0xmovses]

Got it, that's fairly misleading for anyone running the test. Let's patch that later with a nice tracing:info message. Otherwise. this looks fine