Can we archive states for the last N months only?

[jongrun]

In current version, we have two extreme options:

• either run archive node with ALL historical states
• or full node with only recent state (last 128 blocks in momery for mini reorg)

But in archive node case, some too-old states (e.g. 3 months ago) may not be used in most cases. Is there any solution to store states for only the last N months or last N blocks? Looking forward to your reply.

There is a similar question in stackexchange: Run archive node to store states for the last N months only.

[holiman]

Is there any solution to store states for only the last N months or last N blocks?

The answer is No. (see footnote)

However, it is possible to migrate from "one mega-gigantic super-server containing everything" to different models, like for example suggested here: #24461.

(footnote) Actually, if N is small enough, you could mod geth to use more memory and have stuff in memory longer.

[rjl493456442]

In the current storage scheme, it's not easy to retain the last N states. But in the upcoming new storage scheme(path-based), it's totally possible.

The archival node upon path-based can retain the state history(e.g. last N states) and the access to these historical states should be trivial.

But note, it takes time. We must manage to merge the initial version path-based scheme first, then work on the archival mode.

[jongrun]

In the current storage scheme, it's not easy to retain the last N states. But in the upcoming new storage scheme(path-based), it's totally possible.

The archival node upon path-based can retain the state history(e.g. last N states) and the access to these historical states should be trivial.

But note, it takes time. We must manage to merge the initial version path-based scheme first, then work on the archival mode.

@rjl493456442 The path-based scheme uses accountAddress+path as key, and a node can't be reused in two tries. My confusion is how to implement the archival node upon a scheme that trie nodes should not be reused?
If we use the trie history, which is used for deep reorg, as the entrance for reading history states, is it efficient? So I'm wondering if there is any rough idea for the archival node upon pbss? Looking forward to your reply~

[jongrun]

In the current storage scheme, it's not easy to retain the last N states. But in the upcoming new storage scheme(path-based), it's totally possible.
The archival node upon path-based can retain the state history(e.g. last N states) and the access to these historical states should be trivial.
But note, it takes time. We must manage to merge the initial version path-based scheme first, then work on the archival mode.

@rjl493456442 The path-based scheme uses accountAddress+path as key, and a node can't be reused in two tries. My confusion is how to implement the archival node upon a scheme that trie nodes should not be reused? If we use the trie history, which is used for deep reorg, as the entrance for reading history states, is it efficient? So I'm wondering if there is any rough idea for the archival node upon pbss? Looking forward to your reply~

@rjl493456442 Using bloom filter is an idea I can think of to read historical states efficiently from trie history logs.

• First, we divide the trie logs into multiple batches according to a certain block range (e.g. every 1024 blocks).
• Second, we calculate a bloom filter for each batch separately.
• Third, build these bloom filters into a bloom tree. That is, consider the bloom filters as leaf nodes, and combine every two nodes into a new node by merge them. Repeat the process until a single root node is generated.
• When we try to read history states from the trie logs, we can take advantage of bloom filters from the root down to the leaves of the bloom tree.
  However, it's tricky how to upgrade the bloom tree as new leaves join in and old leaves exit when the last N blocks move forward. So I believe there may be a better solution. I would appreciate it if you could share any ideas about archival node upon pbss.

[muang0]

+1 on @jongrun's thoughts- it would be really nice to be able to configure geth to store last N blocks