Suggestion to increase speed of sync

[OSSDeveloper]

Hi,
Right now, the sync process is quiet slow. Reason for this is Node being a single threaded technology, is unable to do utilize the processing power available on machine(in which bitcore-node is running) to the extent it should be utilizing.

Right now, its become difficult to run sync for days(especially for start-ups) & going farward, after an year or two, it will be impossible to do it.

Need to increase the sync speed is obvious(hope you are of the same perspective).

We can achieve this by using node cluster in the following manner :-

The entire bitcore-node should be split into 2 separate node processes.

a. First process reads the data from bitcoin-qt and, passes the result to second process. This will be a single threaded process. No use of cluster needed.

This thread can also be used to store the data to bitcore-node DB and go for next block.

b. Second node process which uses node-cluster and, does all the processing intensive work. This process should by default use "Number of processors available -2".

This passes the entire block along with transactions to the first process which stores the block to bitcore-node DB and, goes to fetch the next block.

By doing this, we won't have the problem of database lock as, single thread will be using the level-db operations(1st process) and, processor intensive work would be delegated to another node-process using cluster. This definitely will increase the sync process.

Hope you find the suggestion helpful.

[braydonf]

Using multiple processes I think is a good idea. Also take a look chainlib-bitcoin, we're using batch for database writes and I think significantly helps, from our tests it takes around 5 hours to sync on testnet3.

[braydonf]

Also you'll notice that blocks with many transactions is a bottleneck, and could benefit from multiple processes, as well as optimizations for transaction serialization.

[OSSDeveloper]

@braydonf I actually did not understand what https://github.com/bitpay/chainlib-bitcoin does from the documentation. Can you pls let me know?

Can I run it in production / livenet mode?

[braydonf]

Yep, you can run livenet and testnet. Docs have been updated with base API methods.

[OSSDeveloper]

Yup, you are right. Batch processing or querying transactions increased the speed considerably. That was a good improvisation.

As the project itself is moving towards becoming all-in-one kind of app, i.e. bitcoin-qt+bitcore-node in one application, it would become essential to use node cluster. Hope you come up with an optimum solution. :)

[braydonf]

Syncronization is done all in bitcoind now.