Skip to content

Commit

Permalink
Optimize RocksDB DB open options (#3316)
Browse files Browse the repository at this point in the history
## Motivation

RocksDB is implemented as an SST. So it has some in memory buffer, that gets flushed to disk when certain criteria is met.
What was happening was that we weren't flushing data to disk frequently enough, and the implementation we're using to get an iterator in `find_keys_by_prefix` flushes some tombstones to disk, if necessary. And since we weren't flushing data to disk frequently enough, it often was.
This was causing an increase in those reads of a few tens of ms, which when trying to make the client achieve high TPS makes a big difference.

## Proposal

Reduce the size of the write buffer in general, so that we flush to disk more often. This overall seems to improve performance.
Based on this https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#flushing-options
We had at most 2 write buffers before of 64 mb each, which is the default settings. Now we have at most 16 write buffers with 8 mb each. Memory usage should stay the same, but disk flushing will be more frequent.
I also changed the compression type to `Lz4`, which should be slightly faster than the default, `Snappy`.

## Test Plan

Ran benchmark locally, time it takes to reach a certain BPS isn't increasing with time anymore. Ran it for almost 4 hours, it stays within the same range of 400-700ms

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
  • Loading branch information
ndr-ds authored Feb 13, 2025
1 parent f77f41e commit a39a38f
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions linera-views/src/backends/rocks_db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ const MAX_VALUE_SIZE: usize = 3221225072;
// 8388608 and so for offset reason we decrease by 400
const MAX_KEY_SIZE: usize = 8388208;

const DB_CACHE_SIZE: usize = 8 * 1024 * 1024; // 8 MiB
const DB_MAX_WRITE_BUFFER_NUMBER: i32 = 16;

/// The RocksDB client that we use.
type DB = rocksdb::DBWithThreadMode<rocksdb::MultiThreaded>;

Expand Down Expand Up @@ -288,6 +291,12 @@ impl RocksDbStoreInternal {
}
let mut options = rocksdb::Options::default();
options.create_if_missing(true);
options.create_missing_column_families(true);
// Flush in-memory buffer to disk more often
options.set_write_buffer_size(DB_CACHE_SIZE);
options.set_max_write_buffer_number(DB_MAX_WRITE_BUFFER_NUMBER);
options.set_compression_type(rocksdb::DBCompressionType::Lz4);

let db = DB::open(&options, path_buf)?;
let executor = RocksDbStoreExecutor {
db: Arc::new(db),
Expand Down

0 comments on commit a39a38f

Please sign in to comment.