How to reduce memory and CPU load for large datasets?

[kushalbakshi]

Hi CaImAn team,

I've run into an issue with a dataset that overwhelms CPU and RAM resources during the CNMF step and I could use your insight on how or if I can optimize parameters to ensure I'm not doing anything wrong.

The dataset was acquired with the Bruker Prairie View system (generating multi-page .ome.tif files). It contains single plane imaging data with 89026 frames acquired at 30 fps for a total recording duration of ~ 49 minutes. The images are 512 x 512 pixels.

In order to process them through CaImAn, I'm using Ubuntu/Linux AWS EC2 r7i.12xlarge instances with a 4th gen Intel Xeon processor and 384 GB RAM. CaImAn is installed using the developer installation approach within the DockerFile. The last logger message I see before I receive a SIGTERM error is Starting CNMF analysis.... I'm able to run everything fine up to that point. I'm not sure which process in the CNMF step is causing the CPU/RAM resources to be overwhelmed.

Also, These are the parameters I'm using to process this dataset:

{'fr': 30,
 'dxy': (2.0, 2.0),
 'decay_time': 0.4,
 'strides': (48, 48),
 'overlaps': (24, 24),
 'max_shifts': (20, 20),
 'max_deviation_rigid': 3,
 'pw_rigid': False,
 'p': 1,
 'nb': 2,
 'rf': 50,
 'K': 5,
 'gSig': array([8, 8], dtype=int64),
 'gSiz': array([17, 17], dtype=int64),
 'stride': 24,
 'method_init': 'greedy_roi',
 'rolling_sum': True,
 'only_init': True,
 'ssub': 1,
 'tsub': 1,
 'merge_thr': 0.85,
 'bas_nonneg': True,
 'min_SNR': 2.0,
 'rval_thr': 0.85,
 'use_cnn': True,
 'min_cnn_thr': 0.99,
 'cnn_lowest': 0.1,
 'use_cuda': False}

Please let me know if you have any suggestions on how this can be optimized and if others have encountered similar situations. I'd be happy to reach out to them if needed to find a solution here.

Thank you!

Kushal

[kushalkolar]

Reduce n_processes, each patch is sent to a process. I think you can determine the number of bytes per patch if you know the shape of each patch, [n_frames, n_rows, n_cols] * 64 bits. If you use fewer processes, fewer patches are processed simultaneously.

[kushalbakshi]

Thank you, @kushalkolar. I tried to reduce the n_processes to use only 30% of available cores but that didn't seem to solve it. I tried an instance with 700+ GB of RAM and I'm still seeing this huge spike in RAM during the CNMF step that terminates the container with SIGTERM. Based on my reading of the docs, the size of each patch is [89000, 72, 72] because patch size is determined by frames x (overlap + strides). Is that correct? If so, I believe the instance has more than enough RAM.

As I was looking through the parameters and the code, it occurred to me if this could have something to do with the CNN classifier? I'm not using an instance that has a GPU / CUDA. Could the initialization of the network be an underlying cause of this?

[kushalkolar]

I think you'll need to run things one step at a time to figure out where you're running out of RAM or if it's a different issue.

[zyyk78]

Maybe you can use this command to check the log:
grep -i 'killed' /var/log/syslog
Then I found this:

************* systemd-oomd[1891]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/gnome-flashback.service due to memory pressure for /user.slice/user-1000.slice/user@1000.service being 77.11% > 50.00% for > 20s with reclaim activity

So you may check your OOM-killer (out of memory), and set a larger memory threshold

[kushalkolar]

The OOM killer is doing it's job, you don't want to touch thag. Just reduce n_processes

[kushalbakshi]

For the session in question, it seemed like each process was allocating ~ 30 GB of RAM. So fewer n_processes was indeed the answer. Thank you @kushalkolar!