High main memory usage and hanging because of reading history since 2.8.5

[saaj]

Amazing stuff -- so much you can do in a terminal!

Last two versions (2.8.5 and 2.8.6) of euporie exhibit a memory problem on my machine. On the
first stroke, say in euporie-console, RSS jumps to ~2.2GiB from expected 70-80MiB while
everything hangs (possibly 1-2 characters of the input appear before hanging). After something
finally loaded, characters appear and the application works, but still quick input has a
visible display lag.

Here's a recording of xterm euporie-console on 2.8.6 (processes: xterm, sh, python
euporie-console, python -m ipykernel_launcher).

euporie installed like pipx install --python python3.11 euporie=={version}.

I quickly ran euporie.console through memray. You have an issue reading IPython
history I assume:

▼ 📂 2.228GB (100.00 %) <ROOT>                                                                                                █▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀█
├── ▼ 📂 2.221GB (99.69 %) _run_tracker  memray.commands.run:65                                                               █                                                                                 █
│   └── ▼ 📂 2.220GB (99.64 %) <module>  euporie.console.__main__:14                                                          █                                                                                 █
│       ├── ▼ 📂 2.220GB (99.64 %) main  euporie.console.__main__:10                                                          █                                                                                 █
│       │   ├── ▼ 📂 2.211GB (99.24 %) main  euporie.core.__main__:42                                                         █                                                                                 █
│       │   │   ├── ▼ 📂 2.201GB (98.78 %) launch  euporie.core.app.app:534                                                   █                                                                                 █
│       │   │   │   └── ▼ 📂 2.201GB (98.78 %) run  prompt_toolkit.application.application:1002                               █                                                                                 █
│       │   │   │       └── ▼ 📂 2.201GB (98.78 %) run  asyncio.runners:190                                                   █                                                                                 █
│       │   │   │           └── ▼ 📂 2.201GB (98.78 %) run  asyncio.runners:118                                               █                                                                                 █
│       │   │   │               └── ▼ 📂 2.201GB (98.78 %) run_until_complete  asyncio.base_events:641                        █                                                                                 █
│       │   │   │                   └── ▼ 📂 2.201GB (98.78 %) run_forever  asyncio.base_events:608                           █                                                                                 █
│       │   │   │                       └── ▼ 📂 2.201GB (98.78 %) _run_once  asyncio.base_events:1936                        █                                                                                 █
│       │   │   │                           └── ▼ 📂 2.201GB (98.78 %) _run  asyncio.events:84                                █                                                                                 █
│       │   │   │                               ├── ▼ 📂 2.191GB (98.35 %) new_coroutine  prompt_toolkit.buffer:1923          █                                                                                 █
│       │   │   │                               │   └── ▼ 📂 2.191GB (98.35 %) async_suggestor  prompt_toolkit.buffer:1851    █                                                                                 █
│       │   │   │                               │       └── ▼ 📂 2.191GB (98.35 %) get_suggestion_async  prompt_toolkit.auto_s█                                                                                 █
│       │   │   │                               │           └── ▼ 📂 2.191GB (98.35 %) get_suggestion_async  euporie.core.sugg█                                                                                 █
│       │   │   │                               │               └── ▼ 📂 2.191GB (98.35 %) get_suggestion_async  prompt_toolki█                                                                                 █
│       │   │   │                               │                   └── ▼ 📂 2.191GB (98.35 %) get_suggestion  euporie.core.su█                                                                                 █
│       │   │   │                               │                       ├── 📄 1.978GB (88.77 %) process_history  euporie.core█                                                                                 █
│       │   │   │                               │                       ├── 📄 127.023MB (5.57 %) process_history  euporie.cor█                                                                                 █
│       │   │   │                               │                       ├── ▶ 📂 72.458MB (3.18 %) process_history  euporie.co█                                                                                 █
│       │   │   │                               │                       ├── 📄 16.000MB (0.70 %) process_history  euporie.core█                                                                                 █
│       │   │   │                               │                       ├── 📄 2.000MB (0.09 %) process_history  euporie.core.█                                                                                 █
│       │   │   │                               │                       └── 📄 1.000MB (0.04 %) process_history  euporie.core.█                                                                                 █
│       │   │   │                               ├── ▶ 📂 7.641MB (0.33 %) run_async  euporie.core.app.app:408                 █                                                                                 █
│       │   │   │                               ├── ▶ 📂 1.040MB (0.05 %) schedule  prompt_toolkit.eventloop.utils:73         █                                                                                 █
│       │   │   │                               ├── ▶ 📂 1.014MB (0.04 %) run_async  euporie.core.app.app:406                 █    💾 Allocations: 198299                                                       █
│       │   │   │                               ├── ▶ 📂 49.250KB (0.00 %) callback_wrapper  prompt_toolkit.input.vt100:162   █                                                                                 █
│       │   │   │                               ├── ▶ 📂 31.562KB (0.00 %) schedule  prompt_toolkit.eventloop.utils:78        █                                                                                 █
│       │   │   │                               ├── 📄 16.500KB (0.00 %) load_history  prompt_toolkit.buffer:394              █    📦 Size: 2.228GB                                                             █
│       │   │   │                               └── ▶ 📂 7.594KB (0.00 %) load_history  prompt_toolkit.buffer:393             █                                                                                 █
│       │   │   ├── ▶ 📂 10.483MB (0.46 %) load  importlib.metadata:202                                                       █                                                                                 █
│       │   │   └── ▶ 📂 9.582KB (0.00 %) launch  euporie.core.app.app:527                                                    █                                                                                 █
│       │   ├── ▶ 📂 5.710MB (0.25 %) main  euporie.core.__main__:32                                                          █                                                                                 █
│       │   └── ▶ 📂 3.403MB (0.15 %) main  euporie.core.__main__:38                                                          █                                                                                 █
│       └── ▶ 📂 178.176KB (0.01 %) main  euporie.console.__main__:8                                                          █                                                                                 █
├── ▶ 📂 1.684MB (0.07 %) _bootstrap  threading:1002                                                                          █                                                                                 █
└── ❓ 5.269MB (0.23 %) 6699 allocations from 4370 locations below the configured threshold                                   █                                                                                 █

Just in case, I use IPython kernels a lot (Jupyter Lab, specific version of Nteract with
reasonable IxD, raw ipython too a bit, all for different things, and was looking for a
faster alternative to Nteract), but I don't have gigabytes of command history.

$ du -sh ~/.ipython/
25M	/home/saaj/.ipython/

[joouha]

Thanks for reporting this!

This is very interesting - I'm almost certain it will be f923682 which causing this issue.

In that commit, I was trying to make auto-suggestions more intelligent by including context when searching the history for lines to suggest. To make this search more performant, I implemented a basic form of indexing of the history lines.

Based on the input lag you're experiencing, I suspect something about your IPython history is causing the index to balloon. My history file is a similar size to yours (~21MB), but I don't see any memory or CPU spikes when the history is being processed.

The indexing only involves basic string processing, so I wonder what is different about our history files that is causing this - does yours contain any extremely long lines by any chance?

I suspect that limiting the length of lines processed, and / or limiting the size of the index will probably prevent the memory issue you're experiencing.

[saaj]

Depends on what you'd consider "extremely long", but there are
indeed a few long samples:

Side note. Not rarely I have a long string (possibly structured)
in the clipboard I want do some computation on. Pasting it in a
cell works, but always inconvenient (wrap in a string and assign
variable, then navigation and scrolling, etc). To the point that
I used ipywidget textarea for input in some cases.

[joouha]

That one 800,000 character entry would do it! I'm able to reproduce this by adding a history entry of similar length.

I've made some optimizations to the history indexing in the dev branch, which will hopefully make it a bit more performant in your case. If you have uv installed, you can test these changes with the following:

uvx --no-cache --from 'git+https://github.com/joouha/euporie.git@373aea0' euporie-console

Hopefully this helps!

[saaj]

I use pipx (reinstalled with
pipx install 'git+https://github.com/joouha/euporie.git@373aea0').
The hanging has been resolved (at least of this machine), thanks!

Here's a comparison. I ran xterm euporie-console, waited 5 seconds, entered
111111 and ran the cell.

euporie 2.8.4

euporie 2.8.6-dev

Thus memory-wise, it depends. I might say it's still wasting ~110MiB with
some aggressive pre-caching of history suggestions (of much smaller original
size). You may say, it is desired because it speeds up suggestion and nobody
cares about hundred megabytes there days. Your call in what you optimise for.