stray Python processes when running mypy, maxing out CPU completely

[ThomasEwaldETH]

Hello there!

Since of today, my Python extension in Visual Studio Code maxes out my CPU. I recently had this issue (maybe a week ago), where I could solve it with
https://stackoverflow.com/questions/71516186/vs-code-uses-100-cpu-even-if-it-is-closed

It worked for a couple of days now. However, as of today, I am quickly again at 100% CPU. Here a couple of things I recognized so far:

• When I start VSC, a Python sub process is generated, which takes up about 25-40% of CPU. This sub process does not get terminated. When it is active, the extension does not work properly, meaning that for example linting does not work. If I terminate it manually, linting immediately works. After terminating, VSC goes down to 0% CPU usage.
• The moment I press 'save' (strg+s) in an arbitrary file in my Python project folder, two sub processes are created: One also taking up 25-40%, the other one quickly going down to zero, but doesn't get terminated. In addition, all linting highlights are gone and the linting seems to not work anymore.
• If I press 'save' a second time (no matter which file I'm in), two new sub processes are created. Again, one quickly falling down to zero, the other one taking up another 25-40% of CPU -> consequently, if I press 'save' four times, my CPU is maxed out.
• If I terminate the CPU draining sub processes, the 'dead' ones with 0% CPU seem to get terminated after a short while.
• I have ALWAYS two 0% CPU sub processes running that are named 'Python'.

In the following a couple of snippets from my Task Manager:

Here, the CPU usage after starting VSC, where the project folder is opened automatically.

Terminated the first sub process manually - remaining two 0% sub processes.

Hit 'save' a single time.

Hit 'save' a second time. And so on...

I am using Win11, about my VSC:
Version: 1.74.3 (user setup)
Commit: 97dec172d3256f8ca4bfb2143f3f76b503ca0534
Date: 2023-01-09T16:59:02.252Z
Electron: 19.1.8
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: 10.2.154.15-electron.0
OS: Windows_NT x64 10.0.22621
Sandboxed: No

I am completely out of ideas on how to get this fixed, since it just appeared out of nowhere today. Maybe someone here has an idea. Searching with Google didn't provide a lot, aside from what I linked above.

I would be grateful for help!
Best regards,
Thomas

[ThomasEwaldETH]

Good morning!
Small addition here: If I disable the extension, and reload once, the python processes are also still running. If I quit VSC, they also do not terminate. I have to manually end them.

Without extension 'Python' everything runs smoothly.

Best regards and have a nice day,
Thomas

[brettcannon]

What have you configured the Python extension to run on your behalf (i.e. linters and formatters)? And when you say, "without extension", does this include Pylance (or are you using Jedi)?

[brettcannon]

And is your installation of Python from python.org or somewhere else?

[ThomasEwaldETH]

Hey!
The Python installation is from python.org.
The extension runs as it comes when downloaded via the VSC Extension tab (so all dependencies are also installed and activated) - everything on basically. When disabeling, I make sure to disable all running parts of the extension (so pylance, isort, etc.).
I don't know about Jedi - is there a link where I can read a little bit about it?

[brettcannon]

You can turn on Jedi by setting your python.languageServer setting to "jedi".

[ThomasEwaldETH]

I did that and found that it does not change the behavior measurably. However, when jedi is active, some colors are missing, e.g., green for classes or yellow for functions. Is that on purpose?

[ThomasEwaldETH]

Hello,
another finding: If I save three .py files at once, it generates three python sub processes, presumable one for each file, which then block the CPU as usual.

In addition, I tried to check out the repo to another folder and tried it there, with the result being the same.

Best regards and have a nice day,
Thomas

[ThomasEwaldETH]

I made a couple further observations:
Frist, it seems that just waiting after hitting 'save' (once) can also solve the problem (sub process terminates). However, it takes quite some time to terminate each individual sub process in some cases. I stopped it and found it was over a minute. Having multiple of those sub processes might also terminate, just after a couple of minutes. However, it quickly maxes out my CPU. In addition, I found that opnening a file also creates there sub processes. Meaning, that if I open 3-4 files at once on startup, my CPU is maxed out immediately and I can either wait 3-4 minutes or terminate them by hand. While CPU is at 100%, coding is not very fluid.

Second, I found that it is depending on the file I am working on. I currently have 6000 lines of code in my project in about 20 modules, where some of the files import a lot of other modules. Those loading many others are most affected by the problem. In contrast, I created a new file, which is not really linked to the rest of the project yet, and it just does not show this problem.

So, after these findings I am a bit confused. Is it possible, that the extension is limited by the size of a project (however, I feel like 6000 lines of code are not that much...) and the structure of the project (I will refactor it a bit, create more modules, less lines of code per module and test that)? Still, I think there is something hiding, because after terminating the sub processes, linting and sorting works as it should, it seems that it is hold back by the sub processes being stuck somewhere (don't know how to best describe that...).

Best regards and have a nice day,
Thomas

[ThomasEwaldETH]

Hello,
and thanks a lot for the kind help!

I extractet the output and filtered it a bit. In the following the relevant parts of the output, with comments from me:

Program starts with this:

poetry env list --full-path
cwd: .
LSP Notebooks experiment is disabled -- Jupyter disabled or not installed
LSP Notebooks interactive window support is disabled -- not in LSP Notebooks experiment
Python interpreter path: ..venv\Scripts\python.exe
Starting Pylance language server.
Extension search path: c:\Users\thoma.vscode\extensions
Extension search path: c:\Users\thoma.vscode\extensions
..venv\Scripts\python.exe ~.vscode\extensions\ms-python.python-2022.20.2\pythonFiles\linter.py -m mypy --follow-imports=silent --ignore-missing-imports --show-column-numbers --no-pretty .\python_optim_tool\meta.py
cwd: .
..venv\Scripts\python.exe ~.vscode\extensions\ms-python.python-2022.20.2\pythonFiles\linter.py -m mypy --follow-imports=silent --ignore-missing-imports --show-column-numbers --no-pretty .\python_optim_tool\windings.py
cwd: .
..venv\Scripts\python.exe ~.vscode\extensions\ms-python.python-2022.20.2\pythonFiles\linter.py -m pylint .\python_optim_tool\meta.py
cwd: .
..venv\Scripts\python.exe ~.vscode\extensions\ms-python.python-2022.20.2\pythonFiles\linter.py -m pylint .\python_optim_tool\windings.py
cwd: .

Then, this is followed by some linting output, that apparently shows me all warning/errors/notifications in all files of the project. I have stopped the time and found, that it takes about 12 seconds until this part of the output is generated/shown in the output pane:

##########Linting Output - pylint##########

[
{
"type": "warning",
"module": "python_optim_tool.meta",
"obj": "",
"line": 236,
"column": 13,
"endLine": null,
"endColumn": null,
"path": "python_optim_tool\meta.py",
"symbol": "fixme",
"message": "TODO what happens if there is only 1 layer?",
"message-id": "W0511"
}, ...
]

##########Linting Output - pylint##########

[
{
"type": "warning",
"module": "python_optim_tool.windings",
"obj": "",
"line": 322,
"column": 1,
"endLine": null,
"endColumn": null,
"path": "python_optim_tool\windings.py",
"symbol": "fixme",
"message": "FIXME erase class 'WindingDimensions' and use actual Foil/Rectangle/Round classes",
"message-id": "W0511"
}, ...
]

Lastly, additional 36 seconds pass by (in total we are at 48 seconds now) until this part appears:

##########Linting Output - mypy##########

python_optim_tool\windings.py:61:13: error: "Parallel" has no attribute "N_l" [attr-defined]
python_optim_tool\windings.py:62:13: error: "Parallel" has no attribute "_x" [attr-defined]
python_optim_tool\windings.py:63:18: error: "Parallel" has no attribute "d" [attr-defined]
python_optim_tool\windings.py:63:31: error: "Parallel" has no attribute "c" [attr-defined]
python_optim_tool\windings.py:63:51: error: "Parallel" has no attribute "e" [attr-defined]
Found 5 errors in 1 file (checked 1 source file)

##########Linting Output - mypy##########

Success: no issues found in 1 source file

Here, the cold start is finished.

So, for me this looks like pylint linting is doing it's job, and searching for everything in the project first. I think taking its time for this is appropriate. During this time, one Python sub process is running. The mypy linting takes additional 36 seconds then. Maybe that's the issue? Still, the same sub process is running.

Thanks a lot and have a nice day,
Thomas

[brettcannon]

OK, so this means you have more settings that you initially shared since mypy and pylint don't run unless you turn them on. Can you turn off the integrations one-by-one and see which one seems to cause the stray process? This may either be our doing (in which case I will probably suggest one of our tool extensions instead of integrated support), or it's the tool itself in which case we can't really do much.

[ThomasEwaldETH]

Hello!
Yes, seems like it. I have to check again on that machine I used to test it. However, I posted the content of the settings.json, and it was that. On another machine, which I am working now and which shows the same problem, I have the following content in the settings.sjon:

{
    "git.autofetch": true,
    "editor.formatOnSave": true,
    "python.linting.pylintEnabled": true,
    "python.linting.mypyEnabled": false,
    "python.formatting.provider": "black",
    "python.terminal.activateEnvInCurrentTerminal": true,
    "python.testing.pytestArgs": [
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "editor.rulers": [
        80
    ],
    "editor.unicodeHighlight.allowedCharacters": {
        "σ": true,
        "α": true,
        "ρ": true
    },
}

If I set mypyEnabled to false, as shown above, I don't get long-lasting stray processes.

I think I don't need both to do my linting, correct?

Best regards and have a nice evening, Thomas

[brettcannon]

Thanks so much for researching this! I am going to convert this to an issue to try and track down if we are somehow leaving the process running by accident.

[brettcannon]

#20668