C extension with stub(pyi) using implicit namespace project does not work

[ileixe]

I found pylsp cannot go to definition in our C extension library using implicit namespace which provides stub (pyi) file.

This is minimal reproduction

• setup.py

from setuptools import setup, Extension, find_namespace_packages
import os

setup(
    name="mypkg",
    version="0.1",
    packages=find_namespace_packages(),  # This will find packages without __init__.py
    ext_modules=[
        Extension("mypkg.example", [os.path.join("mypkg", "example.c")]),
    ],
    package_data={"mypkg": ["py.typed", "example.pyi"]},
)

• mypkg/example.pyi

def add(a: int, b: int) -> int: ...

• mypkg/example.c

(.venv) typing-extension$ cat mypkg/example.c
#include <Python.h>

static PyObject* add(PyObject* self, PyObject* args) {
    int a, b;
    if (!PyArg_ParseTuple(args, "ii", &a, &b)) {
        return NULL;
    }
    return PyLong_FromLong(a + b);
}

static PyMethodDef ExampleMethods[] = {
    {"add", add, METH_VARARGS, "Add two integers."},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef examplemodule = {
    PyModuleDef_HEAD_INIT,
    "mypkg.example",   /* m_name */
    "A simple example C extension module",  /* m_doc */
    -1,
    ExampleMethods
};

PyMODINIT_FUNC PyInit_example(void) {
    return PyModule_Create(&examplemodule);
}

---

This is pylsp (eglot) log

2025-02-12 16:17:46,680 KST - DEBUG - pylsp.config.config -         finish pylsp_lint --> [] [hook]

2025-02-12 16:17:46,680 KST - DEBUG - pylsp.config.config -         finish pylsp_lint --> [] [hook]

2025-02-12 16:17:46,680 KST - DEBUG - pylsp_jsonrpc.endpoint - Sending notification: textDocument/publishDiagnostics {'uri': 'file:///home/ys/Source/practice/python/typing-extension/main.py', 'diagnostics': []}
2025-02-12 16:17:46,680 KST - DEBUG - pylsp.config.config -       finish pylsp_lint --> [] [hook]

2025-02-12 16:17:46,680 KST - DEBUG - pylsp_jsonrpc.endpoint - Sending notification: textDocument/publishDiagnostics {'uri': 'file:///home/ys/Source/practice/python/typing-extension/mypkg/example.pyi', 'diagnostics': []}
2025-02-12 16:17:46,680 KST - DEBUG - pylsp_jsonrpc.endpoint - Sending notification: textDocument/publishDiagnostics {'uri': 'file:///home/ys/Source/practice/python/typing-extension/setup.py', 'diagnostics': []}
2025-02-12 16:17:46,681 KST - DEBUG - pylsp_jsonrpc.endpoint - Sending notification: textDocument/publishDiagnostics {'uri': 'file:///home/ys/Source/practice/python/typing-extension/mypkg/__init__.py', 'diagnostics': []}
2025-02-12 16:17:55,571 KST - DEBUG - pylsp_jsonrpc.endpoint - Handling request from client {'jsonrpc': '2.0', 'id': 6, 'method': 'textDocument/definition', 'params': {'textDocument': {'uri': 'file:///home/ys/Source/practice/python/typing-extension/main.py'}, 'position': {'line': 0, 'character': 13}}}
2025-02-12 16:17:55,571 KST - DEBUG - pylsp.config.config -   pylsp_definitions [hook]
      config: <pylsp.config.config.Config object at 0x75268a6997f0>
      workspace: <pylsp.workspace.Workspace object at 0x75268a6d1970>
      document: file:///home/ys/Source/practice/python/typing-extension/main.py
      position: {'line': 0, 'character': 13}

2025-02-12 16:17:55,571 KST - DEBUG - pylsp.config.config - With configuration: {'plugins': {'pylsp_mypy': {'enabled': True}, 'jedi': {'environment': '~/Source/practice/python/typing-extension/.venv'}, 'pyflakes': {'enabled': None}, 'pycodestyle': {'enabled': None}, 'pylint': {'enabled': False, 'args': [], 'executable': None}, 'preload': {'modules': ['OpenGL', 'PIL', 'array', 'audioop', 'binascii', 'cPickle', 'cStringIO', 'cmath', 'collections', 'datetime', 'errno', 'exceptions', 'gc', 'imageop', 'imp', 'itertools', 'marshal', 'math', 'matplotlib', 'mmap', 'mpmath', 'msvcrt', 'networkx', 'nose', 'nt', 'numpy', 'operator', 'os', 'os.path', 'pandas', 'parser', 'rgbimg', 'scipy', 'signal', 'skimage', 'sklearn', 'statsmodels', 'strop', 'sympy', 'sys', 'thread', 'time', 'wx', 'xxsubtype', 'zipimport', 'zlib']}, 'ruff': {'enabled': True}}, 'rope': {'extensionModules': ['OpenGL', 'PIL', 'array', 'audioop', 'binascii', 'cPickle', 'cStringIO', 'cmath', 'collections', 'datetime', 'errno', 'exceptions', 'gc', 'imageop', 'imp', 'itertools', 'marshal', 'math', 'matplotlib', 'mmap', 'mpmath', 'msvcrt', 'networkx', 'nose', 'nt', 'numpy', 'operator', 'os', 'os.path', 'pandas', 'parser', 'rgbimg', 'scipy', 'signal', 'skimage', 'sklearn', 'statsmodels', 'strop', 'sympy', 'sys', 'thread', 'time', 'wx', 'xxsubtype', 'zipimport', 'zlib']}}
2025-02-12 16:17:55,572 KST - DEBUG - pylsp.plugins.definition - speed: init 9.039801597595215
2025-02-12 16:17:55,572 KST - DEBUG - pylsp.plugins.definition - speed: parsed 9.03991436958313
2025-02-12 16:17:55,572 KST - DEBUG - pylsp.plugins.definition - speed: import (<Name: mypkg@1,7>, <Name: example@1,13>) ModuleContext(<ModuleValue: main@1-2 is_stub=False>) 0.0004303455352783203
2025-02-12 16:17:55,573 KST - DEBUG - pylsp.plugins.definition - dbg: global search_module 'mypkg': <ImplicitNamespaceValue: mypkg>
2025-02-12 16:17:55,574 KST - DEBUG - pylsp.plugins.definition - dbg: search_module 'mypkg.example' in paths ['/home/ys/Source/practice/python/typing-extension/mypkg', '/home/ys/Source/practice/python/typing-extension/./mypkg', '/home/ys/Source/practice/python/typing-extension/.venv/lib/python3.11/site-packages/mypkg']: <CompiledModule: <module 'mypkg.example' from '/home/ys/Source/prac..>
2025-02-12 16:17:55,574 KST - DEBUG - pylsp.plugins.definition - dbg: Start: convert names
2025-02-12 16:17:55,574 KST - DEBUG - pylsp.plugins.definition - dbg: End: convert names
2025-02-12 16:17:55,574 KST - WARNING - pylsp.config.config - Failed to load hook pylsp_definitions: unsupported operand type(s) for -: 'NoneType' and 'int'
Traceback (most recent call last):
  File "/home/ys/Source/project/python/python-lsp-server/pylsp/config/config.py", line 39, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ys/.asdf/installs/python/3.12.9/lib/python3.12/site-packages/pluggy/_manager.py", line 480, in traced_hookexec
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ys/.asdf/installs/python/3.12.9/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/ys/.asdf/installs/python/3.12.9/lib/python3.12/site-packages/pluggy/_result.py", line 62, in from_call
    result = func()
             ^^^^^^
  File "/home/ys/.asdf/installs/python/3.12.9/lib/python3.12/site-packages/pluggy/_manager.py", line 477, in <lambda>
    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ys/.asdf/installs/python/3.12.9/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/ys/.asdf/installs/python/3.12.9/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ys/Source/project/python/python-lsp-server/pylsp/plugins/definition.py", line 72, in pylsp_definitions
    "start": {"line": d.line - 1, "character": d.column},
                      ~~~~~~~^~~
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'
2025-02-12 16:17:55,576 KST - DEBUG - pylsp_jsonrpc.endpoint - Got result from synchronous request handler: []

---

After some investigation, I found jedi does not try to find out module stub under implicit namespace because CompiledModule found here and filtered out: https://github.com/davidhalter/jedi/blob/master/jedi/inference/gradual/typeshed.py#L218

I'm not sure what's exactly filtering from the condition (it not python_value_set), but if I deleted the line so that lookup the CompiledModule's stub, everything works as expected.

Can we delete the line safely?

[davidhalter]

I'm not sure at all what doesn't work. I'm not sure what code you're providing to Jedi (that doesn't work) and I'm definitely not sure what the issue is here.

Can we delete the line safely?

You should probably assume with every line of code that there is a reason behind it. But feel free to play around with it and report your findings.

[ileixe]

Sorry I may not give enough context.

1. What code I gave to Jedi?

Just one line of import with some context

• mypkg is implicit namespace.
• example is C extension module without python code at all, but only stub. (example.pyi)

import mypkg.example 

2. What did not work?

'go to definition' on 'example' does not work. Jedi cannot find stub file.

3. What I was expecting?

Jedi can find stub (example.pyi) and lsp server works.

4. Why it did not work (from my observation)?

• Jedi treated example module as CompiledModule (which makes sense)
• But after that it no longer search stub file
• So lsp server cannot get stub information which have line number (So exception above)

5. What did you try?

• I deleted one line 'if condition' https://github.com/davidhalter/jedi/blob/master/jedi/inference/gradual/typeshed.py#L218 to find stub correctly.
• I found this is working at least but not sure side effect.

---

If you want more context, please let me know.

Thanks