Skip to content

Coverage now fails if there are no covered Python files even if test succeeds #2762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
phst opened this issue Apr 9, 2025 · 1 comment
Labels
core-rules Issues concerning core bin/test/lib rules

Comments

@phst
Copy link
Contributor

phst commented Apr 9, 2025

🐞 bug report

Affected Rule

The issue is caused by the rule: py_test

Is this a regression?

Yes, the previous version in which this bug was not present was: 1.2.0

Description

If there are no covered source files, the coverage.py lcov step fails, causing the entire test to fail when run under coverage (see repro below).

🔬 Minimal Reproduction

See https://github.com/phst/py-cover-bug, and the next section.

🔥 Exception or Error

After cloning the above repository, run bazel coverage --test_output=all //a:test. This fails with an error

[...]
INFO: Using default value for --instrumentation_filter: "^//a[/:]".
[...]
INFO: From Testing //a:test:
==================== Test output for //a:test:
Hello world
[...]/coverage/inorout.py:504: CoverageWarning: Module [...]/_main/bazel-out/darwin_arm64-fastbuild/bin/b was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
[...]/coverage/control.py:894: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")
--
Coverage runner: Not collecting coverage for failed test.
The following commands failed with status 1
[...]/bin/a/test.runfiles/_main/a/test
================================================================================
[...]

Running this with bazel test instead of coverage succeeds. Downgrading to rules_python 1.2.0 also makes it succeed.

Relevant additional output with verbose coverage enabled:

[coveragepy] Instrumented Files:
[...]/test.runfiles/_main/bazel-out/darwin_arm64-fastbuild/bin/b/bin
[...]/test.runfiles/_main/bazel-out/darwin_arm64-fastbuild/bin/b/binruntime_objects_list.txt
[coveragepy] Sources:
[...]/test.runfiles/_main/bazel-out/darwin_arm64-fastbuild/bin/b
Coverage entrypoint: [...]/coverage/__main__.py
[...]
Converting coveragepy database to lcov: /private/var/tmp/_bazel_p/35f69852d1c41a85f4769530320f22fe/sandbox/darwin-sandbox/26/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/_coverage/a/test/test/pylcov.dat
No data to report.
+ TEST_STATUS=1
[...]

Note that no directories containing Python files are listed in "Sources", so no coverage is generated for them. This is expected given the instrumentation filter, but causes coverage.py lcov to err out since it doesn't find any coverage data. My guess is that this is fallout from #2607.

🌍 Your Environment

Operating System:

macOS

Output of bazel version:

8.1.1

Rules_python version:

1.3.0

Anything else relevant?

phst added a commit to phst/rules_elisp that referenced this issue Apr 9, 2025
phst added a commit to phst/rules_elisp that referenced this issue Apr 9, 2025
@aignas aignas added the core-rules Issues concerning core bin/test/lib rules label Apr 10, 2025
@BurnzZ
Copy link
Contributor

BurnzZ commented Apr 14, 2025

Hi @phst ! Cheers for the report!

I can't seem to reproduce your logs, as I get this on mac and linux with bazel coverage --test_output=all //a:test --nocache_test_results:

INFO: Using default value for --instrumentation_filter: "^//a[/:]".
INFO: Override the above default with --instrumentation_filter
INFO: Analyzed target //a:test (83 packages loaded, 3921 targets configured).
INFO: From Testing //a:test:
==================== Test output for //a:test:
Hello world
Apr. 14, 2025 2:44:11 AM com.google.devtools.coverageoutputgenerator.Main runWithArgs
WARNING: There was no coverage found.
================================================================================
INFO: Found 1 test target...
Target //a:test up-to-date:
  bazel-bin/a/test
INFO: Elapsed time: 6.829s, Critical Path: 2.04s
INFO: 7 processes: 15 action cache hit, 5 internal, 2 darwin-sandbox.
INFO: Build completed successfully, 7 total actions
//a:test                                                                 PASSED in 1.9s

Executed 1 out of 1 test: 1 test passes

The response status code after running bazel coverage also looks good:

> echo $?
0

Reverting back to rules_python 1.2.0, 1.1.0, and 1.0.0 also gets the same result on my end.


I was wondering if you might have some env vars set that I'm missing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core-rules Issues concerning core bin/test/lib rules
Projects
None yet
Development

No branches or pull requests

3 participants