Skipping a test by returning godog.ErrSkip does not work?

[juliangp]

👓 What did you see?

When skipping a test by returning godog.ErrSkip the test is marked as failed and not as skipped.

This is what we get when running the test:

✅ What did you expect to see?

We expected to see the first test to pass and the second step to skip?

📦 Which tool/library version are you using?

This is using github.com/cucumber/godog v0.13.0.

🔬 How could we reproduce it?

Steps to reproduce the behavior:

1. Install '...' version '...'
2. Create a file called '....'
3. Run command '....'
4. See error '....'

This is the example that I am running?

skip.zip

📚 Any additional context?

Please let me know if you need more information?

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[vearutop]

I've extracted skip.zip to a directory, created a module there, ran go mod tidy to fetch external deps.
Then

go install github.com/cucumber/godog/cmd/godog@latest
godog --version
Godog version is: v0.13.0

then

godog

Use of godog CLI is deprecated, please use *testing.T instead.
See https://github.com/cucumber/godog/discussions/478 for details.
Use of godog CLI is deprecated, please use *testing.T instead.
See https://github.com/cucumber/godog/discussions/478 for details.
Feature: Test skip error
  User should be able to skip a test.

  Scenario Outline: should skip the tests correctly # features/test_skip.feature:4
    When the test should be "<state>"               # <autogenerated>:1 -> *testContext

    Examples:
      | state |
      | ok    |
      | skip  |
      | ok    |

3 scenarios (3 passed)
3 steps (2 passed, 1 skipped)
390.903µs

Can not reproduce, could it be that your godog runner is an older version?

[juliangp]

@vearutop thanks for looking at this, I am looking to see what is different in our case but it maybe that we are running our tests in parallel.

[juliangp]

@vearutop I am a bit puzzled here - I did what you said, here is my zip with the go.mod skip.zip but it fails when I run like this?

[juliangp]

@vearutop running go test also fails:

[bharatsaradagi]

godog.ErrSkip marks the test as failed in the cucumber report, instead of skipped. is this an issue from godog side ? how can i fix it. i'm using godog v0.14.1 version.

[bharatsaradagi]

for the skipped tests error_message is not showing in the cucumber report, is this a limitation from the godog framework ? can it be fixed ?
the end user won't be knowing that why the tests are getting skipped ?