Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Test retries #27769

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Implement Test retries #27769

wants to merge 3 commits into from

Conversation

jfversluis
Copy link
Member

Description of Change

Adds retries on the test level so tests will be retried 2 times when it fails which will hopefully help overcome some of the flakiness of the tests and make them pass in 1 go without having to manually retry them

@jsuarezruiz
Copy link
Contributor

I understand the changes, and I agree, avoiding manual retries would be great. The only thing that worries me is that the retry could make some flaky tests pass and we only will detect flaky tests when reviewing data in CI, taking a look to tests that usually pass in the second pass. Wouldn't it be better to mark the flaky tests #27772 and try retries only in those cases? And only until the test is reviewed (fix the issue, update the test).

@jfversluis
Copy link
Member Author

Totally agree! I think we should maybe have both. A way to still detect the flaky ones but also it should not stop the CI from going green. I will see if I can get some reporting going on when a test is retried so we can still identify those.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants