Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue Title: How to Determine the End of Bench2Drive Closed-loop evaluation #141

Open
curryqka opened this issue Feb 9, 2025 · 3 comments

Comments

@curryqka
Copy link

curryqka commented Feb 9, 2025

Description

We are currently conducting closed-loop tests using the Bench2Drive system and need a clear method to determine when these tests have successfully concluded. The question is whether we should rely on console output information or if there are other recommended methods for identifying the completion of the test.

Expected Behavior

A clear indication or signal that the Bench2Drive closed-loop test has ended successfully. This could be through console messages, log files, API responses, or any other effective mechanism.

Current Behavior

Uncertainty about relying solely on console information to determine the end of the test. There might be more reliable or additional methods available that we are not aware of.

Possible Needs

  1. Console Output: Relying on specific console messages indicating the successful completion of the test.
  2. Log Files: Checking designated log files for completion markers or status updates.

Steps to Reproduce (if applicable)

  1. Start a Bench2Drive closed-loop test.
  2. Monitor the process until it concludes.
  3. Attempt to identify the conclusion of the test based on current knowledge.

Environment Details

  • Operating System: [e.g., Ubuntu 20.04]

Labels

  • Type: Question
  • Priority: Medium
  • Component: Testing Framework

Could anyone provide guidance on how to accurately determine when a Bench2Drive closed-loop test has ended? Are there preferred methods beyond monitoring console outputs?

@jayyoung0802
Copy link
Member

Hi @curryqka. In the end, It will generate a recorded file, you can check it.

@curryqka
Copy link
Author

Hi @curryqka. In the end, It will generate a recorded file, you can check it.

Hi @jayyoung0802 , I would like to know how it looks like? A json file or the log information in the terminal?

@jayyoung0802
Copy link
Member

Hi @curryqka. In the end, It will generate a recorded file, you can check it.

Hi @jayyoung0802 , I would like to know how it looks like? A json file or the log information in the terminal?

A JSON file in the relevant folder(algo name) which is in the current path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants