Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test file contains sometime forbidden character #442

Open
1 of 5 tasks
vsantele opened this issue Feb 13, 2025 · 1 comment · May be fixed by #448
Open
1 of 5 tasks

Test file contains sometime forbidden character #442

vsantele opened this issue Feb 13, 2025 · 1 comment · May be fixed by #448
Labels
bug Something isn't working

Comments

@vsantele
Copy link
Collaborator

vsantele commented Feb 13, 2025

Involved Module

  • UI
  • EvoSuite
  • LLM
  • Kex
  • Other (please explain)

Describe the bug

A clear and concise description of what the bug is.

If the test class name contains a forbidden character, the file path will be incorrect and will not be created.

To Reproduce

(not very reliable because it depends on the output of an LLM. But we can force it.)

Steps to reproduce the behavior:

  1. Generate a test on a simple hello-world func main()
  2. Manually update the test class name to add a forbidden character on Windows (I don't know if replicable on Linux)
  3. Run the test
  4. See error

Expected behavior

A clear and concise description of what you expected to happen.

We need to sanitize the filename to avoid forbidden characters for all OS.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

The filename path is generated using the name of the generated class. It is excepted to follow the default naming rule [MethodUnderTest][Scenario]Test by default. This convention should not contain any forbidden characters. But if the LLM doesn't follow the rule (happened to me with a low parameters model), it's a possibility.

The only way to add forbidden characters to the name is with the following syntax which doesn't follow the Kotlin conventions.

class `MyClass("extra")`

To avoid this problem, I see two techniques:

  • Ask the LLM to not use this syntax for the class (not very reliable)
  • Simply sanitize the filename in the code. This is far more reliable. And cover the case where a user asks for non-conventional class name.
@vsantele vsantele added the bug Something isn't working label Feb 18, 2025
@vsantele
Copy link
Collaborator Author

The question now is: Where is the best place to sanitize the path?

The easiest place to do it is inside org.jetbrains.research.testspark.tools.TestProcessor.saveGeneratedTest. But this method asks for a filename. So, is it its responsibility to sanitize the filename or the caller?

Because we construct the full path inside this method, I think it's pretty safe to sanitize it inside, too.

What do you think?

@vsantele vsantele linked a pull request Feb 21, 2025 that will close this issue
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant