From a88fceafdeb0097c6900747e02a4047dbe1ab490 Mon Sep 17 00:00:00 2001 From: Arkadii Sapozhnikov <47223481+arksap2002@users.noreply.github.com> Date: Mon, 27 Jan 2025 18:40:38 +0100 Subject: [PATCH] Kex Integration (#431) * minimal attempt to run kex from testspark with harcoded settings Needs debugging and testing * load kex-runner jar from github (build.gradle.kts toplevel) * setup code for kex properties * KexErrorManager based on LLMErrorManager * KexProcessManager based on EvoSuiteProcessManager * Basic UI element (button for running kex) * kex works only for the class codeType (todo funciton and line if possible) * read resource files kex.policy and modules.info * generate Report and series of simplifications for MVP use a provided kex path for now instead of downloading jar and adding dependency use ProcessBuilder (jdk) instead of OSProcessHandler (IJ sdk) use kex.py instead of building java command directly generate Report objects by reading generated java classes * download kex as github release if it doesn't exist * delete commented code Deleted stuff: running Kex through through OSProcessHandler (IJ sdk) running Kex with kex.py downloading kex from github in build.gradle.kts * use javaparser to merge @before and @after generated methods * compiling tests successfully by adding helper method to TestGenData.otherInfo * fold anything but @Test annotated code, but unfortunately only on UI updates * remove python dependency by running kex jar directly * use project's java and add version check * empty implementation of settings classes for kex * allow setting kex arguments from kex settings page * kex test generation for 'method' code type arguably overkill solution of modify PsiMethodWrapper class added parameter names, types and return tyepes explicitly updated implementation for java and kotlin * fold all but @Test methods regardless of declaration order * remove settings for unsupported kex features * folding works even if junit isn't in classpath * use the the subprocess manager from IJ framework * empty kex path download to .cache and LOCALAPPDATA in windows * fix lint. remove wildcard imports * from IJ api, provide correct build directory to kex For multimodule projects using TestSpark the correct module's build directory path is passed based on the location of code for which tests are generated * set maxTests displayed and minimizeTestSuite options * fix code fold to only happen once at the start not every ui update * error handling for errors in kex process manager also fixed a bug with the way options were passed to kex subprocess made them a list of strings instead of a single big space separated string * fix lint * provide signatures with FQNs to kex for java This also includes a simple string based mapping to jvm types through type erasure * refactor. extract functons, add comments, remove redundancies * fix lint * every option in kex cmd is preceeded by --option * use kotlin.time.Duration instead of Int * refactor error handling code, check for non-zero exit code * make generated code manipulation more robust It no longer depends on the order of the methods generated * only check if the correct verison of kex exists * make download url a property and version a user setting * support jdk version 8. no --add-opens * bump up kex version to 0.0.10 * allow kex test generation for classes not in a package * null safety while folding helper code * Merge branch 'development' into edwin1729/improvement/kex-integration * disallow kex and line code type being chosen simultaneously * update description tab and README.md with kex * fix lint * fix run coverage by making method name same as class name This is expected by TestSpark. Relevant file TestProcess.kt:createXmlFromJacoco:109 * undo mistaken removal of addLanguageTextFieldListener before code folding * Add empty LLM response handling (#342) Show a warning if LLM returns a response that cannot be parsed into a test case (e.g., an explanation of this test case rather than a modification). * Restrict TestSpark action to suitable code types and fix generation for a line (#344) * fix update function * create availableForGeneration * ktlint * feat: add javadoc for `JavaPsiHelper.availableForGeneration` * feat: check for nullness of a PSI file in `TestSparkAction.update` * feat: update javadocs in `PsiComponents.kt` * feat: check for a class or method/func in `KotlinPsiHelper.availableForGeneration` * feat: add TODO to `ToolUtils` about a potential bug The bug is reflected in the issue #375. * feat: make `PsiHelper.getSurroundingLineNumber` return 1-based line numbers Before, the `KotlinPsiHelper` returned a 0-based line number which caused an issue with line-based test generation. The generated prompt contained a line above the selected one. * feat: implement line-based test generation with CUT as a context When there is no surrounding method about the selected line, we use the CUT as a context for this line. The CUT must always be present. Otherwise, the generation action should have been disabled for this line. * refactor: apply ktlint * feat: add `See` in TODO * feat: add TODO and surround $NAME in backticks in `linePrompt` template * feat: collect class constructor signatures in `PsiClassWrapper` * feat: remove backticks from `linePrompt` * feat: fill line-based test generation with additional context The line-based test generation that has a method as a context of the line now also accepts constructors of the containing class. * refactor: use `firstOrNull` for `cut` extraction * refactor: apply ktlint * fix: add required parameter to `ClassRepresentation` in tests * publish: core module version `4.0.0` The major version increased due to the change of the public API of `PromptGenerator.generatePromptForLine` method. --------- Co-authored-by: Vladislav Artiukhov * Data class for execution results * Minor refactoring * apply ktlint * fix getJavaVersion * add KexSettingsService to plugin.xml * fix DefaultKexSettingsState * fix KexSettingsState * fix Bundles * Force junit4 for EvoSuite tests * JUnit version forcing * fix getExceptionData * fix tools * fix managers * fix TestCasePanelBuilder.kt * fix TestProcessor.kt * remove unused import * fix ktlint * fix ktlint * remove TestSparkStarter.kt * Add support for language applicability checks in Tool * Add support for language applicability checks in Llm * Add support for language applicability checks in Kex * Add support for language applicability checks in EvoSuite * Filter test generation buttons by language compatibility * Refactor imports for SupportedLanguage in EvoSuite and Llm * Set progress indicator text during Kex test generation --------- Co-authored-by: Edwin Fernando Co-authored-by: Edwin Fernando Co-authored-by: Edwin Fernando Co-authored-by: Vladislav Artiukhov Co-authored-by: Iurii Zaitsev Co-authored-by: Hello-zoka --- README.md | 9 + build.gradle.kts | 3 + .../testspark/core/data/TestGenerationData.kt | 1 + gradle.properties | 1 + .../testspark/java/JavaPsiMethodWrapper.kt | 9 + .../kotlin/KotlinPsiMethodWrapper.kt | 9 + .../testspark/langwrappers/PsiComponents.kt | 3 + .../testspark/actions/TestSparkAction.kt | 50 +++- .../evosuite/EvoSuiteDefaultsBundle.kt | 4 + .../testspark/bundles/kex/KexBundlePaths.kt | 8 + .../bundles/kex/KexDefaultsBundle.kt | 21 ++ .../testspark/bundles/kex/KexLabelsBundle.kt | 17 ++ .../bundles/kex/KexMessagesBundle.kt | 17 ++ .../bundles/kex/KexSettingsBundle.kt | 17 ++ .../research/testspark/data/kex/KexMode.kt | 5 + .../generatedTests/TestCasePanelBuilder.kt | 84 +++++- .../testspark/services/KexSettingsService.kt | 44 +++ .../settings/kex/KexSettingsComponent.kt | 91 +++++++ .../settings/kex/KexSettingsConfigurable.kt | 78 ++++++ .../settings/kex/KexSettingsState.kt | 31 +++ .../testspark/tools/GenerationTool.kt | 1 + .../research/testspark/tools/ToolUtils.kt | 18 ++ .../testspark/tools/evosuite/EvoSuite.kt | 6 + .../generation/EvoSuiteProcessManager.kt | 29 +- .../research/testspark/tools/kex/Kex.kt | 180 +++++++++++++ .../tools/kex/KexSettingsArguments.kt | 116 ++++++++ .../tools/kex/error/KexErrorManager.kt | 60 +++++ .../kex/generation/GeneratedTestsProcessor.kt | 143 ++++++++++ .../tools/kex/generation/KexProcessManager.kt | 254 ++++++++++++++++++ .../research/testspark/tools/llm/Llm.kt | 6 + .../research/testspark/tools/template/Tool.kt | 9 + .../testspark/toolwindow/DescriptionTab.kt | 40 +++ src/main/resources/META-INF/plugin.xml | 9 + .../properties/kex/KexDefaults.properties | 8 + .../properties/kex/KexLabels.properties | 7 + .../properties/kex/KexMessages.properties | 7 + .../properties/kex/KexSettings.properties | 1 + .../properties/plugin/PluginLabels.properties | 1 + .../plugin/PluginMessages.properties | 4 +- 39 files changed, 1362 insertions(+), 39 deletions(-) create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexBundlePaths.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexDefaultsBundle.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexLabelsBundle.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexMessagesBundle.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexSettingsBundle.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/data/kex/KexMode.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/services/KexSettingsService.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsComponent.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsConfigurable.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsState.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/tools/kex/Kex.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/tools/kex/KexSettingsArguments.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/tools/kex/error/KexErrorManager.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/tools/kex/generation/GeneratedTestsProcessor.kt create mode 100644 src/main/kotlin/org/jetbrains/research/testspark/tools/kex/generation/KexProcessManager.kt create mode 100644 src/main/resources/properties/kex/KexDefaults.properties create mode 100644 src/main/resources/properties/kex/KexLabels.properties create mode 100644 src/main/resources/properties/kex/KexMessages.properties create mode 100644 src/main/resources/properties/kex/KexSettings.properties diff --git a/README.md b/README.md index 7ecb96d2b..004dc6999 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,15 @@ TestSpark currently supports two test generation strategies:
  • Generate tests for Java classes, methods, and single lines.
    • +

    Symbolic execution-based test generation

    +

    For this type of test generation, TestSpark uses Kex, supporting symbolic execution for Java Byte Code.

    +
      +
    • Supports up to Java 8 and upwards.
      • +
    • Powered by SMT solvers, it supports really high coverages given larger time frames.
      • +
    • Generated test cases are however not very readable (there are plans to automatically refactor with the help of LLMs).
      • +
    • Generates tests for Java classes and methods.
      • +
    +

    Initially implemented by CISELab at SERG @ TU Delft, TestSpark is currently developed and maintained by ICTL at JetBrains Research.

    ## DISCLAIMER diff --git a/build.gradle.kts b/build.gradle.kts index 625be657d..deb156615 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -170,6 +170,9 @@ dependencies { // https://gitlab.com/mvysny/konsume-xml implementation("com.gitlab.mvysny.konsume-xml:konsume-xml:1.0") + // for merging generated kex tests into a single file + implementation("com.github.javaparser:javaparser-core:3.26.1") + // From the jetbrains repository testImplementation("com.intellij.remoterobot:remote-robot:0.11.13") testImplementation("com.intellij.remoterobot:remote-fixtures:0.11.13") diff --git a/core/src/main/kotlin/org/jetbrains/research/testspark/core/data/TestGenerationData.kt b/core/src/main/kotlin/org/jetbrains/research/testspark/core/data/TestGenerationData.kt index 64c215e93..a1908e285 100644 --- a/core/src/main/kotlin/org/jetbrains/research/testspark/core/data/TestGenerationData.kt +++ b/core/src/main/kotlin/org/jetbrains/research/testspark/core/data/TestGenerationData.kt @@ -16,6 +16,7 @@ data class TestGenerationData( var importsCode: MutableSet = mutableSetOf(), var packageName: String = "", var runWith: String = "", + // Modifications to this code in the tool-window editor are forgotten when apply to test suite var otherInfo: String = "", // changing parameters with a large prompt diff --git a/gradle.properties b/gradle.properties index 513624059..c9a780c0e 100644 --- a/gradle.properties +++ b/gradle.properties @@ -7,6 +7,7 @@ pluginName = TestSpark pluginVersion = 0.3.1 evosuiteVersion = 1.0.5 +kexVersion = 0.0.8 # See https://plugins.jetbrains.com/docs/intellij/build-number-ranges.html # for insight into build numbers and IntelliJ Platform versions. diff --git a/java/src/main/kotlin/org/jetbrains/research/testspark/java/JavaPsiMethodWrapper.kt b/java/src/main/kotlin/org/jetbrains/research/testspark/java/JavaPsiMethodWrapper.kt index fd60cd488..50112188f 100644 --- a/java/src/main/kotlin/org/jetbrains/research/testspark/java/JavaPsiMethodWrapper.kt +++ b/java/src/main/kotlin/org/jetbrains/research/testspark/java/JavaPsiMethodWrapper.kt @@ -37,6 +37,15 @@ class JavaPsiMethodWrapper(private val psiMethod: PsiMethod) : PsiMethodWrapper override val signature: String get() = buildSignature(psiMethod) + override val parameterNames: List + get() = psiMethod.parameterList.parameters.map { it.name } + + override val parameterTypes: List + get() = psiMethod.parameterList.parameters.map { it.type.canonicalText } + + override val returnType: String + get() = psiMethod.returnType?.canonicalText ?: "void" + val parameterList = psiMethod.parameterList val isConstructor: Boolean = psiMethod.isConstructor diff --git a/kotlin/src/main/kotlin/org/jetbrains/research/testspark/kotlin/KotlinPsiMethodWrapper.kt b/kotlin/src/main/kotlin/org/jetbrains/research/testspark/kotlin/KotlinPsiMethodWrapper.kt index 93f39d6ba..a2c2f6449 100644 --- a/kotlin/src/main/kotlin/org/jetbrains/research/testspark/kotlin/KotlinPsiMethodWrapper.kt +++ b/kotlin/src/main/kotlin/org/jetbrains/research/testspark/kotlin/KotlinPsiMethodWrapper.kt @@ -37,6 +37,15 @@ class KotlinPsiMethodWrapper(val psiFunction: KtFunction) : PsiMethodWrapper { override val signature: String get() = buildSignature(psiFunction) + override val parameterNames: List + get() = psiFunction.valueParameters.map { it.name ?: "" } + + override val parameterTypes: List + get() = psiFunction.valueParameters.map { it.typeReference?.text ?: "Any" } + + override val returnType: String + get() = psiFunction.typeReference?.text ?: "Unit" + val parameterList = psiFunction.valueParameterList val isPrimaryConstructor: Boolean = psiFunction is KtPrimaryConstructor diff --git a/langwrappers/src/main/kotlin/org/jetbrains/research/testspark/langwrappers/PsiComponents.kt b/langwrappers/src/main/kotlin/org/jetbrains/research/testspark/langwrappers/PsiComponents.kt index 3f7f1d0c8..71d7cda32 100644 --- a/langwrappers/src/main/kotlin/org/jetbrains/research/testspark/langwrappers/PsiComponents.kt +++ b/langwrappers/src/main/kotlin/org/jetbrains/research/testspark/langwrappers/PsiComponents.kt @@ -25,6 +25,9 @@ interface PsiMethodWrapper { val name: String val methodDescriptor: String val signature: String + val parameterNames: List + val parameterTypes: List + val returnType: String val text: String? val containingClass: PsiClassWrapper? val containingFile: PsiFile? diff --git a/src/main/kotlin/org/jetbrains/research/testspark/actions/TestSparkAction.kt b/src/main/kotlin/org/jetbrains/research/testspark/actions/TestSparkAction.kt index 6304b258d..5f537ac89 100644 --- a/src/main/kotlin/org/jetbrains/research/testspark/actions/TestSparkAction.kt +++ b/src/main/kotlin/org/jetbrains/research/testspark/actions/TestSparkAction.kt @@ -28,21 +28,26 @@ import org.jetbrains.research.testspark.settings.evosuite.EvoSuiteSettingsState import org.jetbrains.research.testspark.settings.llm.LLMSettingsState import org.jetbrains.research.testspark.tools.TestsExecutionResultManager import org.jetbrains.research.testspark.tools.evosuite.EvoSuite +import org.jetbrains.research.testspark.tools.kex.Kex import org.jetbrains.research.testspark.tools.llm.Llm import org.jetbrains.research.testspark.tools.template.Tool import java.awt.BorderLayout import java.awt.CardLayout +import java.awt.Component import java.awt.Dimension import java.awt.Font import java.awt.Toolkit import java.awt.event.WindowAdapter import java.awt.event.WindowEvent +import javax.swing.Box +import javax.swing.BoxLayout import javax.swing.ButtonGroup import javax.swing.JButton import javax.swing.JFrame import javax.swing.JLabel import javax.swing.JPanel import javax.swing.JRadioButton +import javax.swing.SwingConstants /** * Represents an action to be performed in the TestSpark plugin. @@ -110,7 +115,9 @@ class TestSparkAction : AnAction() { private val llmButton = JRadioButton("${Llm().name}") private val evoSuiteButton = JRadioButton("${EvoSuite().name}") + private val kexButton = JRadioButton("${Kex().name}") private val testGeneratorButtonGroup = ButtonGroup() + private val kexForLineCodeTypeErrMsg = JLabel() // The error displayed when if kex and line code type are chosen private val psiHelper: PsiHelper get() { @@ -201,13 +208,17 @@ class TestSparkAction : AnAction() { panelTitle.add(JLabel(TestSparkIcons.pluginIcon)) panelTitle.add(textTitle) - testGeneratorButtonGroup.add(llmButton) - testGeneratorButtonGroup.add(evoSuiteButton) + if (Llm().appliedForLanguage(psiHelper.language)) testGeneratorButtonGroup.add(llmButton) + if (EvoSuite().appliedForLanguage(psiHelper.language)) testGeneratorButtonGroup.add(evoSuiteButton) + if (Kex().appliedForLanguage(psiHelper.language)) testGeneratorButtonGroup.add(kexButton) val testGeneratorPanel = JPanel() testGeneratorPanel.add(JLabel("Select the test generator:")) - testGeneratorPanel.add(llmButton) - testGeneratorPanel.add(evoSuiteButton) + for (button in testGeneratorButtonGroup.elements) testGeneratorPanel.add(button) + if (testGeneratorButtonGroup.elements.toList().size == 1) { + // A single button is selected by default + testGeneratorButtonGroup.elements.toList()[0].isSelected = true + } for ((codeType, codeTypeName) in codeTypes) { val button = JRadioButton(codeTypeName) @@ -236,8 +247,15 @@ class TestSparkAction : AnAction() { .panel val nextButtonPanel = JPanel() + nextButtonPanel.layout = BoxLayout(nextButtonPanel, BoxLayout.Y_AXIS) nextButton.isEnabled = false + nextButton.alignmentX = Component.CENTER_ALIGNMENT + kexForLineCodeTypeErrMsg.alignmentX = Component.CENTER_ALIGNMENT + kexForLineCodeTypeErrMsg.horizontalAlignment = SwingConstants.CENTER + nextButtonPanel.add(kexForLineCodeTypeErrMsg) + nextButtonPanel.add(Box.createVerticalStrut(10)) // Add some space between label and button nextButtonPanel.add(nextButton) + updateNextButton() val cardPanel = JPanel(BorderLayout()) cardPanel.add(panelTitle, BorderLayout.NORTH) @@ -267,6 +285,10 @@ class TestSparkAction : AnAction() { updateNextButton() } + kexButton.addActionListener { + updateNextButton() + } + for ((_, button) in codeTypeButtons) { button.addActionListener { llmSetupPanelFactory.setPromptEditorType(button.text) @@ -286,6 +308,8 @@ class TestSparkAction : AnAction() { cardLayout.next(panel) cardLayout.next(panel) pack() + } else if (kexButton.isSelected) { + startKexGeneration() } else if (evoSuiteButton.isSelected && !evoSuiteSettingsState.evosuiteSetupCheckBoxSelected) { startEvoSuiteGeneration() } else { @@ -387,6 +411,7 @@ class TestSparkAction : AnAction() { dispose() } + private fun startKexGeneration() = startUnitTestGenerationTool(tool = Kex()) private fun startEvoSuiteGeneration() = startUnitTestGenerationTool(tool = EvoSuite()) private fun startLLMGeneration() = startUnitTestGenerationTool(tool = Llm()) @@ -398,12 +423,23 @@ class TestSparkAction : AnAction() { * This method should be called whenever the mentioned above buttons are clicked. */ private fun updateNextButton() { - val isTestGeneratorButtonGroupSelected = llmButton.isSelected || evoSuiteButton.isSelected + val isTestGeneratorButtonGroupSelected = llmButton.isSelected || evoSuiteButton.isSelected || kexButton.isSelected val isCodeTypeButtonGroupSelected = codeTypeButtons.any { it.second.isSelected } - nextButton.isEnabled = isTestGeneratorButtonGroupSelected && isCodeTypeButtonGroupSelected + val kexForCodeLineType = + kexButton.isSelected && codeTypeButtons.any { (codeType, button) -> codeType == CodeType.LINE && button.isSelected } + if (kexForCodeLineType) { + kexForLineCodeTypeErrMsg.text = + "* Kex cannot generate tests for a single line. Please change your selection" + } else { + kexForLineCodeTypeErrMsg.text = "" + } + + nextButton.isEnabled = + isTestGeneratorButtonGroupSelected && isCodeTypeButtonGroupSelected && !kexForCodeLineType if ((llmButton.isSelected && !llmSettingsState.llmSetupCheckBoxSelected && !llmSettingsState.provideTestSamplesCheckBoxSelected) || - (evoSuiteButton.isSelected && !evoSuiteSettingsState.evosuiteSetupCheckBoxSelected) + (evoSuiteButton.isSelected && !evoSuiteSettingsState.evosuiteSetupCheckBoxSelected) || + kexButton.isSelected ) { nextButton.text = PluginLabelsBundle.get("ok") } else { diff --git a/src/main/kotlin/org/jetbrains/research/testspark/bundles/evosuite/EvoSuiteDefaultsBundle.kt b/src/main/kotlin/org/jetbrains/research/testspark/bundles/evosuite/EvoSuiteDefaultsBundle.kt index f32b79165..42246cd08 100644 --- a/src/main/kotlin/org/jetbrains/research/testspark/bundles/evosuite/EvoSuiteDefaultsBundle.kt +++ b/src/main/kotlin/org/jetbrains/research/testspark/bundles/evosuite/EvoSuiteDefaultsBundle.kt @@ -14,4 +14,8 @@ object EvoSuiteDefaultsBundle : DynamicBundle(EvoSuiteBundlePaths.defaults) { */ @Nls fun get(@PropertyKey(resourceBundle = EvoSuiteBundlePaths.defaults) key: String): String = getMessage(key) + // In Intellij Platform version 2, the DynamicBundle returns the whole path and the value at the end in plugin verification. + // Each is separated by "|" (e.g., "|b|properties.llm.LLMDefaults|k|maxLLMRequest|3") + // if we do not split them here, the process will throw java.lang.NumberFormatException + .split("|").last() } diff --git a/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexBundlePaths.kt b/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexBundlePaths.kt new file mode 100644 index 000000000..0db8864c6 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexBundlePaths.kt @@ -0,0 +1,8 @@ +package org.jetbrains.research.testspark.bundles.kex + +object KexBundlePaths { + const val defaults: String = "properties.kex.KexDefaults" + const val messages: String = "properties.kex.KexMessages" + const val labels: String = "properties.kex.KexLabels" + const val settings: String = "properties.kex.KexSettings" +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexDefaultsBundle.kt b/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexDefaultsBundle.kt new file mode 100644 index 000000000..4975663e0 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexDefaultsBundle.kt @@ -0,0 +1,21 @@ +package org.jetbrains.research.testspark.bundles.kex + +import com.intellij.DynamicBundle +import org.jetbrains.annotations.Nls +import org.jetbrains.annotations.PropertyKey + +/** + * Loads the `resources` directory. + */ +object KexDefaultsBundle : DynamicBundle(KexBundlePaths.defaults) { + + /** + * Gets the requested default value. + */ + @Nls + fun get(@PropertyKey(resourceBundle = KexBundlePaths.defaults) key: String): String = getMessage(key) + // In Intellij Platform version 2, the DynamicBundle returns the whole path and the value at the end in plugin verification. + // Each is separated by "|" (e.g., "|b|properties.llm.LLMDefaults|k|maxLLMRequest|3") + // if we do not split them here, the process will throw java.lang.NumberFormatException + .split("|").last() +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexLabelsBundle.kt b/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexLabelsBundle.kt new file mode 100644 index 000000000..6952d17cb --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexLabelsBundle.kt @@ -0,0 +1,17 @@ +package org.jetbrains.research.testspark.bundles.kex + +import com.intellij.DynamicBundle +import org.jetbrains.annotations.Nls +import org.jetbrains.annotations.PropertyKey + +/** + * Loads the `resources` directory. + */ +object KexLabelsBundle : DynamicBundle(KexBundlePaths.labels) { + + /** + * Gets the requested default value. + */ + @Nls + fun get(@PropertyKey(resourceBundle = KexBundlePaths.labels) key: String): String = getMessage(key) +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexMessagesBundle.kt b/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexMessagesBundle.kt new file mode 100644 index 000000000..8eb9b0661 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexMessagesBundle.kt @@ -0,0 +1,17 @@ +package org.jetbrains.research.testspark.bundles.kex + +import com.intellij.DynamicBundle +import org.jetbrains.annotations.Nls +import org.jetbrains.annotations.PropertyKey + +/** + * Loads the `resources` directory. + */ +object KexMessagesBundle : DynamicBundle(KexBundlePaths.messages) { + + /** + * Gets the requested default value. + */ + @Nls + fun get(@PropertyKey(resourceBundle = KexBundlePaths.messages) key: String): String = getMessage(key) +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexSettingsBundle.kt b/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexSettingsBundle.kt new file mode 100644 index 000000000..fc13e27e3 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexSettingsBundle.kt @@ -0,0 +1,17 @@ +package org.jetbrains.research.testspark.bundles.kex + +import com.intellij.DynamicBundle +import org.jetbrains.annotations.Nls +import org.jetbrains.annotations.PropertyKey + +/** + * Loads the `resources` directory. + */ +object KexSettingsBundle : DynamicBundle(KexBundlePaths.settings) { + + /** + * Gets the requested default value. + */ + @Nls + fun get(@PropertyKey(resourceBundle = KexBundlePaths.settings) key: String): String = getMessage(key) +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/data/kex/KexMode.kt b/src/main/kotlin/org/jetbrains/research/testspark/data/kex/KexMode.kt new file mode 100644 index 000000000..bd2b1bad3 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/data/kex/KexMode.kt @@ -0,0 +1,5 @@ +package org.jetbrains.research.testspark.data.kex + +enum class KexMode { + Symbolic, Concolic, +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/display/generatedTests/TestCasePanelBuilder.kt b/src/main/kotlin/org/jetbrains/research/testspark/display/generatedTests/TestCasePanelBuilder.kt index 99c5221c7..b3f76358d 100644 --- a/src/main/kotlin/org/jetbrains/research/testspark/display/generatedTests/TestCasePanelBuilder.kt +++ b/src/main/kotlin/org/jetbrains/research/testspark/display/generatedTests/TestCasePanelBuilder.kt @@ -6,6 +6,7 @@ import com.intellij.notification.NotificationType import com.intellij.openapi.command.WriteCommandAction import com.intellij.openapi.diff.DiffColors import com.intellij.openapi.editor.Editor +import com.intellij.openapi.editor.FoldingModel import com.intellij.openapi.editor.event.DocumentEvent import com.intellij.openapi.editor.event.DocumentListener import com.intellij.openapi.editor.markup.HighlighterLayer @@ -14,11 +15,22 @@ import com.intellij.openapi.progress.ProgressManager import com.intellij.openapi.progress.Task import com.intellij.openapi.project.Project import com.intellij.openapi.ui.ComboBox +import com.intellij.psi.JavaRecursiveElementVisitor +import com.intellij.psi.PsiClassInitializer +import com.intellij.psi.PsiDocumentManager +import com.intellij.psi.PsiElement +import com.intellij.psi.PsiField +import com.intellij.psi.PsiManager +import com.intellij.psi.PsiMethod +import com.intellij.psi.PsiModifierListOwner +import com.intellij.psi.PsiTreeChangeAdapter +import com.intellij.psi.PsiTreeChangeEvent import com.intellij.ui.EditorTextField import com.intellij.ui.JBColor import com.intellij.ui.LanguageTextField import com.intellij.ui.components.JBScrollPane import com.intellij.util.ui.JBUI +import org.jetbrains.kotlin.psi.psiUtil.endOffset import org.jetbrains.research.testspark.bundles.llm.LLMMessagesBundle import org.jetbrains.research.testspark.bundles.plugin.PluginLabelsBundle import org.jetbrains.research.testspark.bundles.plugin.PluginMessagesBundle @@ -66,6 +78,7 @@ import javax.swing.ScrollPaneConstants import javax.swing.SwingUtilities import javax.swing.border.Border import javax.swing.border.MatteBorder +import kotlin.collections.HashMap class TestCasePanelBuilder( private val project: Project, @@ -249,6 +262,11 @@ class TestCasePanelBuilder( addLanguageTextFieldListener(languageTextField) + PsiManager.getInstance(project).addPsiTreeChangeListener(object : PsiTreeChangeAdapter() { + override fun childAdded(event: PsiTreeChangeEvent) { + languageTextField.editor?.foldHelperCode() + } + }) return panel } @@ -358,6 +376,67 @@ class TestCasePanelBuilder( }) } + private fun updateRange(offsets: Pair, element: PsiElement): Pair { + var (start, end) = offsets + start = if (start < element.textRange.startOffset) start else element.textRange.startOffset + end = if (end > element.textRange.endOffset) end else element.textRange.endOffset + return start to end + } + + /** + * Folds helper methods, fields when displaying to user, + * so they can focus on the important @Test annotated methods + */ + private fun Editor.foldHelperCode() { + val document = this.document + val project = this.project ?: return + + // Get the PSI file associated with the document + val psiFile = PsiDocumentManager.getInstance(project).getPsiFile(document) ?: return + + // Run the folding operation + this.foldingModel.runBatchFoldingOperation { + val foldingModel: FoldingModel = this.foldingModel + + var range = document.textLength to 0 + // Find start and end offsets for folding + psiFile.accept(object : JavaRecursiveElementVisitor() { + override fun visitMethod(element: PsiMethod) { + super.visitMethod(element) + if (hasTestAnnotation(element)) { + val (startOffset, endOffset) = range + // apply accumulating folding range + foldingModel.addFoldRegion(startOffset, endOffset, "...")?.isExpanded = false + // reset folding range after end of test method + range = element.endOffset to element.endOffset + } else { + range = updateRange(range, element) + } + } + + override fun visitField(field: PsiField) { + super.visitField(field) + range = updateRange(range, field) + } + + // These are example static initializers which can also be hidden from the user + override fun visitClassInitializer(initializer: PsiClassInitializer) { + super.visitClassInitializer(initializer) + range = updateRange(range, initializer) + } + }) + + val (startOffset, endOffset) = range + val foldRegion = foldingModel.addFoldRegion(startOffset, endOffset, "...") + + foldRegion?.isExpanded = false // Collapsed by default + } + } + + private fun hasTestAnnotation(element: PsiModifierListOwner): Boolean { + return element.modifierList?.annotations?.any { it.qualifiedName?.contains("Test") ?: false } ?: false + } + /** * Updates the user interface based on the provided code. */ @@ -366,7 +445,6 @@ class TestCasePanelBuilder( val lastRunCode = lastRunCodes[currentRequestNumber - 1] languageTextField.editor?.markupModel?.removeAllHighlighters() - resetButton.isEnabled = testCase.testCode != initialCodes[currentRequestNumber - 1] resetToLastRunButton.isEnabled = testCase.testCode != lastRunCode @@ -554,8 +632,8 @@ class TestCasePanelBuilder( indicator.setText("Executing ${testCase.testName}") val fileName = TestAnalyzerFactory.create(language).getFileNameFromTestCaseCode(testCase.testCode) - val junitVersion = - if (generationTool.toolId == "EvoSuite") JUnitVersion.JUnit4 else llmSettingsState.junitVersion + // For LLM JUnit version is taken from settings, while for Kex and EvoSuite only JUnit4 is allowed + val junitVersion = if (generationTool.toolId == "LLM") llmSettingsState.junitVersion else JUnitVersion.JUnit4 val testCompiler = TestCompilerFactory.create( project, diff --git a/src/main/kotlin/org/jetbrains/research/testspark/services/KexSettingsService.kt b/src/main/kotlin/org/jetbrains/research/testspark/services/KexSettingsService.kt new file mode 100644 index 000000000..4afccd2f5 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/services/KexSettingsService.kt @@ -0,0 +1,44 @@ +package org.jetbrains.research.testspark.services + +import com.intellij.openapi.components.PersistentStateComponent +import com.intellij.openapi.components.Service +import com.intellij.openapi.components.State +import com.intellij.openapi.components.Storage +import com.intellij.openapi.project.Project +import org.jetbrains.research.testspark.settings.kex.KexSettingsState + +/** + * This class is responsible for storing the application-level settings persistently. It uses SettingsApplicationState class for that. + */ +@Service(Service.Level.PROJECT) +@State(name = "KexSettingsState", storages = [Storage("KexSettings.xml")]) +class KexSettingsService : PersistentStateComponent { + private var kexSettingsState: KexSettingsState = KexSettingsState() + + /** + * Gets the currently persisted state of the application. + * This method is called every time the values in the EvoSuite Settings page are saved. + * If the values from getState are different from the default values obtained by calling + * the default constructor, the state is persisted (serialised and stored). + */ + override fun getState(): KexSettingsState { + return kexSettingsState + } + + /** + * Loads the state of the application-level settings. + * This method is called after the application-level settings component has been created + * and if the XML file with the state is changes externally. + */ + override fun loadState(state: KexSettingsState) { + kexSettingsState = state + } + + /** + * Returns the service object with a static call. + */ + + companion object { + fun service(project: Project) = project.getService(KexSettingsService::class.java).state + } +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsComponent.kt b/src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsComponent.kt new file mode 100644 index 000000000..9279ccb48 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsComponent.kt @@ -0,0 +1,91 @@ +package org.jetbrains.research.testspark.settings.kex + +import com.intellij.ide.ui.UINumericRange +import com.intellij.openapi.project.Project +import com.intellij.openapi.ui.ComboBox +import com.intellij.ui.JBIntSpinner +import com.intellij.ui.components.JBLabel +import com.intellij.util.ui.FormBuilder +import org.jdesktop.swingx.JXTitledSeparator +import org.jetbrains.research.testspark.bundles.kex.KexLabelsBundle +import org.jetbrains.research.testspark.bundles.kex.KexSettingsBundle +import org.jetbrains.research.testspark.data.kex.KexMode +import org.jetbrains.research.testspark.services.KexSettingsService +import org.jetbrains.research.testspark.settings.template.SettingsComponent +import javax.swing.JPanel +import javax.swing.JTextField + +/** + * This class displays and captures changes to the values of the Settings entries. + */ +class KexSettingsComponent(private val project: Project) : SettingsComponent { + var panel: JPanel? = null + private val kexSettingsState: KexSettingsState + get() = project.getService(KexSettingsService::class.java).state + + private var kexVersionTextField = JTextField() + private var kexPathTextField = JTextField() + private var kexModeSelector = ComboBox(KexMode.entries.toTypedArray()) + private var optionTextField = JTextField() // option triples which override options in kex.ini (see Kex readme) + + // maximum test cases returned by kex when minimization is enabled (enabled by default) + private var maxTestsField = JBIntSpinner(UINumericRange(kexSettingsState.maxTests, 1, Integer.MAX_VALUE)) + private var timeLimitField = JBIntSpinner(UINumericRange(kexSettingsState.timeLimit.inWholeSeconds.toInt(), 1, Integer.MAX_VALUE)) + + var kexVersion: String + get() = kexVersionTextField.text + set(x) { + kexVersionTextField.text = x + } + + var kexPath: String + get() = kexPathTextField.text + set(x) { + kexPathTextField.text = x + } + + var kexMode: KexMode + get() = kexModeSelector.item + set(x) { + kexModeSelector.item = x + } + + var option: String + get() = optionTextField.text + set(x) { + optionTextField.text = x + } + + var maxTests: Int + get() = maxTestsField.number + set(x) { + maxTestsField.number = x + } + + var timeLimit: Int + get() = timeLimitField.number + set(x) { + timeLimitField.number = x + } + + init { + super.initComponent() + } + + override fun stylizePanel() { + kexPathTextField.toolTipText = KexSettingsBundle.get("kexHome") + } + override fun createSettingsPanel() { + panel = FormBuilder.createFormBuilder() + .addComponent(JXTitledSeparator(KexLabelsBundle.get("kexSettings"))) + .addLabeledComponent(JBLabel(KexLabelsBundle.get("kexMode")), kexModeSelector, 10, false) + .addLabeledComponent(JBLabel(KexLabelsBundle.get("maxTests")), maxTestsField, 10, false) + .addLabeledComponent(JBLabel(KexLabelsBundle.get("timeLimit")), timeLimitField, 10, false) + .addLabeledComponent(JBLabel(KexLabelsBundle.get("option")), optionTextField, 10, false) + .addLabeledComponent(JBLabel(KexLabelsBundle.get("kexHome")), kexPathTextField, 10, false) + .addLabeledComponent(JBLabel(KexLabelsBundle.get("kexVersion")), kexVersionTextField, 10, false) + .panel + } + + override fun addListeners() {} +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsConfigurable.kt b/src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsConfigurable.kt new file mode 100644 index 000000000..b057a07a1 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsConfigurable.kt @@ -0,0 +1,78 @@ +package org.jetbrains.research.testspark.settings.kex + +import com.intellij.openapi.project.Project +import org.jetbrains.research.testspark.services.KexSettingsService +import org.jetbrains.research.testspark.settings.template.SettingsConfigurable +import javax.swing.JComponent +import kotlin.time.Duration.Companion.seconds + +class KexSettingsConfigurable(private val project: Project) : SettingsConfigurable { + private val kexSettingsState: KexSettingsState + get() = project.getService(KexSettingsService::class.java).state + + var settingsComponent: KexSettingsComponent? = null + + /** + * Creates a settings component that holds the panel with the settings entries, and returns this panel + * + * @return the panel used for displaying settings + */ + override fun createComponent(): JComponent? { + settingsComponent = KexSettingsComponent(project) + return settingsComponent!!.panel + } + + /** + * Sets the stored state values to the corresponding UI components. This method is called immediately after `createComponent` method. + */ + override fun reset() { + settingsComponent!!.kexVersion = kexSettingsState.kexVersion + settingsComponent!!.kexPath = kexSettingsState.kexHome + settingsComponent!!.kexMode = kexSettingsState.kexMode + settingsComponent!!.option = kexSettingsState.otherOptions + settingsComponent!!.maxTests = kexSettingsState.maxTests + settingsComponent!!.timeLimit = kexSettingsState.timeLimit.inWholeSeconds.toInt() + } + + /** + * Checks if the values of the entries in the settings state are different from the persisted values of these entries. + * + * @return whether any setting has been modified + */ + override fun isModified(): Boolean { + return settingsComponent!!.kexVersion != kexSettingsState.kexVersion || + settingsComponent!!.kexPath != kexSettingsState.kexHome || + settingsComponent!!.kexMode != kexSettingsState.kexMode || + settingsComponent!!.option != kexSettingsState.otherOptions || + settingsComponent!!.maxTests != kexSettingsState.maxTests || + settingsComponent!!.timeLimit != kexSettingsState.timeLimit.inWholeSeconds.toInt() + } + + /** + * Persists the modified state after a user hit Apply button. + */ + override fun apply() { + kexSettingsState.kexVersion = settingsComponent!!.kexVersion + kexSettingsState.kexHome = settingsComponent!!.kexPath + kexSettingsState.kexMode = settingsComponent!!.kexMode + kexSettingsState.otherOptions = settingsComponent!!.option + kexSettingsState.maxTests = settingsComponent!!.maxTests + kexSettingsState.timeLimit = settingsComponent!!.timeLimit.seconds + } + + /** + * Returns the displayed name of the Settings tab. + * + * @return the name displayed in the menu (settings) + */ + override fun getDisplayName(): String { + return "TestSpark" + } + + /** + * Disposes the UI resources. It is called when a user closes the Settings dialog. + */ + override fun disposeUIResources() { + settingsComponent = null + } +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsState.kt b/src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsState.kt new file mode 100644 index 000000000..9da06ee32 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/settings/kex/KexSettingsState.kt @@ -0,0 +1,31 @@ +package org.jetbrains.research.testspark.settings.kex + +import org.jetbrains.research.testspark.bundles.kex.KexDefaultsBundle +import org.jetbrains.research.testspark.data.kex.KexMode +import kotlin.time.Duration +import kotlin.time.Duration.Companion.seconds + +/** + * This class is the actual data class that stores the values of the Kex Settings entries. + */ +data class KexSettingsState( + var kexVersion: String = DefaultKexSettingsState.kexVersion, + var kexHome: String = DefaultKexSettingsState.kexHome, + var kexMode: KexMode = DefaultKexSettingsState.kexMode, + var otherOptions: String = DefaultKexSettingsState.otherOptions, + var timeLimit: Duration = DefaultKexSettingsState.timeLimit, + var maxTests: Int = DefaultKexSettingsState.maxTests, +) { + + /** + * Default values of SettingsKexState. + */ + object DefaultKexSettingsState { + val kexVersion: String = KexDefaultsBundle.get("kexVersion") + val kexHome: String = KexDefaultsBundle.get("kexHome") + val kexMode: KexMode = KexMode.Concolic + val otherOptions: String = KexDefaultsBundle.get("otherOptions") + val timeLimit: Duration = KexDefaultsBundle.get("timeLimit").toInt().seconds + val maxTests = KexDefaultsBundle.get("maxTests").toInt() + } +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/GenerationTool.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/GenerationTool.kt index 38dcceb68..76e053ce1 100644 --- a/src/main/kotlin/org/jetbrains/research/testspark/tools/GenerationTool.kt +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/GenerationTool.kt @@ -6,6 +6,7 @@ package org.jetbrains.research.testspark.tools enum class GenerationTool(val toolId: String) { EvoSuite("EvoSuite"), LLM("LLM"), + Kex("Kex"), ; companion object { diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/ToolUtils.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/ToolUtils.kt index 226fff5ba..d8047dd3d 100644 --- a/src/main/kotlin/org/jetbrains/research/testspark/tools/ToolUtils.kt +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/ToolUtils.kt @@ -11,6 +11,7 @@ import org.jetbrains.research.testspark.core.generation.llm.getClassWithTestCase import org.jetbrains.research.testspark.core.monitor.ErrorMonitor import org.jetbrains.research.testspark.core.progress.CustomProgressIndicator import org.jetbrains.research.testspark.core.test.SupportedLanguage +import org.jetbrains.research.testspark.core.utils.CommandLineRunner import org.jetbrains.research.testspark.core.utils.DataFilesUtil import org.jetbrains.research.testspark.data.IJTestCase import org.jetbrains.research.testspark.testmanager.TestGeneratorFactory @@ -20,6 +21,13 @@ object ToolUtils { val sep = File.separatorChar val pathSep = File.pathSeparatorChar + /** + * Concatenate strings with OS specific path seperators + */ + fun osJoin(vararg strings: String): String { + return strings.joinToString(sep.toString()) + } + /** * Saves the data related to test generation in the specified project's workspace. * @@ -100,6 +108,16 @@ object ToolUtils { return buildPath } + /** + * @param path path of the java executable as String. Refactor to type Path eventually + * @return version of java which is used in the project for which tests are generated by TestSpark + */ + fun getJavaVersion(path: String): Int? { + return """(1.)?(\d+)""".toRegex().find(CommandLineRunner.run(arrayListOf(path, "-version")).executionMessage)?.let { + it.groupValues[2].toInt() + } + } + /** * Checks if the process has been stopped. * diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/evosuite/EvoSuite.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/evosuite/EvoSuite.kt index 7539e4d8a..1236d087f 100644 --- a/src/main/kotlin/org/jetbrains/research/testspark/tools/evosuite/EvoSuite.kt +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/evosuite/EvoSuite.kt @@ -5,6 +5,7 @@ import com.intellij.openapi.diagnostic.Logger import com.intellij.openapi.project.Project import com.intellij.openapi.roots.ProjectRootManager import org.jetbrains.research.testspark.actions.controllers.TestGenerationController +import org.jetbrains.research.testspark.core.test.SupportedLanguage import org.jetbrains.research.testspark.core.test.data.CodeType import org.jetbrains.research.testspark.data.FragmentToTestData import org.jetbrains.research.testspark.display.TestSparkDisplayManager @@ -151,6 +152,11 @@ class EvoSuite(override val name: String = "EvoSuite") : Tool { ) } + override fun appliedForLanguage(language: SupportedLanguage): Boolean { + // EvoSuite is a Java test generation tool + return language == SupportedLanguage.Java + } + /** * Creates a pipeline object for the given project, psiFile, caret, and fileUrl. * The packageName is determined based on the projectClassPath and the buildPath from the project settings. diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/evosuite/generation/EvoSuiteProcessManager.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/evosuite/generation/EvoSuiteProcessManager.kt index d4fbd12d0..fc7ec61e6 100644 --- a/src/main/kotlin/org/jetbrains/research/testspark/tools/evosuite/generation/EvoSuiteProcessManager.kt +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/evosuite/generation/EvoSuiteProcessManager.kt @@ -21,7 +21,6 @@ import org.jetbrains.research.testspark.core.monitor.ErrorMonitor import org.jetbrains.research.testspark.core.progress.CustomProgressIndicator import org.jetbrains.research.testspark.core.test.SupportedLanguage import org.jetbrains.research.testspark.core.test.data.CodeType -import org.jetbrains.research.testspark.core.utils.CommandLineRunner import org.jetbrains.research.testspark.data.FragmentToTestData import org.jetbrains.research.testspark.data.IJReport import org.jetbrains.research.testspark.data.ProjectContext @@ -83,33 +82,7 @@ class EvoSuiteProcessManager( try { if (ToolUtils.isProcessStopped(errorMonitor, indicator)) return null - val regex = Regex("version \"(.*?)\"") - - // Resolving relative java path, if it's possible - if (evoSuiteSettingsState.javaPath.startsWith("~/")) { - val pathEvalCommandResult = - CommandLineRunner.run(arrayListOf("eval", "ls", evoSuiteSettingsState.javaPath)) - if (!pathEvalCommandResult.isSuccessful()) { - evoSuiteErrorManager.errorProcess( - EvoSuiteMessagesBundle.get("incorrectJavaVersion"), - project, - errorMonitor, - ) - return null - } - evoSuiteSettingsState.javaPath = pathEvalCommandResult.executionMessage.trim('\n') - } - - val versionCommandResult = CommandLineRunner.run(arrayListOf(evoSuiteSettingsState.javaPath, "-version")) - - log.info("Version command result: exit code '${versionCommandResult.exitCode}', message '${versionCommandResult.executionMessage}'") - val version = regex.find(versionCommandResult.executionMessage) - ?.groupValues - ?.get(1) - ?.split(".") - ?.get(0) - ?.toInt() - + val version = ToolUtils.getJavaVersion(evoSuiteSettingsState.javaPath) if (version == null || version > 11) { evoSuiteErrorManager.errorProcess(EvoSuiteMessagesBundle.get("incorrectJavaVersion"), project, errorMonitor) return null diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/Kex.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/Kex.kt new file mode 100644 index 000000000..bed6dc3c2 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/Kex.kt @@ -0,0 +1,180 @@ +package org.jetbrains.research.testspark.tools.kex + +import com.intellij.openapi.components.service +import com.intellij.openapi.diagnostic.Logger +import com.intellij.openapi.project.Project +import com.intellij.openapi.roots.ProjectRootManager +import org.jetbrains.research.testspark.actions.controllers.TestGenerationController +import org.jetbrains.research.testspark.core.test.SupportedLanguage +import org.jetbrains.research.testspark.core.test.data.CodeType +import org.jetbrains.research.testspark.data.FragmentToTestData +import org.jetbrains.research.testspark.display.TestSparkDisplayManager +import org.jetbrains.research.testspark.langwrappers.PsiHelper +import org.jetbrains.research.testspark.langwrappers.PsiMethodWrapper +import org.jetbrains.research.testspark.services.PluginSettingsService +import org.jetbrains.research.testspark.tools.Pipeline +import org.jetbrains.research.testspark.tools.TestsExecutionResultManager +import org.jetbrains.research.testspark.tools.ToolUtils +import org.jetbrains.research.testspark.tools.kex.generation.KexProcessManager +import org.jetbrains.research.testspark.tools.template.Tool + +class Kex(override val name: String = "Kex") : Tool { + private val log = Logger.getInstance(this::class.java) + + /** + * Returns a new instance of KexProcessManager for the given project. + * + * @param project The IntelliJ IDEA project for which the KexProcessManager is created. + * @return The KexProcessManager instance created for the given project. + */ + private fun getKexProcessManager(project: Project): KexProcessManager { + val projectClassPath: String = ProjectRootManager.getInstance(project).contentRoots.first().path + val settingsProjectState = project.service().state + val buildPath = ToolUtils.osJoin(projectClassPath, settingsProjectState.buildPath) + return KexProcessManager(project, buildPath) + } + + /** + * Generates tests for a class using Kex. + * + * @param project The current project. + * @param psiFile The PsiFile containing the class to generate tests for. + * @param caret The caret position within the PsiFile. + * @param fileUrl The URL of the file being tested. + * @param testSamplesCode The code to be used as test samples. + */ + override fun generateTestsForClass( + project: Project, + psiHelper: PsiHelper, + caretOffset: Int, + fileUrl: String?, + testSamplesCode: String, + testGenerationController: TestGenerationController, + testSparkDisplayManager: TestSparkDisplayManager, + testsExecutionResultManager: TestsExecutionResultManager, + ) { + log.info("Starting tests generation for class by Kex") + createPipeline( + project, + psiHelper, + caretOffset, + fileUrl, + testGenerationController, + testSparkDisplayManager, + testsExecutionResultManager, + ).runTestGeneration( + getKexProcessManager(project), + FragmentToTestData( + CodeType.CLASS, + ), + ) + } + + override fun generateTestsForMethod( + project: Project, + psiHelper: PsiHelper, + caretOffset: Int, + fileUrl: String?, + testSamplesCode: String, + testGenerationController: TestGenerationController, + testSparkDisplayManager: TestSparkDisplayManager, + testsExecutionResultManager: TestsExecutionResultManager, + ) { + log.info("Starting tests generation for method by Kex") + val psiMethod: PsiMethodWrapper = psiHelper.getSurroundingMethod(caretOffset)!! + createPipeline( + project, + psiHelper, + caretOffset, + fileUrl, + testGenerationController, + testSparkDisplayManager, + testsExecutionResultManager, + ).runTestGeneration( + getKexProcessManager(project), + FragmentToTestData( + CodeType.METHOD, + // remove generics due to type erasure at jvm and the bool is a strange requirement by kex + removeGenerics("${psiMethod.name}(${psiMethod.parameterTypes.joinToString(",")}):${psiMethod.returnType}").replace( + "boolean", + "bool", + ), + ), + ) + } + + override fun generateTestsForLine( + project: Project, + psiHelper: PsiHelper, + caretOffset: Int, + fileUrl: String?, + testSamplesCode: String, + testGenerationController: TestGenerationController, + testSparkDisplayManager: TestSparkDisplayManager, + testsExecutionResultManager: TestsExecutionResultManager, + ) { + } + + override fun appliedForLanguage(language: SupportedLanguage): Boolean { + // Kex is a Java test generation tool + return language == SupportedLanguage.Java + } + + /** + * Creates a pipeline object for the given project, psiFile, caret, and fileUrl. + * The packageName is determined based on the projectClassPath and the buildPath from the project settings. + * + * @param project The project for which to create the pipeline. + * @param psiFile The psiFile associated with the pipeline. + * @param caret The caret position in the psiFile. + * @param fileUrl The URL of the file associated with the pipeline. Can be null. + * @return The created pipeline object. + */ + + private fun createPipeline( + project: Project, + psiHelper: PsiHelper, + caretOffset: Int, + fileUrl: String?, + testGenerationController: TestGenerationController, + testSparkDisplayManager: TestSparkDisplayManager, + testsExecutionResultManager: TestsExecutionResultManager, + ): Pipeline { + val projectClassPath: String = ProjectRootManager.getInstance(project).contentRoots.first().path + + val settingsProjectState = project.service().state + val packageName = "$projectClassPath/${settingsProjectState.buildPath}" + + return Pipeline( + project, + psiHelper, + caretOffset, + fileUrl, + packageName, + testGenerationController, + testSparkDisplayManager, + testsExecutionResultManager, + name, + ) + } + + /** + * Removes the generic type arguments from a string. Any characters between angle brackets (<>) are removed, + * along with the angle brackets themselves. The resulting string does not contain any generic type information. + * + * @receiver The string from which to remove the generic type arguments. + * @return The resulting string with generic type arguments removed. + */ + private fun removeGenerics(typeString: String): String { + val s = StringBuilder() + var stack: Int = 0 + for (c in typeString) { + if (c == '<') { + ++stack + } else if (c == '>') { + --stack + } else if (stack == 0) s.append(c) + } + return s.toString() + } +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/KexSettingsArguments.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/KexSettingsArguments.kt new file mode 100644 index 000000000..cf1162a8a --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/KexSettingsArguments.kt @@ -0,0 +1,116 @@ +package org.jetbrains.research.testspark.tools.kex + +import com.intellij.openapi.module.Module +import com.intellij.openapi.roots.CompilerModuleExtension +import com.intellij.openapi.roots.ModuleRootManager +import com.intellij.openapi.vfs.VfsUtil +import org.jetbrains.research.testspark.bundles.kex.KexDefaultsBundle +import org.jetbrains.research.testspark.settings.kex.KexSettingsState +import org.jetbrains.research.testspark.tools.kex.generation.GeneratedTestsProcessor +import java.io.File +import java.nio.file.Path +import java.nio.file.Paths + +class KexSettingsArguments( + private val javaExecPath: String, + private val javaVersion: Int, + private val module: Module, + private val target: String, + private val resultName: String, + private val kexSettingsState: KexSettingsState, + private val kexExecPath: String, + private val kexHome: String, +) { + fun buildCommand(): MutableList { + return mutableListOf(javaExecPath) + .setJvmProperties() + .setAddOpensArgs(javaVersion) + .setKexParams() + .setKexIniOptions() + } + + private fun MutableList.setJvmProperties(): MutableList { + return this.addAll( + "-Xmx${KexDefaultsBundle.get("heapSize")}g", + "-Djava.security.manager", + "-Djava.security.policy==$kexHome/kex.policy", + "-Dlogback.statusListenerClass=ch.qos.logback.core.status.NopStatusListener", + ) + } + + private fun MutableList.setAddOpensArgs(javaVersion: Int): MutableList { + if (javaVersion > 8) { + File("$kexHome/runtime-deps/modules.info").readLines().forEach { this.addAll("--add-opens", it) } + } + return this + } + + /** + * @see kex readme + */ + private fun MutableList.setKexParams(): MutableList { + return this.addAll( + "-jar", + kexExecPath, + "--classpath", + getBuildOutputDirectory(module)!!.toString(), + "--target", + "\"$target\"", + "--output", + resultName, + "--mode", + kexSettingsState.kexMode.toString(), + ) + } + + /** + * Kex uses a configuration file kex.ini. + * This can effectively be modified on an invocation through the --option cmd line arg + */ + private fun MutableList.setKexIniOptions(): MutableList { + // Add options provided with help of settings ui + val optionStrings = mutableListOf( + listOf("kex", "minimizeTestSuite", KexDefaultsBundle.get("minimizeTestSuite")), + listOf("testGen", "maxTests", kexSettingsState.maxTests.toString()), + listOf("concolic", "timeLimit", "${kexSettingsState.timeLimit.inWholeSeconds}"), + listOf("symbolic", "timeLimit", "${kexSettingsState.timeLimit.inWholeSeconds}"), + listOf("testGen", "defaultPackageName", GeneratedTestsProcessor.DEFAULT_PACKAGE_NAME), + ) + .map { it.joinToString(":") } + .toMutableList() + + // adding explicitly provided user option + // break into a list of options if multiple are provided + if (kexSettingsState.otherOptions.isNotBlank()) { + optionStrings.addAll(kexSettingsState.otherOptions.splitToSequence(' ')) + } + + // add --option before every option + val separator = "--option" + val interspersed = optionStrings.fold(mutableListOf()) { acc, item -> + acc.add(separator) + acc.add(item) + acc + } + this.addAll(interspersed) + return this + } + + /** + * Retrieves the build output directory for the given module. + * kex works correctly even with multimodule projects + * + * @param module The module for which to retrieve the build output directory. + * @return The path to the build output directory, or null if it cannot be determined. + */ + private fun getBuildOutputDirectory(module: Module): Path? { + val moduleRootManager = ModuleRootManager.getInstance(module) + val compilerProjectExtension = moduleRootManager.getModuleExtension(CompilerModuleExtension::class.java) + return compilerProjectExtension?.compilerOutputUrl?.let { Paths.get(VfsUtil.urlToPath(it)) } + } +} + +private fun MutableList.addAll(vararg elems: E): MutableList { + this.addAll(elems) + return this +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/error/KexErrorManager.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/error/KexErrorManager.kt new file mode 100644 index 000000000..52bb38f88 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/error/KexErrorManager.kt @@ -0,0 +1,60 @@ +package org.jetbrains.research.testspark.tools.kex.error + +import com.intellij.execution.process.OSProcessHandler +import com.intellij.notification.NotificationGroupManager +import com.intellij.notification.NotificationType +import com.intellij.openapi.diagnostic.Logger +import com.intellij.openapi.project.Project +import org.jetbrains.research.testspark.bundles.kex.KexMessagesBundle +import org.jetbrains.research.testspark.bundles.plugin.PluginMessagesBundle +import org.jetbrains.research.testspark.core.monitor.ErrorMonitor +import org.jetbrains.research.testspark.tools.template.error.ErrorManager + +class KexErrorManager : ErrorManager { + + private var output = StringBuilder() + + fun addLineToKexOutput(line: String) { + output.append(line + "\n") + } + + private val log = Logger.getInstance(this::class.java) + override fun errorProcess(message: String, project: Project, errorMonitor: ErrorMonitor) { + if (errorMonitor.notifyErrorOccurrence()) { + log.warn("Error in Test Generation: $message") + NotificationGroupManager.getInstance() + .getNotificationGroup("LLM Execution Error") + .createNotification( + PluginMessagesBundle.get("kexErrorTitle"), + message, + NotificationType.ERROR, + ) + .notify(project) + } + } + + override fun warningProcess(message: String, project: Project) { + log.warn("Error in Test Generation: $message") + NotificationGroupManager.getInstance() + .getNotificationGroup("Kex Execution Error") + .createNotification( + PluginMessagesBundle.get("kexWarningTitle"), + message, + NotificationType.WARNING, + ) + .notify(project) + } + + fun isProcessCorrect( + handler: OSProcessHandler, + project: Project, + errorMonitor: ErrorMonitor, + ): Boolean { + // check for non-zero exit code + if (handler.exitCode != 0) { + errorProcess("${KexMessagesBundle.get("nonZeroExitCode")}\n$output", project, errorMonitor) + return false + } + return true + } +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/generation/GeneratedTestsProcessor.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/generation/GeneratedTestsProcessor.kt new file mode 100644 index 000000000..5d7de0d81 --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/generation/GeneratedTestsProcessor.kt @@ -0,0 +1,143 @@ +package org.jetbrains.research.testspark.tools.kex.generation + +import com.github.javaparser.StaticJavaParser +import com.github.javaparser.ast.NodeList +import com.github.javaparser.ast.body.FieldDeclaration +import com.github.javaparser.ast.body.MethodDeclaration +import com.github.javaparser.ast.stmt.BlockStmt +import com.github.javaparser.ast.stmt.Statement +import com.intellij.openapi.project.Project +import org.jetbrains.research.testspark.bundles.kex.KexMessagesBundle +import org.jetbrains.research.testspark.core.data.TestCase +import org.jetbrains.research.testspark.core.data.TestGenerationData +import org.jetbrains.research.testspark.core.generation.llm.getImportsCodeFromTestSuiteCode +import org.jetbrains.research.testspark.core.monitor.ErrorMonitor +import org.jetbrains.research.testspark.data.IJReport +import org.jetbrains.research.testspark.data.ProjectContext +import org.jetbrains.research.testspark.tools.TestsExecutionResultManager +import org.jetbrains.research.testspark.tools.ToolUtils +import org.jetbrains.research.testspark.tools.kex.error.KexErrorManager +import java.io.File + +class GeneratedTestsProcessor( + private val project: Project, + private val errorMonitor: ErrorMonitor, + private val kexErrorManager: KexErrorManager, + private val testsExecutionResultManager: TestsExecutionResultManager, +) { + + companion object { + const val EQUALITY_UTILS = "EqualityUtils" + const val REFLECTION_UTILS = "ReflectionUtils" + + // can be any arbitrary name. used when no package is provided + const val DEFAULT_PACKAGE_NAME = "example" + } + + fun process( + resultName: String, + classFQN: String, + generatedTestsData: TestGenerationData, + projectContext: ProjectContext, + ) { + val report = IJReport() + val imports = mutableSetOf() + val packageStr = classFQN.substringBeforeLast('.', missingDelimiterValue = DEFAULT_PACKAGE_NAME) + + val generatedTestsDir = File( + ToolUtils.osJoin(resultName, "tests", packageStr.replace('.', '/')), + ) + if (generatedTestsDir.exists() && generatedTestsDir.isDirectory) { // collect all generated tests into a report + for ((index, file) in generatedTestsDir.listFiles()!!.withIndex()) { + val testCode = file.readText() + + if (file.name.equals("$EQUALITY_UTILS.java") || file.name.equals("$REFLECTION_UTILS.java")) { + // collecting all methods + generatedTestsData.otherInfo += "${getHelperClassBody(testCode)}\n" + } else { + // Example.java -> example. example() is the test method name and testcase name expected by TestSpark + val testCaseName = file.name.substringBefore('.').replaceFirstChar { it.lowercaseChar() } + // merge @before and @test annotated methods into a single method + val testMethod = extractTestMethod(file, testCaseName) ?: continue + report.testCaseList[index] = + TestCase( + index, + testCaseName, + testMethod.toString(), + setOf(), + ) + } + // extracting just imports out of the test code + // remove any imports referring to the helper classes + imports.addAll( + getImportsCodeFromTestSuiteCode(testCode, projectContext.classFQN!!) + .filterNot { it.contains(EQUALITY_UTILS) || it.contains(REFLECTION_UTILS) }, + ) + } + } else { + kexErrorManager.errorProcess( + KexMessagesBundle.get("testsDontExist"), + project, + errorMonitor, + ) + } + + ToolUtils.transferToIJTestCases(report) + ToolUtils.saveData( + project, + report, + packageStr, + imports, + projectContext.fileUrlAsString!!, + generatedTestsData, + testsExecutionResultManager, + ) + } + + /** + * Precondition: there is only one class or interface per helper file (ie, EqualityUtils or ReflectionUtils) + */ + private fun getHelperClassBody(testCode: String) = + testCode.substringAfter('{').substringBeforeLast('}') + + /** + * Merge the fields, @Before, @Test into a single @Test annotated method + */ + private fun extractTestMethod(file: File, testCaseName: String): MethodDeclaration? { + val compilationUnit = StaticJavaParser.parse(file) + val methods = compilationUnit.findAll(MethodDeclaration::class.java) + val beforeAnnMethod = methods.getMethodWithAnnotation("@Before", file.name) ?: return null + val testAnnMethod = methods.getMethodWithAnnotation("@Test", file.name) ?: return null + + val collatedMethodBody = NodeList() + + // convert fields to local variables + collatedMethodBody.addAll( + compilationUnit.findAll(FieldDeclaration::class.java) + .filter { it.annotations.isEmpty() } // drops the unwanted timeout field + .map { it.setModifiers(NodeList()) } // remove all modifiers (including access specifiers like public and private) + .map { it.toString() } + .map { StaticJavaParser.parseStatement(it) }, + ) + collatedMethodBody.addAll(beforeAnnMethod.statements()) + collatedMethodBody.addAll(testAnnMethod.statements()) + testAnnMethod.setBody(BlockStmt(collatedMethodBody)) + testAnnMethod.setName(testCaseName) + return testAnnMethod + } + + private fun MethodDeclaration.statements(): NodeList? = + body.map { it.statements }.orElse(NodeList()) + + private fun MutableList.getMethodWithAnnotation(ann: String, fileName: String): MethodDeclaration? = + firstOrNull { it.annotations.map { it.toString() }.contains(ann) }.apply { + if (this == null) { + kexErrorManager.warningProcess( + """${KexMessagesBundle.get("unexpectedGeneratedTestStructure")} + There is no $ann annotated method in file $fileName + """.trimMargin(), + project, + ) + } + } +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/generation/KexProcessManager.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/generation/KexProcessManager.kt new file mode 100644 index 000000000..ead7a4aae --- /dev/null +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/kex/generation/KexProcessManager.kt @@ -0,0 +1,254 @@ +package org.jetbrains.research.testspark.tools.kex.generation + +import com.intellij.execution.configurations.GeneralCommandLine +import com.intellij.execution.process.OSProcessHandler +import com.intellij.execution.process.ProcessAdapter +import com.intellij.execution.process.ProcessEvent +import com.intellij.execution.process.ProcessOutput +import com.intellij.execution.process.ProcessTerminatedListener +import com.intellij.openapi.diagnostic.Logger +import com.intellij.openapi.project.Project +import com.intellij.openapi.roots.ProjectRootManager +import com.intellij.openapi.util.Key +import org.jetbrains.research.testspark.bundles.kex.KexDefaultsBundle +import org.jetbrains.research.testspark.bundles.kex.KexMessagesBundle +import org.jetbrains.research.testspark.bundles.plugin.PluginMessagesBundle +import org.jetbrains.research.testspark.core.data.TestGenerationData +import org.jetbrains.research.testspark.core.monitor.ErrorMonitor +import org.jetbrains.research.testspark.core.progress.CustomProgressIndicator +import org.jetbrains.research.testspark.core.test.data.CodeType +import org.jetbrains.research.testspark.data.FragmentToTestData +import org.jetbrains.research.testspark.data.ProjectContext +import org.jetbrains.research.testspark.data.UIContext +import org.jetbrains.research.testspark.services.KexSettingsService +import org.jetbrains.research.testspark.settings.kex.KexSettingsState +import org.jetbrains.research.testspark.tools.TestsExecutionResultManager +import org.jetbrains.research.testspark.tools.ToolUtils +import org.jetbrains.research.testspark.tools.kex.KexSettingsArguments +import org.jetbrains.research.testspark.tools.kex.error.KexErrorManager +import org.jetbrains.research.testspark.tools.llm.generation.StandardRequestManagerFactory +import org.jetbrains.research.testspark.tools.template.generation.ProcessManager +import java.io.File +import java.io.IOException +import java.net.URL +import java.nio.charset.Charset +import java.util.zip.ZipInputStream +import kotlin.io.path.Path +import kotlin.io.path.createDirectories + +class KexProcessManager( + private val project: Project, + private val projectPath: String, +) : ProcessManager { + + private val kexErrorManager: KexErrorManager = KexErrorManager() + private val log = Logger.getInstance(this::class.java) + + private val kexVersion = kexSettingsState.kexVersion + private val kexUrl = "${KexDefaultsBundle.get("kexUrlBase")}$kexVersion/kex-$kexVersion.zip" + private var kexHome: String = kexSettingsState.kexHome + + init { + // use default cache location if not explicitly provided + if (kexHome.isBlank()) { + val userHome = System.getProperty("user.home") + val kexHomeFile = when { + // On Windows, use the LOCALAPPDATA environment variable + // TODO windows stuff is untested + System.getProperty("os.name").startsWith("Windows") -> System.getenv("LOCALAPPDATA") + ?.let { File(it, ToolUtils.osJoin("JetBrains", "TestSpark", "kex")) } + // On Unix-like systems, use the ~/.cache directory + else -> File(userHome, ToolUtils.osJoin(".cache", "JetBrains", "TestSpark", "kex")) + } + // Ensure the cache directory exists + if (kexHomeFile != null && !kexHomeFile.exists()) { + kexHomeFile.mkdirs() + } + kexHome = kexHomeFile.toString() + } + } + private val kexSettingsState: KexSettingsState + get() = project.getService(KexSettingsService::class.java).state + private var kexExecPath = + ToolUtils.osJoin(kexHome, "kex-runner", "target", "kex-runner-$kexVersion-jar-with-dependencies.jar") + + override fun runTestGenerator( + indicator: CustomProgressIndicator, + codeType: FragmentToTestData, + packageName: String, + projectContext: ProjectContext, + generatedTestsData: TestGenerationData, + errorMonitor: ErrorMonitor, + testsExecutionResultManager: TestsExecutionResultManager, + ): UIContext? { + try { + if (ToolUtils.isProcessStopped(errorMonitor, indicator)) return null + + val classFQN = projectContext.classFQN!! + val resultName = ToolUtils.osJoin(generatedTestsData.resultPath, "KexResult") + val projectSdk = ProjectRootManager.getInstance(project).projectSdk!! + val javaExecPath = ToolUtils.osJoin(projectSdk.homePath!!, "bin", "java") + + indicator.setText(PluginMessagesBundle.get("searchMessage")) + + // set target argument for kex subprocess. ensure not Line codetype which is unsupported + val target: String = when (codeType.type!!) { + CodeType.CLASS -> classFQN + CodeType.METHOD -> "$classFQN::${codeType.objectDescription}" + CodeType.LINE -> return null // This is impossible since this combination is disallowed in UI (see TestSparkAction.kt) + } + + // Disallow old java versions + val version = ToolUtils.getJavaVersion(javaExecPath) + if (version == null || version < 8) { + kexErrorManager.errorProcess(KexMessagesBundle.get("incorrectJavaVersion"), project, errorMonitor) + return null + } + Path(generatedTestsData.resultPath).createDirectories() + + // Download kex executable if not present + if (!ensureKexExists()) { + kexErrorManager.errorProcess(KexMessagesBundle.get("invalidUrl"), project, errorMonitor) + return null + } + + val cmd = KexSettingsArguments( + javaExecPath, + version, + projectContext.cutModule!!, + target, + resultName, + kexSettingsState, + kexExecPath, + kexHome, + ).buildCommand() + + val cmdString = cmd.fold(String()) { acc, e -> acc.plus(e).plus(" ") } + log.info("Starting Kex with arguments: $cmdString") + + // run kex as subprocess + if (!runKex(cmd, errorMonitor, indicator)) return null + + log.info("Save generated test suite and test cases into the project workspace") + + GeneratedTestsProcessor(project, errorMonitor, kexErrorManager, testsExecutionResultManager).process( + resultName, + classFQN, + generatedTestsData, + projectContext, + ) + } catch (e: Exception) { + kexErrorManager.errorProcess( + KexMessagesBundle.get("kexErrorCommon").format(e.message), + project, + errorMonitor, + ) + e.printStackTrace() + } + + return UIContext( + projectContext, + generatedTestsData, + StandardRequestManagerFactory(project).getRequestManager(project), + errorMonitor, + ) + } + + /** + * @return false if the subprocess didn't run successfully + */ + private fun runKex( + cmd: MutableList, + errorMonitor: ErrorMonitor, + indicator: CustomProgressIndicator, + ): Boolean { + try { + val kexProcess = GeneralCommandLine(cmd) + kexProcess.charset = Charset.forName("UTF-8") + kexProcess.workDirectory = File(kexHome) + kexProcess.environment["KEX_HOME"] = kexHome + + val handler = OSProcessHandler(kexProcess) + val output = ProcessOutput() + ProcessTerminatedListener.attach(handler) + + // rerouting stdout and stderr + handler.addProcessListener(object : ProcessAdapter() { + override fun onTextAvailable(event: ProcessEvent, outputType: Key<*>) { + if (ToolUtils.isProcessStopped(errorMonitor, indicator)) { + handler.destroyProcess() + return + } + kexErrorManager.addLineToKexOutput(event.text) + if (outputType.toString() == "stdout") { + output.appendStdout(event.text) + } else if (outputType.toString() == "stderr") { + output.appendStderr(event.text) + } + } + + override fun processTerminated(event: ProcessEvent) { + log.info("Process terminated with exit code: ${event.exitCode}") + log.info("Output from Kex stdout:\n ${output.stdout}") + log.info("Output from Kex stderr:\n ${output.stderr}") + } + }) + + handler.startNotify() + handler.waitFor() // no timeout provided since kex has builtin timeouts which we pass as args + return kexErrorManager.isProcessCorrect(handler, project, errorMonitor) + } catch (e: IOException) { + kexErrorManager.errorProcess( + KexMessagesBundle.get("kexErrorSubprocess").format(e.message), + project, + errorMonitor, + ) + e.printStackTrace() + log.error("Error running KEX process", e) + return false + } + } + + /** + * Ensures that the necessary files for Kex execution exist. If the files already exist, this method will return true. + * If any required file is missing, this method will attempt to download the necessary files. + * + * @return False if kex doesn't exist and files couldn't be downloaded + */ + private fun ensureKexExists(): Boolean { + val kexDir = File(kexHome) + if (!kexDir.exists()) { + kexDir.mkdirs() + } + + if (File(kexExecPath).exists()) { + log.info("Specified version found, skipping update") + return true + } + + log.info("Kex executable and helper files not found, downloading Kex") + val stream = + try { + URL(kexUrl).openStream() + } catch (e: Exception) { + log.error("Error fetching latest kex custom release - $e") + return false + } + + // this can fail unexpectedly if a file with the same name as a required directory exists + // inside the given kexHome path. + ZipInputStream(stream).use { zipInputStream -> + generateSequence { zipInputStream.nextEntry } + .filterNot { it.isDirectory } + .forEach { entry -> + val file = File("$kexHome/${entry.name}") + file.parentFile.mkdirs() // makes any directories required + file.outputStream().use { output -> + zipInputStream.copyTo(output) + } + } + } + log.info("Latest kex project successfully downloaded") + return true + } +} diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/llm/Llm.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/llm/Llm.kt index 068f433ef..214afe414 100644 --- a/src/main/kotlin/org/jetbrains/research/testspark/tools/llm/Llm.kt +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/llm/Llm.kt @@ -7,6 +7,7 @@ import com.intellij.openapi.project.Project import org.jetbrains.research.testspark.actions.controllers.TestGenerationController import org.jetbrains.research.testspark.bundles.plugin.PluginMessagesBundle import org.jetbrains.research.testspark.core.exception.TestSparkException +import org.jetbrains.research.testspark.core.test.SupportedLanguage import org.jetbrains.research.testspark.core.test.data.CodeType import org.jetbrains.research.testspark.data.FragmentToTestData import org.jetbrains.research.testspark.display.TestSparkDisplayManager @@ -183,6 +184,11 @@ class Llm(override val name: String = "LLM") : Tool { ) } + override fun appliedForLanguage(language: SupportedLanguage): Boolean { + // LLM test generation applied for all languages + return true + } + /** * Generates tests for a given code type. * diff --git a/src/main/kotlin/org/jetbrains/research/testspark/tools/template/Tool.kt b/src/main/kotlin/org/jetbrains/research/testspark/tools/template/Tool.kt index fc1127704..8f770cc7c 100644 --- a/src/main/kotlin/org/jetbrains/research/testspark/tools/template/Tool.kt +++ b/src/main/kotlin/org/jetbrains/research/testspark/tools/template/Tool.kt @@ -3,6 +3,7 @@ package org.jetbrains.research.testspark.tools.template import com.intellij.openapi.project.Project import com.intellij.psi.PsiFile import org.jetbrains.research.testspark.actions.controllers.TestGenerationController +import org.jetbrains.research.testspark.core.test.SupportedLanguage import org.jetbrains.research.testspark.display.TestSparkDisplayManager import org.jetbrains.research.testspark.langwrappers.PsiHelper import org.jetbrains.research.testspark.tools.TestsExecutionResultManager @@ -78,4 +79,12 @@ interface Tool { testSparkDisplayManager: TestSparkDisplayManager, testsExecutionResultManager: TestsExecutionResultManager, ) + + /** + * Checks whether the tool is applicable for the specified programming language. + * + * @param language The programming language to check, represented by an instance of SupportedLanguage. + * @return True if the tool is applicable for the provided language, false otherwise. + */ + fun appliedForLanguage(language: SupportedLanguage): Boolean } diff --git a/src/main/kotlin/org/jetbrains/research/testspark/toolwindow/DescriptionTab.kt b/src/main/kotlin/org/jetbrains/research/testspark/toolwindow/DescriptionTab.kt index 1718e3b5c..e647360e2 100644 --- a/src/main/kotlin/org/jetbrains/research/testspark/toolwindow/DescriptionTab.kt +++ b/src/main/kotlin/org/jetbrains/research/testspark/toolwindow/DescriptionTab.kt @@ -7,6 +7,7 @@ import com.intellij.util.ui.FormBuilder import org.jetbrains.research.testspark.bundles.plugin.PluginLabelsBundle import org.jetbrains.research.testspark.display.TestSparkIcons import org.jetbrains.research.testspark.settings.evosuite.EvoSuiteSettingsConfigurable +import org.jetbrains.research.testspark.settings.kex.KexSettingsConfigurable import org.jetbrains.research.testspark.settings.llm.LLMSettingsConfigurable import org.jetbrains.research.testspark.settings.plugin.PluginSettingsConfigurable import java.awt.Desktop @@ -58,6 +59,16 @@ class DescriptionTab(private val project: Project) { } } + private val testSparkKexDescription = JTextPane().apply { + isEditable = false + contentType = "text/html" + addHyperlinkListener { evt -> + if (HyperlinkEvent.EventType.ACTIVATED == evt.eventType) { + Desktop.getDesktop().browse(evt.url.toURI()) + } + } + } + private val testSparkDisclaimerDescription = JTextPane().apply { isEditable = false contentType = "text/html" @@ -74,6 +85,9 @@ class DescriptionTab(private val project: Project) { // Link to EvoSuite settings private val evoSuiteSettingsButton = JButton(PluginLabelsBundle.get("evoSuiteSettingsLink"), TestSparkIcons.settings) + // Link to Kex settings + private val kexSettingsButton = JButton(PluginLabelsBundle.get("kexSettingsLink"), TestSparkIcons.settings) + // Link to open settings private val settingsButton = JButton(PluginLabelsBundle.get("settingsLink"), TestSparkIcons.settings) @@ -107,6 +121,13 @@ class DescriptionTab(private val project: Project) { ShowSettingsUtil.getInstance().showSettingsDialog(project, EvoSuiteSettingsConfigurable::class.java) } + // Kex settings button setup + kexSettingsButton.isOpaque = false + kexSettingsButton.isContentAreaFilled = false + kexSettingsButton.addActionListener { + ShowSettingsUtil.getInstance().showSettingsDialog(project, KexSettingsConfigurable::class.java) + } + // Settings button setup settingsButton.isOpaque = false settingsButton.isContentAreaFilled = false @@ -135,6 +156,10 @@ class DescriptionTab(private val project: Project) { testSparkEvoSuiteDescription.isOpaque = false testSparkEvoSuiteDescription.text = getEvoSuiteDescriptionText(getContent().preferredSize.width) + // testSpark Kex description setup + testSparkKexDescription.isOpaque = false + testSparkKexDescription.text = getKexDescriptionText(getContent().preferredSize.width) + // testSpark disclaimer description setup testSparkDisclaimerDescription.isOpaque = false testSparkDisclaimerDescription.text = getDisclaimerText(getContent().preferredSize.width) @@ -153,6 +178,8 @@ class DescriptionTab(private val project: Project) { .addComponent(llmSettingsButton, 10) .addComponent(testSparkEvoSuiteDescription, 20) .addComponent(evoSuiteSettingsButton, 10) + .addComponent(testSparkKexDescription, 20) + .addComponent(kexSettingsButton, 10) .addComponent(testSparkDisclaimerDescription, 20) .addComponent(settingsButton, 10) .addComponent(documentationButton, 10) @@ -209,6 +236,19 @@ class DescriptionTab(private val project: Project) { "However, it only supports projects implemented by Java versions 8 to 11." } + /** + * Returns the descriptive text for Kex test generation. + * + * @param width The width of the text body in pixels. + * @return The formatted HTML string containing the description of test generation through symbolic execution using Kex. + */ + private fun getKexDescriptionText(width: Int): String { + return "" + + "Symbolic Execution-based test generation

    " + + "Uses Kex. You can generate tests with this tool locally.
    " + + "However, it only supports projects implemented by Java versions 8 and upwards.    " + } + /** * Returns the disclaimer text based on the given width. * diff --git a/src/main/resources/META-INF/plugin.xml b/src/main/resources/META-INF/plugin.xml index f1655eb08..219708171 100644 --- a/src/main/resources/META-INF/plugin.xml +++ b/src/main/resources/META-INF/plugin.xml @@ -210,6 +210,8 @@ serviceImplementation="org.jetbrains.research.testspark.services.PluginSettingsService"/> + + + ) +maxTests=Max test cases to display (higher coverage tests prioritised) +timeLimit=Timeout in seconds +kexVersion=A valid newer version tag of newer github release \ No newline at end of file diff --git a/src/main/resources/properties/kex/KexMessages.properties b/src/main/resources/properties/kex/KexMessages.properties new file mode 100644 index 000000000..cd7cb290c --- /dev/null +++ b/src/main/resources/properties/kex/KexMessages.properties @@ -0,0 +1,7 @@ +kexErrorCommon=An Error occurred in Kex based test generation +kexErrorSubprocess=An Error occurred while running the Kex subprocess +testsDontExist=Generated tests don't exist. Must have had a problem running Kex +incorrectJavaVersion=Java version must be 8 or greater +invalidUrl=Error downloading kex executable. Perhaps our url may be invalid or server may be down. You may try deleting ~/.cache/JetBrains/TestSpark/kex (linux/macos only) or downloading kex manually and setting path in settings +nonZeroExitCode=Kex Subprocess failed with a non-zero exit code +unexpectedGeneratedTestStructure=Some or all generated tests did not have the expected file structure and could not be used. Please report the error to developers. \ No newline at end of file diff --git a/src/main/resources/properties/kex/KexSettings.properties b/src/main/resources/properties/kex/KexSettings.properties new file mode 100644 index 000000000..bd7ee9d5f --- /dev/null +++ b/src/main/resources/properties/kex/KexSettings.properties @@ -0,0 +1 @@ +kexHome=Provide the path to directory to where kex should be downloaded diff --git a/src/main/resources/properties/plugin/PluginLabels.properties b/src/main/resources/properties/plugin/PluginLabels.properties index 07c77a161..5b7d51dcc 100644 --- a/src/main/resources/properties/plugin/PluginLabels.properties +++ b/src/main/resources/properties/plugin/PluginLabels.properties @@ -9,6 +9,7 @@ showCoverageDescription=Plugin Setting: Coverage Visualisation settingsLink=Go to Advanced Settings llmSettingsLink=Go to LLM Settings evoSuiteSettingsLink=Go to EvoSuite Settings +kexSettingsLink=Go to Kex Settings documentationLink=Go to Documentation quickAccess=TestSpark Plugin !TestSpark Window diff --git a/src/main/resources/properties/plugin/PluginMessages.properties b/src/main/resources/properties/plugin/PluginMessages.properties index 1ba7b008d..4958f3e8a 100644 --- a/src/main/resources/properties/plugin/PluginMessages.properties +++ b/src/main/resources/properties/plugin/PluginMessages.properties @@ -9,6 +9,7 @@ collectingClassesToTest=Collecting Classes To Test !Build error titles buildErrorTitle=Build error evosuiteErrorTitle=EvoSuite error +kexErrorTitle=Kex error llmErrorTitle=Large Language Model error llmWarningTitle=Large Language Model warning generationWindowWarningTitle=TestSpark window warning @@ -26,4 +27,5 @@ testCaseCopied=Test case copied commonBuildErrorMessage=Please make sure that IntelliJ can build your project without any issues or provide the correct build command in the settings sendingFeedback=Sending your request to Large Language Model !Run caution message -runCautionMessage=By clicking "OK" you agree to run this code on your machine \ No newline at end of file +runCautionMessage=By clicking "OK" you agree to run this code on your machine +kexWarningTitle=Kex warning \ No newline at end of file