Merge pull request #109 from boozallen/82

#82 Address caching defects in downstream html report generation

Author: cpointe-ibllanos

extensions/extensions-helm/extensions-helm-spark-infrastructure/LICENSE.txt

@@ -0,0 +1,49 @@
+Booz Allen Public License v1.0
+------------------------------
+
+INTRODUCTION
+The Booz Allen Public License allows government, non-profit academic, other non-profit, and commercial entities access to distinctive, disruptive, and robust code with the goal of Empowering People to Change the World(SM). Products licensed under the Booz Allen Public License are founded on the basis that collective ingenuity can make the largest impact in the community.
+
+DEFINITIONS
+* **Commercial Entity.** “Commercial Entity” means any individual or entity other than a government, non-profit academic, or other non-profit entity.
+* **Derivative.** “Derivative” means any work of authorship in Source Code or Object Code form that results from an addition to, deletion from, or modification of the Source Code of the Product.
+* **License.**  “License” means this Booz Allen Public License.
+* **Object Code.** “Object Code” means the form resulting from transformation or translation of Source Code into machine readable code, including but not limited to, compiled object code.
+* **Originator.** “Originator” means each individual or legal entity that creates, contributes to the creation of, or owns the Product. 
+* **Patent Claims.** “Patent Claims” means any patent claim(s) in any patent to which Originator has a right to grant a license that would be infringed by Your making, using, selling, offering for sale, having made, or importing of the Product, but for the grant of this License. 
+* **Product.** “Product” means the Source Code of the software which the initial Originator made available under this License, and any Derivative of such Source Code. 
+* **Source Code.** “Source Code” means software in human-readable form.
+* **You.** “You” means either an individual or an entity (if you are taking this license on behalf of an entity) that exercises the rights granted under this License.  
+  
+LICENSE
+**Government/Non-Profit Academic/Other Non-Profit.**  
+This Section applies if You are not a Commercial Entity.  
+
+* **License.** Subject to the terms and conditions of this License, each Originator hereby grants You a perpetual, worldwide, non-exclusive, royalty-free license to reproduce, display, perform, modify, distribute and otherwise use the Product and Derivatives, in Source Code and Object Code form, in accordance with the terms and conditions of this License in order to support the general public good and for your internal business purposes.
+* **Distribution.** You may distribute to third parties copies of the Product, including any Derivative that You create, in Source Code or Object Code form.  If You distribute copies of the Product, including any Derivative that You create, in Source Code form, such distribution must be under the terms of this License and You must inform recipients of the Source Code that the Product is governed under this License and how they can obtain a copy of this License.  You may distribute to third parties copies of the Product, including any Derivative that You create, in Object Code form, or allow third parties to access or use the Product, including any Derivative that You create, under a license of Your choice.
+* **Commercial Sales.** You may not distribute, or allow third parties to access or use, the Product or any Derivative for a fee, unless You first obtain permission from the Originator. If Booz Allen Hamilton is the Originator, please contact Booz Allen Hamilton at  <opensource@bah.com>.  
+
+**Commercial Entities**.
+This Section applies if You are a Commercial Entity.
+
+* **License.** Subject to the terms and conditions of this License, each Originator hereby grants You a perpetual, worldwide, non-exclusive, royalty-free license to reproduce, display, perform, modify, distribute and otherwise use the Product and Derivatives, in Source Code and Object Code form, in accordance with the terms and conditions of this License for the sole purpose of Your internal business purposes and the provision of services to government, non-profit academic, and other non-profit entities. 
+* **Distribution and Derivatives.** You may distribute to third parties copies of the Product, including any Derivative that You create, in Source Code or Object Code form.  If You distribute copies of the Product, including any Derivative that You create, in Source Code form, such distribution must be under the terms of this License and You must inform recipients of the Source Code that the Product is governed under this License and how they can obtain a copy of this License.  You may distribute to third parties copies of the Product, including any Derivative that You create, in Object Code form, or allow third parties to access or use the Product, including any Derivative that You create, under a license of Your choice, provided that You make available, and inform the recipient of such distribution how they can obtain, a copy of the Source Code thereof, at no charge, and inform the recipient of the Source Code that the Product is governed under this License and how they can obtain a copy of this License.
+* **Commercial Sales.** You may not distribute, or allow third parties to access or use, the Product or any Derivative for a fee, unless You first obtain permission from the Originator. If Booz Allen Hamilton, please contact Booz Allen Hamilton at <opensource@bah.com>.   
+
+ 
+**Patent Claim(s)**.  
+This Section applies regardless of whether You are a government, non-profit academic, or other non-profit entity or a Commercial Entity. 
+
+* **Patent License.** Subject to the limitations in the Sections above, each Originator hereby grants You a perpetual, worldwide, non-exclusive, royalty-free license under Patent Claims of such Originator to make, use, sell, offer for sale, have made, and import the Product.  The foregoing patent license does not apply (a) to any code that an Originator has removed from the Product, or (b) for infringement caused by Your modifications of the Product or the combination of any Derivative created by You or on Your behalf with other software.      
+
+GENERAL TERMS
+This Section applies regardless of whether You are a government, non-profit academic, or other non-profit entity or a Commercial Entity.
+
+* **Required Notices.** If You distribute the Product or a Derivative, in Object Code or Source Code form, You shall not remove or otherwise modify any proprietary markings or notices contained within or placed upon the Product or any Derivative. Any distribution of the Product or a Derivative, in Object Code or Source Code form, shall contain a clear and conspicuous Originator copyright and license reference in accordance with the below:
+	* *Unmodified Product Notice*: “This software package is licensed under the Booz Allen Public License. Copyright © 20__ [Copyright Holder Name].  All Rights Reserved.”
+	* *Derivative Notice*: “This software package is licensed under the Booz Allen Public License. Portions of this code are Copyright © 20__ [Copyright Holder Name].  All Rights Reserved.”
+* **Compliance with Laws.** You agree that You shall not reproduce, display, perform, modify, distribute and otherwise use the Product in any way that violates applicable law or regulation or infringes or violates the rights of others, including, but not limited to, third party intellectual property, privacy, and publicity rights.
+* **Disclaimer.** You understand that the Product is licensed to You, and not sold. The Product is provided on an “As Is” basis, without any warranties, representations, and guarantees, whether oral or written, express, implied or statutory, with regard to the Product, including without limitation, warranties of merchantability, fitness for a particular purpose, title, non-infringement, non-interference, and warranties arising from course of dealing or usage of trade, to the maximum extent permitted by applicable law.  Originator does not warrant that (i) the Product will meet your needs; (ii) the Product will be error-free or accessible at all times; or (iii) the use or the results of the use of the Product will be correct, accurate, timely, or otherwise reliable. You acknowledge that the Product has not been prepared to meet Your individual requirements, whether or not such requirements have been communicated to Originator. You assume all responsibility for use of the Product.
+* **Limitation of Liability.** Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Originator, or anyone who distributes the Product in accordance with this License, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if informed of the possibility of such damages.
+* **Export Control.** The Product is subject to U.S. export control laws and may be subject to export or import regulations in other countries.  You agree to strictly comply with all such laws and regulations and acknowledges that You are responsible for obtaining such licenses to export, re-export, or import as may be required.
+* **Severability.** If the application of any provision of this License to any particular facts or circumstances shall be held to be invalid or unenforceable, then the validity and enforceability of other provisions of this License shall not in any way be affected or impaired thereby.  

foundation/foundation-archetype/src/main/resources/archetype-resources/.mvn/maven-build-cache-config.xml

@@ -43,9 +43,9 @@
                 <dirName>dockerbuild</dirName>
                 <dirName>krausening</dirName>
                 <!-- Reports needed on CI to ensure consistent reporting of QA metrics: -->
-                <dirName>cucumber-reports</dirName>
-                <dirName>surefire-reports</dirName>
-                <dirName>failsafe-reports</dirName>
+                <dirName>../target/cucumber-reports</dirName>
+                <dirName>../target/surefire-reports</dirName>
+                <dirName>../target/failsafe-reports</dirName>
             </dirNames>
         </attachedOutputs>
         <local>

foundation/foundation-archetype/src/test/resources/test-project-archetype-script/maven-build-cache-config.xml

@@ -49,9 +49,9 @@
                 <dirName>quarkus-app</dirName>
                 <dirName>helm</dirName>
                 <!-- Reports needed on CI to ensure consistent reporting of QA metrics: -->
-                <dirName>cucumber-reports</dirName>
-                <dirName>surefire-reports</dirName>
-                <dirName>failsafe-reports</dirName>
+                <dirName>../target/cucumber-reports</dirName>
+                <dirName>../target/surefire-reports</dirName>
+                <dirName>../target/failsafe-reports</dirName>
             </dirNames>
         </attachedOutputs>
         <local>

foundation/foundation-mda/src/main/resources/templates/general-mlflow/pipeline.step.pom.xml.vm

@@ -95,13 +95,22 @@
             </plugin>
 
             <plugin>
-				<groupId>org.codehaus.mojo</groupId>
-				<artifactId>license-maven-plugin</artifactId>
-				<configuration>
-					<licenseFile>LICENSE</licenseFile>
-				</configuration>
-			</plugin>
+                <groupId>org.codehaus.mojo</groupId>
+                <artifactId>license-maven-plugin</artifactId>
+                <configuration>
+                    <licenseFile>LICENSE</licenseFile>
+                </configuration>
+            </plugin>
+            <plugin>
+                <groupId>net.masterthought</groupId>
+                <artifactId>maven-cucumber-reporting</artifactId>
+            </plugin>
         </plugins>
     </build>
-
+    <dependencies>
+        <dependency>
+            <groupId>net.masterthought</groupId>
+            <artifactId>cucumber-reporting</artifactId>
+        </dependency>
+    </dependencies>
 </project>

foundation/foundation-mda/src/test/java/com/boozallen/aiops/mda/generator/DataRecordsModuleStep.java

@@ -7,14 +7,17 @@
  * This software package is licensed under the Booz Allen Public License. All Rights Reserved.
  * #L%
  */
+
+import com.boozallen.aiops.mda.generator.util.PipelineUtils;
+
 import com.boozallen.aiops.mda.metamodel.element.AbstractModelInstanceSteps;
+import com.boozallen.aiops.mda.metamodel.element.BaseStepDecorator;
+import com.boozallen.aiops.mda.metamodel.element.RecordElement;
 import com.boozallen.aiops.mda.metamodel.element.BasePipelineDecorator;
 import com.boozallen.aiops.mda.metamodel.element.Pipeline;
-import com.boozallen.aiops.mda.metamodel.element.RecordElement;
 import com.boozallen.aiops.mda.metamodel.element.python.PythonRecord;
 import io.cucumber.java.Before;
 import io.cucumber.java.Scenario;
-import io.cucumber.java.en.And;
 import io.cucumber.java.en.Given;
 import io.cucumber.java.en.Then;
 import io.cucumber.java.en.When;
@@ -42,6 +45,7 @@
 import java.util.List;
 import java.util.Map;
 import java.util.regex.Pattern;
+
 import static org.junit.Assert.assertTrue;
 
 public class DataRecordsModuleStep extends AbstractModelInstanceSteps {
@@ -97,13 +101,9 @@ public void a_pre_existing_semantic_data_module_called_with(String moduleName, S
     public void the_profile_is_generated(String profileName) throws Exception {
         readMetadata(projectName);
         Map<String, ExpandedProfile> profiles = loadProfiles();
-        GenerateSourcesHelper.performSourceGeneration(profileName, profiles,
-                this::createGenerationContext,
-                (missingProfile, foundProfiles) -> {
-                    throw new RuntimeException("Missing profile: " + missingProfile);
-                },
-                new Slf4jDelegate(logger),
-                projectDir.toFile());
+        GenerateSourcesHelper.performSourceGeneration(profileName, profiles, this::createGenerationContext, (missingProfile, foundProfiles) -> {
+            throw new RuntimeException("Missing profile: " + missingProfile);
+        }, new Slf4jDelegate(logger), projectDir.toFile());
     }
 
     @Then("a semantic-data module is generated with {string} under {string}")
@@ -137,8 +137,7 @@ public void the_user_is_notified_that_the_module_must_be_added_to_the_parent_pom
     @Then("no module is generated under {string}")
     public void no_module_is_generated_under(String moduleName) {
         Path module = projectDir.resolve(moduleName);
-        assertTrue("Language-specific module created when existing module should have been reused: " + moduleName,
-                Files.notExists(module));
+        assertTrue("Language-specific module created when existing module should have been reused: " + moduleName, Files.notExists(module));
     }
 
     @Then("the pipeline POM has a dependency on {string}")
@@ -151,6 +150,16 @@ public void the_pipeline_pom_has_the_plugin(String plugin) throws Exception {
         the_pipeline_pom_has_the_plugin(pipeline.getName(), plugin);
     }
 
+    @Then("the pipeline's child POMs have a dependency on {string}")
+    public void the_pipelines_child_poms_have_a_dependency_on(String dataModule) throws Exception {
+        the_pipeline_child_poms_have_a_dependency_on(pipeline.getName(), dataModule);
+    }
+
+    @Then("the pipeline's child POMs has the plugin {string}")
+    public void the_pipelines_child_poms_have_the_plugin(String plugin) throws Exception {
+        the_pipeline_child_poms_have_the_plugin(pipeline.getName(), plugin);
+    }
+
     @Then("{string} has a dependency on {string}")
     public void has_a_dependency_on(String module, String dependency) throws Exception {
         Path pom = projectDir.resolve(module).resolve("pom.xml");
@@ -175,19 +184,36 @@ public void the_pipeline_pom_has_a_dependency_on(String pipelineName, String dat
         has_a_dependency_on(decoratedPipeline.deriveArtifactIdFromCamelCase(), dataModule);
     }
 
+    @Then("the {string} pipeline child POMs have a dependency on {string}")
+    public void the_pipeline_child_poms_have_a_dependency_on(String pipelineName, String dataModule) throws Exception {
+        Pipeline pipeline = pipelines.get(pipelineName);
+        BasePipelineDecorator decoratedPipeline = new BasePipelineDecorator(pipeline);
+        for (BaseStepDecorator step : decoratedPipeline.getSteps()) {
+            has_a_dependency_on(PipelineUtils.deriveArtifactIdFromCamelCase(step.getName()), dataModule);
+        }
+    }
+
     @Then("the {string} pipeline POM has the plugin {string}")
     public void the_pipeline_pom_has_the_plugin(String pipelineName, String dataModule) throws Exception {
         Pipeline pipeline = pipelines.get(pipelineName);
         BasePipelineDecorator decoratedPipeline = new BasePipelineDecorator(pipeline);
         has_the_plugin(decoratedPipeline.deriveArtifactIdFromCamelCase(), dataModule);
     }
 
+    @Then("the {string} pipeline child POMs have the plugin {string}")
+    public void the_pipeline_child_poms_have_the_plugin(String pipelineName, String dataModule) throws Exception {
+        Pipeline pipeline = pipelines.get(pipelineName);
+        BasePipelineDecorator decoratedPipeline = new BasePipelineDecorator(pipeline);
+        for (BaseStepDecorator step : decoratedPipeline.getSteps()) {
+            has_the_plugin(PipelineUtils.deriveArtifactIdFromCamelCase(step.getName()), dataModule);
+        }
+    }
+
     @Then("the pyproject.toml file has a dependency on {string}")
     public void the_pyproject_toml_file_has_a_dependency_on(String dataModule) throws IOException {
         Path pyproject = projectDir.resolve("pyproject.toml");
         Pattern monoRepoPattern = Pattern.compile(dataModule + " *= *\\{path *= \".*?" + dataModule + "\".*?}");
-        boolean hasDependency = Files.lines(pyproject)
-                .anyMatch(monoRepoPattern.asMatchPredicate());
+        boolean hasDependency = Files.lines(pyproject).anyMatch(monoRepoPattern.asMatchPredicate());
         assertTrue("Dependency " + dataModule + " not found in " + pyproject, hasDependency);
     }
 
@@ -251,8 +277,7 @@ private Notification getNotification(String file, String notificationType) {
         Map<String, Map<String, Notification>> notifications = NotificationCollector.getNotifications();
         assertTrue("No notifications for file " + file, notifications.containsKey(file));
         Map<String, Notification> fileNotifications = notifications.get(file);
-        assertTrue("No notifications of type " + notificationType + " for " + file,
-                fileNotifications.containsKey(file + "_" + notificationType));
+        assertTrue("No notifications of type " + notificationType + " for " + file, fileNotifications.containsKey(file + "_" + notificationType));
         return fileNotifications.get(file + "_" + notificationType);
     }
 
@@ -261,7 +286,7 @@ private Notification getNotification(String file, String notificationType) {
      * expected content.
      *
      * @param pom     the POM file
-     * @param query    the XPath query
+     * @param query   the XPath query
      * @param content the expected content
      * @return true if the content is found
      * @throws Exception if the POM cannot be read

foundation/foundation-mda/src/test/resources/specifications/data-records-generation.feature

@@ -181,7 +181,7 @@ Feature: Generating pipeline resources
       | data-flow | data-delivery-pyspark |
 
   @pipeline-generation @module-generation
-  Scenario Outline: Habushu pipelines integrate test report generation by default
+  Scenario Outline: Spark and Pyspark pipelines integrate test report generation by default
     Given a project named "example"
     And a "<type>" pipeline using "<implementation>"
     And a dictionary and 0 or more record models
@@ -194,6 +194,15 @@ Feature: Generating pipeline resources
       | data-flow                 | data-delivery-pyspark   | aissemble-maven-modules |
       | data-flow                 | data-delivery-spark     | aissemble-maven-modules |
 
+  @pipeline-generation @module-generation
+  Scenario: Machine learning pipelines integrate test report generation in child modules by default
+    Given a project named "example"
+    And a "machine-learning" pipeline using "machine-learning-mlflow"
+    And a dictionary and 0 or more record models
+    When the profile "aissemble-maven-modules" is generated
+    Then the pipeline's child POMs have a dependency on "cucumber-reporting"
+    And the pipeline's child POMs has the plugin "maven-cucumber-reporting"
+
   @pipeline-generation @module-generation
   Scenario: Pipelines depend on data module by default - two language support
     Given a project named "example"