Merge pull request #126 from boozallen/111-image-names-baton-migration

#111 Baton Script to Update Image Names

Author: colinpalmer-pro

DRAFT_RELEASE_NOTES.md

@@ -90,21 +90,22 @@ The following steps will upgrade your project to 1.7. These instructions consist
 ## Automatic Upgrades
 To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version.  These migrations run automatically when you build your project, and are included by default when you update the `build-parent` version in your root POM.  Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.
 
-| Migration Name                                        | Description                                                                                                                                                                                                                                            |
-|-------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| upgrade-tiltfile-aissemble-version-migration          | Updates the aiSSEMBLE version within your project's Tiltfile                                                                                                                                                                                           |
-| upgrade-v2-chart-files-aissemble-version-migration    | Updates the helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE                                                                             |
-| upgrade-v1-chart-files-aissemble-version-migration    | Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE                                                                                   |
-| upgrade-mlflow-v2-external-s3-migration               | Update the mlflow V2 deployment (if present) in your project to utilize Localstack for local development and SealedSecrets for remote deployments                                                                                                      |
-| upgrade-spark-application-s3-migration                | Update the pipeline SparkApplication(s) (if present) in your project to utilize Localstack for local development and SealedSecrets for remote deployments                                                                                              |
-| upgrade-foundation-extension-python-package-migration | Updates the pyproject.toml files within your projects pipelines folder (<YOUR_PROJECT>-pipelines) to use the updated aiSSEMBLE foundation and extension Python packages with the latest naming convention                                              | 
-| upgrade-helm-chart-names-migration                    | Updates the Chart.yaml and values*.yaml files within your project's deploy folder (<YOUR_PROJECT>-deploy) to use the new Helm chart naming convention (`aissemble-<chart-name>-chart`).                                                                |
-| upgrade-helm-module-names-migration                   | Updates the Chart.yaml and values*.yaml files within your project's deploy folder (<YOUR_PROJECT>-deploy) to use the new Helm module naming convention (`aissemble-<chart-name>-chart`)                                                                |
-| upgrade-helm-chart-repository-url-migration           | Updates the Helm repository URL within your project's deploy Chart.yaml file to point to ghcr.io. Only runs if the previous Helm chart repository URL is passed in through the `oldHelmRepositoryUrl` system property (using `-DoldHelmRepositoryUrl`) |
-| upgrade-dockerfile-pip-install-migration              | Updates dockerfiles such that python dependency installations fail during the build, rather than at runtime                                                                                                                                            |
-| enable-habushu-build-cache-migration                  | Updates the `pom.xml` file for any Habushu-managed modules to ensure that the build directory is specified.                                                                                                                                            |
-| data-lineage-package-import-migration                 | Updates the package imports for all java files that are referencing `com.boozallen.aissemble.data.lineage`.                                                                                                                                            |
-| upgrade-spark-application-exec-migration.          | Fixes the exec-maven-plugin executions in pipeline POMs to use the new ghcr.io aissemble-spark-application-chart package                                                                                                                                      |
+| Migration Name                                             | Description                                                                                                                                                                                                                                            |
+|------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| upgrade-tiltfile-aissemble-version-migration               | Updates the aiSSEMBLE version within your project's Tiltfile                                                                                                                                                                                           |
+| upgrade-v2-chart-files-aissemble-version-migration         | Updates the helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE                                                                             |
+| upgrade-v1-chart-files-aissemble-version-migration         | Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE                                                                                   |
+| upgrade-mlflow-v2-external-s3-migration                    | Update the mlflow V2 deployment (if present) in your project to utilize Localstack for local development and SealedSecrets for remote deployments                                                                                                      |
+| upgrade-spark-application-s3-migration                     | Update the pipeline SparkApplication(s) (if present) in your project to utilize Localstack for local development and SealedSecrets for remote deployments                                                                                              |
+| upgrade-foundation-extension-python-package-migration      | Updates the pyproject.toml files within your projects pipelines folder (<YOUR_PROJECT>-pipelines) to use the updated aiSSEMBLE foundation and extension Python packages with the latest naming convention                                              | 
+| upgrade-helm-chart-names-migration                         | Updates the Chart.yaml and values*.yaml files within your project's deploy folder (<YOUR_PROJECT>-deploy) to use the new Helm chart naming convention (`aissemble-<chart-name>-chart`).                                                                |
+| upgrade-helm-module-names-migration                        | Updates the Chart.yaml and values*.yaml files within your project's deploy folder (<YOUR_PROJECT>-deploy) to use the new Helm module naming convention (`aissemble-<chart-name>-chart`)                                                                |
+| upgrade-helm-chart-repository-url-migration                | Updates the Helm repository URL within your project's deploy Chart.yaml file to point to ghcr.io. Only runs if the previous Helm chart repository URL is passed in through the `oldHelmRepositoryUrl` system property (using `-DoldHelmRepositoryUrl`) |
+| upgrade-dockerfile-pip-install-migration                   | Updates dockerfiles such that python dependency installations fail during the build, rather than at runtime                                                                                                                                            |
+| enable-habushu-build-cache-migration                       | Updates the `pom.xml` file for any Habushu-managed modules to ensure that the build directory is specified.                                                                                                                                            |
+| data-lineage-package-import-migration                      | Updates the package imports for all java files that are referencing `com.boozallen.aissemble.data.lineage`.                                                                                                                                            |
+| upgrade-spark-application-exec-migration.                  | Fixes the exec-maven-plugin executions in pipeline POMs to use the new ghcr.io aissemble-spark-application-chart package                                                                                                                               |
+| upgrade-project-specific-image-naming-convention-migration | Updates the project specific aiSSEMBLE generated image names by removing the `boozallen/` prefix                                                                                                                                                       |
 
 To deactivate any of these migrations, add the following configuration to the `baton-maven-plugin` within your root `pom.xml`:
 

docs/modules/ROOT/pages/containers.adoc

@@ -100,7 +100,7 @@ Example:
     Example:
     # airflow
     docker_build(
-        ref='boozallen/example-airflow-docker',
+        ref='example-airflow-docker',
         context='example-docker/example-airflow-docker',
         build_args=build_args,
         dockerfile='example-docker/example-airflow-docker/src/main/resources/docker/Dockerfile'

docs/modules/ROOT/pages/guides/guides-efficient-debugging.adoc

@@ -193,7 +193,7 @@ sync_properties = sync(
 
 # project-name-quick-inference-docker
 docker_build_with_restart(
-   ref='boozallen/project-name-quick-inference-docker',
+   ref='project-name-quick-inference-docker',
    context='project-name-docker/project-name-quick-inference-docker',
    live_update=[sync_properties,
       run('cd /modules/quick-inference; for x in *.whl; do pip install $x --no-cache-dir --no-deps --force-reinstall; done')
@@ -223,7 +223,7 @@ tests a module called `quick inference` and copies the resulting `dist` director
    remote_path='/modules/quick-inference')`: This specifies the locations that need to be synchronized. It ensures that
 the `dist` directory from the previous step is kept in sync with a specific directory on the remote target.
 * `docker_build_with_restart(
-   ref='boozallen/project-name-quick-inference-docker',
+   ref='project-name-quick-inference-docker',
    context='project-name-docker/project-name-quick-inference-docker',...)`: This section is referenced earlier in the
 code in `load('ext://restart_process', 'docker_build_with_restart')`. The configuration includes the image reference,
 file location, and additional options and defines the setup of a Docker container with live update functionality.

extensions/extensions-helm/README.md

@@ -33,7 +33,7 @@ to leverage the new Helm pattern for the aiSSEMBLE module at hand:
     * `local('helm dependency build <project-name>-deploy/src/main/resources/apps/<module-name>') `
   * Below the module's corresponding `k8s_resource()` call, add the following line to manually build the docker image 
     for the migrated module:
-    * `local('docker build -f <project-name>-docker/<project-name>-<module-name>-docker/target/Dockerfile -t boozallen/<project-name>-<module-name>:latest <project-name>-docker/<project-name>-<module-name>-docker/ --build-arg VERSION_AISSEMBLE=<current aiSSEMBLE version>') `
+    * `local('docker build -f <project-name>-docker/<project-name>-<module-name>-docker/target/Dockerfile -t <project-name>-<module-name>:latest <project-name>-docker/<project-name>-<module-name>-docker/ --build-arg VERSION_AISSEMBLE=<current aiSSEMBLE version>') `
       * Note: this step is not always necessary, and only needed when the chart is dependent on an image being built in 
         a downstream project
 

foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/generator/InferenceDockerResourcesGenerator.java

@@ -34,10 +34,12 @@ public class InferenceDockerResourcesGenerator extends AbstractResourcesGenerato
 
 
     private static final Logger logger = LoggerFactory.getLogger(InferenceDockerResourcesGenerator.class);
+    protected static final String DOCKER_PROJECT_REPOSITORY_URL = "dockerProjectRepositoryUrl";
 
     @Override
     public void generate(GenerationContext context) {
         String rootModuleName = context.getRootArtifactId();
+        String dockerProjectRepositoryUrl = context.getPropertyVariables().get(DOCKER_PROJECT_REPOSITORY_URL);
 
         if (rootModuleName == null || rootModuleName.isEmpty()) {
             logger.error("Root module could not be determined!");
@@ -49,6 +51,7 @@ public void generate(GenerationContext context) {
             VelocityContext vc = getNewVelocityContext(context);
             vc.put(VelocityProperty.INFERENCE_MODULE, inferenceModule);
             vc.put(VelocityProperty.INFERENCE_MODULE_SNAKE_CASE, PythonGeneratorUtils.normalizeToPythonCase(inferenceModule));
+            vc.put(VelocityProperty.DOCKER_PROJECT_REPOSITORY_URL, dockerProjectRepositoryUrl);
 
             generateFile(context, vc);
         }

foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/generator/SparkApplicationGenerator.java

@@ -46,6 +46,7 @@ public class SparkApplicationGenerator extends AbstractResourcesGenerator {
     protected ManualActionNotificationService manualActionNotificationService = new ManualActionNotificationService();
 
     private final SparkDependencyConfiguration config = SparkDependencyConfiguration.getInstance();
+    protected static final String DOCKER_PROJECT_REPOSITORY_URL = "dockerProjectRepositoryUrl";
 
     private void handlePySpark(VelocityContext vc, GenerationContext context) {
         Pipeline pipeline = PipelineUtils.getTargetedPipeline(context, metadataContext);
@@ -134,10 +135,12 @@ public void generate(GenerationContext context) {
         }
 
         final String projectName = context.getRootArtifactId();
+        String dockerProjectRepositoryUrl = context.getPropertyVariables().get(DOCKER_PROJECT_REPOSITORY_URL);
 
         vc.put(VelocityProperty.SPARK_APPLICATION_NAME, context.getArtifactId());
         vc.put(VelocityProperty.PROJECT_NAME, projectName);
         vc.put(VelocityProperty.PIPELINE, pipeline.getName());
+        vc.put(VelocityProperty.DOCKER_PROJECT_REPOSITORY_URL, dockerProjectRepositoryUrl);
 
         if (!"test".equalsIgnoreCase(context.getArtifactType()) && SparkStorageEnum.S3LOCAL == metamodelRepository.getDeploymentConfigurationManager().getSparkDeploymentConfiguration().getStorageType()) {
             vc.put(VelocityProperty.USE_S3_LOCAL, true);

foundation/foundation-mda/src/main/resources/templates/data-delivery-pyspark/pipeline.pom.xml.vm

@@ -29,6 +29,7 @@
                         <targetPipeline>${pipeline.name}</targetPipeline>
                         <behaveFeature>data_delivery</behaveFeature>
                         <aissembleVersion>${version.aissemble}</aissembleVersion>
+                        <dockerProjectRepositoryUrl>${docker.project.repository.url}</dockerProjectRepositoryUrl>
                     </propertyVariables>
                     <!-- see ${parentArtifactId} for base configuration settings -->
                 </configuration>

foundation/foundation-mda/src/main/resources/templates/data-delivery-spark/pipeline.pom.xml.vm

@@ -27,6 +27,7 @@
                     <propertyVariables>
                         <targetPipeline>${pipeline.name}</targetPipeline>
                         <aissembleVersion>${version.aissemble}</aissembleVersion>
+                        <dockerProjectRepositoryUrl>${docker.project.repository.url}</dockerProjectRepositoryUrl>
                     </propertyVariables>
                     <!-- see ${parentArtifactId} for base configuration settings -->
                 </configuration>

foundation/foundation-mda/src/main/resources/templates/general-docker/inference.docker.pom.xml.vm

@@ -25,6 +25,9 @@
 				<configuration>
 					<basePackage>${basePackage}</basePackage>
 					<profile>aissemble-inference-docker</profile>
+					<propertyVariables>
+						<dockerProjectRepositoryUrl>${docker.project.repository.url}</dockerProjectRepositoryUrl>
+					</propertyVariables>
 				</configuration>
 			</plugin>
 			<plugin>

foundation/foundation-mda/src/main/resources/templates/inference/inference.perceptor.yaml.vm

@@ -4,5 +4,5 @@ wrapper: prime
 language: python
 
 name: "${artifactId}" 
-repository: "ghcr.io/boozallen/${artifactId}"
+repository: "${dockerProjectRepositoryUrl}${artifactId}"
 tag: "${version}"

foundation/foundation-upgrade/src/main/java/com/boozallen/aissemble/upgrade/migration/v1_7_0/ProjectSpecificImageNameMigration.java

@@ -0,0 +1,67 @@
+package com.boozallen.aissemble.upgrade.migration.v1_7_0;
+
+/*-
+ * #%L
+ * aiSSEMBLE::Foundation::Upgrade
+ * %%
+ * Copyright (C) 2021 Booz Allen
+ * %%
+ * This software package is licensed under the Booz Allen Public License. All Rights Reserved.
+ * #L%
+ */
+
+import com.boozallen.aissemble.upgrade.migration.AbstractAissembleMigration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.technologybrewery.baton.BatonException;
+
+import java.io.File;
+import java.io.IOException;
+
+import static com.boozallen.aissemble.upgrade.util.FileUtils.*;
+
+/**
+ * Baton migration used to update the naming convention of project specific aiSSEMBLE generated images.
+ */
+public class ProjectSpecificImageNameMigration extends AbstractAissembleMigration {
+
+    public static final Logger logger = LoggerFactory.getLogger(ProjectSpecificImageNameMigration.class);
+
+    public static final String ORGANIZATION_PRE_1_7_0 = "boozallen/";
+    private String projectName;
+
+    /**
+     * Function to check whether the migration is necessary.
+     * @param file file to check
+     * @return shouldExecute - whether the migration is necessary.
+     */
+    @Override
+    protected boolean shouldExecuteOnFile(File file) {
+        boolean shouldExecute;
+        try {
+            projectName = getProjectPrefix();
+            shouldExecute = hasRegExMatch(ORGANIZATION_PRE_1_7_0 + projectName, file);
+        } catch (IOException e){
+            throw new BatonException("Unable to determine if project specific image name migration must be executed", e);
+        }
+        return shouldExecute;
+    }
+
+    /**
+     * Performs the migration if the shouldExecuteOnFile() returns true.
+     * @param file file to migrate
+     * @return isMigrated - Whether the file was migrated successfully.
+     */
+    @Override
+    protected boolean performMigration(File file) {
+        boolean isMigrated;
+
+        try {
+            isMigrated = replaceLiteralInFile(file, ORGANIZATION_PRE_1_7_0 + projectName, projectName);
+        } catch (Exception e) {
+            logger.error("Error in migrating project specific image names.");
+            throw new BatonException(e);
+        }
+        return isMigrated;
+    }
+}

foundation/foundation-upgrade/src/main/resources/migrations.json

@@ -109,6 +109,21 @@
             "includes": ["src/**/*.java", "src/**/*.properties"]
           }
         ]
+      },
+      {
+        "name": "upgrade-project-specific-image-naming-convention-migration",
+        "implementation": "com.boozallen.aissemble.upgrade.migration.v1_7_0.ProjectSpecificImageNameMigration",
+        "fileSets": [
+          {
+            "includes": [
+              "*-deploy/src/main/resources/apps/*/values*.yaml",
+              "*-deploy/src/main/resources/apps/spark-worker-image/spark-worker-image.yaml",
+              "*-deploy/src/main/resources/apps/aissemble-machine-learning-training-image/aissemble-machine-learning-training-image.yaml",
+              "*-pipelines/*/src/**/resources/apps/*values.yaml",
+              "Tiltfile"
+            ]
+          }
+        ]
       }
     ]
   },