|
1 | 1 | # Major Additions |
2 | 2 |
|
3 | | -## Service account support for spark-infrastructure |
4 | | -To have a more flexible and secure way to authenticate with AWS services, the spark-infrastructure helm chart has been enhanced to support [AWS IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IAM Roles for Service Accounts) authentication. See the _**How to Upgrade**_ for more information. |
5 | | - |
6 | | -## Path to Production Alignment |
7 | | -To better align development processes with processes in CI/CD and higher environments, we no longer recommend using Tilt live-reloading. As such, upgrading projects should consider narrowing the scope of their Tiltfile. These changes will also help smooth the transition as further alignment is brought to the path to production. See _**How to Upgrade**_ for more information. |
8 | | - |
9 | | -## Data Access Upgrade |
10 | | -Data access through [GraphQL](https://graphql.org/) has been deprecated and replaced with [Trino](https://trino.io/). Trino is optimized for performing queries against large datasets by leveraging a distributed architecture that processes queries in parallel, enabling fast and scalable data retrieval. |
11 | | - |
12 | | -## Spark Upgrade |
13 | | -Spark and PySpark have been upgraded from version 3.5.2 to 3.5.4. |
14 | | - |
15 | | -## Record Relation |
16 | | -To enable nested data records, we have added a new relation feature to the record metamodel. This allows records to reference other records. For more details, refer to the [Record Relation Options](https://boozallen.github.io/aissemble/aissemble/current-dev/record-metamodel.html#_record_relation_options). |
17 | | -Several features are still a work in progress: |
18 | | -- PySpark and Spark schema based validation for relations will only validate the record and not its relations. Object based validation for relations is available. |
19 | | - |
20 | | -## Helm Charts Resource Specification |
21 | | -The following Helm charts have been updated to include the configuration options for specifying container resource requests/limits: |
22 | | -- `aissemble-spark-history-chart` |
23 | | -- `aissemble-quarkus-chart` |
24 | | - |
25 | | -See the[official Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for more details. |
26 | | - |
27 | 3 | # Breaking Changes |
28 | 4 | _Note: instructions for adapting to these changes are outlined in the upgrade instructions below._ |
29 | 5 |
|
30 | | - - The following Java classes have been renamed: |
31 | | - | Old Java Class | New Java Class | |
32 | | - |-------------------------------|------------------------------------| |
33 | | - | `AIOpsModelInstanceRepostory` | `AissembleModelInstanceRepository` | |
34 | | - | `AiopsMdaJsonUtils` | `AissembleMdaJsonUtils` | |
35 | | - - To improve the development cycle and docker build consistency, we have deprecated the docker_build() and local_resources() functions in the Tilt and enable maven docker build for the docker modules. Follow the instruction in the `Finalizing the Upgrade` to avoid duplicated docker image build. |
36 | | - - In an attempt to harden the `aissemble-hive-service` image, several changes were made that may impact projects with Hive customization |
37 | | - |
38 | | - |
39 | 6 | # Known Issues |
40 | 7 |
|
41 | 8 | ## Docker Module Build Failures |
@@ -64,20 +31,11 @@ The following steps will upgrade your project to `1.11`. These instructions cons |
64 | 31 | ## Automatic Upgrades |
65 | 32 | To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the `build-parent` version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE. |
66 | 33 |
|
67 | | -| Migration Name | Description | |
68 | | -|----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
69 | | -| upgrade-tiltfile-aissemble-version-migration | Updates the aiSSEMBLE version within your project's Tiltfile | |
70 | | -| upgrade-v2-chart-files-aissemble-version-migration | Updates the Helm chart dependencies within your project's deployment resources (`<YOUR_PROJECT>-deploy/src/main/resources/apps/`) to use the latest version of the aiSSEMBLE | |
71 | | -| upgrade-v1-chart-files-aissemble-version-migration | Updates the docker image tags within your project's deployment resources (`<YOUR_PROJECT>-deploy/src/main/resources/apps/`) to use the latest version of the aiSSEMBLE | |
72 | | -| pipeline-invocation-service-template-migrtion | Include the helm.valueFiles param to ArgoCD pipeline-invocation-service template | |
73 | | -| docker-module-pom-dependency-type-migration | Updates the maven pipeline dependency type within your project's sub docker module pom file(`<YOUR_PROJECT>-docker/*-docker/pom.xml`) to fix the build cache checksum calculation issue | |
74 | | -| enable-maven-docker-build-migration | Remove the maven fabric8 plugin `skip` configuration within your project's docker module pom file(`<YOUR_PROJECT>-docker/pom.xml`) to enable the maven docker build | |
75 | | -| ml-pipeline-docker-pom-migration | Adds pipeline ML pipeline dependencies to relevant docker POMs to improve the Maven build cache functionality | |
76 | | -| training-api-image-tag-migration | Update training docker image tags to use project version | |
77 | | -| inference-docker-image-tag-migration | Update inference docker image tags to use project version | |
78 | | -| spark-worker-docker-image-tag-migration | Updates Spark docker image tags to use project version | |
79 | | -| spark-infrastructure-server-side-label-migration | Enables the Server-Side Diff Strategy within ArgoCD for the Spark Infrastructure resource so that changes made by the Universal Config Store mutating webhook are ignored | |
80 | | -| data-access-default-migration | Migrates Record metamodels that were relying on the default Data Access settings to preserve semantics with the updated default value | |
| 34 | +| Migration Name | Description | |
| 35 | +|----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| 36 | +| upgrade-tiltfile-aissemble-version-migration | Updates the aiSSEMBLE version within your project's Tiltfile | |
| 37 | +| upgrade-v2-chart-files-aissemble-version-migration | Updates the Helm chart dependencies within your project's deployment resources (`<YOUR_PROJECT>-deploy/src/main/resources/apps/`) to use the latest version of the aiSSEMBLE | |
| 38 | +| upgrade-v1-chart-files-aissemble-version-migration | Updates the docker image tags within your project's deployment resources (`<YOUR_PROJECT>-deploy/src/main/resources/apps/`) to use the latest version of the aiSSEMBLE | |
81 | 39 |
|
82 | 40 | To deactivate any of these migrations, add the following configuration to the `baton-maven-plugin` within your root `pom.xml`: |
83 | 41 |
|
@@ -113,44 +71,8 @@ To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.11.0 |
113 | 71 | </parent> |
114 | 72 | ``` |
115 | 73 |
|
116 | | -### Tilt Docker Builds |
117 | | -To avoid duplicate docker builds, remove all the related `docker_build()` and `local_resources()` functions from your Tiltfile. Also, the `spark-worker-image.yaml` is no longer used so the `-deploy/src/main/resources/apps/spark-worker-image` directory and the related `k8s_yaml()` function from your Tiltfile can be removed. |
118 | | - |
119 | 74 | ## Conditional Steps |
120 | 75 |
|
121 | | -### For projects that have customized the Hive service |
122 | | -Several changes were made to both the Hive service Docker image and the Hive service chart included as part of the Spark Infrastructure chart of a project. The defaults have been adjusted so that these changes should be transparent, however due to the nature of some possible customizations this may not always hold true. The following changes may impact the function of your customizations and may need to be accounted for: |
123 | | - - The image is now only the Hive Standalone Metastore service and cannot function as a full [Hive Server](https://hive.apache.org/development/quickstart/) |
124 | | - - The Java installation at `/opt/java` is no longer symlinked to `/opt/jre` -- `JAVA_HOME` has been adjusted accordingly by default |
125 | | - - The default working directory for the `aissemble-hive-service` image was changed from `/opt` to `/opt/hive` |
126 | | - - Schema initialization is no longer done as part of an `initContainer` in the `aissemble-hive-service-chart` and is instead done in a new `entrypoint` script. This is consistent with the [official `apache/hive` Docker image](https://hub.docker.com/r/apache/hive). |
127 | | - |
128 | | -### AWS IRSA (IAM Roles Service Account) Authentication |
129 | | -This is not a required step but a recommended way to authenticate AWS service |
130 | | -1. [Create an IAM OIDC provider for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) |
131 | | -2. Follow the [Assign IAM roles to Kubernetes service accounts](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) document but **skip** the step that creates the service account |
132 | | -3. In the spark-infrastructure chart template, add the service account create configuration as below: |
133 | | - |
134 | | -aissemble-spark-history-chart |
135 | | -```yaml |
136 | | -aissemble-spark-history-chart: |
137 | | - serviceAccount: |
138 | | - name: service-account-name |
139 | | - enabled: true |
140 | | - metadata: |
141 | | - annotations: |
142 | | - # Ref: IAM roles arn from step 2 |
143 | | - eks.amazonaws.com/role-arn: arn:aws:iam::aws-id:role/iam-role-name |
144 | | -``` |
145 | | -
|
146 | | -aissemble-thrift-server-chart: |
147 | | -```yaml |
148 | | -aissemble-thrift-server-chart: |
149 | | - deployment: |
150 | | - # service account name must match the service account name specified in the IAM roles trust relationships |
151 | | - serviceAccountName: service-account-name |
152 | | -``` |
153 | | -
|
154 | 76 | ## Final Steps - Required for All Projects |
155 | 77 | ### Finalizing the Upgrade |
156 | 78 | 1. Run `./mvnw org.technologybrewery.baton:baton-maven-plugin:baton-migrate` to apply the automatic migrations |
|
0 commit comments