Preparing 3.6.2 Release Changelog

.github/CONTRIBUTING.md

@@ -4,7 +4,7 @@
 
 When reporting issues please include as much detail as possible about your
 operating environment, schemachange version and python version. Whenever possible, please
-also include a brief, self-contained code example that demonstrates the problem. 
+also include a brief, self-contained code example that demonstrates the problem.
 
 We have included [issue templates for reporting bugs, requesting features and seeking clarifications.](https://github.com/Snowflake-Labs/schemachange/issues/new/choose)
 Choose the appropriate issue template to contribute to the repository.
@@ -21,50 +21,50 @@ Thanks for your interest in contributing code to schemachange!
 1. If you are a first-time contributor
     + Go to [Snowflake-Labs/Schemachange](https://github.com/Snowflake-Labs/schemachange) and click the "fork" button to create your own copy of the project.
     + [Clone](https://github.com/git-guides/git-clone) the project to your local computer
-  
+
     ```shell
     git clone https://github.com/your-username/schemachange.git
-    
+
     ```
-    
+
     + Change the directory
-   
+
     ```shell
     cd schemachange
-    
+
     ```
     + Add upstream repository:
 
     ```shell
     git remote add upstream https://github.com/Snowflake-Labs/schemachange
-    
+
     ```
-    
+
     + Now, `git remote -v` will show two [remote](https://github.com/git-guides/git-remote) repositories named:
       + `upstream`, which refers to the `schemachange` repository
-      + `origin`, which refers to your personal fork 
+      + `origin`, which refers to your personal fork
     + [Pull](https://github.com/git-guides/git-pull) the latest changes from upstream, including tags:
-      
+
     ```shell
     git checkout main
     git pull upstream main --tags
-    
+
     ```
-    
+
 3. Develop your contribution
     + Create a branch for the features you want to work on. Since the branch name will appear in the merge message, use a sensible name such as 'update-build-library-dependencies':
-      
+
     ```shell
     git checkout -b update-build-library-dependencies
     ```
-    
+
     + Commit locally as you progress ( [git add](https://github.com/git-guides/git-add) and [git commit](https://github.com/git-guides/git-commit) ). Use a properly formatted commit message. Be sure to document any changed behavior.
 4. To submit your contribution
     + [Push](https://github.com/git-guides/git-push) your changes back to your fork on GitHub
 
     ```shell
     git push origin update-build-library-dependencies
-    
+
     ```
 
     + Go to GitHub. The new branch will show up with a green [Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#initiating-the-pull-request) button. Make sure the title and message are clear, concise and self explanatory. Then click the button to submit it.

.github/workflows/dev-pytest.yml

@@ -1,73 +1,91 @@
 name: dev-pytest
 
+defaults:
+  run:
+    shell: bash
+
 on:
-  push:
-    branches:
-      - dev  
-
-  workflow_dispatch: 
+  workflow_dispatch:
 
   pull_request:
-    types: [opened, reopened]
+    branches:
+      - "*"
+    paths-ignore:
+      - '**/*.md'
+      - '**/*.MD'
+      - '*.md'
+      - '**.MD'
+
+  push:
+    paths-ignore:
+      - '**/*.md'
+      - '**/*.MD'
+      - '*.md'
+      - '*.MD'
+    tags:
+      - "v*" # Push events to matching v*, i.e. v3.6.2, v4.0.0
+    branches:
+      - main
 
 jobs:
   build:
+    strategy:
+      fail-fast: false
+      matrix:
+        scenario-name: [ 'basics_demo', 'citibike_demo', 'citibike_demo_jinja']
+        os: ["ubuntu-latest"]
+        python-version: ["3.11"]
+    runs-on: ${{ matrix.os }}
+
     env:
       SNOWFLAKE_PASSWORD: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_PASSWORD }}
       SNOWFLAKE_USER: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_USER }}
       SNOWFLAKE_ACCOUNT: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_ACCOUNT }}
-      SNOWFLAKE_CONNECTIONS_MYCONNECTION_PASSWORD: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_PASSWORD }}
-      SNOWFLAKE_CONNECTIONS_MYCONNECTION_ACCOUNT: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_ACCOUNT }}
-      SNOWFLAKE_CONNECTIONS_MYCONNECTION_USER: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_USER }}
-    runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-
+      SNOWFLAKE_DATABASE: SCHEMACHANGE_DEMO
+      SNOWFLAKE_WAREHOUSE: SCHEMACHANGE_DEMO_WH
+      SNOWFLAKE_ROLE: '"SCHEMACHANGE_DEMO-DEPLOY"'
+      MY_TARGET_SCHEMA: ${{ matrix.scenario-name }}_${{ github.run_number }}_${{ strategy.job-index }}
+      SCENARIO_NAME: ${{ matrix.scenario-name }}
     steps:
       - uses: actions/checkout@v4
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v5
         with:
-          python-version: "3.11"
+          python-version: ${{ matrix.python-version }}
       - name: Update pip
         run: |
           python -m pip install --upgrade pip
       - name: Install dependencies
         run: |
           python -m pip install -e .[dev]
       - name: Test with pytest
+        id: pytest
         run: |
           pytest
-      - name: Setup SnowCLI connection file
-        run: |
-          echo "[connections]" > config.toml
-          echo "[connections.myconnection]" >> config.toml
-          chmod 0600 config.toml
-      - name: Install snowcli
+      - name: Test Schemachange on ${{ matrix.os }} targeting ${{ env.SNOWFLAKE_DATABASE }}.${{ env.MY_TARGET_SCHEMA }} schema
         run: |
-          pip install snowflake-cli-labs
-      - name: Test Default snowcli connection
-        run: |
-          snow --version
-          echo "::group::Config Contents"
-          more config.toml
+          echo "::group::Setting up ${MY_TARGET_SCHEMA}"
+          schemachange deploy --config-folder ./demo/setup/${SCENARIO_NAME}
           echo "::endgroup::"
-          snow --config-file config.toml connection test -c myconnection
-          echo "Tested SnowCLI connection successfully!"
-          echo "Set default connection"
-          snow --config-file config.toml connection set-default "myconnection"
-      - name: Test Basic Schemachange
-        run: |
-          snow --config-file config.toml sql -f ./demo/setup/setup_basics_demo.sql
-          schemachange --config-folder ./demo/basics
-          snow --config-file config.toml sql -f ./demo/teardown/teardown_basics_demo.sql
-      - name: Test Citibike Demo
-        run: |
-          snow --config-file config.toml sql -f ./demo/setup/setup_citibike_demo.sql
-          schemachange --config-folder ./demo/citibike
-          snow --config-file config.toml sql -f ./demo/teardown/teardown_citibike_demo.sql
-      - name: Test Citibike Demo Jinja Template
-        run: |
-          snow --config-file config.toml sql -f ./demo/setup/setup_citibike_demo_jinja.sql
-          schemachange --config-folder ./demo/citibike_jinja
-          snow --config-file config.toml sql -f ./demo/teardown/teardown_citibike_demo_jinja.sql
+          echo "::group::Testing Rendering to ${MY_TARGET_SCHEMA}"
+          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/A__render.sql
+          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/R__render.sql
+          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/V1.0.0__render.sql
+          echo "::endgroup::"
+          echo "::group::Testing Deployment using ${MY_TARGET_SCHEMA}"
+          set +e
+          schemachange deploy --config-folder ./demo/${SCENARIO_NAME}
+          RESULT=$?
+          if [ $RESULT -eq 0 ]; then
+            echo "Deployment Completed!"
+          else
+            echo "Deployment Failed. Proceeding to Teardown."
+          fi
+          echo "::endgroup::"
+          set -e
+          echo "::group::Tearing down up ${MY_TARGET_SCHEMA}"
+          schemachange deploy --config-folder ./demo/teardown/${SCENARIO_NAME}
+          echo "::endgroup::"
+          if [ $RESULT -ne 0 ]; then
+            exit 1
+          fi

.github/workflows/master-pytest.yml

@@ -1,32 +1,50 @@
 name: master-pytest
 
+defaults:
+  run:
+    shell: bash
+
 on:
+  pull_request:
+    branches:
+      - main
+    paths-ignore:
+      - '**/*.md'
+      - '**/*.MD'
+      - '*.md'
+      - '*.MD'
   push:
+    tags:
+      - "v*" # Push events to matching v*, i.e. v3.6.2, v4.0.0
     branches:
-      - master
-
-  pull_request:
-    types: [opened, reopened]
+      - main
+    paths-ignore:
+      - '**/*.md'
+      - '**/*.MD'
+      - '*.md'
+      - '*.MD'
 
 jobs:
   build:
-    env:
-      SNOWFLAKE_PASSWORD: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_PASSWORD }}
-      SNOWFLAKE_USER: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_USER }}
-      SNOWFLAKE_ACCOUNT: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_ACCOUNT }}
-      SNOWFLAKE_CONNECTIONS_MYCONNECTION_PASSWORD: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_PASSWORD }}
-      SNOWFLAKE_CONNECTIONS_MYCONNECTION_ACCOUNT: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_ACCOUNT }}
-      SNOWFLAKE_CONNECTIONS_MYCONNECTION_USER: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_USER }}
-    runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
+        scenario-name: [ 'basics_demo', 'citibike_demo', 'citibike_demo_jinja']
         os: ["ubuntu-latest", "macos-latest", "windows-latest"]
         python-version: ["3.8", "3.9", "3.10", "3.11"]
-
+    runs-on: ${{ matrix.os }}
+    env:
+      SNOWFLAKE_PASSWORD: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_PASSWORD }}
+      SNOWFLAKE_USER: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_USER }}
+      SNOWFLAKE_ACCOUNT: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_ACCOUNT }}
+      SNOWFLAKE_DATABASE: SCHEMACHANGE_DEMO
+      SNOWFLAKE_WAREHOUSE: SCHEMACHANGE_DEMO_WH
+      SNOWFLAKE_ROLE: '"SCHEMACHANGE_DEMO-DEPLOY"'
+      MY_TARGET_SCHEMA: ${{ matrix.scenario-name }}_${{ github.run_number }}_${{ strategy.job-index }}
+      SCENARIO_NAME: ${{ matrix.scenario-name }}
     steps:
       - uses: actions/checkout@v4
-      - name: Set up Python ${{ matrix.python-version }} on ${{ matrix.os }}
+      - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
@@ -39,36 +57,30 @@ jobs:
       - name: Test with pytest
         run: |
           pytest
-      - name: Setup SnowCLI connection file
+      - name: Test Schemachange on ${{ matrix.os }} targeting ${{ env.SNOWFLAKE_DATABASE }}.${{ env.MY_TARGET_SCHEMA }} schema
         run: |
-          echo "[connections]" > config.toml
-          echo "[connections.myconnection]" >> config.toml
-          chmod 0600 config.toml
-      - name: Install snowcli
-        run: |
-          pip install snowflake-cli-labs
-      - name: Test Default snowcli connection
-        run: |
-          snow --version
-          echo "::group::Config Contents"
-          more config.toml
+          echo "::group::Setting up ${MY_TARGET_SCHEMA}"
+          schemachange deploy --config-folder ./demo/setup/${SCENARIO_NAME}
           echo "::endgroup::"
-          snow --config-file config.toml connection test -c myconnection
-          echo "Tested SnowCLI connection successfully!"
-          echo "Set default connection"
-          snow --config-file config.toml connection set-default "myconnection"
-      - name: Test Basic Schemachange
-        run: |
-          snow --config-file config.toml sql -f ./demo/setup/setup_basics_demo.sql
-          schemachange --config-folder ./demo/basics
-          snow --config-file config.toml sql -f ./demo/teardown/teardown_basics_demo.sql
-      - name: Test Citibike Demo
-        run: |
-          snow --config-file config.toml sql -f ./demo/setup/setup_citibike_demo.sql
-          schemachange --config-folder ./demo/citibike
-          snow --config-file config.toml sql -f ./demo/teardown/teardown_citibike_demo.sql
-      - name: Test Citibike Demo Jinja Template
-        run: |
-          snow --config-file config.toml sql -f ./demo/setup/setup_citibike_demo_jinja.sql
-          schemachange --config-folder ./demo/citibike_jinja
-          snow --config-file config.toml sql -f ./demo/teardown/teardown_citibike_demo_jinja.sql
+          echo "::group::Testing Rendering to ${MY_TARGET_SCHEMA}"
+          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/A__render.sql
+          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/R__render.sql
+          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/V1.0.0__render.sql
+          echo "::endgroup::"
+          echo "::group::Testing Deployment using ${MY_TARGET_SCHEMA}"
+          set +e
+          schemachange deploy --config-folder ./demo/${SCENARIO_NAME}
+          RESULT=$?
+          if [ $RESULT -eq 0 ]; then
+            echo "Deployment Completed!"
+          else
+            echo "Deployment Failed. Proceeding to Teardown."
+          fi
+          echo "::endgroup::"
+          set -e
+          echo "::group::Tearing down up ${MY_TARGET_SCHEMA}"
+          schemachange deploy --config-folder ./demo/teardown/${SCENARIO_NAME}
+          echo "::endgroup::"
+          if [ $RESULT -ne 0 ]; then
+            exit 1
+          fi

.gitignore

@@ -131,4 +131,4 @@ dmypy.json
 .idea
 
 # Mac files
-**/.DS_Store
+**/.DS_Store

.pre-commit-config.yaml

@@ -29,4 +29,11 @@ repos:
       - id: ruff
         args: [ --fix ]
       # Run the formatter.
-      - id: ruff-format
+      - id: ruff-format
+
+  - repo: git@github.com:GitGuardian/ggshield.git
+    rev: v1.28.0
+    hooks:
+      - id: ggshield
+        language_version: python3
+        stages: [commit]

CHANGELOG.md

@@ -3,9 +3,16 @@ All notable changes to this project will be documented in this file.
 
 *The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).*
 
-## [3.6.2] - 2023-12-07
+## [3.6.2] - 2024-05-03 (TBD)
 ### Changed
-- Aligning with snowflake [identifier requirements](https://docs.snowflake.com/en/sql-reference/identifiers-syntax). 
+- Aligning with snowflake [identifier requirements](https://docs.snowflake.com/en/sql-reference/identifiers-syntax) in the configuration settings
+- Added Session ID as part of the initial connection successful message to be visible in the logs
+- Removed pandas library dependency to improve schemachange install footprint
+- Fixed the bug with `Missing default warehouse`
+- Fixed Demo examples resulting from change in public data set location
+- Improved unit test coverage
+- Updated Github Actions Workflow to check PRs and Merges does not break the demo examples.
+- Updated Docs related to latest Demo content included in schemachange
 
 ## [3.6.1] - 2023-11-15
 ### Added

README.md

@@ -168,7 +168,7 @@ A secret is just a standard variable that has been tagged as a secret. This is d
 ### Jinja templating engine
 schemachange uses the Jinja templating engine internally and supports: [expressions](https://jinja.palletsprojects.com/en/3.0.x/templates/#expressions), [macros](https://jinja.palletsprojects.com/en/3.0.x/templates/#macros), [includes](https://jinja.palletsprojects.com/en/3.0.x/templates/#include) and [template inheritance](https://jinja.palletsprojects.com/en/3.0.x/templates/#template-inheritance).
 
-These files can be stored in the root-folder but schemachange also provides a separate modules folder `--modules-folder`. This allows common logic to be stored outside of the main changes scripts. The [demo/citibike_jinja](demo/citibike_jinja) has a simple example that demonstrates this.
+These files can be stored in the root-folder but schemachange also provides a separate modules folder `--modules-folder`. This allows common logic to be stored outside of the main changes scripts. The [demo/citibike_demo_jinja](demo/citibike_demo_jinja) has a simple example that demonstrates this.
 
 The Jinja autoescaping feature is disabled in schemachange, this feature in Jinja is currently designed for where the output language is HTML/XML. So if you are using schemachange with untrusted inputs you will need to handle this within your change scripts.
 
@@ -225,7 +225,7 @@ Default [Password](https://docs.snowflake.com/en/user-guide/python-connector-exa
 [Key Pair](https://docs.snowflake.com/en/user-guide/python-connector-example.html#using-key-pair-authentication) Authenticator| `snowflake`
 [External Oauth](https://docs.snowflake.com/en/user-guide/oauth-external.html) | `oauth`
 [Browser based SSO](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-use.html#setting-up-browser-based-sso) | `externalbrowser`
-[Programmatic SSO](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-use.html#native-sso-okta-only) (Okta Only) | Okta URL endpoing for your Okta account typically in the form `https://<okta_account_name>.okta.com` OR `https://<okta_account_name>.oktapreview.com`
+[Programmatic SSO](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-use.html#native-sso-okta-only) (Okta Only) | Okta URL endpoint for your Okta account typically in the form `https://<okta_account_name>.okta.com` OR `https://<okta_account_name>.oktapreview.com`
 
 If an authenticator is unsupported, then schemachange will default to `snowflake`. If the authenticator is `snowflake`, and both password and key pair values are provided then schemachange will use the password over the key pair values.