Unit tests do not respect macro variables

[m1hawkgsm]

We're on SQLMesh version 0.164.0, and we have a test of the form:

test_claim_model:
    model: "{{ var('silver') }}.client_x.claim"
    inputs:
      "BRONZE.client_x.CLAIM":
        file: tests/silver/client_x/data/bronze_claim.csv
      "@{SILVER}.client_x.INSURANCE_PLAN":
        file: tests/silver/client_x/data/silver_insurance_plan.csv
    outputs:
      query:
        file: tests/silver/client_x/data/silver_claim.csv

Now, silver is a variable that is well defined in our config.py, and I can verify this because every single model has a database parameterization on a variable such as silver, and sqlmesh plan/sqlmesh run have worked fine.

However, running sqlmesh test, consistently we get:

[WARNING] Model '"NONE"."client_x"."CLAIM"' was not found at /Users/gesus/Repos/core/adonis-core/adonis_sqlmesh/integrations/tests/silver/client_x/test_claim.yaml

As you can see, as well, we follow this document and even mix different modalities of using variables.

Am I missing something obvious or do we have a bug here?

[georgesittas]

I can't reproduce what you're seeing against the latest main branch. Variable replacement works fine for a project like this:

# test.yaml
test_claim_model:
  model: "{{ var('silver') }}.client_x.claim"
  inputs:
    "bronze.client_x.claim":
      - x: 1
    "{{ var('silver') }}.client_x.insurance_plan":
      - x: 1
  outputs:
    query:
      - x: 1

-- test.sql
MODEL (
  name @silver.client_x.claim,
  kind FULL
);

SELECT 1 AS x
FROM bronze.client_x.claim AS t1, @silver.client_x.insurance_plan AS t2
WHERE t1.x = 1 AND t2.x = 1

# config.yaml
gateways:
  duckdb:
    connection:
      type: duckdb
      database: db.db

default_gateway: duckdb

model_defaults:
  dialect: duckdb
  start: 2025-03-25

variables:
  silver: silver

Going to close this as a non-issue, but happy to continue discussing. My guess is there's probably some configuration issue on your side. Otherwise, it may be a dialect-specific issue, but will need more details on that to debug it.

[m1hawkgsm]

Have you tested on 0.164.0? I mention it very specifically for a reason.

[viniciusnunest-torc]

I just tried the same snippet that you sent @georgesittas and didn't work for me !

Version : 0.164.0

[WARNING] Model '"NONE"."x"."CLAIM"' was not found at /Users/viniciustavares/core/adonis-core/adonis_sqlmesh/integrations/tests/gold/test.yaml

----------------------------------------------------------------------
Ran 0 tests in 0.000s```

And the yaml :

test_claim_model:
model: "{{ var('SILVER') }}.ripplcare_33.claim"
inputs:
"bronze.ripplcare_33.claim":
- x: 1
"{{ var('SILVER') }}.ripplcare_33.insurance_plan":
- x: 1
outputs:
query:
- x: 1

[georgesittas]

Test runs fine for me in 0.164.0 as well:

(.venv) ➜  test_env sqlmesh test
.
----------------------------------------------------------------------
Ran 1 test in 0.012s

OK
(.venv) ➜  test_env pip show sqlmesh
Name: sqlmesh
Version: 0.164.0

Both with duckdb and snowflake as the default dialect:

# config.yaml
gateways:
  duckdb:
    connection:
      type: duckdb
      database: db.db

default_gateway: duckdb

model_defaults:
  dialect: snowflake
  start: 2025-03-25

variables:
  silver: silver

I think I'll need more details about your setup to see if I can repro this.

[viniciusnunest-torc]

@georgesittas This is our var setup : (The strange thing is, work for models, but not for unit tests)

if environment == Environment.PRODUCTION:
    DEFAULT_DEV_ENVIRONMENT = "prod"
    gateway_variables_dictionary = {
        "gold": "GOLD",
        "GOLD": "GOLD",
        "silver": "SILVER",
        "SILVER": "SILVER",
        "bronze": "BRONZE",
        "BRONZE": "BRONZE",
    }
else:  # Dev for now, in the future we'll have dev AND staging for real.
    DEFAULT_DEV_ENVIRONMENT = raw_target_db_config.user
    print_statement_color = "bright_green"
    gateway_variables_dictionary = {x: "DEV_SQLMESH" for x in ["gold", "GOLD", "silver", "SILVER", "bronze", "BRONZE"]} 


config: Config = Config(
    cicd_bot=GithubCICDBotConfig(
        invalidate_environment_after_deploy=True,
        skip_pr_backfill=False,
        auto_categorize_changes=CategorizerConfig(
            external=AutoCategorizationMode.FULL,
            python=AutoCategorizationMode.FULL,
            sql=AutoCategorizationMode.FULL,
            seed=AutoCategorizationMode.FULL,
        ),
        pr_environment_name=PR_ENV_PREFIX,
    ),
    default_gateway=environment.value,
    default_target_environment=DEFAULT_DEV_ENVIRONMENT,
    environment_ttl=ENVIRONMENT_TTL,
    gateways={
        environment.value: GatewayConfig(
            connection=sqlmesh_target_db_config,
            state_connection=sqlmesh_state_db_config,
            scheduler=BuiltInSchedulerConfig(),
            state_schema=STATE_SCHEMA,
            variables=gateway_variables_dictionary,
        )
    },