Skip to content

Commit 5ec13d2

Browse files
authored
Feat: Enable INCREMENTAL_UNMANAGED models in native projects (#3943)
1 parent dba7f02 commit 5ec13d2

File tree

3 files changed

+65
-1
lines changed

3 files changed

+65
-1
lines changed

docs/concepts/models/model_kinds.md

+28-1
Original file line numberDiff line numberDiff line change
@@ -1552,4 +1552,31 @@ Depending on the target engine, models of the `INCREMENTAL_BY_PARTITION` kind ar
15521552
| BigQuery | DELETE by partitioning key, then INSERT |
15531553
| Redshift | DELETE by partitioning key, then INSERT |
15541554
| Postgres | DELETE by partitioning key, then INSERT |
1555-
| DuckDB | DELETE by partitioning key, then INSERT |
1555+
| DuckDB | DELETE by partitioning key, then INSERT |
1556+
1557+
## INCREMENTAL_UNMANAGED
1558+
1559+
The `INCREMENTAL_UNMANAGED` model kind exists to support append-only tables. It's "unmanaged" in the sense that SQLMesh doesnt try to manage how the data is loaded. SQLMesh will just run your query on the configured cadence and append whatever it gets into the table.
1560+
1561+
!!! question "Should you use this model kind?"
1562+
1563+
Some patterns for data management, such as Data Vault, may rely on append-only tables. In this situation, `INCREMENTAL_UNMANAGED` is the correct type to use.
1564+
1565+
In most other situations, you probably want `INCREMENTAL_BY_TIME_RANGE` or `INCREMENTAL_BY_UNIQUE_KEY` because they give you much more control over how the data is loaded.
1566+
1567+
Usage of the `INCREMENTAL_UNMANAGED` model kind is straightforward:
1568+
1569+
```sql linenums="1" hl_lines="3"
1570+
MODEL (
1571+
name db.events,
1572+
kind INCREMENTAL_UNMANAGED,
1573+
);
1574+
```
1575+
1576+
Since it's unmanaged, it doesnt support the `batch_size` and `batch_concurrency` properties to control how data is loaded like the other incremental model types do.
1577+
1578+
!!! warning "Only full restatements supported"
1579+
1580+
Similar to `INCREMENTAL_BY_PARTITION`, attempting to [restate](../plans.md#restatement-plans) an `INCREMENTAL_UNMANAGED` model will trigger a full restatement. That is, the model will be rebuilt from scratch rather than from a time slice you specify.
1581+
1582+
This is because an append-only table is inherently non-idempotent. Restating `INCREMENTAL_UNMANAGED` models may lead to data loss and should be performed with care.

sqlmesh/core/dialect.py

+1
Original file line numberDiff line numberDiff line change
@@ -594,6 +594,7 @@ def parse(self: Parser) -> t.Optional[exp.Expression]:
594594
ModelKindName.INCREMENTAL_BY_TIME_RANGE,
595595
ModelKindName.INCREMENTAL_BY_UNIQUE_KEY,
596596
ModelKindName.INCREMENTAL_BY_PARTITION,
597+
ModelKindName.INCREMENTAL_UNMANAGED,
597598
ModelKindName.SEED,
598599
ModelKindName.VIEW,
599600
ModelKindName.SCD_TYPE_2,

tests/core/test_model.py

+36
Original file line numberDiff line numberDiff line change
@@ -3003,6 +3003,42 @@ def test_incremental_unmanaged_validation():
30033003
model.validate_definition()
30043004

30053005

3006+
def test_incremental_unmanaged():
3007+
expr = d.parse(
3008+
"""
3009+
MODEL (
3010+
name foo,
3011+
kind INCREMENTAL_UNMANAGED
3012+
);
3013+
3014+
SELECT x.a AS a FROM test.x AS x
3015+
"""
3016+
)
3017+
3018+
model = load_sql_based_model(expressions=expr)
3019+
3020+
assert isinstance(model.kind, IncrementalUnmanagedKind)
3021+
assert not model.kind.insert_overwrite
3022+
3023+
expr = d.parse(
3024+
"""
3025+
MODEL (
3026+
name foo,
3027+
kind INCREMENTAL_UNMANAGED (
3028+
insert_overwrite true
3029+
),
3030+
partitioned_by a
3031+
);
3032+
3033+
SELECT x.a AS a FROM test.x AS x
3034+
"""
3035+
)
3036+
3037+
model = load_sql_based_model(expressions=expr)
3038+
assert isinstance(model.kind, IncrementalUnmanagedKind)
3039+
assert model.kind.insert_overwrite
3040+
3041+
30063042
def test_custom_interval_unit():
30073043
assert (
30083044
load_sql_based_model(

0 commit comments

Comments
 (0)