Docs: update incremental by partition concepts (#3688)

Author: treysp

docs/concepts/models/model_kinds.md

@@ -121,9 +121,9 @@ WHERE
 ```
 
 ### Idempotency
-It is recommended that queries of models of this kind are [idempotent](../glossary.md#idempotency) to prevent unexpected results during data [restatement](../plans.md#restatement-plans).
+We recommend making sure incremental by time range model queries are [idempotent](../glossary.md#idempotency) to prevent unexpected results during data [restatement](../plans.md#restatement-plans).
 
-Note, however, that upstream models and tables can impact a model's idempotency. For example, referencing an upstream model of kind [FULL](#full) in the model query automatically causes the model to be non-idempotent.
+Note, however, that upstream models and tables can impact a model's idempotency. For example, referencing an upstream model of kind [FULL](#full) in the model query automatically causes the model to be non-idempotent because its data could change on every model execution.
 
 ### Materialization strategy
 Depending on the target engine, models of the `INCREMENTAL_BY_TIME_RANGE` kind are materialized using the following strategies:
@@ -142,11 +142,25 @@ Depending on the target engine, models of the `INCREMENTAL_BY_TIME_RANGE` kind a
 
 Models of the `INCREMENTAL_BY_PARTITION` kind are computed incrementally based on partition. A set of columns defines the model's partitioning key, and a partition is the group of rows with the same partitioning key value.
 
-This model kind is designed for the scenario where data rows should be loaded and updated as a group based on their shared value for the partitioning key. This kind may be used with any SQL engine; SQLMesh will automatically create partitioned tables on engines that support explicit table partitioning (e.g., [BigQuery](https://cloud.google.com/bigquery/docs/creating-partitioned-tables), [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-partition.html)).
+!!! info "Should you use this model kind?"
 
-If a partitioning key in newly loaded data is not present in the model table, the new partitioning key and its data rows are inserted. If a partitioning key in newly loaded data is already present in the model table, **all the partitioning key's existing data rows in the model table are replaced** with the partitioning key's data rows in the newly loaded data. If a partitioning key is present in the model table but not present in the newly loaded data, the partitioning key's existing data rows are not modified and remain in the model table.
+    Any model kind can use a partitioned table by specifying the [`partitioned_by` key](../models/overview.md#partitioned_by) in the `MODEL` DDL. The "partition" in `INCREMENTAL_BY_PARTITION` is about how the data is **loaded** when the model runs.
 
-This kind is a good fit for datasets that have the following traits:
+    `INCREMENTAL_BY_PARTITION` models are inherently [non-idempotent](../glossary.md#idempotency), so restatements and other actions can cause data loss. This makes them more complex to manage than other model kinds.
+
+    In most scenarios, an `INCREMENTAL_BY_TIME_RANGE` model can meet your needs and will be easier to manage. The `INCREMENTAL_BY_PARTITION` model kind should only be used when the data must be loaded by partition (usually for performance reasons).
+
+This model kind is designed for the scenario where data rows should be loaded and updated as a group based on their shared value for the partitioning key.
+
+It may be used with any SQL engine. SQLMesh will automatically create partitioned tables on engines that support explicit table partitioning (e.g., [BigQuery](https://cloud.google.com/bigquery/docs/creating-partitioned-tables), [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-partition.html)).
+
+New rows are loaded based on their partitioning key value:
+
+- If a partitioning key in newly loaded data is not present in the model table, the new partitioning key and its data rows are inserted.
+- If a partitioning key in newly loaded data is already present in the model table, **all the partitioning key's existing data rows in the model table are replaced** with the partitioning key's data rows in the newly loaded data.
+- If a partitioning key is present in the model table but not present in the newly loaded data, the partitioning key's existing data rows are not modified and remain in the model table.
+
+This kind should only be used for datasets that have the following traits:
 
 * The dataset's records can be grouped by a partitioning key.
 * Each record has a partitioning key associated with it.
@@ -183,12 +197,22 @@ MODEL (
 );
 ```
 
-This is a fuller example of how you would use this model kind in practice to avoid backfilling too many partitions and/or limiting the partitions to backfill based on time ranges. 
+!!! warning "Only full restatements supported"
+
+    Partial data [restatements](../plans.md#restatement-plans) are used to reprocess part of a table's data (usually a limited time range).
+
+    Partial data restatement is not supported for `INCREMENTAL_BY_PARTITION` models. If you restate an `INCREMENTAL_BY_PARTITION` model, its entire table will be recreated from scratch.
+
+    Restating `INCREMENTAL_BY_PARTITION` models may lead to data loss and should be performed with care.
+
+### Example
+
+This is a fuller example of how you would use this model kind in practice. It limits the number of partitions to backfill based on time range in the `partitions_to_update` CTE.
 
 ```sql linenums="1"
 MODEL (
   name demo.incremental_by_partition_demo,
-  kind INCREMENTAL_BY_PARTITION, 
+  kind INCREMENTAL_BY_PARTITION,
   partitioned_by user_segment,
 );
 
@@ -221,7 +245,7 @@ SELECT
   usage_count,
   feature_utilization_score,
   user_segment,
-  CASE 
+  CASE
     WHEN usage_count > 100 AND feature_utilization_score > 0.7 THEN 'Power User'
     WHEN usage_count > 50 THEN 'Regular User'
     WHEN usage_count IS NULL THEN 'New User'
@@ -357,18 +381,18 @@ Redshift supports only the `UPDATE` or `DELETE` actions for the `WHEN MATCHED` c
 
 ### Merge Filter Expression
 
-The `MERGE` statement typically induces a full table scan of the existing table, which can be problematic with large data volumes. 
+The `MERGE` statement typically induces a full table scan of the existing table, which can be problematic with large data volumes.
 
 Prevent a full table scan by passing filtering conditions to the `merge_filter` parameter.
 
-The `merge_filter` accepts a single or a conjunction of predicates to be used in the `ON` clause of the `MERGE` operation: 
+The `merge_filter` accepts a single or a conjunction of predicates to be used in the `ON` clause of the `MERGE` operation:
 
 ```sql linenums="1" hl_lines="5"
 MODEL (
   name db.employee_contracts,
   kind INCREMENTAL_BY_UNIQUE_KEY (
     unique_key id,
-    merge_filter source._operation IS NULL AND target.contract_date > dateadd(day, -7, current_date) 
+    merge_filter source._operation IS NULL AND target.contract_date > dateadd(day, -7, current_date)
   )
 );
 ```
@@ -935,7 +959,7 @@ GROUP BY
 
 ### Reset SCD Type 2 Model (clearing history)
 
-SCD Type 2 models are designed by default to protect the data that has been captured because it is not possible to recreate the history once it has been lost. 
+SCD Type 2 models are designed by default to protect the data that has been captured because it is not possible to recreate the history once it has been lost.
 However, there are cases where you may want to clear the history and start fresh.
 For this use use case you will want to start by setting `disable_restatement` to `false` in the model definition.
 
@@ -949,9 +973,9 @@ MODEL (
 );
 ```
 
-Plan/apply this change to production. 
+Plan/apply this change to production.
 Then you will want to [restate the model](../plans.md#restatement-plans).
-    
+
 ```bash
 sqlmesh plan --restate-model db.menu_items
 ```