Model provenance - including init feature vector (#403)

Author: o-alex

docs/assets/images/guides/fs/provenance/provenance_fv.png

@@ -1,10 +1,28 @@
-# Provenance 
+# Provenance
 
-## Introduction 
+## Introduction
 
-Hopsworks feature store allows users to track provenance (lineage) between storage connectors, feature groups, feature views, training datasets and models. Tracking lineage allows users to determine where/if a feature group is being used. You can track if feature groups are being used to create additional (derived) feature groups or feature views.
+Hopsworks allows users to track provenance (lineage) between:
 
-You can interact with the provenance graph using the UI and the APIs.
+- storage connectors
+- feature groups
+- feature views
+- training datasets
+- models
+
+In the provenance pages we will call a provenance artifact or shortly artifact, any of the five entities above.
+
+With the following provenance graph:
+
+```
+storage connector -> feature group -> feature group -> feature view -> training dataset -> model
+```
+
+we will call the parent, the artifact to the left, and the child, the artifact to the right. So a feature view has a number of feature groups as parents and can have a number of training datasets as children.
+
+Tracking provenance allows users to determine where and if an artifact is being used. You can track, for example, if feature groups are being used to create additional (derived) feature groups or feature views, or if their data is eventually used to train models.
+
+You can interact with the provenance graph using the UI or the APIs.
 
 ## Step 1: Storage connector lineage
 
@@ -28,7 +46,7 @@ The relationship between storage connectors and feature groups is captured autom
 
 ### Using the APIs
 
-Starting from a feature group metadata object, you can traverse upstream the provenance graph to retrieve the metadata objects of the storage connectors that are part of the feature group. To do so, you can use the [get_storage_connector_provenance](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#get_storage_connector_provenance) method.
+Starting from a feature group metadata object, you can traverse upstream the provenance graph to retrieve the metadata objects of the storage connectors that are part of the feature group. To do so, you can use the [get_storage_connector_provenance](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#get_storage_connector_provenance) method.
 
 === "Python"
 
@@ -53,7 +71,7 @@ Starting from a feature group metadata object, you can traverse upstream the pro
     user_profiles_fg.get_storage_connector()
     ```
 
-To traverse the provenance graph in the opposite direction (i.e. from the storage connector to the feature group), you can use the [get_feature_groups_provenance](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/storage_connector_api/#get_feature_groups_provenance) method. When navigating the provenance graph downstream, the `deleted` feature groups are not tracked by provenance, as such, the `deleted` property will always return an empty list.
+To traverse the provenance graph in the opposite direction (i.e. from the storage connector to the feature group), you can use the [get_feature_groups_provenance](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/storage_connector_api/#get_feature_groups_provenance) method. When navigating the provenance graph downstream, the `deleted` feature groups are not tracked by provenance, as such, the `deleted` property will always return an empty list.
 
 === "Python"
 
@@ -79,15 +97,15 @@ To traverse the provenance graph in the opposite direction (i.e. from the storag
 
 ### Assign parents to a feature group
 
-When creating a feature group, it is possible to specify a list of feature groups used to create the derived features. For example, you could have an external feature group defined over a Snowflake or Redshift table, which you use to compute the features and save them in a feature group. You can mark the external feature group as parent of the feature group you are creating by using the `parents` parameter in the [get_or_create_feature_group](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#get_or_create_feature_group) or [create_feature_group](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#create_feature_group) methods:
+When creating a feature group, it is possible to specify a list of feature groups used to create the derived features. For example, you could have an external feature group defined over a Snowflake or Redshift table, which you use to compute the features and save them in a feature group. You can mark the external feature group as parent of the feature group you are creating by using the `parents` parameter in the [get_or_create_feature_group](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#get_or_create_feature_group) or [create_feature_group](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#create_feature_group) methods:
 
 === "Python"
 
     ```python
     # Retrieve the feature group
     profiles_fg = fs.get_external_feature_group("user_profiles", version=1)
 
-    # Do feature engineering 
+    # Do feature engineering
     age_df = transaction_df.merge(profiles_fg.read(), on="cc_num", how="left")
     transaction_df["age_at_transaction"] = (age_df["datetime"] - age_df["birthdate"]) / np.timedelta64(1, "Y")
 
@@ -103,7 +121,7 @@ When creating a feature group, it is possible to specify a list of feature group
     transaction_fg.insert(transaction_df)
     ```
 
-Another example use case for derived feature group is if you have a feature group containing features with daily resolution and you are using the content of that feature group to populate a second feature group with monthly resolution: 
+Another example use case for derived feature group is if you have a feature group containing features with daily resolution and you are using the content of that feature group to populate a second feature group with monthly resolution:
 
 === "Python"
 
@@ -112,7 +130,7 @@ Another example use case for derived feature group is if you have a feature grou
     daily_transaction_fg = fs.get_feature_group("daily_transaction", version=1)
     daily_transaction_df = daily_transaction_fg.read()
 
-    # Do feature engineering 
+    # Do feature engineering
     cc_group = daily_transaction_df[["cc_num", "amount", "datetime"]] \
                     .groupby("cc_num") \
                     .rolling("1M", on="datetime")
@@ -132,7 +150,7 @@ Another example use case for derived feature group is if you have a feature grou
 
 ### List feature group parents
 
-You can query the provenance graph of a feature group using the UI and the APIs. From the APIs you can list the parent feature groups by calling the method [get_parent_feature_groups](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#get_parent_feature_groups)
+You can query the provenance graph of a feature group using the UI and the APIs. From the APIs you can list the parent feature groups by calling the method [get_parent_feature_groups](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#get_parent_feature_groups)
 
 === "Python"
 
@@ -151,7 +169,7 @@ You can query the provenance graph of a feature group using the UI and the APIs.
 
 A parent is marked as `deleted` (and added to the deleted list) if the parent feature group was deleted. `inaccessible` if you no longer have access to the parent feature group (e.g. the parent feature group belongs to a project you no longer have access to).
 
-To traverse the provenance graph in the opposite direction (i.e. from the parent feature group to the child), you can use the [get_generate_feature_groups](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#get_generated_feature_groups) method. When navigating the provenance graph downstream, the `deleted` feature groups are not tracked by provenance, as such, the `deleted` property will always return an empty list.
+To traverse the provenance graph in the opposite direction (i.e. from the parent feature group to the child), you can use the [get_generate_feature_groups](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#get_generated_feature_groups) method. When navigating the provenance graph downstream, the `deleted` feature groups are not tracked by provenance, as such, the `deleted` property will always return an empty list.
 
 === "Python"
 
@@ -180,7 +198,7 @@ The relationship between feature groups and feature views is captured automatica
 
 ### Using the APIs
 
-Starting from a feature view metadata object, you can traverse upstream the provenance graph to retrieve the metadata objects of the feature groups that are part of the feature view. To do so, you can use the [get_parent_feature_groups](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_view_api/#get_parent_feature_groups) method.
+Starting from a feature view metadata object, you can traverse upstream the provenance graph to retrieve the metadata objects of the feature groups that are part of the feature view. To do so, you can use the [get_parent_feature_groups](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/feature_view_api/#get_parent_feature_groups) method.
 
 === "Python"
 
@@ -204,14 +222,37 @@ You can also traverse the provenance graph in the opposite direction. Starting f
     ```python
     lineage = transaction_fg.get_generated_feature_views()
 
-    # List all accessible downstream feature views 
+    # List all accessible downstream feature views
     lineage.accessible
 
-    # List all the inaccessible downstream feature views 
+    # List all the inaccessible downstream feature views
     lineage.inaccessible
     ```
 
-### Using the UI 
+Users can call the [get_models_provenance](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/feature_view_api/#get_models_provenance) method which will return a [Link](#provenance-links) object.
+
+You can also retrive directly the accessible models, without the need to extract them from the provenance links object:
+=== "Python"
+
+    ```python
+    #List all accessible models
+    models = fraud_fv.get_models()
+
+    #List accessible models trained from a specific training dataset version
+    models = fraud_fv.get_models(training_dataset_version: 1)
+    ```
+
+Also we added a utility method to retrieve from the user's accessible models, the last trained one. Last is determined based on timestamp when it was saved into the model registry.
+=== "Python"
+
+    ```python
+    #Retrieve newest model from all user's accessible models based on this feature view
+    model = fraud_fv.get_newest_model()
+    #Retrieve newest model from all user's accessible models based on this training dataset version
+    model = fraud_fv.get_newest_model(training_dataset_version: 1)
+    ```
+
+### Using the UI
 
 In the feature view overview UI you can explore the provenance graph of the feature view:
 
@@ -221,3 +262,11 @@ In the feature view overview UI you can explore the provenance graph of the feat
     <figcaption>Feature view provenance graph</figcaption>
   </figure>
 </p>
+
+## Provenance Links
+
+All the `_provenance` methods return a `Link` dictionary object that contains `accessible`, `inaccesible`, `deleted` lists.
+
+- `accessible` - contains any artifact from the result, that the user has access to.
+- `inaccessible` - contains any artifacts that might have been shared at some point in the past, but where this sharing was retracted. Since the relation between artifacts is still maintained in the provenance, the user will only have access to limited metadata and the artifacts will be included in this `inaccessible` list.
+- `deleted` - contains artifacts that are deleted with children stil present in the system. There is minimum amount of metadata for the deleted allowing for some limited human readable identification.

docs/assets/images/guides/mlops/provenance/provenance_model.png

@@ -0,0 +1,109 @@
+# Provenance
+
+## Introduction
+
+Hopsworks allows users to track provenance (lineage) between:
+
+- storage connectors
+- feature groups
+- feature views
+- training datasets
+- models
+
+In the provenance pages we will call a provenance artifact or shortly artifact, any of the five entities above.
+
+With the following provenance graph:
+
+```
+storage connector -> feature group -> feature group -> feature view -> training dataset -> model
+```
+
+we will call the parent, the artifact to the left, and the child, the artifact to the right. So a feature view has a number of feature groups as parents and can have a number of training datasets as children.
+
+Tracking provenance allows users to determine where and if an artifact is being used. You can track, for example, if feature groups are being used to create additional (derived) feature groups or feature views, or if their data is eventually used to train models.
+
+You can interact with the provenance graph using the UI or the APIs.
+
+## Model provenance
+
+The relationship between feature views and models is captured in the model [constructor](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/model_registry/model_api/#create_model). If you do not provide at least the feature view object to the constructor, the provenance will not capture this relation and you will not be able to navigate from model to the feature view it used or from the feature view to this model.
+
+You can provide the feature view object and have the training dataset version be inferred.
+
+=== "Python"
+
+    ```python
+    # this fv object will be provided to the model constructor
+    fv = hsfs.get_feature_view(...)
+
+    # when calling trainig data related methods on the feature view, the training dataset version is cached in the feature view and is implicitly provided to the model constructor
+    X_train, X_test, y_train, y_test = feature_view.train_test_split(...)
+
+    # provide the feature_view object in the model constructor
+    hsml.model_registry.ModelRegistry.python.create_model(
+        ...
+        feature_view = fv
+        ...)
+    ```
+
+You can of course explicitly provide the training dataset version.
+=== "Python"
+
+    ```python
+    # this object will be provided to the model constructor
+    fv = hsfs.get_feature_view(...)
+
+    # this training dataset version will be provided to the model constructor
+    X_train, X_test, y_train, y_test = feature_view.get_train_test_split(training_dataset_version=1)
+
+    # provide the feature_view object in the model constructor
+    hsml.model_registry.ModelRegistry.python.create_model(
+        ...
+        feature_view = fv,
+        training_dataset_version = 1,
+        ...)
+    ```
+
+Once the relation is stored in the provenance graph, you can navigate the graph from model to feature view or training dataset and the other way around.
+
+Users can call the [get_feature_view_provenance(https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/model_registry/model_api/#get_feature_view_provenance) method or the [get_training_dataset_provenance(https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/model_registry/model_api/#get_training_dataset_provenance) method which will each return a [Link](#provenance-links) object.
+
+You can also retrieve directly the parent feature view object, without the need to extract them from the provenance links object, using the [get_feature_view(https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/model_registry/model_api/#get_feature_view ) method
+
+=== "Python"
+
+    ```python
+    feature_view = model.get_feature_view()
+    ```
+
+This utility method also has the options to initialize the required components for batch or online retrieval of feature vectors.
+
+=== "Python"
+
+    ```python
+    model.get_feature_view(init: bool = True, online: Optional[bool]: None)
+    ```
+
+By default, the base init for feature vector retrieval is enabled. In case you have a workflow that requires more particular options, you can disable this base init by setting the `init` to `false`.
+The method detects if it is running within a deployment and will initialize the feature vector retrieval for the serving.
+If the `online` argument is provided and `true` it will initialize for online feature vector retrieval.
+If the `online` argument is provided and `false` it will initialize the feature vector retrieval for batch scoring.
+
+### Using the UI
+
+In the model overview UI you can explore the provenance graph of the model:
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/mlops/provenance/provenance_model.png" alt="Model provenance graph">
+    <figcaption>Provenance graph of derived feature groups</figcaption>
+  </figure>
+</p>
+
+## Provenance Links
+
+All the `_provenance` methods return a `Link` dictionary object that contains `accessible`, `inaccesible`, `deleted` lists.
+
+- `accessible` - contains any artifact from the result, that the user has access to.
+- `inaccessible` - contains any artifacts that might have been shared at some point in the past, but where this sharing was retracted. Since the relation between artifacts is still maintained in the provenance, the user will only have access to limited metadata and the artifacts will be included in this `inaccessible` list.
+- `deleted` - contains artifacts that are deleted with children stil present in the system. There is minimum amount of metadata for the deleted allowing for some limited human readable identification.

docs/user_guides/fs/provenance/provenance.md

@@ -195,6 +195,7 @@ nav:
               - API Protocol: user_guides/mlops/serving/api-protocol.md
               - Troubleshooting: user_guides/mlops/serving/troubleshooting.md
           - Vector Database: user_guides/mlops/vector_database/index.md
+          - Provenance: user_guides/mlops/provenance/provenance.md
       - Migration:
           - 3.X to 4.0: user_guides/migration/40_migration.md
   - Setup and Administration: