Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FSTORE-1147] Online feature store notification system #343

Merged
merged 11 commits into from
Feb 16, 2024
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
111 changes: 111 additions & 0 deletions docs/user_guides/fs/feature_group/notification.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
---
description: Documentation on how notifications work for feature groups in Hopsworks.
---

# How notifications work for a Feature Group

## Introduction

Changes to online-enabled feature groups can be captured by listening to events on specified topics.
This optimizes the user experience by allowing users to proactively make predictions as soon as there is an update on the features.

In this guide you will learn how to enable notifications for online feature groups within Hopsworks, showing examples in HSFS APIs as well as the user interface.

## Prerequisites

Before you begin this guide we suggest you read the [Feature Group](../../../concepts/fs/feature_group/fg_overview.md) concept page to understand what a feature group is and how it fits in the ML pipeline.

### Create notification topic

To begin with, you will need to have a topic where the notifications will be sent.
To create a topic in Hopsworks you can navigate to the `Kafka` subsection of project `settings` and click on `Add a new topic`.

<p align="center">
<figure>
<img src="../../../../assets/images/guides/feature_group/manage_kafka.png" alt="Manage Kafka">
</figure>
</p>

After specifying topic details it can be created by clicking on the `Create Topic` button.

<p align="center">
<figure>
<img src="../../../../assets/images/guides/feature_group/create_kafka_topic.png" alt="Create kafka topic">
</figure>
</p>

## Using HSFS APIs

### Create a Feeature Group with a notification topic

To enable notifications for an online-enabled feature group using the HSFS APIs you need to [create a feature group](./create.md) and set the `notification_topic_name` properties value to the previously created topic.

=== "Python"

```python
fg = fs.create_feature_group(name="feature_group_name", version=feature_group_version, primary_key=feature_group_primary_keys, online_enabled=True, notification_topic_name="notification_topic_name")
```

### Update Feature Group notification topic

The notification topic name can be changed after the creation of the feature group.
By setting the `notification_topic_name` value to `None` or empty string notification will be disabled.
It can take some time for these changes to take place since the onlinefs service internally caches feature groups for a set amount of time.

=== "Python"

```python
fg.update_notification_topic_name(notification_topic_name="new_notification_topic_name")
```

## Using UI

### Create a Feeature Group with a notification topic

During the creation of the feature group enable online feature serving.
When enabled you will be able to set the `CDC topic name` property.
CDC stands for change data capture.

<p align="center">
<figure>
<img src="../../../../assets/images/guides/feature_group/create_online_enabled_feature_group.png" alt="Create online enabled feature group">
</figure>
</p>

### Update Feeature Group with notification topic

The notification topic name can be changed after creation by editing the feature group.
By setting the `CDC topic name` value to empty the notifications will be disabled.
It can take some time for changes to notifications to take place.

<p align="center">
<figure>
<img src="../../../../assets/images/guides/feature_group/edit_online_enabled_feature_group.png" alt="Edit online enabled feature group">
</figure>
</p>

## Example of notification event

Once properly set up the online feature store service will produce events to the provided notification topic when data ingestion is completed for records.

Here is an example output:

```
{
"projectId":119,
"featureStoreId":67,
"featureGroupId":14,
"entry":{
"id":"15",
"text":"test"
},
"featureViews":[
{
"id":9,
"name":"test",
"version":1,
"featurestoreId":67
}
]
}
```
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ nav:
- Create: user_guides/fs/feature_group/create.md
- Create External: user_guides/fs/feature_group/create_external.md
- Deprecating: user_guides/fs/feature_group/deprecation.md
- Notification: user_guides/fs/feature_group/notification.md
- Data Types and Schema management: user_guides/fs/feature_group/data_types.md
- Statistics: user_guides/fs/feature_group/statistics.md
- Data Validation: user_guides/fs/feature_group/data_validation.md
Expand Down