Skip to content

Commit 6c2cf1d

Browse files
davitbzhdavitbzh
authored
[FSTORE-1008] enable interacting with java client to hopsworks - 3.9 (#432)
1 parent b751d8d commit 6c2cf1d

File tree

4 files changed

+121
-15
lines changed

4 files changed

+121
-15
lines changed

‎docs/user_guides/fs/compute_engines.md

+18-14
Original file line numberDiff line numberDiff line change
@@ -4,31 +4,31 @@ In order to execute a feature pipeline to write to the Feature Store, as well as
44
Hopsworks Feature Store APIs are built around dataframes, that means feature data is inserted into the Feature Store from a Dataframe and likewise when reading data from the Feature Store, it is returned
55
as a Dataframe.
66

7-
As such, Hopsworks supports three computational engines:
7+
As such, Hopsworks supports five computational engines:
88

99
1. [Apache Spark](https://spark.apache.org): Spark Dataframes and Spark Structured Streaming Dataframes are supported, both from Python environments (PySpark) and from Scala environments.
1010
2. [Python](https://www.python.org/): For pure Python environments without dependencies on Spark, Hopsworks supports [Pandas Dataframes](https://pandas.pydata.org/) and [Polars Dataframes](https://pola.rs/).
1111
3. [Apache Flink](https://flink.apache.org): Flink Data Streams are currently supported as an experimental feature from Java/Scala environments.
12-
3. [Apache Beam](https://beam.apache.org/) *experimental*: Beam Data Streams are currently supported as an experimental feature from Java/Scala environments.
12+
4. [Apache Beam](https://beam.apache.org/) *experimental*: Beam Data Streams are currently supported as an experimental feature from Java/Scala environments.
13+
5. [Java](https://www.java.com): For pure Java environments without dependencies on Spark, Hopsworks supports writing using List of POJO Objects.
1314

1415
Hopsworks supports running [compute on the platform itself](../../concepts/dev/inside.md) in the form of [Jobs](../projects/jobs/pyspark_job.md) or in [Jupyter Notebooks](../projects/jupyter/python_notebook.md).
1516
Alternatlively, you can also connect to Hopsworks using Python or Spark from [external environments](../../concepts/dev/outside.md), given that there is network connectivity.
1617

1718
## Functionality Support
1819

19-
Hopsworks is aiming to provide funtional parity between the computational engines, however, there are certain Hopsworks functionalities which are exclusive to the engines.
20+
Hopsworks is aiming to provide functional parity between the computational engines, however, there are certain Hopsworks functionalities which are exclusive to the engines.
2021

21-
| Functionality | Method | Spark | Python | Flink | Beam | Comment |
22-
| ----------------------------------------------------------------- | ------ | ----- | ------ | ------ | ------ | ------- |
23-
| Feature Group Creation from dataframes | [`FeatureGroup.create_feature_group()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#create_feature_group) | :white_check_mark: | :white_check_mark: | - | - | Currently Flink/Beam doesn't support registering feature group metadata. Thus it needs to be pre-registered before you can write real time features computed by Flink/Beam.|
24-
| Training Dataset Creation from dataframes | [`TrainingDataset.save()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/training_dataset_api/#save) | :white_check_mark: | - | - | - | Functionality was deprecated in version 3.0 |
25-
| Data validation using Great Expectations for streaming dataframes | [`FeatureGroup.validate()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#validate) [`FeatureGroup.insert_stream()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#insert_stream) | - | - | - | - | `insert_stream` does not perform any data validation even when a expectation suite is attached. |
26-
| Stream ingestion | [`FeatureGroup.insert_stream()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#insert_stream) | :white_check_mark: | - | :white_check_mark: | :white_check_mark: | Python/Pandas/Polars has currently no notion of streaming. |
27-
| Stream ingestion | [`FeatureGroup.insert_stream()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#insert_stream) | :white_check_mark: | - | :white_check_mark: | :white_check_mark: | Python/Pandas/Polars has currently no notion of streaming. |
28-
| Reading from Streaming Storage Connectors | [`KafkaConnector.read_stream()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/storage_connector_api/#read_stream) | :white_check_mark: | - | - | - | Python/Pandas/Polars has currently no notion of streaming. For Flink/Beam only write operations are supported |
29-
| Reading training data from external storage other than S3 | [`FeatureView.get_training_data()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_view_api/#get_training_data) | :white_check_mark: | - | - | - | Reading training data that was written to external storage using a Storage Connector other than S3 can currently not be read using HSFS APIs, instead you will have to use the storage's native client. |
30-
| Reading External Feature Groups into Dataframe | [`ExternalFeatureGroup.read()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/external_feature_group_api/#read) | :white_check_mark: | - | - | - | Reading an External Feature Group directly into a Pandas/Polars Dataframe is not supported, however, you can use the [Query API](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/query_api/) to create Feature Views/Training Data containing External Feature Groups. |
31-
| Read Queries containing External Feature Groups into Dataframe | [`Query.read()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/query_api/#read) | :white_check_mark: | - | - | - | Reading a Query containing an External Feature Group directly into a Pandas/Polars Dataframe is not supported, however, you can use the Query to create Feature Views/Training Data and write the data to a Storage Connector, from where you can read up the data into a Pandas/Polars Dataframe. |
22+
| Functionality | Method | Spark | Python | Flink | Beam | Java | Comment |
23+
| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------ | ---------------------- | ------------------ | ------------------ |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
24+
| Feature Group Creation from dataframes | [`FeatureGroup.create_feature_group()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#create_feature_group) | :white_check_mark: | :white_check_mark: | - | - | - | Currently Flink/Beam/Java doesn't support registering feature group metadata. Thus it needs to be pre-registered before you can write real time features computed by Flink/Beam. |
25+
| Training Dataset Creation from dataframes | [`TrainingDataset.save()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/training_dataset_api/#save) | :white_check_mark: | - | - | - | - | Functionality was deprecated in version 3.0 |
26+
| Data validation using Great Expectations for streaming dataframes | [`FeatureGroup.validate()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#validate) <br/> [`FeatureGroup.insert_stream()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#insert_stream) | - | - | - | - | - | `insert_stream` does not perform any data validation even when a expectation suite is attached. |
27+
| Stream ingestion | [`FeatureGroup.insert_stream()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_group_api/#insert_stream) | :white_check_mark: | - | :white_check_mark: | :white_check_mark: | :white_check_mark: | Python/Pandas/Polars has currently no notion of streaming. |
28+
| Reading from Streaming Storage Connectors | [`KafkaConnector.read_stream()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/storage_connector_api/#read_stream) | :white_check_mark: | - | - | - | - | Python/Pandas/Polars has currently no notion of streaming. For Flink/Beam/Java only write operations are supported |
29+
| Reading training data from external storage other than S3 | [`FeatureView.get_training_data()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/feature_view_api/#get_training_data) | :white_check_mark: | - | - | - | - | Reading training data that was written to external storage using a Storage Connector other than S3 can currently not be read using HSFS APIs, instead you will have to use the storage's native client. |
30+
| Reading External Feature Groups into Dataframe | [`ExternalFeatureGroup.read()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/external_feature_group_api/#read) | :white_check_mark: | - | - | - | - | Reading an External Feature Group directly into a Pandas/Polars Dataframe is not supported, however, you can use the [Query API](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/query_api/) to create Feature Views/Training Data containing External Feature Groups. |
31+
| Read Queries containing External Feature Groups into Dataframe | [`Query.read()`](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/generated/api/query_api/#read) | :white_check_mark: | - | - | - | - | Reading a Query containing an External Feature Group directly into a Pandas/Polars Dataframe is not supported, however, you can use the Query to create Feature Views/Training Data and write the data to a Storage Connector, from where you can read up the data into a Pandas/Polars Dataframe. |
3232

3333
## Python
3434

@@ -77,3 +77,7 @@ Apache Beam integration with Hopsworks feature store was only tested using Dataf
7777

7878
For more details head over to the [Getting Started Guide](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/integrations/java/beam).
7979

80+
## Java
81+
It is also possible to interact to Hopsworks feature store using pure Java environments without dependencies on Spark, Flink or Beam.
82+
83+
For more details head over to the [Getting Started Guide](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/java).

‎docs/user_guides/integrations/index.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,10 @@
33
Hopsworks is an open platform aiming to be accessible from a variety of tools. Learn in this section how to connect to Hopsworks from
44

55
- [Python](python)
6+
- [Java](java)
67
- [Databricks](databricks/networking)
78
- [AWS SageMaker](sagemaker)
89
- [AWS EMR](emr/networking)
910
- [Azure HDInsight](hdinsight)
1011
- [Azure Machine Learning](mlstudio_designer)
11-
- [Apache Spark](spark)
12+
- [Apache Spark](spark)

‎docs/user_guides/integrations/java.md

+100
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
---
2+
description: Documentation on how to connect to Hopsworks from a Java client.
3+
---
4+
5+
# Java client
6+
7+
Starting from version 3.9.0-RC13, HSFS provides a pure Java client. This guide explains how to use the client to connect to Hopsworks and read or write feature data.
8+
9+
## Generate an API key
10+
11+
For instructions on how to generate an API key follow this [user guide](../projects/api_key/create_api_key.md). For the Java client to work correctly make sure you add the following scopes to your API key:
12+
13+
1. featurestore
14+
2. project
15+
3. job
16+
4. kafka
17+
18+
## Add the HSFS dependency to your project:
19+
20+
The HSFS library is available on the Hopsworks' Maven repository. If you are using Maven as build tool, you can add the following in your pom.xml file:
21+
22+
```xml
23+
<repositories>
24+
<repository>
25+
<id>Hops</id>
26+
<name>Hops Repository</name>
27+
<url>https://archiva.hops.works/repository/Hops/</url>
28+
<releases>
29+
<enabled>true</enabled>
30+
</releases>
31+
<snapshots>
32+
<enabled>true</enabled>
33+
</snapshots>
34+
</repository>
35+
</repositories>
36+
```
37+
38+
The artifactId for the HSFS Java build is hsfs, if you are using Maven as build tool, you can add the following dependency:
39+
40+
```xml
41+
<dependency>
42+
<groupId>com.logicalclocks</groupId>
43+
<artifactId>hsfs</artifactId>
44+
<version>${hsfs.version}</version>
45+
</dependency>
46+
```
47+
48+
!!!note "Java Version"
49+
50+
Please note that the Java client has been tested with Java versions up to Java 17
51+
52+
## Connecting to the Feature Store
53+
54+
You are now ready to connect to the Hopsworks Feature Store from a Java client:
55+
56+
```Java
57+
//Import necessary classes
58+
import com.logicalclocks.hsfs.FeatureStore;
59+
import com.logicalclocks.hsfs.FeatureView;
60+
import com.logicalclocks.hsfs.HopsworksConnection;
61+
62+
//Establish connection with Hopsworks.
63+
HopsworksConnection hopsworksConnection = HopsworksConnection.builder()
64+
.host("my_instance") // DNS of your Feature Store instance
65+
.port(443) // Port to reach your Hopsworks instance, defaults to 443
66+
.project("my_project") // Name of your Hopsworks Feature Store project
67+
.apiKeyValue("api_key") // The API key to authenticate with the feature store
68+
.hostnameVerification(false) // Disable for self-signed certificates
69+
.build();
70+
71+
//get feature store handle
72+
FeatureStore fs = hopsworksConnection.getFeatureStore();
73+
74+
//get feature view handle
75+
FeatureView fv = fs.getFeatureView(fvName, fvVersion);
76+
77+
// get feature vector
78+
List<Object> singleVector = fv.getFeatureVector(new HashMap<String, Object>() {{
79+
put("id", 100);
80+
}});
81+
```
82+
83+
### Update feature data
84+
85+
The Java client allows you to update data on existing feature groups using the streaming interface. You can provide a list of POJO objects to the [insertStream](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/javadoc/com/logicalclocks/hsfs/StreamFeatureGroup.html#insertStream-java.util.List-) method.
86+
87+
The feature group should exists already (can be created using the Python client) and the POJO objects should be serializable with the feature group's AVRO schema.
88+
89+
Please see the [tutorial](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/integrations/java/java) for a code example on how to write data.
90+
91+
### Limitations
92+
93+
Currently using the Java client to retrieve feature vectors have the following limitations:
94+
95+
* Only the SQL interface is supported. It is not possible to retrieve feature vectors using the REST API Interface
96+
* Feature Views with model dependent transformations attached are not applied. If your feature view has model dependent transformations, please use the Python client.
97+
98+
## Next Steps
99+
100+
You can find more information on how to interact from Java client in the [JavaDoc](https://docs.hopsworks.ai/feature-store-api/{{{ hopsworks_version }}}/javadoc/) or this [tutorial](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/integrations/java/java)

‎mkdocs.yml

+1
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,7 @@ nav:
119119
- Apache Spark: user_guides/integrations/spark.md
120120
- Apache Flink: user_guides/integrations/flink.md
121121
- Apache Beam: user_guides/integrations/beam.md
122+
- Java: user_guides/integrations/java.md
122123
- Sharing: user_guides/fs/sharing/sharing.md
123124
- Tags: user_guides/fs/tags/tags.md
124125
- Provenance: user_guides/fs/provenance/provenance.md

0 commit comments

Comments
 (0)