Home

Spark-Transformers

Library for exporting spark models in Java ecosystem.

Goal of this library is to :

• Provide a way to export Spark models/transformations into a custom format which can be imported back into a java object.
• Provide a way to do model predictions in java ecosystem.

#Usage

Add jar to classpath

http://spark.apache.org/docs/latest/programming-guide.html#using-the-shell

./bin/spark-shell --master local --jars adapters-V1.0-SNAPSHOT.jar

Train, export in spark. Import and predict in java.

//Train model in spark
LogisticRegressionModel lrmodel = new LogisticRegressionWithSGD().run(data.rdd());

//Export this model
byte[] exportedModel = ModelExporter.export(lrmodel, null);

//Import and get Transformer
Transformer transformer = ModelImporter.importAndGetTransformer(exportedModel);

//predict
double predicted = (double) transformer.transform(new Double[] {0.3, 0.5. 0.6});

For detailed usage see unit tests. https://github.com/flipkart-incubator/spark-transformers/blob/master/adapters/src/test/java/com/flipkart/fdp/ml/adapter/BucketizerBridgeTest.java

Getting help

For help regarding usage, drop an email to fdp-ml-dev@flipkart.com