Merge pull request #35 from mdreves/master

Project import generated by Copybara.

Author: mdreves

README.md

@@ -55,13 +55,14 @@ The following table is the TFMA package versions that are compatible with each
 other. This is determined by our testing framework, but other *untested*
 combinations may also work.
 
-|tensorflow-model-analysis  |tensorflow    |apache-beam[gcp]|
-|---------------------------|--------------|----------------|
-|GitHub master              |1.9           |2.6.0           |
-|0.9.2                      |1.9           |2.6.0           |
-|0.9.1                      |1.10          |2.6.0           |
-|0.9.0                      |1.9           |2.5.0           |
-|0.6.0                      |1.6           |2.4.0           |
+|tensorflow-model-analysis  |tensorflow          |apache-beam[gcp]|
+|---------------------------|--------------------|----------------|
+|GitHub master              |1.11                |2.8.0           |
+|0.11.0                     |1.11                |2.8.0           |
+|0.9.2                      |1.9                 |2.6.0           |
+|0.9.1                      |1.10                |2.6.0           |
+|0.9.0                      |1.9                 |2.5.0           |
+|0.6.0                      |1.6                 |2.4.0           |
 
 ## Questions
 

RELEASE.md

@@ -1,3 +1,77 @@
+# Release 0.11.0
+
+## Major Features and Improvements
+
+* We now support unsupervised models which have `model_fn`s that do not take a
+  `labels` argument.
+* Improved performance by using `make_callable` instead of repeated
+  `session.run` calls.
+* Improved performance by better choice of default "combine" batch size.
+* We now support passing in custom extractors in the model_eval_lib API.
+* Added support for models which have multiple examples per raw input (e.g.
+  input is a compressed example which expands to multiple examples when parsed
+  by the model). For such models, you must specify an `example_ref` parameter
+  to your `EvalInputReceiver`. This 1-D integer Tensor should be batch aligned
+  with features, predictions and labels and each element in it is an index in
+  the raw input tensor to identify which input each feature / prediction /
+  label came from. See
+  `eval_saved_model/example_trainers/fake_multi_examples_per_input_estimator.py`
+  for an example.
+* Added support for metrics with string `value_op`s.
+* Added support for metrics whose `value_op`s return multidimensional arrays.
+* We now support including your serving graph in the EvalSavedModel. You can
+  do this by passing a `serving_input_receiver_fn` to `export_eval_savedmodel`
+  or any of the TFMA Exporters.
+
+## Bug fixes and other changes
+
+* Depends on `apache-beam[gcp]>=2.8,<3`.
+* Depends on `tensorflow-transform>=0.11,<1`.
+* Requires pre-installed TensorFlow >=1.11,<2.
+* Factor our utility functions for adding sliceable "meta-features" to FPL.
+* Added public API docs
+* Add an extractor to add sliceable "meta-features" to FPL.
+* Potentially improved performance by fanning out large slices.
+* Add support for assets_extra in `tfma.exporter.FinalExporter`.
+* Add a light-weight library that includes only the export-related modules for
+  TFMA for use in your Trainer. See docstring in
+  `tensorflow_model_analysis/export_only/__init__.py` for usage.
+* Update `EvalInputReceiver` so the TFMA collections written to the graph only
+  contain the results of the last call if multiple calls to `EvalInputReceiver`
+  are made.
+* We now finalize the graph after it's loaded and post-export metrics are added,
+  potentially improving performance.
+* Fix a bug in post-export PrecisionRecallAtK where labels with only 1 dimension
+  were not correctly handled.
+* Fix an issue where we were not correctly wrapping SparseTensors for `features`
+  and `labels` in `tf.identity`, which could cause TFMA to encounter
+  TensorFlow issue #17568 if there were control dependencies on these `features`
+  or `labels`.
+* We now correctly preserve `dtypes` when splitting and concatenating
+  SparseTensors internally. The failure to do so previously could result in
+  unexpectedly large memory usage if string values were involved due to the
+  inefficient pickling of NumPy string arrays with a large number of elements.
+
+## Breaking changes
+
+* Requires pre-installed TensorFlow >=1.11,<2.
+* We now require that `EvalInputReceiver`, `export_eval_savedmodel`,
+  `make_export_strategy`, `make_final_exporter`, `FinalExporter` and
+  `LatestExporter` be called with keyword arguments only.
+* Removed `check_metric_compatibility` from `EvalSavedModel`.
+* We now enforce that the `receiver_tensors` dictionary for `EvalInputReceiver`
+  contains exactly one key named `examples`.
+* Post-export metrics have now been moved up one level to
+  `tfma.post_export_metrics`. They should now be accessed via
+  `tfma.post_export_metrics.auc` instead of
+  `tfma.post_export_metrics.post_export_metrics.auc` as they were before.
+* Separated extraction from evaluation. `EvaluteAndWriteResults` is now called
+  `ExtractEvaluateAndWriteResults`.
+* Added `EvalSharedModel` type to encapsulate `model_path` and
+  `add_metrics_callbacks` along with a handle to a shared model instance.
+
+## Deprecations
+
 # Release 0.9.2
 
 ## Major Features and Improvements
@@ -21,7 +95,6 @@
 ## Bug fixes and other changes
 
 * Depends on `apache-beam[gcp]>=2.6,<3`.
-* Requires pre-installed TensorFlow >=1.10,<2.
 * Updated ExampleCount to use the batch dimension as the example count. It
   also now tries a few fallbacks if none of the standard keys are found in the
   predictions dictionary: the first key in sorted order in the predictions
@@ -34,6 +107,8 @@
 
 ## Breaking changes
 
+*   Requires pre-installed TensorFlow >=1.10,<2.
+
 ## Deprecations
 
 # Release 0.9.0

examples/chicago_taxi/chicago_taxi_client.py

@@ -48,7 +48,7 @@ def _do_local_inference(host, port, serialized_examples):
                                               shape=[len(serialized_examples)],
                                               dtype=tf.string)
   # The name of the input tensor is 'examples' based on
-  # https://github.com/tensorflow/tensorflow/blob/r1.9/tensorflow/python/estimator/export/export.py#L290
+  # https://github.com/tensorflow/tensorflow/blob/r1.11/tensorflow/python/estimator/export/export.py#L306
   request.inputs['examples'].CopyFrom(tfproto)
   print(stub.Predict(request, _LOCAL_INFERENCE_TIMEOUT_SECONDS))
 

examples/chicago_taxi/chicago_taxi_tfma_local_playground.ipynb

@@ -429,7 +429,7 @@
     "\n",
     "    run_experiment(hparams)\n",
     "\n",
-    "print('Done.')"
+    "print('Done')"
    ]
   },
   {
@@ -445,7 +445,7 @@
     "                     num_layers=4,\n",
     "                     first_layer_size=100,\n",
     "                     scale_factor=0.7)\n",
-    "print('Done.')"
+    "print('Done')"
    ]
   },
   {
@@ -496,6 +496,9 @@
     "    \"\"\"\n",
     "    eval_model_base_dir = os.path.join(get_tf_output_dir(tf_run_id), EVAL_MODEL_DIR)\n",
     "    eval_model_dir = os.path.join(eval_model_base_dir, next(os.walk(eval_model_base_dir))[1][0])\n",
+    "    eval_shared_model = tfma.default_eval_shared_model(\n",
+    "        eval_saved_model_path=eval_model_dir,\n",
+    "        add_metrics_callbacks=add_metrics_callbacks)\n",
     "    schema = taxi.read_schema(schema_file)\n",
     "    \n",
     "    print(eval_model_dir)\n",
@@ -518,12 +521,13 @@
     "            raw_data\n",
     "            | 'ToSerializedTFExample' >> beam.Map(coder.encode))\n",
     "\n",
-    "        _ = raw_data | 'EvaluateAndWriteResults' >> tfma.EvaluateAndWriteResults(\n",
-    "            eval_saved_model_path=eval_model_dir,\n",
-    "            slice_spec=slice_spec,\n",
-    "            output_path=get_tfma_output_dir(tfma_run_id),\n",
-    "            add_metrics_callbacks=add_metrics_callbacks,\n",
-    "            display_only_data_location=input_csv)\n",
+    "        _ = (raw_data\n",
+    "             | 'ExtractEvaluateAndWriteResults' >>\n",
+    "             tfma.ExtractEvaluateAndWriteResults(\n",
+    "                 eval_shared_model=eval_shared_model,\n",
+    "                 slice_spec=slice_spec,\n",
+    "                 output_path=get_tfma_output_dir(tfma_run_id),\n",
+    "                 display_only_data_location=input_csv))\n",
     "\n",
     "    return tfma.load_eval_result(output_path=get_tfma_output_dir(tfma_run_id))\n",
     "    \n",
@@ -593,7 +597,7 @@
     "                         tfma_run_id=1,\n",
     "                         slice_spec=ALL_SPECS,\n",
     "                         schema_file=get_schema_file())\n",
-    "print('Done.')\n"
+    "print('Done')\n"
    ]
   },
   {
@@ -667,12 +671,12 @@
     "                    add_metrics_callbacks=[\n",
     "                        # calibration_plot_and_prediction_histogram computes calibration plot and prediction\n",
     "                        # distribution at different thresholds.\n",
-    "                        tfma.post_export_metrics.post_export_metrics.calibration_plot_and_prediction_histogram(),\n",
+    "                        tfma.post_export_metrics.calibration_plot_and_prediction_histogram(),\n",
     "                        # auc_plots enables precision-recall curve and ROC visualization at different thresholds.\n",
-    "                        tfma.post_export_metrics.post_export_metrics.auc_plots()\n",
+    "                        tfma.post_export_metrics.auc_plots()\n",
     "                    ])\n",
     "\n",
-    "print('Done.')"
+    "print('Done')"
    ]
   },
   {
@@ -823,7 +827,7 @@
     "                     first_layer_size=240,\n",
     "                     scale_factor=0.5)\n",
     "\n",
-    "print('Done.')"
+    "print('Done')"
    ]
   },
   {
@@ -845,7 +849,7 @@
     "                         tfma_run_id=3,\n",
     "                         slice_spec=ALL_SPECS,\n",
     "                         schema_file=get_schema_file())\n",
-    "print('Done.')"
+    "print('Done')"
    ]
   },
   {

examples/chicago_taxi/process_tfma.py

@@ -25,12 +25,8 @@
 import tensorflow as tf
 
 import tensorflow_model_analysis as tfma
-from tensorflow_model_analysis.eval_saved_model.post_export_metrics import post_export_metrics
-
 from trainer import taxi
 
-from tensorflow_model_analysis.slicer import slicer
-
 
 def process_tfma(eval_result_dir,
                  schema_file,
@@ -53,7 +49,6 @@ def process_tfma(eval_result_dir,
       None.
     eval_model_dir: A directory where the eval model is located.
     max_eval_rows: Number of rows to query from BigQuery.
-
     pipeline_args: additional DataflowRunner or DirectRunner args passed to the
       beam pipeline.
 
@@ -66,12 +61,19 @@ def process_tfma(eval_result_dir,
         'one of --input_csv or --big_query_table should be provided.')
 
   slice_spec = [
-      slicer.SingleSliceSpec(),
-      slicer.SingleSliceSpec(columns=['trip_start_hour'])
+      tfma.SingleSliceSpec(),
+      tfma.SingleSliceSpec(columns=['trip_start_hour'])
   ]
 
   schema = taxi.read_schema(schema_file)
 
+  eval_shared_model = tfma.default_eval_shared_model(
+      eval_saved_model_path=eval_model_dir,
+      add_metrics_callbacks=[
+          tfma.post_export_metrics.calibration_plot_and_prediction_histogram(),
+          tfma.post_export_metrics.auc_plots()
+      ])
+
   with beam.Pipeline(argv=pipeline_args) as pipeline:
     if input_csv:
       csv_coder = taxi.make_csv_coder(schema)
@@ -97,13 +99,10 @@ def process_tfma(eval_result_dir,
     _ = (
         raw_data
         | 'ToSerializedTFExample' >> beam.Map(coder.encode)
-        | 'EvaluateAndWriteResults' >> tfma.EvaluateAndWriteResults(
-            eval_saved_model_path=eval_model_dir,
+        |
+        'ExtractEvaluateAndWriteResults' >> tfma.ExtractEvaluateAndWriteResults(
+            eval_shared_model=eval_shared_model,
             slice_spec=slice_spec,
-            add_metrics_callbacks=[
-                post_export_metrics.calibration_plot_and_prediction_histogram(),
-                post_export_metrics.auc_plots()
-            ],
             output_path=eval_result_dir))
 
 
@@ -130,8 +129,7 @@ def main():
       default=None,
       type=int)
   parser.add_argument(
-      '--schema_file',
-      help='File holding the schema for the input data')
+      '--schema_file', help='File holding the schema for the input data')
 
   known_args, pipeline_args = parser.parse_known_args()
 

examples/chicago_taxi/setup.py

@@ -14,22 +14,22 @@
 """Setup dependencies for local and cloud deployment."""
 import setuptools
 
-TF_VERSION = '1.9.0'
+TF_VERSION = '1.11.0'
 
 if __name__ == '__main__':
   setuptools.setup(
       name='tfx_chicago_taxi',
-      version='0.9.2',
+      version='0.11.0',
       packages=setuptools.find_packages(),
       install_requires=[
-          'apache-beam[gcp]==2.6.0',
+          'apache-beam[gcp]==2.8.0',
           'jupyter==1.0',
           'numpy==1.13.3',
           'protobuf==3.6.0',
-          'tensorflow=='+TF_VERSION,
-          'tensorflow-data-validation==0.9.0',
+          'tensorflow==' + TF_VERSION,
+          'tensorflow-data-validation==0.11.0',
           'tensorflow-metadata==0.9.0',
-          'tensorflow-model-analysis==0.9.2',
-          'tensorflow-serving-api==1.9.0',
+          'tensorflow-model-analysis==0.11.0',
+          'tensorflow-serving-api==1.11.0',
           'tensorflow-transform==0.11.0',
       ])

examples/chicago_taxi/start_model_server_mlengine.sh

@@ -31,7 +31,9 @@ gsutil ls $WORKING_DIR/serving_model_dir/export/chicago-taxi/
 MODEL_BINARIES=$(gsutil ls $WORKING_DIR/serving_model_dir/export/chicago-taxi/ \
   | sort | grep '\/[0-9]*\/$' | tail -n1)
 
+TF_VERSION=1.10
+
 gcloud ml-engine versions create v1 \
   --model chicago_taxi \
   --origin $MODEL_BINARIES \
-  --runtime-version 1.6
+  --runtime-version $TF_VERSION

examples/chicago_taxi/train_mlengine.sh

@@ -61,10 +61,12 @@ EVAL_FILE=$TFT_OUTPUT_PATH/train_transformed-*
 TRAIN_STEPS=100000
 EVAL_STEPS=1000
 
+TF_VERSION=1.10
+
 gcloud ml-engine jobs submit training $JOB_ID \
                                     --stream-logs \
                                     --job-dir $MODEL_DIR \
-                                    --runtime-version 1.9 \
+                                    --runtime-version $TF_VERSION \
                                     --module-name trainer.task \
                                     --package-path trainer/ \
                                     --region us-central1 \

g3doc/get_started.md

@@ -110,13 +110,15 @@ results.  The results can be loaded for visualization using
 
 ```python
 # To run the pipeline.
+eval_shared_model = tfma.default_eval_shared_model(
+    model_path='/path/to/eval/saved/model')
 with beam.Pipeline(runner=...) as p:
   _ = (p
        # You can change the source as appropriate, e.g. read from BigQuery.
        | 'ReadData' >> beam.io.ReadFromTFRecord(data_location)
        | 'ExtractEvaluateAndWriteResults' >>
        tfma.ExtractEvaluateAndWriteResults(
-            eval_saved_model_path='/path/to/eval/saved/model',
+            eval_shared_model=eval_shared_model,
             output_path='/path/to/output',
             display_only_data_location=data_location))
 

g3doc/index.md

@@ -58,7 +58,8 @@ combinations may also work.
 
 |tensorflow-model-analysis  |tensorflow    |apache-beam[gcp]|
 |---------------------------|--------------|----------------|
-|GitHub master              |1.9           |2.6.0           |
+|GitHub master              |1.11          |2.8.0           |
+|0.11.0                     |1.11          |2.8.0           |
 |0.9.2                      |1.9           |2.6.0           |
 |0.9.1                      |1.10          |2.6.0           |
 |0.9.0                      |1.9           |2.5.0           |