Skip to content

Commit f8e5a90

Browse files
Prepare for release 0.3.0 (#29)
Prepares Maggy for the release of 0.3.0 - Adds developer documentation - Fixes bugs - Adds Ablation example
1 parent 799d8b2 commit f8e5a90

12 files changed

+651
-28
lines changed

README.rst

+24-13
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,28 @@
11
Maggy
22
=====
33

4+
|Downloads| |PypiStatus| |PythonVersions| |Docs|
5+
6+
47
Maggy is a framework for efficient asynchronous optimization of expensive
58
black-box functions on top of Apache Spark. Compared to existing frameworks,
69
maggy is not bound to stage based optimization algorithms and therefore it is
710
able to make extensive use of early stopping in order to achieve efficient
811
resource utilization.
912

1013
Right now, maggy supports asynchronous hyperparameter tuning of machine
11-
learning and deep learning models, but other use cases include ablation studies
12-
and asynchronous distributed training.
14+
learning and deep learning models, and ablation studies on neural network
15+
layers as well as input features.
1316

1417
Moreover, it provides a developer API that allows advanced usage by
1518
implementing custom optimization algorithms and early stopping criteria.
1619

17-
In order to make decisions on early stopping, the Spark executors are sending
18-
heart beats with the current performance of the model they are training to the
19-
maggy experiment driver which is running on the Spark driver. We call the
20-
process of training a model with a certain hyperparameter combination a
21-
*trial*. The experiment driver then uses all information of finished trials and
22-
the currently running ones to check in a specified interval, which of the
23-
trials should be stopped early.
24-
Subsequently, the experiment driver provides a new trial to the Spark
25-
executor.
20+
To accomodate asynchronous algorithms, support for communication between the
21+
Driver and Executors via RPCs through Maggy was added. The Optimizer that guides
22+
hyperparameter search is located on the Driver and it assigns trials to
23+
Executors. Executors periodically send back to the Driver the current
24+
performance of their trial, and the Optimizer can decide to early-stop any
25+
ongoing trial and send the Executor a new trial instead.
2626

2727
Quick Start
2828
-----------
@@ -31,8 +31,8 @@ To Install:
3131

3232
>>> pip install maggy
3333

34-
The programming model is that you wrap the code containing the model training
35-
inside a wrapper function. Inside that wrapper function provide all imports and
34+
The programming model consists of wrapping the code containing the model training
35+
inside a function. Inside that wrapper function provide all imports and
3636
parts that make up your experiment.
3737

3838
There are three requirements for this wrapper function:
@@ -94,4 +94,15 @@ see the Jupyter Notebook in the `examples` folder.
9494
Documentation
9595
-------------
9696

97+
Read our `blog post <https://www.logicalclocks.com/blog/scaling-machine-learning-and-deep-learning-with-pyspark-on-hopsworks>`_ for more details.
98+
9799
API documentation is available `here <https://maggy.readthedocs.io/en/latest/>`_.
100+
101+
.. |Downloads| image:: https://pepy.tech/badge/maggy/month
102+
:target: https://pepy.tech/project/maggy
103+
.. |PypiStatus| image:: https://img.shields.io/pypi/v/maggy?color=blue
104+
:target: https://pypi.org/project/hops
105+
.. |PythonVersions| image:: https://img.shields.io/pypi/pyversions/maggy.svg
106+
:target: https://pypi.org/project/hops
107+
.. |Docs| image:: https://img.shields.io/readthedocs/maggy
108+
:target: https://maggy.readthedocs.io/en/latest/

docs/developer.rst

+18-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,19 @@
11
Maggy Developer API
2-
===================
2+
===================
3+
4+
As a developer you have the possibility to implement your custom optimizers
5+
or ablators. For that you can implement an abstract method, which you can then
6+
pass as an argument when launching the experiment. For examples, please look at
7+
existing optimizers and ablators.
8+
9+
maggy.optimizer module
10+
-----------------------
11+
12+
.. autoclass:: maggy.optimizer.AbstractOptimizer
13+
:members:
14+
15+
maggy.ablation.ablator module
16+
-----------------------------
17+
18+
.. autoclass:: maggy.ablation.ablator.abstractablator.AbstractAblator
19+
:members:

docs/user.rst

+8
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,11 @@ maggy.callbacks module
2323

2424
.. autoclass:: maggy.callbacks.KerasEpochEnd
2525
:members:
26+
27+
maggy.ablation module
28+
---------------------
29+
30+
.. autoclass:: maggy.ablation.AblationStudy
31+
:members:
32+
33+
.. automethod:: __init__

examples/maggy-ablation-titanic-example.ipynb

+469
Large diffs are not rendered by default.

maggy/ablation/ablationstudy.py

+90-4
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,91 @@
11
class AblationStudy(object):
2-
def __init__(self, training_dataset_name, training_dataset_version,
3-
label_name, **kwargs):
2+
"""The `AblationStudy` object is the entry point to define an ablation
3+
study with maggy. This object can subsequently be passed as an argument
4+
when the experiment is launched with `experiment.lagom()`.
5+
6+
Sample usage:
7+
8+
>>> from maggy.ablation import AblationStudy
9+
>>> ablation_study = AblationStudy('titanic_train_dataset',
10+
>>> label_name='survived')
11+
12+
Define your study by including layers and features, which should be
13+
ablated:
14+
15+
>>> ablation_study.features.include('pclass', 'fare')
16+
>>> ablation_study.model.layers.include('my_dense_two',
17+
>>> 'my_dense_three')
18+
19+
You can also add a layer group using a list:
20+
21+
>>> ablation_study.model.layers.include_groups(['my_dense_two',
22+
>>> 'my_dense_four'])
23+
24+
Or add a layer group using a prefix:
25+
26+
>>> ablation_study.model.layers.include_groups(prefix='my_dense')
27+
28+
Next you should define a base model function using the layer and feature
29+
names you previously specified:
30+
31+
>>> # you only need to add the `name` parameter to layer initializers
32+
>>> def base_model_generator():
33+
>>> model = tf.keras.Sequential()
34+
>>> model.add(tf.keras.layers.Dense(64, activation='relu'))
35+
>>> model.add(tf.keras.layers.Dense(..., name='my_dense_two', ...)
36+
>>> model.add(tf.keras.layers.Dense(32, activation='relu'))
37+
>>> model.add(tf.keras.layers.Dense(..., name='my_dense_sigmoid', ...)
38+
>>> # output layer
39+
>>> model.add(tf.keras.layers.Dense(1, activation='linear'))
40+
>>> return model
41+
42+
Make sure to include the generator function in the study:
43+
44+
>>> ablation_study.model.set_base_model_generator(base_model_generator)
45+
46+
Last but not least you can define your actual training function:
47+
48+
>>> from maggy import experiment
49+
>>> from maggy.callbacks import KerasBatchEnd
50+
51+
>>> def training_function(dataset_function, model_function, reporter):
52+
>>> import tensorflow as tf
53+
>>> epochs = 5
54+
>>> batch_size = 10
55+
>>> tf_dataset = dataset_function(epochs, batch_size)
56+
>>> model = model_function()
57+
>>> model.compile(optimizer=tf.train.AdamOptimizer(0.001),
58+
>>> loss='binary_crossentropy',
59+
>>> metrics=['accuracy'])
60+
>>> ### Maggy REPORTER
61+
>>> callbacks = [KerasBatchEnd(reporter, metric='acc')]
62+
>>> history = model.fit(tf_dataset, epochs=5, steps_per_epoch=30)
63+
>>> return float(history.history['acc'][-1])
64+
65+
Lagom the experiment:
66+
67+
>>> result = experiment.lagom(map_fun=training_function,
68+
>>> experiment_type='ablation',
69+
>>> ablation_study=ablation_study,
70+
>>> ablator='loco',
71+
>>> name='Titanic-LOCO',
72+
>>> hb_interval=5)
73+
"""
74+
75+
def __init__(
76+
self, training_dataset_name, training_dataset_version, label_name,
77+
**kwargs):
78+
"""Initializes the ablation study.
79+
80+
:param training_dataset_name: Name of the training dataset in the
81+
featurestore.
82+
:type training_dataset_name: str
83+
:param training_dataset_version: Version of the training dataset to be
84+
used.
85+
:type training_dataset_version: int
86+
:param label_name: Name of the target prediction label.
87+
:type label_name: str
88+
"""
489
self.features = Features()
590
self.model = Model()
691
self.hops_training_dataset_name = training_dataset_name
@@ -11,8 +96,9 @@ def __init__(self, training_dataset_name, training_dataset_version,
1196
def to_dict(self):
1297
"""
1398
Returns the ablation study configuration as a Python dictionary.
14-
:return: A dictionary with ablation study configuration parameters as keys
15-
(i.e. 'training_dataset_name', 'included_features', etc.)
99+
100+
:return: A dictionary with ablation study configuration parameters as
101+
keys (i.e. 'training_dataset_name', 'included_features', etc.)
16102
:rtype: dict
17103
"""
18104
ablation_dict = {

maggy/ablation/ablator/__init__.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
from maggy.ablation.ablator.abstractablator import AbstractAblator
2-
from maggy.ablation.ablator.loco import LOCO
2+
# from maggy.ablation.ablator.loco import LOCO

maggy/ablation/ablator/abstractablator.py

+3-1
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ def get_number_of_trials(self):
1313
"""
1414
If applicable, calculate and return the total number of trials of the ablation experiment.
1515
Make sure to also include the base (reference) trial in the count.
16+
1617
:return: total number of trials of the ablation study experiment
1718
:rtype: int
1819
"""
@@ -25,7 +26,7 @@ def get_dataset_generator(self, ablated_feature, dataset_type='tfrecord'):
2526
The returned function will be executed on the executor per each trial.
2627
2728
:param ablated_feature: the name of the feature to be excluded from the training dataset.
28-
Must match a feature name in the corresponding feature group in the feature store.
29+
Must match a feature name in the corresponding feature group in the feature store.
2930
:type ablated_feature: str
3031
:param dataset_type: type of the dataset. For now, we only support 'tfrecord'.
3132
:return: A function that generates a TFRecordDataset
@@ -53,6 +54,7 @@ def get_trial(self, ablation_trial=None):
5354
The trial should contain a dataset generator and a model generator.
5455
Depending on the ablator policy, the trials could come from a list (buffer) of pre-made trials,
5556
or generated on the fly.
57+
5658
:rtype: `Trial` or `None`
5759
"""
5860
pass

maggy/core/experimentdriver.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,8 @@
1515
from maggy.earlystop import AbstractEarlyStop, MedianStoppingRule, NoStoppingRule
1616
from maggy.searchspace import Searchspace
1717

18-
from maggy.ablation.ablator import AbstractAblator, LOCO
18+
from maggy.ablation.ablator import AbstractAblator
19+
from maggy.ablation.ablator.loco import LOCO
1920
from maggy.ablation.ablationstudy import AblationStudy
2021

2122
from hops import constants as hopsconstants

maggy/core/rpc.py

+3-3
Original file line numberDiff line numberDiff line change
@@ -329,7 +329,6 @@ def start(self, exp_driver):
329329
app_id = str(sc.applicationId)
330330

331331
method = hopsconstants.HTTP_CONFIG.HTTP_POST
332-
connection = hopsutil._get_http_connection(https=True)
333332
resource_url = hopsconstants.DELIMITERS.SLASH_DELIMITER + \
334333
hopsconstants.REST_CONFIG.HOPSWORKS_REST_RESOURCE + hopsconstants.DELIMITERS.SLASH_DELIMITER + \
335334
"maggy" + hopsconstants.DELIMITERS.SLASH_DELIMITER + "drivers"
@@ -341,8 +340,9 @@ def start(self, exp_driver):
341340
headers = {hopsconstants.HTTP_CONFIG.HTTP_CONTENT_TYPE: hopsconstants.HTTP_CONFIG.HTTP_APPLICATION_JSON}
342341

343342
try:
344-
response = hopsutil.send_request(connection, method, resource_url, body=json_embeddable, headers=headers)
345-
if (response.status != 200):
343+
response = hopsutil.send_request(method, resource_url, data=json_embeddable, headers=headers)
344+
345+
if (response.status_code // 100) != 2:
346346
print("No connection to Hopsworks for logging.")
347347
exp_driver._log("No connection to Hopsworks for logging.")
348348
except Exception as e:

maggy/optimizer/abstractoptimizer.py

+13
Original file line numberDiff line numberDiff line change
@@ -10,12 +10,25 @@ def __init__(self):
1010

1111
@abstractmethod
1212
def initialize(self):
13+
"""
14+
A hook for the developer to initialize the optimizer.
15+
"""
1316
pass
1417

1518
@abstractmethod
1619
def get_suggestion(self, trial=None):
20+
"""
21+
Return a `Trial` to be assigned to an executor, or `None` if there are
22+
no trials remaining in the experiment.
23+
24+
:rtype: `Trial` or `None`
25+
"""
1726
pass
1827

1928
@abstractmethod
2029
def finalize_experiment(self, trials):
30+
"""
31+
This method will be called before finishing the experiment. Developers
32+
can implement this method e.g. for cleanup or extra logging.
33+
"""
2134
pass

maggy/version.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
__version__ = '0.2'
1+
__version__ = '0.3.0'

setup.py

+19-3
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,31 @@ def read(fname):
1111
name='maggy',
1212
version=__version__,
1313
install_requires=[
14-
'numpy'
14+
'numpy==1.16.5'
1515
],
16+
extras_require={
17+
'pydoop': ['pydoop'],
18+
'tf': ['tensorflow==1.14.0'],
19+
'docs': [
20+
'sphinx==1.8.5',
21+
'sphinx-autobuild',
22+
'recommonmark',
23+
'sphinx_rtd_theme',
24+
'jupyter_sphinx_theme'
25+
],
26+
'test': [
27+
'pylint',
28+
'pytest',
29+
],
30+
'spark': ['pyspark==2.4.3']
31+
},
1632
author='Moritz Meister',
1733
author_email='meister.mo@gmail.com',
18-
description='',
34+
description='Efficient asynchronous optimization of expensive black-box functions on top of Apache Spark',
1935
license='GNU Affero General Public License v3',
2036
keywords='Hyperparameter, Optimization, Auto-ML, Hops, Hadoop, TensorFlow, Spark',
2137
url='https://github.com/logicalclocks/maggy',
22-
# download_url = 'http://snurran.sics.se/hops/hops-util-py/hops-' + __version__ + '.tar.gz',
38+
download_url='http://snurran.sics.se/hops/maggy/maggy-' + __version__ + '.tar.gz',
2339
packages=find_packages(),
2440
long_description=read('README.rst'),
2541
classifiers=[

0 commit comments

Comments
 (0)