Train like keras's fit_generator

[aishwarya-agrawal]

Feature request

What is the expected behavior?
For large datasets that do not fit into RAM, can we have keras fit_generator type training ?

What is motivation or use case for adding/changing the behavior?

How should this be implemented in your opinion?

Are you willing to work on this yourself?
yes

[Optimox]

This feature has already been mentioned in features before, I think it’s an important feature but I think we need a list of the most common files type used (hdf5, parquet...).

What would be your usage?

I think it would not be too difficult to implement in an easy way for users, like giving the path to the file instead of a numpy matrix as X_train, X_valid...
But there are quite a lot of side effects that would probably make things a bit difficult but it’s probably worth the work.

[aishwarya-agrawal]

I have data in parquet format which is split across multiple files which is why a fit_generator type of functionality works best for me where I modify the data in the generator and yield the expected the values.

[Optimox]

Ok, I’m not familiar with Keras fit_generator, I’ll take a look to get some inspiration

[Optimox]

I'm closing this one as it's directly related to #143, feel free to discuss more in depth in the other issue. Thanks!

[cadama]

+1 for this. I would also be happy with any hack that makes this possible.