Extending ImageDataGenerator

[tetmin]

Is there an easy way to write generator extensions for Keras? I'd like to use some of the ImageDataGenerator preprocessing steps but also add some of my own such as randomly occluding areas of the image, adding noise etc. If not I can add some of these things to the ImageDataGenerator class in Keras, is that something that would be useful?

Also can I just double check that the ImageDataGenerator does indeed generate batches of examples in a non-locking way, thereby not causing any GPU training bottlenecks?

[oeway]

I would be interested if someone can make the ImageDataGenerator extendable.

Plus, another feature I would like to have is to separate the random variables generator part which used by ImageDataGenerator, thus the variables can be saved somewhere and make the process reproducible, you don't need to actually store the transformed images, just store the random variables, as a dictionary maybe. These random variables can be feed to into a ImageDataGenerator to produce the results.

This going to be super useful in case of dense prediction(such as segmentation), in which we need to do data augmentation on both input image and output image, we can make two ImageDataGenerator, and use the separated random generator to produce the random variables, and then feed into two ImageDataGenerator, so the spatial transform such as rotation are synchronized on both input and output.

I would begin to work on this, please let me know if anyone else have ideas regarding this.

EDIT: perhaps you would think there is a seed option in flow and flow_from_directory, yes, indeed, you can reproduce the image by controlling the seed, but the problem comes if you want to apply different configuration to two ImageDataGenerator, for example, add noise on the input image but not on the output image, in that case, even you use the same seed, the spatial transformation could be different. In that sense, perhaps, we should use different seed for different type of data augmentation function, what do you think?

[oeway]

I am working on improve ImageDataGenerator but would be great anyone interested could provide suggestions and feedbacks.
The main idea to make it more flexible, and support dense prediction dataset. Here are a bunch of ideas I would like to discuss:

• provide a customizable pipeline( a list of functions) to let user add/change their own preprocessing. There will be a pipeline contains random_transform and standardize by default, and user can insert any function in any position of the pipeline. ( For example, Add target_shape and resize to ImageDataGenerator #2738 can be resolved, and people can use opencv scikit-image or other libraries)
• reproducible random spatial transform by using a separate random seed from the keyword, so dense prediction data has image-in and image-out can use two different ImageDataGenerator which have the same random spatial transform. (related to ImageDataGenerator with masks as labels #3059)
• provide featurewise_standardize_axis and samplewise_standardize_axis for more flexible standardization, for example, you can pass (0, 2, 3) to do a channel-wise normalization(related to issue ImageDataGenerator samplewise_center standardize method subtracting mean over image channels #2559 and PR Update ImageDataGenerator to allow channelwise standardization #2879, also mentioned in Rewriting image augmenter  #2446).

For flow_from_directory():

• remove the limit for channel number and color mode, by using PIL, we can just provide the PIL image mode as a parameter which gives more flexibility on the image format, channel data type and channel number. Image with ARGB mode png file, or F mode(float) tiff which is widely used in many field should be naturally supported. (also related to ImageDataGenerator - feature-request support for 'BGR' #3140)
• provide an optional image file reader keyword to replace the underlying PIL image reader (for flow_from_directory). With a file reader function which takes a path as input and return a numpy array, we can provide much more flexible on the image format. We can read hdf5, npz, pickle file, or for example, you can use a library named openslide to read histopathological images. (also related to ImageDataGenerator for time distributed image data #1819 for reading time distributed data)

For fit():

• fit is required for feature-wise standardization and ZCA , and it only takes an array as parameter, there is no fit for directory. For now, we need to manually read a subset of the image to do this fit for a directory. One idea is we can change fit() to accept the generator itself(flow_from_directory), of course, standardization should be disabled during fit.

[pengpaiSH]

@oeway When it comes to visual semantic segmentation, I find out that extending ImageDataGenerator is really necessary! Current ImageDataGenerator will only conduct transformations on X instead of Y. However, in image segmentation scenarios, input image X and output mask Y should be transformed simultaneously.

[oeway]

@pengpaiSH I have a working version right now, you are welcome to try it out. https://github.com/oeway/keras/tree/extendImageDataGenerator (branch: extendImageDataGenerator)

For now, you can customize the pipeline and synchronize two ImageDataGenerator by a __add__(+) operator. You can see more extensions in the following example:

from keras.preprocessing.image import ImageDataGenerator,standardize,random_transform
# input generator with standardization on
datagenX = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    featurewise_standardize_axis=(0, 2, 3),
    rotation_range=180,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    fill_mode='reflect',
    seed=0,
    verbose=1)

# output generator with standardization off
datagenY = ImageDataGenerator(
    featurewise_center=False,
    featurewise_std_normalization=False,
    rotation_range=180,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    fill_mode='reflect',
    seed=0)

def center_crop(x, center_crop_size, **kwargs):
    centerw, centerh = x.shape[1]//2, x.shape[2]//2
    halfw, halfh = center_crop_size[0]//2, center_crop_size[1]//2
    return x[:, centerw-halfw:centerw+halfw,centerh-halfh:centerh+halfh]

def random_crop(x, random_crop_size, sync_seed=None, **kwargs):
    np.random.seed(sync_seed)
    w, h = x.shape[1], x.shape[2]
    rangew = (w - random_crop_size[0]) // 2
    rangeh = (h - random_crop_size[1]) // 2
    offsetw = 0 if rangew == 0 else np.random.randint(rangew)
    offseth = 0 if rangeh == 0 else np.random.randint(rangeh)
    return x[:, offsetw:offsetw+random_crop_size[0], offseth:offseth+random_crop_size[1]]

datagenX.config['random_crop_size'] = (800, 800)
datagenY.config['random_crop_size'] = (800, 800)
datagenX.config['center_crop_size'] = (512, 512)
datagenY.config['center_crop_size'] = (360, 360)

# customize the pipeline
datagenX.set_pipeline([random_crop, random_transform, standardize, center_crop])
datagenY.set_pipeline([random_crop, random_transform, center_crop])

# flow from directory is extended to support more format and also you can even use your own reader function
# here is an example of reading image data saved in csv file
# datagenX.flow_from_directory(csvFolder, image_reader=csvReaderGenerator, read_formats={'csv'}, reader_config={'target_size':(572,572),'resolution':20, 'crange':(0,100)}, class_mode=None, batch_size=1)

dgdx= datagenX.flow_from_directory(inputDir, class_mode=None, read_formats={'png'}, batch_size=2)
dgdy= datagenY.flow_from_directory(outputDir,  class_mode=None, read_formats={'png'}, batch_size=2)

# you can now fit a generator as well
datagenX.fit_generator(dgdx, nb_iter=100)

# here we sychronize two generator and combine it into one
train_generator = dgdx+dgdy

model.fit_generator(
        train_generator,
        samples_per_epoch=2000,
        nb_epoch=50,
        validation_data=validation_generator,
        nb_val_samples=800)

[pengpaiSH]

@oeway Thank you for your quick response and your enhancement for ImageGenerator which I think is really really necessary. Below are my confusions:

1. As far as I understand, center_crop and random_crop is our customized augmented transformation, right?
2. In the defined pipeline, datagenX.set_pipeline([random_crop, random_transform, standardize, center_crop]), where does random_transform and standardize come from? They are not even defined such as center_crop.
3. I don't understand why datagenX should be fit_generator()? I think only model should have fit_generator()

[oeway]

Sorry for the confusion, I will write documents about it when I have
time.

1, yes, you just need to provide a function and put in the pipeline,
center_crop is one example, it shows how you can define the function and
how to set arguments for your function.

2, I forgot to put the import line, so random_transform and standardize are
to predefined function which I split from the previous ImageDataGenerator,
they are from the same module as ImageDataGenerator. Just do:

from keras.preprocessing.image import standardize,random_transform

3, fit_generator and fit is for some augment function such as feature_wise
standardization which require a pre computed mean and std value. I added
fit_generator for flow_from_directory(see my previous post for why we need
that), and I generalized the fit interface so that you can now add
customized function which support pre-fit.

I will document all the features if there are more people interested, and
perhaps make a PR. In the meantime, welcome to try and provide feedback.

On Tue, Aug 9, 2016 at 10:49 Pai Peng notifications@github.com wrote:

@oeway https://github.com/oeway Thank you for your quick response and
your enhancement for ImageGenerator which I think is really really
necessary. Below are my confusions:

1. As far as I understand, center_crop and random_crop is our
   customized augmented transformation, right?
2. In the defined pipeline, datagenX.set_pipeline([random_crop,
   random_transform, standardize, center_crop]), where does
   random_transform and standardize come from? They are not even defined
   such as center_crop.
3. I don't understand why datagenX should be fit_generator()? I think
   only model should have fit_generator()

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3338 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdNy2ZFmGvxVU_i2jfVxSHOREIHs7dGks5qeD8ogaJpZM4JXAo4
.

[pengpaiSH]

@oeway Thank you for your clarifying. Perfect! By the way, the transformations defined in the pipeline come in order or not ?

[oeway]

@pengpaiSH Welcome, of course, the idea is use an ordered list to defined the pipeline, and you can change the order of the pipeline.
Let me know if you have any problem.

[landersson]

I also need random cropping, better scaling etc and was just about to start implementing something like this myself. Looks very useful. Will you try to get this merged into the main repo?

[burgalon]

Any plans to merge @oeway branch? I'm specifically interested in the CSV reader option

[ncullen93]

@burgalon It's very straight-forward .. like maybe 10 lines of code.. to extend the data generator to read csv.. Still agree it should be merged but in case you dont want to wait :) . I attached some very unpolished code which works perfectly for me (see 'flow_np_from_directory' function and 'NumpyDirectoryIterator' class specifically) to read in .npy files in the same directory structure as the image version, e.g.:

• dir/
  -class1/
  ... file1.npy
  ... file2.npy
  -class2/
  ... file3.npy

image.zip

[burgalon]

Thanks @ncullen93

[johnny5550822]

@oeway @burgalon @ncullen93 also, I notice that .flow_from_directory has a way to make the original input to target_size, but .flow() does not....maybe we should add this functionality to the ImageDataGenerator so the for example I can resize my image size of 21x21 to 224x224.

I don't find any function like what I have pointed out above.

[ncullen93]

Hm yeah.. I noticed that too. That's probably because the image resizing is done directly on the image object before converting to an array using PIL's resize method (see line 302 in preprocessing/image.py)... .flow() would require resampling a numpy array, probably using scipy's functionality, so I guess it's assumed you will just resize it yourself since the data already fits in memory.

[johnny5550822]

@ncullen93 yup, exactly, but sometime your data may be large array but not image. So it would be nice to provide a way to do resize. I think cv2 can do that nicely. if this functionality is included, I don't need to write a custom datagenerator to do the resize using cv2.

[ncullen93]

true. i don't make those decisions unfortunately. In the meantime, here's a very slightly modified image.py file with a big commented section on line 719 ( look for the "### ADD CODE HERE TO RESHAPE ARRAY TO TARGET SIZE ####") for you to add one line of code to upsample/reshape your array however you like! Just add code to upsample/reshape your array and then you can use "target_size" on the flow() function just like flow_from_directory(). It's very simple. Then replace this image.py file with that in your keras directory, and reinstall. That's all I personally can offer.

image.zip

[johnny5550822]

@ncullen93 Wow, thanks! I will look at that.

[stratospark]

I extended @oeway 's ImageDataGenerator fork to use a multiprocessing Pool. I got a pretty good speedup: https://github.com/stratospark/food-101-keras/blob/master/tools/image_gen_extended.py

This was to fully utilize a Titan X GPU as I write about here: http://blog.stratospark.com/deep-learning-applied-food-classification-deep-learning-keras.html

[ncullen93]

Can you comment a little more on it. Does it support both arrays and directory sampling? Seems to only be arrays from my quick look. Directory sampling is probably the biggest bottleneck in my opinion, since bigger nets typically coincide w/ datasets that dont fit in memory - so the speedup margins on using this w/ directory sampling would be much greater. anyways, implementing just the multiprocessing speedup on the actual keras image generator without the pipeline stuff (since it's not standardized) would be a huge contribution in my opinion

[stratospark]

@ncullen93 I'm planning on cleaning up the code soon and posting a PR so it can hopefully be merged into the main Keras branch. Enough to maintain compatibility with the mainline Keras features.

It was a quick hack to support my use case, augmenting images in memory fast enough to keep my GPU close to full utilization. I had to disable any augmentation related to fitting the model, such as normalization and zca. This was due to the locks not being able to be pickled when doing a multiprocessing map.

Another change I had to make was having to explicitly pass a multiprocessing.Pool. This is due to the fact that Python multiprocessing forks the process, and thus it is easy to run into out of memory errors if you fork after loading your images. I have to create the pool at the very beginning of the script so it uses as little resources as possible. I'm pretty new at this, so there is probably a more elegant way of handling this. One thing that I would like to fix is when interrupting the training in Jupyter Notebook, it kills the processes so I have to restart the kernel and load up all the images to do another run. This really breaks the flow of trying different models and parameters!

To support reproducible results, I also had to create separate random number generators for each process, so each process didn't return the same random numbers (and thus the exact same images as each other). I know I left a few numpy default random generators, so I need to fix that up.

[joeyearsley]

This would fit nicely in the contrib library: https://github.com/farizrahman4u/keras-contrib

[ahundt]

@stratospark I agree with @joeyearsley you should add a PR for keras-contrib, particularly now that keras-2 is out.

[jkjung-avt]

I shared how I extended Keras' ImageDataGenerator to support random cropping in this blog post:
Extending Keras' ImageDataGenerator to Support Random Cropping.
https://jkjung-avt.github.io/keras-image-cropping/

I think the same technique could be easily adapted as a solution to the original question.

"Is there an easy way to write generator extensions for Keras? I'd like to use some of the ImageDataGenerator preprocessing steps but also add some of my own such as randomly occluding areas of the image, adding noise etc. If not I can add some of these things to the ImageDataGenerator class in Keras, is that something that would be useful?"

[ChristianEschen]

I am doing semantic segmentation, thus I have both input images X and label images Y of size 1440x1920.
I would like to train a model with 224x224 input size. In order not to
lose information it would be nice to random crop the images of size
224x224.
Furthermore, I would like to do random crop sampling from all
categories from the label image Y in order to have batches with equal class
distributions.
I have investigated https://jkjung-avt.github.io/keras-image-cropping/#comment-3887776756 for random crop.
The problem is that the information for doing random crop from the classes in the label images comes from the label image Y. Thus I guess we need to store the x_index and y_index for random crop from classes in label image Y.
However, I can't see how this is possible.
Eg. you have 2 generators.
datagenY = ImageDataGenerator()
datagenX = ImageDataGenerator()

and I do:

dgdy= datagenY.flow_np_from_directory()
dgdx= datagenX.flow_np_from_directory()

And syncronizing the generators:
training_generator=zip(dgdx,dgdy)

And if I modify image.py I can add a random_crop_from_categories.
However, I do not have access to the pixel index from datagenY in datagenX.

Is it possible to implement a random_crop_from_categories in the image.py file to sample equally from the categories in Y and apply the same crop in X?

[gledsonmelotti]

In the keras command, you have to define the image type, that is, color_mode = 'rgb' or 'grayscale' when using the command flow_from_directory with model.fit_generator. So how do I change color_mode in flow_from_directory to use with 2, 4, 5 and 6 channels?

I thank you for your attention,
Gledson

[jkjung-avt]

@ChristianEschen, I copied my reply on Disqus over here.

You can define your paired_crop_generator() to crop the same region from X (images) and Y (masks) simultaneously. The code below is just quick and dirty hack, but hopefully you get the idea.

def paired_crop_generator(batches, crop_length=224):
    '''
    Assuming batch_X.shape is (N,1440,1920,3) and batch_Y.shape is (N,1440,1920,1),
    this function would crop the same (224,224) region from the XY pair for each image
    in the batch, then return (yield) the cropped batch.
    '''
    while True:
        batch_X, batch_Y = next(batches)
        batch_Xcrops = np.zeros((batch_X.shape[0], crop_length, crop_length, 3))
        batch_Ycrops = np.zeros((batch_X.shape[0], crop_length, crop_length, 1))
        height, width = batch_X.shape[1], batch_X.shape[2]
        dy, dx = random_crop_length, random_crop_length
        for i in range(batch_X.shape[0]):  # loop through all images in the batch
            # select a random top-left corner of the crop, for current image in the batch
            x = np.random.randint(0, width - dx + 1)
            y = np.random.randint(0, height - dy + 1)
            # take the crop (same region) from both X and Y
            batch_Xcrops[i] = batch_X[i][y:(y+dy), x:(x+dx), :]
            batch_Ycrops[i] = batch_Y[i][y:(y+dy), x:(x+dx), :]
        yield (batch_Xcrops, batch_Ycrops)

......
training_generator = zip(dgdx,dgdy)
train_crops = paired_crop_generator(training_generator, 224)

net_final.fit_generator(train_crops, ......)

[jkjung-avt]

@gledsonmelotti, I believe Keras' flow_from_directory() API could only handle 'rgb' or 'grayscale' images. For training data with 2, 4, 5, 6, ... channels, you'd need to implement your own data loading code.

[tinalegre]

@oeway I've a model with multiple-output loss functions and need to provide for each of those losses the ground-truth masks, sth. like.: train_generator = imGen+maskGen+maskGen+maskGen, for three multiple-output loss functions. Any idea, on how to achieve that with the extension you did to the ImageGenerator class? i.e. how to combine the multiple generators in a correct way?

[mbenami]

Hey @tinalegre
have you try to load the 3 mask you need and insert them into 1 list?
then you just need to extract the element in custom loss function that handles this.

[oeway]

@tinalegre @mbenami If your output images are with the same size, what I would do is to combine these them into one image with multiple channels, so you can generate the data as if there are one input image and one output image. with my extension, most of those transformation functions they can support any number of channels.
For the network itself, you will need to use K.concatenate to to combine your outputs into one output image with multiple channels, and then, I will add a customized objective function:

def customized_objective(y_true, y_pred):
        # split the tensor into different channels
        ch1_true = y_true[:, :, :, :1]
        ch2_true = y_true[:, :, :, 1:2]
        ...
        return mse(ch1_true, ch1_pred) + L1(ch2_true, ch2_pred)

At least, this is how I do it for outputs, good luck.

[tinalegre]

@mbenami @oeway thanks for your feedback, I ended up following the solution of presented here and it worked, basically I created a custom function that given the generators of the form (x,y), it returns the output in the form (x, [y,y,y])

[gledsonmelotti]

@jkjung-avt Thank you very much.

[tinalegre]

@oeway @mbenami thank you for your feedback, my previous solution was actually wrong. I followed your suggestion @oeway and I changed my network by using a Lambda function to concatenate the outputs as follows:

    def concatenate_activations(x):
		return K.concatenate(x)

    ....
    outputs = Lambda(concatenate_activations, name='Concatenate_activations')(outputs)
    model = Model(inputs=inputs, outputs=outputs)

without the Lambda it was not working as Keras expected a Tensor as output parameter. When compiling the model, I'm doing: model.compile(loss=multi_loss, optimizer=optimizer, metrics=metrics)

def multi_loss(y_true, y_preds, nb_classes=2, nb_combloss=10, single_weight=1.0, combine_weight=1.0):
    y_combine_preds = y_preds[:, :, 0:(nb_classes*nb_combloss)]
    y_single_preds = y_preds[:, :, (nb_classes*nb_combloss):(nb_classes*nb_combloss+nb_classes)]

    def combine_loss(preds):
        loss = 0
        for ii in range(1, nb_combloss+1):
            ii_curr = (nb_classes*(ii-1))
            pred = preds[:,:, ii_curr:ii_curr+nb_classes]
            loss += categorical_crossentropy(y_true, pred)
        return loss

    def single_loss(preds):
        loss = categorical_crossentropy(y_true, preds)
        return loss

    return single_weight * single_loss(y_single_preds) + combine_weight * combine_loss(y_combine_preds)

where y_preds is of shape [:,65536,22], y_single_preds is of shape [:,65536,2] and y_combine_preds is of shape [:, 65536,20]. It seems to be that I did something wrong, as I'm getting an error of: Invalid argument: Incompatible shapes: [8,65536,2] vs. [8,65536,22], any ideas?

[oeway]

@tinalegre It seems to me that you need to make y_true has the same shape as y_pred, by padding with zeros for example. I guess Keras expect you will have the same shape for y_true and y_pred, so you just need to pass the check of the Keras engine.

[tinalegre]

@oeway the problem is somehow in the metrics = ['accuracy', 'mean_squared_error'] functions, for some reason, for instance, the mean_squared_error that I've only one output. Basically, what I've a binary segmentation problem with binary masks as one_hot_vectors and I'm summing up a set of categorical_crossentropy losses. Any ideas whether I need to modify the mean_squared_error so that it averages the mean_squared_error from all the losses?

[oeway]

@tinalegre I think mean_squared_error expect you to have the same shape for y_pred and y_true. It doesn't make sense to me that why you need mean_squared_error if it's a classification problem. I would simply remove mean_squared_error. Or if you want to somehow have it, you definately need to make your own customized metric function which accepts different shapes for y_pred and y_true.

[XiaoTonyLuo]

@oeway Hi I am confused on the memory allocation scheme of ImageDataGenerator. I write a generator, the structure is similar to @jagiella 's example sulution. But when I use fit_generator to train my network I found that the memory of the generator accumulates as epoch increases and finally stopped with out of memory error, and when I try ImageDataGenerator there is no such problem. Could you please give me some advice on it? Thank you!

[amineio]

For Transforming Images and masks (synchronously) for segmentation for example, take a look on keras documentation (checked recently) 👍
Example of transforming images and masks together

[tinalegre]

@amineio thanks for the info. did you use it? and how would you manage one-hot encoding for the masks?

[amineio]

@tinalegre there is only two classes (black and white masks) the final layer looks like that :
outputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)
You can refer to many Unet implementations for further details, exp : zhixuhao

[bernardohenz]

@oeway
first, I want to say that I've been using your code, and it feels so good the ability to control the order of transformations on the pipeline (and even create new transformations and add them to the pipeline).

Currently, I am using the ImageDataGenerator` as a pre-processing step before prediction, so I must run them even when testing the models accuracy. My question is: when using evaluate_generatorwithuse_multiprocessing=True```, is it guarantee that each sample will be evaluated only once?

I've been search in the documentation, but the only thing I could find is

The use of keras.utils.Sequence guarantees the ordering and guarantees the single use of every input per epoch when using use_multiprocessing=True.

Does your code account for that?

[oeway]

@bernardohenz thanks for your feedback. I don't have time to look in more details right now. I think probably it's not guaranteed, Keras introduced keras.utils.Sequence to work with multiprocessing(a warning related to that), and my implementation do not use that. I think you will need to investigate on how to change the generator to inherit form keras.utils.Sequence. Maybe you want to look at this tutorial:
https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly.html

[bernardohenz]

hi @oeway

I've checked this link already, but the generator already offers so much features (flow_from_directory, several transformations, pipeline ordering) that would take some time to implement in the keras.utils.Sequence. I believe that adapting the Sequence inside the generator would be faster, but right now I don't know if it is possible. I'll take a look into that.

[bernardohenz]

@oeway I've adapted your code to use the keras.utils.Sequence. It is working both fow flow and flow_from_directory. I created a repo with the Datagen and test files: https://github.com/bernardohenz/ExtendableImageDatagen

[wt-huang]

Closing as this is resolved

[rstml]

Just for the reference, here's another way of adding image manipulation (random & center crop in my case) to the pipeline before Keras resizes images. It's done by monkey patching keras_preprocessing.image.utils.loag_img function as I couldn't find any other way.

https://gist.github.com/rstml/bbd491287efc24133b90d4f7f3663905

[Nestak2]

@stratospark @oeway Are any of the scripts here merged into the main keras? 3 years have gone and default keras still doesn't allow the use of csv/npy ...

[tinalegre]

@oeway in your code you showed us how to sychronize two generator and combine it into one: train_generator = dgdx+dgdy. Now, imagine, I would like to combine 3 generators, i.e. 1x for the input image (dgdx), and 2x outputs: segmentation mask (dgdx) and depth mask (dgdz). I was therefore trying to do something like: train_generator = dgdx+dgdy+dgdz but it didn't work.
=> TypeError: unsupported operand type(s) for +: 'zip' and 'DirectoryIterator'
Any idea, how could I integrate this into my model, i.e. 2 outputs with your generator?

@pengpaiSH I have a working version right now, you are welcome to try it out. https://github.com/oeway/keras/tree/extendImageDataGenerator (branch: extendImageDataGenerator)

For now, you can customize the pipeline and synchronize two ImageDataGenerator by a __add__(+) operator. You can see more extensions in the following example:

from keras.preprocessing.image import ImageDataGenerator,standardize,random_transform
# input generator with standardization on
datagenX = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    featurewise_standardize_axis=(0, 2, 3),
    rotation_range=180,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    fill_mode='reflect',
    seed=0,
    verbose=1)

# output generator with standardization off
datagenY = ImageDataGenerator(
    featurewise_center=False,
    featurewise_std_normalization=False,
    rotation_range=180,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    fill_mode='reflect',
    seed=0)

def center_crop(x, center_crop_size, **kwargs):
    centerw, centerh = x.shape[1]//2, x.shape[2]//2
    halfw, halfh = center_crop_size[0]//2, center_crop_size[1]//2
    return x[:, centerw-halfw:centerw+halfw,centerh-halfh:centerh+halfh]

def random_crop(x, random_crop_size, sync_seed=None, **kwargs):
    np.random.seed(sync_seed)
    w, h = x.shape[1], x.shape[2]
    rangew = (w - random_crop_size[0]) // 2
    rangeh = (h - random_crop_size[1]) // 2
    offsetw = 0 if rangew == 0 else np.random.randint(rangew)
    offseth = 0 if rangeh == 0 else np.random.randint(rangeh)
    return x[:, offsetw:offsetw+random_crop_size[0], offseth:offseth+random_crop_size[1]]

datagenX.config['random_crop_size'] = (800, 800)
datagenY.config['random_crop_size'] = (800, 800)
datagenX.config['center_crop_size'] = (512, 512)
datagenY.config['center_crop_size'] = (360, 360)

# customize the pipeline
datagenX.set_pipeline([random_crop, random_transform, standardize, center_crop])
datagenY.set_pipeline([random_crop, random_transform, center_crop])

# flow from directory is extended to support more format and also you can even use your own reader function
# here is an example of reading image data saved in csv file
# datagenX.flow_from_directory(csvFolder, image_reader=csvReaderGenerator, read_formats={'csv'}, reader_config={'target_size':(572,572),'resolution':20, 'crange':(0,100)}, class_mode=None, batch_size=1)

dgdx= datagenX.flow_from_directory(inputDir, class_mode=None, read_formats={'png'}, batch_size=2)
dgdy= datagenY.flow_from_directory(outputDir,  class_mode=None, read_formats={'png'}, batch_size=2)

# you can now fit a generator as well
datagenX.fit_generator(dgdx, nb_iter=100)

# here we sychronize two generator and combine it into one
train_generator = dgdx+dgdy

model.fit_generator(
        train_generator,
        samples_per_epoch=2000,
        nb_epoch=50,
        validation_data=validation_generator,
        nb_val_samples=800)

[zachwarner]

Hi all. Many thanks to @oeway and @bernardohenz for their very helpful solutions. Just checking back in two years later: is there a similar adaptation of ImageDataGenerator now that the Keras team have refactored this code?

The specific functionality I'm looking for is just precise reproducibility; I'd like ImageDataGenerator to be seedable but do not need the other features. For reference, my workflow is ImageDataGenerator -> flow -> fit_generator and I'm running Keras 2.3.1 with TF 2.0 backend in python 3.6.9 in a docker container with base Ubuntu 18.04.

[prarshah]

@ChristianEschen, I copied my reply on Disqus over here.

You can define your paired_crop_generator() to crop the same region from X (images) and Y (masks) simultaneously. The code below is just quick and dirty hack, but hopefully you get the idea.

def paired_crop_generator(batches, crop_length=224):
    '''
    Assuming batch_X.shape is (N,1440,1920,3) and batch_Y.shape is (N,1440,1920,1),
    this function would crop the same (224,224) region from the XY pair for each image
    in the batch, then return (yield) the cropped batch.
    '''
    while True:
        batch_X, batch_Y = next(batches)
        batch_Xcrops = np.zeros((batch_X.shape[0], crop_length, crop_length, 3))
        batch_Ycrops = np.zeros((batch_X.shape[0], crop_length, crop_length, 1))
        height, width = batch_X.shape[1], batch_X.shape[2]
        dy, dx = random_crop_length, random_crop_length
        for i in range(batch_X.shape[0]):  # loop through all images in the batch
            # select a random top-left corner of the crop, for current image in the batch
            x = np.random.randint(0, width - dx + 1)
            y = np.random.randint(0, height - dy + 1)
            # take the crop (same region) from both X and Y
            batch_Xcrops[i] = batch_X[i][y:(y+dy), x:(x+dx), :]
            batch_Ycrops[i] = batch_Y[i][y:(y+dy), x:(x+dx), :]
        yield (batch_Xcrops, batch_Ycrops)

......
training_generator = zip(dgdx,dgdy)
train_crops = paired_crop_generator(training_generator, 224)

net_final.fit_generator(train_crops, ......)

@jkjung-avt @XiaoTonyLuo
Thankyou, I tried it, but using this method the model.predict method never completes, any suggestions how to tackle it?