Style Transfer error

[alienatorZ]

Issue Description

Please describe your issue, along with:

• expected behavior
  Run until execution finished.

• encountered behavior
  Error stops execution.

Stack Trace:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:66)
Caused by: java.lang.RuntimeException: NativeOps::execPairwiseTransform both operands must have same data type; Expected: [DOUBLE]; Actual: [FLOAT]
at org.nd4j.nativeblas.Nd4jCuda$NativeOps.execPairwiseTransform(Native Method)
at org.nd4j.linalg.jcublas.ops.executioner.CudaExecutioner.invoke(CudaExecutioner.java:1420)
at org.nd4j.linalg.jcublas.ops.executioner.CudaExecutioner.exec(CudaExecutioner.java:635)
at org.nd4j.linalg.api.ndarray.BaseNDArray.addi(BaseNDArray.java:3818)
at org.nd4j.linalg.api.ndarray.BaseNDArray.addi(BaseNDArray.java:3788)
at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.createCombinationImage(NeuralStyleTransfer.java:192)
at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.transferStyle(NeuralStyleTransfer.java:132)
at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.main(NeuralStyleTransfer.java:124)
... 5 more

Version Information

Please indicate relevant versions, including, if relevant:

• Deeplearning4j version
  1.0.0beta4
• platform information (OS, etc)
  Windows 10
• CUDA version, if used
  10.1
• NVIDIA driver version, if in use
  430.86

Contributing

If you'd like to help us fix the issue by contributing some code, but would
like guidance or help in doing so, please mention it!

[martinsbrn]

I'm having the same issue.

[saudet]

/cc @klevis

[mohamad7788]

Hi,

May this could help ? deeplearning4j/deeplearning4j#7841

[vera-dobryanskaya]

Transform both operands must have same data type; Expected: [DOUBLE]; Actual: [FLOAT] can be resolved with combination.muli(NOISE_RATION).addi(content.muli(1 - NOISE_RATION).castTo(DataType.DOUBLE)); at line 194.

However, after the change is made, I see a different exception which seems more serious:

---

Exception in thread "main" java.lang.IllegalStateException: Cannot perform backprop: Dropout mask array is absent (already cleared?)
	at org.nd4j.base.Preconditions.throwStateEx(Preconditions.java:641)
	at org.nd4j.base.Preconditions.checkState(Preconditions.java:268)
	at org.deeplearning4j.nn.conf.dropout.Dropout.backprop(Dropout.java:167)
	at org.deeplearning4j.nn.layers.AbstractLayer.backpropDropOutIfPresent(AbstractLayer.java:302)
	at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.backpropGradient(ConvolutionLayer.java:192)
	at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.backPropagate(NeuralStyleTransfer.java:292)
	at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.backPropagateStyles(NeuralStyleTransfer.java:170)
	at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.transferStyle(NeuralStyleTransfer.java:145)
	at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.main(NeuralStyleTransfer.java:126)

---

Any thoughts on how to fix that? Does this mean the layers list/order is incorrect?

[vera-dobryanskaya]

Ok, I kind of was able to run the style transfer example, but at this point I'm not sure if it is mathematically correct/efficient. Here is the patch content / intellij 2019.2:

Index: dl4j-examples/src/main/java/org/deeplearning4j/examples/styletransfer/NeuralStyleTransfer.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- dl4j-examples/src/main/java/org/deeplearning4j/examples/styletransfer/NeuralStyleTransfer.java	(revision 20225a8a70b8932677538d55be55ccacafa01333)
+++ dl4j-examples/src/main/java/org/deeplearning4j/examples/styletransfer/NeuralStyleTransfer.java	(date 1567049714186)
@@ -19,11 +19,13 @@
 import org.datavec.image.loader.NativeImageLoader;
 import org.deeplearning4j.examples.download.DownloaderUtility;
 import org.deeplearning4j.nn.api.Layer;
+import org.deeplearning4j.nn.conf.dropout.IDropout;
 import org.deeplearning4j.nn.graph.ComputationGraph;
 import org.deeplearning4j.nn.workspace.LayerWorkspaceMgr;
 import org.deeplearning4j.zoo.PretrainedType;
 import org.deeplearning4j.zoo.ZooModel;
 import org.deeplearning4j.zoo.model.VGG16;
+import org.nd4j.linalg.api.buffer.DataType;
 import org.nd4j.linalg.api.ndarray.INDArray;
 import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization;
 import org.nd4j.linalg.dataset.api.preprocessor.VGG16ImagePreProcessor;
@@ -145,7 +147,7 @@
             INDArray backPropContent = backPropagateContent(vgg16FineTune, activationsContentMap, activationsCombMap);
             INDArray backPropAllValues = backPropContent.muli(ALPHA).addi(styleBackProb.muli(BETA));
             adamUpdater.applyUpdater(backPropAllValues, iteration, 0);
-            combination.subi(backPropAllValues);
+            combination.subi(backPropAllValues.castTo(DataType.DOUBLE));
 
             log.info("Total Loss: " + totalLoss(activationsStyleMap, activationsCombMap, activationsContentMap));
             if (iteration % SAVE_IMAGE_CHECKPOINT == 0) {
@@ -190,7 +192,7 @@
         INDArray content = LOADER.asMatrix(new File(dataLocalPath,CONTENT_FILE));
         IMAGE_PRE_PROCESSOR.transform(content);
         INDArray combination = createCombineImageWithRandomPixels();
-        combination.muli(NOISE_RATION).addi(content.muli(1 - NOISE_RATION));
+        combination.muli(NOISE_RATION).addi(content.muli(1 - NOISE_RATION).castTo(DataType.DOUBLE));
         return combination;
     }
 
@@ -288,7 +290,17 @@
 
         for (int i = startFrom; i > 0; i--) {
             Layer layer = vgg16FineTune.getLayer(ALL_LAYERS[i]);
-            dLdANext = layer.backpropGradient(dLdANext, LayerWorkspaceMgr.noWorkspaces()).getSecond();
+            org.deeplearning4j.nn.conf.layers.Layer conf = (org.deeplearning4j.nn.conf.layers.Layer)layer.getConfig();
+
+            try {
+
+                ((org.deeplearning4j.nn.conf.layers.Layer) layer.getConfig()).setIDropout(null);
+
+                dLdANext = layer.backpropGradient(dLdANext, LayerWorkspaceMgr.noWorkspaces())
+                                .getSecond();
+            }catch (Exception e){
+                log.error("Error {} => {}", layer.getConfig().getClass().getSimpleName(), layer.getConfig().getLayerName(), e);
+            }
         }
         return dLdANext;
     }

 

[saudet]

@AlexDBlack What do you think?

[verunder]

This issue still open. I was able to do style transfer just fine with the diff fix, even on other works(custom drawings). So, should I create an MR for my fix from above?

Thank you
vera-dobryanskaya