Use originating mirror for types instantiated from TypeTag. #27

metasim · 2018-03-10T23:41:54Z

Fix for #26

witold-jedrzejewski · 2018-03-13T10:04:38Z

seahorse-workflow-executor/deeplang/src/main/scala/ai/deepsense/deeplang/TypeUtils.scala

@@ -57,10 +58,10 @@ object TypeUtils {
  }

  def instanceOfType[T](typeTag: ru.TypeTag[T]): T = {
-    val constructorT = constructorForType(typeTag.tpe).getOrElse {
+    val clazz = typeTag.mirror.runtimeClass(typeTag.tpe)
+    Try(clazz.newInstance.asInstanceOf[T]).getOrElse {


Perhaps we could rework classToType and typeToClass methods to use appropriate mirrors instead?

Tried to minimize changes more than a single type away from TypeUtils. Signed-off-by: Simeon H.K. Fitch <fitch@astraea.io>

metasim · 2018-03-13T13:41:44Z

@witold-jedrzejewski-codilime I attempted to propagate the use of TypeTags as far as I could without it spreading really far into the core classes. If we want to go that route, we'd have to replace most uses of Type with TypeTag and figure out how to do some of the parent class operations that are currently done with Type in that modality, or start passing the runtime.Mirror along as a parameter.

witold-jedrzejewski · 2018-03-13T13:40:46Z

...ow-executor/deeplang/src/main/scala/ai/deepsense/deeplang/catalogs/doperable/TraitNode.scala

@@ -17,7 +17,7 @@
 package ai.deepsense.deeplang.catalogs.doperable

 import scala.reflect.runtime.{universe => ru}
-
+import ru.{TypeTag, typeTag}


Sorry. One of those mornings.

witold-jedrzejewski · 2018-03-13T13:44:10Z

seahorse-workflow-executor/deeplang/src/main/scala/ai/deepsense/deeplang/TypeUtils.scala

 import ai.deepsense.deeplang.params.exceptions.NoArgumentConstructorRequiredException
 import ai.deepsense.sparkutils

+import scala.util.Try


Signed-off-by: Simeon H.K. Fitch <fitch@astraea.io>

jaroslaw-osmanski · 2018-03-14T14:30:10Z

@metasim @witold-jedrzejewski-codilime this change makes testing sdk-example fail (from ./build/build_and_run_tests.sh). I'm working on other issues now but I'll have time tomorrow . We should wait with merge.

metasim · 2018-03-15T14:13:44Z

@jaroslaw-osmanski Is this issue you were having the same as #35? If so, it's from something further downstream.

jaroslaw-osmanski · 2018-03-16T07:23:38Z

@metasim it is:

[WARN ] [2018-03-15 08:09:32,558] [pool-8-thread-10-ScalaTest-running-RandomSplitSpec]  org.apache.spark.sql.SparkSession$Builder - Using an existing SparkSession; some configuration may not take effect.
[info] IdentityOperationSpec:
[info] IdentityOperation
[info] - should output DataFrame provided as input *** FAILED ***
[info]   java.lang.ClassNotFoundException: ai.deepsense.deeplang.DOperable
[info]   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
[info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[info]   at java.lang.Class.forName0(Native Method)
[info]   at java.lang.Class.forName(Class.java:348)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror.javaClass(JavaMirrors.scala:555)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToJava$1.apply(JavaMirrors.scala:1211)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToJava$1.apply(JavaMirrors.scala:1203)
[info]   at scala.reflect.runtime.TwoWayCaches$TwoWayCache$$anonfun$toJava$1.apply(TwoWayCaches.scala:49)
[info]   at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
[info]   ...
[info] RandomSplitSpec:
[info] RandomSplit
[info] - should split DataFrame using given proportion *** FAILED ***
[info]   java.lang.ClassNotFoundException: ai.deepsense.deeplang.DOperable
[info]   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
[info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[info]   at java.lang.Class.forName0(Native Method)
[info]   at java.lang.Class.forName(Class.java:348)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror.javaClass(JavaMirrors.scala:555)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToJava$1.apply(JavaMirrors.scala:1211)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToJava$1.apply(JavaMirrors.scala:1203)
[info]   at scala.reflect.runtime.TwoWayCaches$TwoWayCache$$anonfun$toJava$1.apply(TwoWayCaches.scala:49)
[info]   at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
[info]   ...
[info] Run completed in 6 seconds, 404 milliseconds.

metasim · 2018-03-16T15:06:57Z

@jaroslaw-osmanski Yeh, this PR likely caused that. I'll work on it this weekend. Need to understand why sbt test didn't catch that...

metasim · 2018-03-18T14:52:17Z

I think part of the problem has to do with the API now being a Provided dependency, class loaders being more precisely controlled and catalog registration still occurring even when the test is instantiating it directly from the test classloader. The error is coming from the registration process, not the actual test execution. Still working out specifics.

Here's the fuller stack trace (provided by testOptions in Test += Tests.Argument("-oF")):

[info] - should split DataFrame using given proportion *** FAILED ***
[info]   java.lang.ClassNotFoundException: ai.deepsense.deeplang.DOperable
[info]   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
[info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[info]   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[info]   at java.lang.Class.forName0(Native Method)
[info]   at java.lang.Class.forName(Class.java:348)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror.javaClass(JavaMirrors.scala:555)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToJava$1.apply(JavaMirrors.scala:1211)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToJava$1.apply(JavaMirrors.scala:1203)
[info]   at scala.reflect.runtime.TwoWayCaches$TwoWayCache$$anonfun$toJava$1.apply(TwoWayCaches.scala:49)
[info]   at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
[info]   at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
[info]   at scala.reflect.runtime.TwoWayCaches$TwoWayCache.toJava(TwoWayCaches.scala:44)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror.classToJava(JavaMirrors.scala:1203)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:194)
[info]   at scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:54)
[info]   at ai.deepsense.deeplang.TypeUtils$.typeToClass(TypeUtils.scala:34)
[info]   at ai.deepsense.deeplang.catalogs.doperable.DOperableCatalog.register(DOperableCatalog.scala:90)
[info]   at ai.deepsense.deeplang.catalogs.doperable.DOperableCatalog.<init>(DOperableCatalog.scala:35)
[info]   at ai.deepsense.deeplang.CatalogRecorder.createDOperableCatalog(CatalogRecorder.scala:254)
[info]   at ai.deepsense.deeplang.CatalogRecorder.catalogs$lzycompute(CatalogRecorder.scala:266)
[info]   at ai.deepsense.deeplang.CatalogRecorder.catalogs(CatalogRecorder.scala:265)
[info]   at ai.deepsense.sdk.example.HelperMock$.executionContext$lzycompute(HelperMock.scala:48)
[info]   at ai.deepsense.sdk.example.HelperMock$.executionContext(HelperMock.scala:45)
[info]   at ai.deepsense.sdk.example.RandomSplitSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(RandomSplitSpec.scala:36)
[info]   at ai.deepsense.sdk.example.RandomSplitSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(RandomSplitSpec.scala:27)
[info]   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.WordSpecLike$$anon$1.apply(WordSpecLike.scala:1078)
[info]   at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
[info]   at org.scalatest.WordSpec.withFixture(WordSpec.scala:1881)
[info]   at org.scalatest.WordSpecLike$class.invokeWithFixture$1(WordSpecLike.scala:1075)
[info]   at org.scalatest.WordSpecLike$$anonfun$runTest$1.apply(WordSpecLike.scala:1088)
[info]   at org.scalatest.WordSpecLike$$anonfun$runTest$1.apply(WordSpecLike.scala:1088)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
[info]   at org.scalatest.WordSpecLike$class.runTest(WordSpecLike.scala:1088)
[info]   at org.scalatest.WordSpec.runTest(WordSpec.scala:1881)
[info]   at org.scalatest.WordSpecLike$$anonfun$runTests$1.apply(WordSpecLike.scala:1147)
[info]   at org.scalatest.WordSpecLike$$anonfun$runTests$1.apply(WordSpecLike.scala:1147)
[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
[info]   at scala.collection.immutable.List.foreach(List.scala:381)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:373)
[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:410)
[info]   at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
[info]   at scala.collection.immutable.List.foreach(List.scala:381)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
[info]   at org.scalatest.WordSpecLike$class.runTests(WordSpecLike.scala:1147)
[info]   at org.scalatest.WordSpec.runTests(WordSpec.scala:1881)
[info]   at org.scalatest.Suite$class.run(Suite.scala:1147)
[info]   at org.scalatest.WordSpec.org$scalatest$WordSpecLike$$super$run(WordSpec.scala:1881)
[info]   at org.scalatest.WordSpecLike$$anonfun$run$1.apply(WordSpecLike.scala:1192)
[info]   at org.scalatest.WordSpecLike$$anonfun$run$1.apply(WordSpecLike.scala:1192)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
[info]   at org.scalatest.WordSpecLike$class.run(WordSpecLike.scala:1192)
[info]   at org.scalatest.WordSpec.run(WordSpec.scala:1881)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:472)
[info]   at sbt.TestRunner.runTest$1(TestFramework.scala:76)
[info]   at sbt.TestRunner.run(TestFramework.scala:85)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
[info]   at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
[info]   at sbt.TestFunction.apply(TestFramework.scala:207)
[info]   at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
[info]   at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
[info]   at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
[info]   at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
[info]   at sbt.std.Transform$$anon$4.work(System.scala:63)
[info]   at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
[info]   at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
[info]   at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
[info]   at sbt.Execute.work(Execute.scala:235)
[info]   at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
[info]   at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
[info]   at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
[info]   at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:748)

…n ClassLoader. Signed-off-by: Simeon H.K. Fitch <fitch@astraea.io>

metasim · 2018-03-18T15:41:28Z

@jaroslaw-osmanski I believe the SDK test is fixed now.

Use originating mirror for types instantiated from TypeTag.

f799d1a

Fix for #26

witold-jedrzejewski reviewed Mar 13, 2018

View reviewed changes

Propagated TypeTag/mirror provenance further through APIs.

5e17972

Tried to minimize changes more than a single type away from TypeUtils. Signed-off-by: Simeon H.K. Fitch <fitch@astraea.io>

witold-jedrzejewski approved these changes Mar 13, 2018

View reviewed changes

Removed unused imports.

38ff34d

Signed-off-by: Simeon H.K. Fitch <fitch@astraea.io>

Jaroslaw Osmanski and others added 2 commits March 18, 2018 11:28

Change version to 1.4.3-SNAPSHOT

7e4a7dc

Fixed incorrect selection of "boot" ClassLoader instead of applicatio…

87d58fa

…n ClassLoader. Signed-off-by: Simeon H.K. Fitch <fitch@astraea.io>

jaroslaw-osmanski merged commit 96d7902 into deepsense-ai:master Mar 22, 2018

metasim deleted the astraea/fix-26 branch March 22, 2018 15:06

metasim mentioned this pull request Mar 29, 2018

Fixed another TypeTag-mirror related issue in AbstractChoiceParam #46

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use originating mirror for types instantiated from TypeTag. #27

Use originating mirror for types instantiated from TypeTag. #27

metasim commented Mar 10, 2018

witold-jedrzejewski Mar 13, 2018

metasim commented Mar 13, 2018

witold-jedrzejewski Mar 13, 2018

metasim Mar 13, 2018

witold-jedrzejewski Mar 13, 2018

jaroslaw-osmanski commented Mar 14, 2018

metasim commented Mar 15, 2018

jaroslaw-osmanski commented Mar 16, 2018

metasim commented Mar 16, 2018

metasim commented Mar 18, 2018

metasim commented Mar 18, 2018

Use originating mirror for types instantiated from TypeTag. #27

Use originating mirror for types instantiated from TypeTag. #27

Conversation

metasim commented Mar 10, 2018

witold-jedrzejewski Mar 13, 2018

Choose a reason for hiding this comment

metasim commented Mar 13, 2018

witold-jedrzejewski Mar 13, 2018

Choose a reason for hiding this comment

metasim Mar 13, 2018

Choose a reason for hiding this comment

witold-jedrzejewski Mar 13, 2018

Choose a reason for hiding this comment

jaroslaw-osmanski commented Mar 14, 2018

metasim commented Mar 15, 2018

jaroslaw-osmanski commented Mar 16, 2018

metasim commented Mar 16, 2018

metasim commented Mar 18, 2018

metasim commented Mar 18, 2018