[BUG]: In WriteStream not run ForeachWriter

[parad74]

Describe the bug
I created simple code

   var foreachWriter = new TestForeachWriter();

           SparkSession spark = SparkSession
                   .Builder()
                   .AppName("itur")
                   .GetOrCreate();


           var mySchema = new Microsoft.Spark.Sql.Types.StructType(new[]
                {
                   new StructField("IturCode", new Microsoft.Spark.Sql.Types.StringType()),
                   new StructField("IturERP", new Microsoft.Spark.Sql.Types.StringType()) ,
                   new StructField("QuantityEdit", new Microsoft.Spark.Sql.Types.StringType()),
                   new StructField("PartialQuantity", new Microsoft.Spark.Sql.Types.StringType())
               });

           DataFrame lines = spark
                   .ReadStream()
                  .Schema(mySchema)
                 .Csv(streamInputPath);

           s_query = lines
                .WriteStream()
                 .Foreach(foreachWriter)
                .Trigger(Trigger.ProcessingTime(5000))
                .Start();


           s_query.AwaitTermination();

not run

public virtual void Process(Row value)
      {
         Console.WriteLine(" == yes ==");
     }

from it

To Reproduce
Add sample project to look at
https://github.com/parad74/StreamingSparkTest

It possible add working simple code or look at my project
Thanks

[imback82]

@suhsteve Can you take a look?

@parad74 I cannot get to your repo. Can you share?

[suhsteve]

Are there any error messages you can share with us ?

[parad74]

I cannot get to your repo. Can you share?
https://github.com/parad74/StreamingSparkTest

[suhsteve]

@parad74 can you provide your error logs, the spark-submit command you used, and your the DOTNET_ASSEMBLY_SEARCH_PATHS environment variable if it is set ?

[parad74]

spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local C:\Count4U\S3\SparkTest\StreamingSparkTest\src\StreamingSparkTest\bin\Debug\netcoreapp3.1\microsoft-spark-2.4.x-0.10.0.jar debug

DOTNET_ASSEMBLY_SEARCH_PATHS not set

System.Exception: 'JVM method execution failed: Nonstatic method awaitTermination failed for class 21 when called with no arguments'
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName, Object[] args)
at Microsoft.Spark.Interop.Ipc.JvmObjectReference.Invoke(String methodName, Object[] args)
at Microsoft.Spark.Sql.Streaming.StreamingQuery.AwaitTermination()
at WebAPI.Model.AWS.Program.RunSparkStream(String streamInputPath) in C:\Count4U\S3\SparkTest\StreamingSparkTest\src\StreamingSparkTest\Program.cs:line 75
at WebAPI.Model.AWS.Program.Main(String[] args) in C:\Count4U\S3\SparkTest\StreamingSparkTest\src\StreamingSparkTest\Program.cs:line 24

[suhsteve]

@parad74 what's the working directory where you're running the spark-submit from?

[imback82]

@parad74 In addition to what @suhsteve is asking, can you share the full log for your run? (there should have been error messages before the ones you pasted above).

[suhsteve]

@parad74 can you also try setting the DOTNET_ASSEMBLY_SEARCH_PATHS env variable to the directory that your UDF dlls are located (probably C:\Count4U\S3\SparkTest\StreamingSparkTest\src\StreamingSparkTest\bin\Debug\netcoreapp3.1)

https://github.com/dotnet/spark/blob/master/docs/deploy-worker-udf-binaries.md#1-question-when-i-run-a-spark-app-with-udfs-i-get-the-following-error-what-should-i-do