Float DataType no longer supported

[ElioMonaco]

Describe the bug
When attempting to write data from a Delta table with a float column to a a table in Azure Data Explorer using the spark kusto connector, the following error occurs:

(com.microsoft.azure.kusto.data.exceptions.DataServiceException) DataType "float" is unsupported for entities of kind "Column". Reason="Column: Name="WarrantySaving" uses unsupported data type, DataType="float""

To Reproduce
Steps to reproduce the behavior:
pyKusto = SparkSession.builder.appName("kustoPySpark").getOrCreate()
dataframe_input.write.
format("com.microsoft.kusto.spark.datasource").
option("kustoCluster",connection_input["kustoCluster"]).
option("kustoDatabase",connection_input["kustoDatabase"]).
option("kustoTable", connection_input["kustoTable"]).
option("kustoAadAppId",connection_input["kustoAadAppId"]).
option("kustoAadAppSecret",connection_input["kustoAadAppSecret"]).
option("kustoAadAuthorityID",connection_input["kustoAadAuthorityID"]).
mode("Append").
save()

Expected behavior
The data is appended to the Kusto table without any error

Additional context
Spark cluster
com.microsoft.azure.kusto:kusto-spark_3.0_2.12:3.1.15 on 13.1 (includes Apache Spark 3.4.0, Scala 2.12)
Kusto cluster
Standard_L8s_v3

[ag-ramachandran]

Hello @ElioMonaco , Couple of quick questions. can you do a printSchema and show the output of that please. Also, we had a similar issue where we mapped missing numeric types to real/decimal. Just to confirm if there is a fix already, would you be able to use say v5.2.7 the latest version to see if that works.

As a temporary work around, changing the type of the field to double on the dataframe and give it a try.

[ElioMonaco]

Hello @ag-ramachandran,
thank you for looking into this.

Here is the result of the printSchema:

root
|-- col01: string (nullable = true)
|-- col02: string (nullable = true)
|-- col03: integer (nullable = true)
|-- col04: string (nullable = true)
|-- col05: timestamp (nullable = true)
|-- col06: integer (nullable = true)
|-- col07: timestamp (nullable = true)
|-- col08: boolean (nullable = true)
|-- col09: string (nullable = true)
|-- col10: integer (nullable = true)
|-- col11: string (nullable = true)
|-- col12: boolean (nullable = true)
|-- col13: string (nullable = true)
|-- col14: string (nullable = true)
|-- col15: string (nullable = true)
|-- WarrantySaving: float (nullable = true)
|-- col18: float (nullable = true)
|-- col19: boolean (nullable = true)
|-- col20: string (nullable = true)
|-- col21: timestamp (nullable = true)
|-- col22: integer (nullable = true)
|-- col23: integer (nullable = true)
|-- col24: integer (nullable = true)
|-- col25: long (nullable = true)
|-- col26: double (nullable = true)
|-- col27: string (nullable = true)
|-- col28: timestamp (nullable = false)

I have tried casting all of the float columns to double, but the same exact error DataType "float" is unsupported for entities of kind "Column" when attempting to write.
I have also tried to use a different cluster with more recent spark version (15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)) where I installed the more recent version of the kusto connector (com.microsoft.azure.kusto:kusto-spark_3.0_2.12:5.2.7), but again the same error occurred.

The issue was eventually solved for me by changing the data type of the column in the Azure Data Explorer sink table itself from real to double.
Thank you,