Question on using a UDT as a struct field from another project

[barrettc]

I'm relatively new to Kotlin and Spark so apologies if this is a newbie question.

We have a Scala based Spark job that we'd like to covert to Kotlin for language uniformity. It uses a UDT called EmbeddedList that you can see here.

Here's a test example:

    val struct = StructType(Seq(
      StructField("name", StringType),
      StructField("out", EmbeddedListType)
    ))
    val df: DataFrame = sqlContext.read
      .format("org.apache.spark.orientdb.graphs")
      .option("dburl", dbUrl)
      .option("user", user)
      .option("password", password)
      .option("vertextype", "character")
      .option("query", "select name, out('appears_in').title from character")
      .schema(struct)
      .load()

My question is how can I go about defining this struct schema correctly in Kotlin with this UDT? I've tried the following:

    val struct = DataTypes.createStructType(
        arrayOf(
            DataTypes.createStructField(
                "name",
                DataTypes.StringType,
                false
            ),
            DataTypes.createStructField(
                "out",
                EmbeddedListType,
                false
            )
        )
    )

The compilation error is Classifier 'EmbeddedListType' does not have a companion object, and thus must be initialized here. Thanks for any hints about how to resolve.

[Jolanrensen]

Would you also like to convert the EmbeddedListType and EmbeddedList to Kotlin? I suspect the trouble comes from mixing Scala and Kotlin as those two languages tend to clash a bit.
You can create UDTs in Kotlin too, for example: https://github.com/Kotlin/kotlin-spark-api/blob/release/examples/src/main/kotlin/org/jetbrains/kotlinx/spark/examples/UdtRegistration.kt

[barrettc]

It's an option but one I was hoping to avoid. :)

I debugged this with a colleague and we found this workaround by getting a reference to the Scala singleton with reflection. A bit ugly but works at the moment.

        val c = Class.forName("org.apache.spark.orientdb.udts.EmbeddedListType$")
        val f = c.getField("MODULE$")
        val s = f.get(null) as DataType
        val struct = DataTypes.createStructType(
            arrayOf(
                DataTypes.createStructField(
                    "name",
                    DataTypes.StringType,
                    false
                ),
                DataTypes.createStructField(
                    "out",
                    s,
                    false
                )
            )
        )

Once I get the rest of the Scala job ported over to Kotlin, I can loop back and look at converting the UDT to Kotlin.

[Jolanrensen]

So, it all comes down to getting the instance of the Scala singleton from Kotlin, right?
Well, as you've discovered, Scala is a bit less friendly towards other JVM languages compared to Kotlin. Using MODULE$ is actually the official way to access singleton instances.

However, since we're using Kotlin, we can make it a tiny bit simpler, without using reflection, thanks to being able to backtick names:

val struct = DataTypes.createStructType(
    arrayOf(
        DataTypes.createStructField(
            "name",
            DataTypes.StringType,
            false
        ),
        DataTypes.createStructField(
            "out",
            `EmbeddedListType$`.`MODULE$`,
            false
        )
    )
)

[barrettc]

Ahh brilliant - thanks, that's exactly what I was hoping for!