[BUG] java.lang.NoSuchMethodError when using synapse.ml.lightgbm

[PranavPavan]

SynapseML version

synapseml_2.12:1.0.4

System information

• Language version (e.g. python 3.8, scala 2.12):
• Spark Version (e.g. 3.4.1):
• Spark Platform ( Databricks):
• Databricks Runtime Version (13.3 LTS ML (includes Apache Spark 3.4.1, Scala 2.12))

Describe the problem

I have installed Synapse ML onto my databricks cluster from Maven using cooradinates com.microsoft.azure:synapseml_2.12:1.0.4 which should support spark 3.4.

In my notebook, I have imported the LightGBMRegressor using the code
from synapse.ml.lightgbm.LightGBMRegressor import LightGBMRegressor

But when I call the regressor in the notebook, it throws me an error message

Error Message:
Error encountered: An error occurred while calling None.com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.
: java.lang.NoSuchMethodError: org.apache.spark.ml.ComplexParamsWritable.$init$(Lorg/apache/spark/ml/ComplexParamsWritable;)V
at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.(LightGBMRegressor.scala:40)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397)
at py4j.Gateway.invoke(Gateway.java:257)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199)
at py4j.ClientServerConnection.run(ClientServerConnection.java:119)
at java.lang.Thread.run(Thread.java:750)

This is happening because of Spark version mismatch but SynapseML package I installed is expected to support Spark cluster has at least Spark 3.2 and Scala 2.12. This according to theinstallation page of SynapseML. https://microsoft.github.io/SynapseML/docs/Get%20Started/Install%20SynapseML/.

Kindly help me how to resolve it?

Code to reproduce issue

try:
from synapse.ml.lightgbm.LightGBMRegressor import LightGBMRegressor
lgbm_cv = LightGBMRegressor()
except Exception as e:
print(f"Error encountered: {e}")

Other info / logs

No response

What component(s) does this bug affect?

• area/cognitive: Cognitive project
• area/core: Core project
• area/deep-learning: DeepLearning project
• area/lightgbm: Lightgbm project
• area/opencv: Opencv project
• area/vw: VW project
• area/website: Website
• area/build: Project build system
• area/notebooks: Samples under notebooks folder
• area/docker: Docker usage
• area/models: models related issue

What language(s) does this bug affect?

• language/scala: Scala source code
• language/python: Pyspark APIs
• language/r: R APIs
• language/csharp: .NET APIs
• language/new: Proposals for new client languages

What integration(s) does this bug affect?

• integrations/synapse: Azure Synapse integrations
• integrations/azureml: Azure ML integrations
• integrations/databricks: Databricks integrations