Simplest example

[Volodymyr128]

Could you please provide with simplest working example (with JAR).

My own JARs (which work through standard Java API) failed. Job status contains null as message, so I can't figure out what's going wrong.

[ywilkof]

Hi @Volodymyr128,
I just validated that the examples for requesting job status in the README file work as expected, so you can use them. I validated now with Spark 2.0 and message field was not null and contained a detailed error. It sounds like there might be something problematic with your setup? If you give more details I might be able to help.

[mahmoudparsian]

Can you please share your script and error messages. Thank you, best regards, Mahmoud

…

On May 21, 2017, at 3:37 AM, Yonatan Wilkof ***@***.***> wrote: Hi @Volodymyr128 <https://github.com/volodymyr128>, I just validated that the examples for requesting job status in the README file work as expected, so you can use them. I validated now with Spark 2.0 and message field was not null and contained a detailed error. It sounds like there might be something problematic with your setup? If you give more details I might be able to help. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACdE0NwXC49yzf5J-bJl429OgWq6E0Faks5r8BPYgaJpZM4Nclko>.

[ywilkof]

Tested this with a cluster, which consists of one Master an one Executor, both local, run in Standalone mode.

In this example the Jar Spark should use does not exists.

          @Test
    public void submitAndRequestStatus() throws FailedSparkRequestException {
        SparkRestClient client = SparkRestClient.builder()
                .masterHost("ywilkof-XPS-13-9343")
                .sparkVersion("2.0.0")
                .build();

        final String submissionId = client.prepareJobSubmit()
                .appName("MySparkJob!")
                .appResource("file:/not/exist.jar")
                .mainClass("com.sparkjobs.Test")
                .submit();

        final DriverState driverState = client
                .checkJobStatus()
                .withSubmissionId(submissionId);

        System.out.println(driverState);

        JobStatusResponse jobStatus =
                client
                        .checkJobStatus()
                        .withSubmissionIdFullResponse(submissionId);

        System.out.println(jobStatus.getWorkerHostPort());
        System.out.println(jobStatus.getMessage());

    }

And following is the output:

ERROR
192.168.178.96:46367
Exception from the cluster:
java.nio.file.NoSuchFileException: /not/exist.jar
	sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
	sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
	java.nio.file.Files.copy(Files.java:1274)
	org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:607)
	org.apache.spark.util.Utils$.copyFile(Utils.scala:578)
	org.apache.spark.util.Utils$.doFetchFile(Utils.scala:663)
	org.apache.spark.util.Utils$.fetchFile(Utils.scala:462)
	org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:154)
	org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:83)

[mahmoudparsian]

Can you please include your whole program rather than just a method. Thanks, best regards, Mahmoud

…

On May 21, 2017, at 11:15 PM, Yonatan Wilkof ***@***.***> wrote: Tested this with a cluster, which consist of one Master an one executor, both local, run in Standalone mode. In this example the Jar Spark should use does not exists. @test public void submitAndRequestStatus() throws FailedSparkRequestException { SparkRestClient client = SparkRestClient.builder() .masterHost("ywilkof-XPS-13-9343") .sparkVersion("2.0.0") .build(); final String submissionId = client.prepareJobSubmit() .appName("MySparkJob!") .appResource("file:/not/exist.jar") .mainClass("com.sparkjobs.Test") .submit(); final DriverState driverState = client .checkJobStatus() .withSubmissionId(submissionId); System.out.println(driverState); JobStatusResponse jobStatus = client .checkJobStatus() .withSubmissionIdFullResponse(submissionId); System.out.println(jobStatus.getWorkerHostPort()); System.out.println(jobStatus.getMessage()); } And following is the output: ERROR 192.168.178.96:46367 Exception from the cluster: java.nio.file.NoSuchFileException: /not/exist.jar sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526) sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253) java.nio.file.Files.copy(Files.java:1274) org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:607) org.apache.spark.util.Utils$.copyFile(Utils.scala:578) org.apache.spark.util.Utils$.doFetchFile(Utils.scala:663) org.apache.spark.util.Utils$.fetchFile(Utils.scala:462) org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:154) org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:83) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACdE0D7mX9KSiIFR7Qera_t67gcwxGivks5r8SgYgaJpZM4Nclko>.

[ywilkof]

Hi @mahmoudparsian ,
I am not sure what a full Jar would supply that above example does not already provide. Are you interested in something specific? Is there some setup that does not function properly for you?

[NizarNi]

It doesn't work! I am trying to sumpit spark pi job to the cluster I always get :

this is workersHostPort: Optional.empty
this is job statusMsg: null

and the submitted job gonna be found in Running Drivers and without any worker