Using a Windows client together with a Linux cluster

If your Hadoop client runs on Windows and the Hadoop cluster runs on Linux then there are a couple of things you need to be aware of.

You need to:

• use a Hadoop version of 2.4.0 or later since that's when these issues were first fixed.

• set an extra configuration option of:

mapreduce.app-submission.cross-platform=true

• specify the Hadoop home directory that contains the winutils.exe program

  • If you don't have a local Hadoop install on Windows you can download winutils.exe and then:
    • create a Hadoop home directory
    • place winutils.exe in a bin directory under that Hadoop home directory
  • use a system property -Dhadoop.home.dir to point to the Hadoop home directory when you start the Java process. An example:
    java -D"hadoop.home.dir=C:\Users\<username>\Hadoop" -jar my.jar
  • you can use an environment variable named JAVA_OPTS if you are running Spring XD or a batch file generated by the Maven appassembler plugin. Here is an example:
    set JAVA_OPTS=-Dhadoop.home.dir=C:\Users\<username>\Hadoop