breeze-viz gives HeadlessException under Databricks

[sujitpal]

I tried importing breeze-viz-2.10-0.11.2 into a Databricks environment. Here is a code snippet I used to test basic operations.

import breeze.linalg._
import breeze.plot._

val f = Figure()
val p = f.subplot(0)
val x = linspace(0.0,1.0)
p += plot(x, x :^ 2.0)
p += plot(x, x :^ 3.0, '.')
p.xlabel = "x axis"
p.ylabel = "y axis"
f.saveas("lines.png")

It gives me the following exception:

java.awt.HeadlessException: 
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
    at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:207)
    at java.awt.Window.<init>(Window.java:535)
    at java.awt.Frame.<init>(Frame.java:420)
    at javax.swing.JFrame.<init>(JFrame.java:218)
    at breeze.plot.Figure.breeze$plot$Figure$$frame$lzycompute(Figure.scala:88)
    at breeze.plot.Figure.breeze$plot$Figure$$frame(Figure.scala:87)
    at breeze.plot.Figure.refresh(Figure.scala:132)
    at breeze.plot.Figure$$anon$1.refresh(Figure.scala:181)
    at breeze.plot.Plot$$anonfun$refresh$3.apply(Plot.scala:51)
    at breeze.plot.Plot$$anonfun$refresh$3.apply(Plot.scala:51)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at breeze.plot.Plot.refresh(Plot.scala:51)
    at breeze.plot.Plot.$plus$eq(Plot.scala:33)

I read some Stack Overflow posts advising to explicitly set java.awt.headless to true, but it doesn't seem to have any effect. Also java.awt.GraphicsEnvironment.isHeadless() returns true.

My environment is Databricks version 2.0.1 running Scala 2.10.4 and Oracle Java 1.7.0_65.

Is breeze-viz suitable for use in a notebook environment such as Databricks? It would be nice to use with Scala notebooks (similar to matplotlib with Python notebooks).

[dlwh]

set f.visible = false before doing anything else

On Mon, Aug 24, 2015 at 11:54 AM, Sujit Pal [email protected]
wrote:

I tried importing breeze-viz-2.10-0.11.2 into a Databricks environment.
Here is a code snippet I used to test basic operations.

import breeze.linalg.import breeze.plot.
val f = Figure()val p = f.subplot(0)val x = linspace(0.0,1.0)
p += plot(x, x :^ 2.0)
p += plot(x, x :^ 3.0, '.')
p.xlabel = "x axis"
p.ylabel = "y axis"
f.saveas("lines.png")

It gives me the following exception:

java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:207)
at java.awt.Window.(Window.java:535)
at java.awt.Frame.(Frame.java:420)
at javax.swing.JFrame.(JFrame.java:218)
at breeze.plot.Figure.breeze$plot$Figure$$frame$lzycompute(Figure.scala:88)
at breeze.plot.Figure.breeze$plot$Figure$$frame(Figure.scala:87)
at breeze.plot.Figure.refresh(Figure.scala:132)
at breeze.plot.Figure$$anon$1.refresh(Figure.scala:181)
at breeze.plot.Plot$$anonfun$refresh$3.apply(Plot.scala:51)
at breeze.plot.Plot$$anonfun$refresh$3.apply(Plot.scala:51)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at breeze.plot.Plot.refresh(Plot.scala:51)
at breeze.plot.Plot.$plus$eq(Plot.scala:33)

I read some Stack Overflow posts advising to explicitly set
java.awt.headless to true, but it doesn't seem to have any effect. Also
java.awt.GraphicsEnvironment.isHeadless() returns true.

My environment is Databricks version 2.0.1 running Scala 2.10.4 and Oracle
Java 1.7.0_65.

Is breeze-viz suitable for use in a notebook environment such as
Databricks? It would be nice to use with Scala notebooks (similar to
matplotlib with Python notebooks).

—
Reply to this email directly or view it on GitHub
#9.

[sujitpal]

Thanks David. I did as you suggested, adding your suggestion after the Figure instantiation, ie:

val f = Figure()
f.visible = false
...

The stack trace is a bit different now, looks like it now fails the headless check at the visible setting line (ie the line you suggested).

java.awt.HeadlessException: 
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
    at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:207)
    at java.awt.Window.<init>(Window.java:535)
    at java.awt.Frame.<init>(Frame.java:420)
    at javax.swing.JFrame.<init>(JFrame.java:218)
    at breeze.plot.Figure.breeze$plot$Figure$$frame$lzycompute(Figure.scala:88)
    at breeze.plot.Figure.breeze$plot$Figure$$frame(Figure.scala:87)
    at breeze.plot.Figure.visible_$eq(Figure.scala:74)

[dlwh]

I guess it doesn't work :(

It would be good to be fixed, but i don't have bandwidth to work on viz
right now.

On Mon, Aug 24, 2015 at 1:56 PM, Sujit Pal [email protected] wrote:

Thanks David. I did as you suggested, adding your suggestion after the
Figure instantiation, ie:

val f = Figure()
f.visible = false
...

The stack trace is a bit different now, looks like it now fails the
headless check at the visible setting line (ie the line you suggested).

java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:207)
at java.awt.Window.(Window.java:535)
at java.awt.Frame.(Frame.java:420)
at javax.swing.JFrame.(JFrame.java:218)
at breeze.plot.Figure.breeze$plot$Figure$$frame$lzycompute(Figure.scala:88)
at breeze.plot.Figure.breeze$plot$Figure$$frame(Figure.scala:87)
at breeze.plot.Figure.visible_$eq(Figure.scala:74)

—
Reply to this email directly or view it on GitHub
#9 (comment).

[sujitpal]

Yes, I guess I will go back to doing what I am doing now. Either use the somewhat anemic visualization built into Databricks around DataFrames, or save the output data and build a separate Python notebook to visualize. Not ideal, but works. I can close the issue if you want unless this serves as a reminder to come back to it when you have some time?

[dlwh]

yeah, let's leave it open.

I have hopes someone will come and fork it and be done with it.

On Mon, Aug 24, 2015 at 3:56 PM, Sujit Pal [email protected] wrote:

Yes, I guess I will go back to doing what I am doing now. Either use the
somewhat anemic visualization built into Databricks around DataFrames, or
save the output data and build a separate Python notebook to visualize. Not
ideal, but works. I can close the issue if you want unless this serves as a
reminder to come back to it when you have some time?

—
Reply to this email directly or view it on GitHub
#9 (comment).

[praveen-srinivasan]

I got this working (you don't need to change anything in Breeze).

The result of the code below is an HTML snippet that can be embedded in a webpage to show a plot image inline. Credit goes to https://github.com/wookietreiber/scala-chart for helping me learn how to do it (they have a good library for Scala vizualization as well, also using JFreeChart).

The key is to make sure you don't use Figure from Breeze since that tries to display stuff.

import breeze.plot._
import breeze.linalg._
val x = linspace(0.0,1.0,101)
val p = new Plot()
p +=plot(x, x :^ 2.0)
p.xlabel = "x axis"
p.ylabel = "y axis"
val img = p.chart.createBufferedImage(600,600,java.awt.image.BufferedImage.TYPE_INT_ARGB,null )
import org.jfree.chart.encoders.EncoderUtil
val imgDisplaySz=(600,600)

import org.apache.commons.codec.binary.Base64

val imageDataString =  Base64.encodeBase64String(EncoderUtil.encode(img, "png"))
val imgHeight = imgDisplaySz._1
val imgWidth = imgDisplaySz._2
val imgTag = s"""<img height="$imgHeight" width="$imgWidth" src="data:image/png;base64,$imageDataString" />"""

[sujitpal]

@praveen-srinivasan: Did you get this working on Databricks? If so, can you please share the Databricks version and if you had to do anything special with the environment, such as importing extra libraries, etc? I ask because I tried with and without importing the breeze-viz libraries and I get back an error that I saw long back which indicates that there is some classpath conflicts between the breeze built in with Spark and the one I added.

[praveen-srinivasan]

Not on databricks - didn't try there.
On Feb 18, 2016 3:10 PM, "Sujit Pal" [email protected] wrote:

@praveen-srinivasan https://github.com/praveen-srinivasan: Did you get
this working on Databricks? If so, can you please share the Databricks
version and if you had to do anything special with the environment, such as
importing extra libraries, etc? I ask because I tried with and without
importing the breeze-viz libraries and I get back an error that I saw long
back which indicates that there is some classpath conflicts between the
breeze built in with Spark and the one I added.

—
Reply to this email directly or view it on GitHub
#9 (comment).

[sarariazi]

I used X virtual framebuffer and it works perfectly.
Xvfb :1 &
export DISPLAY=:1
let java.awt.Headless remains false and set f.visible = false

[devpog]

$ sudo vi /etc/init.d/Xvfb_go 

#! /bin/sh
# /etc/init.d/Xvfb_go

echo "This is Xvfb's starting script"

case "$1" in
  start)
    echo "Starting Xvfb_go"
    Xvfb :1 &
    echo "Xvfb_go has started..."
    ;;
  stop)
    echo "Stopping Xvfb_go"
    kill -9 `ps -ef | grep -i xvfb | grep -v grep | awk '{print $2}'`
    echo "Xvfb_go has stopped..."
    ;;
  *)
    echo "Usage: /etc/init.d/Xvfb_go {start|stop}"
    exit 1
    ;;
esac

exit 0

$ sudo chmod 755 /etc/init.d/Xvfb_go
$ sudo update-rc.d Xvfb_go defaults

After that it is easy to:

/etc/init.d/Xvfb_go start

and...

/etc/init.d/Xvfb_go stop

[MichaelPolla]

I'm running into the same issue in an Apache Zeppelin notebook.
@praveen-srinivasan did you ever get this to work ?

[rockreel]

If you want to save plot as an image file, I use the following code

val width = ...
val height = ...
val file = new java.io.File("plot.png")
val out = new FileOutputStream(file)
val plot = new Plot()
// draw plot
// ...
def drawPlot(g2d : Graphics2D) {
  plot.chart.draw(g2d, new java.awt.Rectangle(0, 0, width, height))   
}
ExportGraphics.writePNG(
  out,
  draw = drawPlot,
  width = width,
  height =  height,
  dpi = dpi)
}

with this code, you don't need to create Figure, and don't need to mess up with system display environment.

[kongkongk]

@zhoulu312 It works!