Discuss Beholder potentially becoming an officially supported TensorBoard plugin

[chrisranderson]

To: @jart @dandelionmane @wchargin

I think @jart and @dandelionmane both mentioned potentially merging this repo into TensorBoard. Here are some of my questions.

1. Would you actually like to merge the repo? What if it was left separate, and I just waited for the plugin system to be changed so that installation was easier? A summary of your view of pros and cons for both parties would be nice.
2. What changes in the code would have to happen before a transfer?
3. What would the timeline be?
4. How will my influence be limited? How would I contribute afterwards? It's been fun to build, and it's a bit sad to me to consider not being able to work on it as much.
5. What will happen to this repo?

[teamdandelion]

Hi Chris,

First of all - nice work! @jart, @wchargin have poked around the repository and run the demo and we're all really impressed. The streaming image response with multipart response content type is really neat, none of us have seen that before.

To answer your questions:

1. Would you actually like to merge the repo? What if it was left separate, and I just waited for the plugin system to be changed so that installation was easier? A summary of your view of pros and cons for both parties would be nice.

Whether we want to merge the repo: we're not sure yet, because we aren't sure how broad interest in the plugin would be. It'd be helpful for us to show it to a bunch of researchers within Google and see if it excites them; if it does, that would be a strong indicator for including it.

Right now, the documentation is a bit sparse. Could you document the API, and create an example that shows how it is practically useful for TF users' workflows? A guided tour kind of like my Dev Summit tutorial would be sweet.

Here are my views of the pros and cons:

Merge: Pros

• Immediately available to the whole TF community; much wider audience
  • We don't have near-term plans to improve the plugin distribution story, so it might be hard to find otherwise
• TB team is on the hook for maintaining it
  • Maybe more overhead for @chrisranderson further improving it

Keep Separate: Pros

• More realistic long-term strategy for plugins, we can't merge everything
  • Eventually, we will want to improve plugin distribution and integration; in the short term, maybe we can sanction an official "TensorBoard-Plus" distro that includes core tensorboard as well as every plugin that looks good to us
• Clearer ownership for @chrisranderson

2. What changes in the code would have to happen before a transfer?

We'd need to review the code more thoroughly to say. The copy+pasted ffmpeg code might be an issue for synchronizing into Google (although you did a nice job keeping the licensing clear). We'd need to change the way Pillow is depended on. Overall, I don't expect drastic changes.

3. What would the timeline be?

Probably it would take us a week or two to decide, after you provided more documentation / a motivating demo.

4. How will my influence be limited? How would I contribute afterwards? It's been fun to build, and it's a bit sad to me to consider not being able to work on it as much.

After we merged the code, you would continue to be the owner of the plugin. We would do code reviews, but mostly defer to you on directions the plugin should take.

5. What will happen to this repo?

Probably it would be best to mothball it with a message at the top of the readme pointing to its new home in TensorBoard.

[chrisranderson]

Thanks! I'm glad you like it. :)

In regards to how it is practically useful, I started the why-is-this-even-useful section of my project report (part of my Master's requirements) today. Turns out it's kinda useful! Vanishing gradients could be pretty easy to detect, based on one contrived example I tried.

For CNNs, I reshape conv weights so that each row of [kernel height, kernel width] blocks represents an input channel and each column of blocks represents an output channel. Because of this, you can e.g. look at a glance to see if there is high variance among convolutional filters - in this untrained GAN, there is not:

In a pre-trained VGG network, there is more variance (though some redundancy might lead us to believe we don't need as many output channels for this layer):

It brings up interesting exploratory questions:

This one is from the same pre-trained VGG network... but why is there that one channel of a filter that is so much higher than the rest? Why is that input channel (row) so distinctive?

I also added color images to the frame option, which would have been nice when I was playing around with style transfer a while ago. Visualize your network with one option and switch to see what the image looks like so far (this version of style transfer optimized an image that was a variable) with another option, hit the record button to get a better idea of how things change over time.

There's still a lot of potential enhancements, but it's kinda fun to play with as is. I think I'll include some examples like this in the README. I also got a suggestion to include some presets (I'm going to have so many options I won't know where to put them all) for common issues. For example, to detect vanishing gradients you would want to see variance over time, and use the "all sections" image scaling option.

Pillow would be installed during pip install . after the repo is cloned. This allows people to use it outside of the plugin directory. I don't know much about how to package all this the right way. What would be better?

I haven't tested Mac or Windows.

At this point, I'd lean towards keeping it in this repo for a few weeks, and then passing it off sometime after that if I can (given interest, etc.), because I'll likely be very busy September through December trying to cram classes in to graduate a bit early.

I imagine researchers at Google will have all kinds of ideas on what they would like to see different; please pass those on!

[jart]

This is very useful information @chrisranderson. Thank you.

Regarding what @dandelionmane said earlier about a guided tour, one easy way you could do that on a Mac is by opening QuickTime Player and clicking File | New Screen Recording. You can then draw a rectangle on your screen, which it records along with your voice and mouse cursor. The ideal length would probably be five to ten minutes. Then upload it to YouTube.

I think this will be the most effective way to communicate to our researchers why they should feel enthusiastic about what your wrote. After all, Beholder is basically a video stream into TensorFlow. Explaining it with a video just feels like the right thing to do. But most importantly, a video would stand a better chance of convincing them to sit down and fiddle with Beholder for a few hours with their own models.

Regarding Pillow, have you considered using TensorFlow instead? Your _frame_generator function could be something like the following:

with tf.Session() as sess:
  x = tf.placeholder(tf.uint8, name='x')
  encode = tf.image.encode_png(x)
  while True:
    yield sess.run(encode, feed_dict={x: self._fetch_current_frame()})

Ignoring initialization, each sess.run should take a couple milliseconds. It also scales to multiple cores, across WSGI request threads.

[chrisranderson]

@jart @dandelionmane You're both right about a video. Maybe someday I'll work up the courage to put something out there... for now, I've added a bunch of info to the README. I could use that as an outline for a video.

@jart I have not thought about using TensorFlow, but I guess it isn't a bad idea to reduce the number of dependencies. I'll add an issue.

[chrisranderson]

@jart How goes feedback gathering?

[wchargin]

(Justine is OOO until Monday.)

We posted a link to your video and this issue in the Brain chat. There was a general positive response ("real time view of things would be really cool," etc.). One thing that someone noted was that network overhead could be an issue if data is on a remote filesystem—but our progress toward SQL-mode could help with this. That's all I have for you so far. :-)

[jart]

We discussed Beholder at our team meeting today. There is unanimous support for merging it into TensorBoard. It's a fantastic contribution. Everyone really loves it. There's only two small things we want to see happen beforehand:

1. Add a disclaimer to the web UI that it currently only works well on local file systems.
2. Make your is_active method work really well. Then have an informative inactive page that explains to the user how to set up beholder in their training script. Be sure to include the disclaimer there too.

Everything else can be done by iterating after the merge, like transient tensor delivery over the network. We're still not 100% sure how we want to approach it. But we don't want that to stop you from bringing your work to the community.

[chrisranderson]

That's great news! Thank you for getting back to me on this. It will be fun to share this when I present my project to the committee on Wednesday. I'll add issues for what you mentioned, add at least one of my own, and put them on a milestone.

[chrisranderson]

Here's the milestone: https://github.com/chrisranderson/beholder/milestone/1.

[chrisranderson]

I'm still (slowly) working on this. I almost have is_active and the inactive plugin page done. Development from me on Beholder has (and will be) slowed down dramatically, given that I have a job search and school to do.

@jart From the milestone, what do you think is necessary before handing everything over? How will that work? Will git history be preserved?

[chrisranderson]

It begins! Here is the pull request to start merging: tensorflow/tensorboard#613