-
Notifications
You must be signed in to change notification settings - Fork 10
Pixar Tractor
A How To that is a Little Easier to understand than the Tractor Documentation (Hopefully)
Hello. Welcome to the how to about Tractor. When I first started working on this I found that the biggest headache that I had was that nobody had written anything down, and the documentation was a little hard to understand, and Tractor is not used by a huge group of people so searching on Google yielded little results. So after a lot of work I was able to piece together what on earth is going on. Now that I have I find that the documentation make a lot more sense. So instead of going through and giving a dumbed down description of how Tractor works I will try and explain some things that made the whole thing start to make more sense. And then you should go read the documentation and hopefully this will make it make more sense. So this will be a walk through of the process from Houdini to final render. I will mostly focus on all of the things that tripped me up along the way. Let's start in Houdini.
In Houdini we have a script that will send the renders to the farm; we will call it Houdini Script. (More about this script and how to add it to Houdini later.) So basically what happens is that this script will render the scenes out in a series of IFD files. IFDs are handy little files that know everything about the scene at a specific frame. So with just this file Mantra can render the frame outside of Houdini. There will be one for every frame in the rendered sequence. (NB that in order to work the script will need to have access to the mantra nodes. So if you, for example, have a bunch of mantra nodes in a subnet or digital asset in order to do all of the layers or whatever other reason, you will need to either enable editing of contents or just make the nodes editable in the Node tab of the Type Properties window for the digital asset.) So from here the script will send a command to the farm to render all of those IFDs. My big question was how on earth does the farm know what to do? So it turns out that you can call mantra from outside of Houdini by using a terminal command. This command accepts a path to an IFD file and then uses that IFD to determine how to render the image.
You can run Mantra in the terminal by going to /opt/hfsXX.X.XXX/bin/ in which folder you should be able to find and run mantra. ./mantra -h should give you a good start with the options and arguments. Additionally these two pages will give you a good idea about all of the option that are available. https://www.sidefx.com/docs/houdini/render/batch http://www.sidefx.com/docs/houdini12.0/rendering/commandline. In my experience I have found that that ./mantra -V a4 -f /groups/film/ifds/path/file_0001.ifd works pretty well. Breaking it down we have the -V option which controls the output for the terminal. The a argument for that option make the output compatible with Alfred. Why? I don't know but I left it in there because who ever was working on this before thought that was good so why not just leave it? The number controls the amount of output. 1 is like just a "Hey we are starting to render" and 9 is "Let me give you information for every single pixel we are rendering" After some trial and error I felt like 4 was a good level. It gives us some interesting information without flooding the log with info for every pixel. Then the last thing is the path to the IFD. NB The path to the output is defined as part of the IFD. Like I said it literally has everything you need which is simultaneously convenient and unnerving. You can override that output by adding the output path as another argument. I think you just tack it onto the end. At any rate with the IFDs and that command in hand you are ready to render all of the frames from outside of Houdini! Each of the blades on the farm should have Houdini and Mantra installed we can send that mantra command to the farm and it will be able to render the frame associated with each IFD. The Houdini Script should generate all of the IFDs you need and save them in the groups folder. Then it will send a command to the farm to render each of those IFDs.
NB It is extraordinarily important that all of the blades are running the same version of Houdini and Mantra as the lab computers. The renders could crash or be funky if they are not the same versions. You can check this by either asking the CSR's or using the render user to ssh into one of the blades and check the version. The version of Mantra should be the same as the version of Houdini so you just need to cd into /opt and see what version of Houdini is installed. Or if you want to be thorough you could do some ./mantra -h and that will tell you the version of Mantra.
So how do we send the command to the farm? Part of the Tractor is a python script that you run to send commands to the farm; we will call this script the Spool Script. Essentially you call the spool script and pass the terminal command you want to execute as the argument. This command could be anything from a simple ls or a mantra command. You can look up the exact syntax in the tractor documentation. It looks like it is slightly different for Tractor 2 than it is for Tractor 1 so make sure you get the right one. The Tractor has a user account to do all of it's work The user name is "render" This user will need to have permission to the groups folder for the film. If it doesn't go talk to the System Administrators and they can add "render" to the group. If you have everything setup correctly then the farm and all of the blades on it have access to the IFD so you don't need to send the files over you just need to send the path as part of the mantra command. All of the rest of the information will be stored in the IFD so with just that path the farm should have everything thing that it need in order to render. So you just need to call that Spool Script for each frame in the film! JK the Houdini Script will take call the Spool Script for each frame.
So what will happen from there is that the Spool script will send that command to the Tractor Engine, the Tractor Engine will then look at all of the Tractor Blades and find one that isn't busy and send the command to it. Then you can go into the Tractor Dashboard and you will be able to see all of the blades and jobs and processes and logs in there. To access the tractor dashboard go to renderfarm.cs.byu.edu or renderfarm/ animfarm.cs.byu.edu or 192.168.3.4 they should all be the same thing. It seems like you can only access the dashboard if you are on one of the lab machines.
You may have noticed that in the Dashboard there is no "Add Blade" button. So how do you add more machines to the farm? Good question. So the Tractor Engine is a server and will just wait for blades to join it. The Tractor documentation should walk you through that but I just wanted to tell you this because it was the source of much confusion for me to not understand how more blades are added. It seems a little backwards but I guess it makes sense.
- Blade - see Tractor Blade
- Engine - see Tractor Engine
- Farm - see Render Farm
- IFD - Instantaneous Frame Description (a file that has all of the information needed to render a single frame. Without anything else Mantra can take this file and render the scene. Based on my understanding of things these IFD's may contain references to geometry and textures so that it doesn't take up as much space. Or at least it should. You may have to enable that. But the Houdini documentation should explain how to do that alright and they even have some demo files that you can look at. On of the side effects of this however is that you may need to make sure that the Render Farm user has access to those referenced files. As long as everything is in the groups folder for the film I don't think there should be and issue with that. https://www.sidefx.com/docs/houdini/render/ifd_workflows
- Job - This is the top level process for the Render Farm to do. It can be broken down into tasks and sub-tasks which can down in parallel to each other. Typically we will have a job for each the shots and then a tasks for each of the layers and then sub tasks for each of the frames in those layers. Machine - see Tractor Blade Mantra - Houdini's Renderer
- Render Farm - A bunch of computers working together to render stuff. If you have 50 computers rendering 50 frames then you get done in the time it takes one computer to render one frame. Instead of one computer taking 50 times longer.
- Render User - this is the user that the farm uses to work. This user needs to have permissions to access all of the folders for the all of the assets. The username is "render" and you will have to talk to Seth Holladay to get the passwords. There are two users apparently. One is a local user for the blades and only works on the blades and the other is a network user
- Spool - Sending a job to the tractor.
- Task - See also Job. Its pretty much like a sub job. I think.
- Tractor - software made by Pixar to enable you to make and maintain a render farm.
- Tractor Blade - The remote execution server process on each render farm host. Each blade monitors its own host to determine when it is ready to accept new work, then it fetches a task definition from the central queue and launches the specified commands, tracking status and reporting results. [Basically the blades are the computers that actually do the rendering]. NB While blade technically refers to any computer that is working on the farm it also could refer to the computers that are in the Talmage
- Tractor Dashboard - The web-based user interface, providing status and control from any modern browser.
- Tractor Engine - The central job queue and task assigner. [It tells the blades what to do]
- Tractor Monitor - A subsystem of the engine that acts as a job status clearinghouse for connected user interfaces. [I still don't really know what this is or what it does]
NOTE: In it's current state the render script for Houdini will ignore the current frame option and only render the frame range specified
As of Nov 14, 2016 we are still using Tractor 1.7.2 even though Tractor 2 has been out since 2014. When you are looking into the documentation be sure you are reading the right version. You can find the current version of Tractor going to renderfarm/ then click on settings.