How To: Render .blend Files on the ResPower Super/Farm™

Overview

The ResPower Super/Farm™ proudly supports Blender rendering using the “Blender Internal” and “YAFRAY” render engines.  We also support the “Cycles” render engine starting with Blender 2.62.

This tutorial discusses how to use the ResPower Super/Farm™ for rendering .blend files created using Blender 2.46 through 2.62.

There are a few basic steps to take, which will be discussed in detail below:

  • Prepare your scene file
  • Subscribe
  • Upload your content
  • Submit your job for rendering
  • Monitor your job’s progress
  • Download the frames

 

Blender Community Involvement

ResPower seeks to be an active member in the community of Blender developers, testers, and debuggers. Part of this effort is our commitment to providing tutorials on how to use Blender productively. Here is a sampling of the tutorials we provide:

 

This is a list of patches ResPower has contributed to the Blender project:

  • Patch #3760 – Blender 2.41+ This patch allows a python script to call the renderer when the UI is not active. Without the patch, blender will segfault since G.vd is NULL.
  • Patch #4934 - Blender 2.42.x+ An enhancement to the Python API to allow setting the “texturesdir” variable from a Python Script. Extended by campbellbarton to support all path variables.
  • Patch #6258 - OdePhysicsEnvironment.cpp does not compile. A simple bug-fix.
  • Patch #6395 - Minor bug in source/blender/blenkernel/intern/material.c – unchecked return value led to Blender crash.
  • Currently “Open”
    • Patch #6161 - Blender Internals Tracing – This patch is what ResPower uses to provide you with detailed progress logs.
  • Rejected
    • Patch #6196 Small threading optimization. This patch removed two if statements in favor of a faster table indexing. Rejected on the grounds that the if statements are not broken.
    • Patch #4933 Progress in console. This patch shows how ResPower used to provide detailed progress logs. Rejected on the grounds that it was too verbose for typical user. Re-worked as Patch #6161, a Python extension that is completely silent by default.

 

And here is a list of bugs reported by ResPower:

 

Prepare Your Scene File

Configure Render Engine: To configure the scene for a specific renderer, click the current renderer and select from the list that pops up.
Specify Output File Type:Configure your scene to output a specific file type. Choose still formats only, e.g., JPEG, BMP, TGA, PNG, etc. Make a note of the type.
Pack Data: Pack your data before uploading to ensure textures and other files required by your scene can be located during a render. YafRay users: When we first started the service, we told you not to pack. Now we’re changing our minds – pack your textures!

Note:  Cycles Render Engine - If you are using Cycles materials from a library file, you will need to to use the Super/Cache feature on our farm.  See the note below on Fluids, Particles Cloth Simulations, etc. for how this works.

Note:  Camera Names – You will need to specify a camera name in the Render Wizard, so note the camera you wish to use for your scene.   If you use the “bind camera to marker” function to change cameras in your animation, then you will not need to specify a specific camera in this field.  At the moment, that field is still required, so you can type in junk text there and it will work just fine.

Note:  Scene Names – There is an option to specify a scene on the Render Wizard. If no scene is specified, the last one that was active when your scene was saved will be rendered. Please note that both the Scene name and Camera name are very sensitive. Any typos will cause the render to fail, so be careful.

Note on Fluids, Particles, Cloth Simulations, etc.Recent versions of Blender (2.46 at the time of this sections’ writing) use external data files to store the result of certain simulations. These data files are expected to be in the same location as the scene file. ResPower has created an option called Super/Cache which can place these files in the same location where the .blend file will be during a render. To use it, create a .zip file containing the Blender simulation files. Upload this .zip file with your .blend, and use the Render Wizard to mark the .zip file as a “file to cache.” Just prior to render, the Super/Farm will copy the .blend file and any files to cache to “c:\respower\temp” on each render node. It will unpack the .blend and unzip the .zip, thereby reproducing the proper directory layout.

Configure Render Engine – Select a render engine to use for your scene in the Scene options section inside Blender. ResPower currently supports the Blender Internal renderer and the YafRay renderer. If you are using YafRay, be sure to turn OFF the export-to-xml option. You want to use the plug-in version of YafRay, not the external version.

Select Ouput File Type – Double check that you have selected a still file type in the Format window of the Options tab in Blender. Quicktime .MOV and AVI file formats are not supported on the ResPower Super/Farm™

Use Relative Paths – If your scene contains Textures of type Image, make sure that your image files are located using relative paths. When loading an image for a texture, there is a button that says “Relative Paths.” Make sure it is selected because using relative paths is the only way to ensure your image-based textures will render properly.

Turn on Threads – Turning on Threads in the render options of your Blender scene ensures that Blender will take full advantage of our dual processor render nodes. This results in speedups of as much as 20-30% per frame and ensures that you get the maximum processing power possible with your subscription.

Pack Data – Blender Internal Only! Be sure to use the Pack Data option under File in the main menu. Packing your data into the .blend file ensures that your textures, images, and other files necessary to render properly can be found by your scene. YafRay users: When we first started the service, we told you not to pack. Now we’re changing our minds – pack your textures!

Turn Extensions On – Extensions are turned on by default in Blender, but if you regularly turn them off for some reason, turn them back on before uploading your scene. Without extensions, the names of your finished frames will be incorrect and if you submit any split frame jobs, the restitcher won’t work.

Save Your Scene – This probably seems obvious, but you must save the changes to your scene file prior to uploading it for the render to reflect your design. We recommend always appending a version number, and increasing that number with each save, e.g., MySceneV01.blend, MySceneV02.blend, etc.

Sign Up

Create a ResPower User Name – You must have an account with the ResPower Super/Farm™ before you can upload any content. Sign up here.

Blender users may use the Metered Service, or purchase a subscription. You can add a subscription to the ResPower Super/Blender Rendering Service from your account status page. There are several subscription levels available with varying priorities, and frame time limits. While the Metered Service is available, Blender users will find that subscriptions are considerablyless expensive than even small jobs with the Metered Service. We suggest only using the Metered Service if you have exceptionally long render times.

Upload Your Content

The server name iswww.respower.com, and the user name/password is the same as for the web site.
FileZilla – to upload content, simply drag from the file list on the left into the “source” folder on the right.

Create a ResPower User Name – You must have an account with the ResPower Super/Farm before you can upload any content. Sign up here.

Open your FTP client – Although ResPower is officially FTP-client and platform neutral, we will focus on using FileZilla in this HOWTO. FileZilla is free, fast, and reliable – all Good Things. If you are using another FTP client, the concepts are the same, but the screens will look different. Here is an incomplete list of available FTP clients:

Please note that while using Internet Explorer as an FTP client is possible, it is not recommended. Many users experience unreliable connections and inexplicable behavior when using IE. Because of this, ResPower cannot provide technical assistance under these circumstances. If you have problems connecting with IE, please use one of the fine FTP clients listed above.

 

Connect via FTP – The user name and password are the same as the one you use to log in to the web site. The server name is www.respower.com. The site profile looks like the one shown in the diagram; you can download a FileZilla profile here and import it by selecting File->Import from the site manager menu. To connect, select the ResPower connection and click the “Connect” button.

Upload your content – This is simply a matter of finding your content in the “local” file system – the one on your computer – and dragging it over to the “remote” file system, the one at ResPower. ResPower recommends that you pack your .blend files before upload with both Blender Internal and Yafray. This keeps all the necessary files in one central location and ensures your scene will have no problems finding the resources it needs to render properly. If you choose not to pack your .blend file, you must mimic the ResPower file system when creating your content and upload your textures separately when you are ready to render.

Submit your job for rendering

Click on “Render Wizard”.
Fill in the Render Wizard.

Render Wizard™ – Log in to the ResPower web site using the Login panel at the top left of the page here. From there, choose the “Render Wizard” link.

Fill in the Render Wizard – Required options show up in a pale red. Camera names are sensitive and if the wrong name is entered your scene will fail. ResPower recommends copying the name of the camera you wish to use directly from you scene file and pasting it into this field.

Split-Frame Rendering - Basically, this takes a single frame and divides it into a grid. Each cell in the grid, or Bucket, is then sent to a separate computer where it is rendered. This is not supported for animations; use it only to render out stills. At this time, automated re-assembly of the final image is in Beta. In the event it does not work properly, you will need to stitch the frame back together using photo-editing software like Photoshop.

To specify the way in which the scene is split, please set the “Rows” and “Columns” values to the number of rows and columns that the farm should split the render into. You should set the pixel overlap to its maximum value in order to avoid the possibility of any artifacts around the edge of each bucket.

The start/end row and column fields are advanced settings that should normally be left blank. These values will reduce the number of buckets rendered to the rectangle that includes only the rows and columns you specify. For example, if you specify a 5×5 split, and set both the start and end column to 2, then the center column of buckets will be rendered.

Monitor your job’s progress

Once your job has been submitted, you can monitor its progress and perform special actions using the Jobs page.

To open the jobs page, click the “Jobs” link at the top of any Super/Farm page. Each row contains a job that you have submitted. Jobs include rendering submissions, frames, and certain other commands, notably: unpack, rar, and zip.

The table has several columns:

  • Job #
    This is a unique number identifying your job. These numbers increase sequentially – at the time of this writing, there were 9,438,784 jobs.
  • User
    This should reflect your user name. If you have turned on “Show Other Users”, this column and the “Description” column will be merged for jobs belonging to other users and will contain the phrase “Other User”
  • Description
    This is the description that you provided on the Render Wizard, or one generated during the submission of other types of commands.
  • Status
    This set of columns tells you the current state of your job. If the job is a render, it will be divided further into the columns listed below. Each column will contain a number indicating the number of sub-jobs (frames, buckets, etc) that are currently in that particular state. For example, if you submit a render with 1000 frames, the “waiting” column will have 1000 in it until computers start running those frames. 

    • waiting
      Jobs in this column are waiting for computers to process them.
    • running
      Jobs in this column are being processed by computers.
    • aborted
      Jobs in this column are aborted.
    • dead
      Jobs in this column have failed multiple times, and the Super/Farm has decided not to try any further.
    • complete
      Jobs in this column have finished running.
    • total
      This is the total number of sub-jobs.

    You can drill down to see details on your job by clicking the number of sub-jobs in a particular column. For example, if you have submitted a 1000 frame job, and 5 frames have died, you can click on the “5″ in the “dead” column to get a list of the dead frames.

  • Credits
    This is the number of credits used by a job. Credits are calculated with a formula that can be found on the “Purchase Discounted Credits” page. For subscription users, this value is essentially irrelevant. All costs associated with running a subscription job are included in the price of the subscription.
  • Actions
    This column contains a “Go” button and a drop-down list of commands you can perform on a job. Certain commands only show up in certain situations. For example, if your job has no waiting or running sub-jobs, the “abort” command does not appear. 

    • View Log
      Brings you to the job’s log page. If the job has sub-jobs (e.g., frames or buckets), you can get to their logs using the table at the bottom of the log page.
    • What’s Wrong
      Generates a customized report to explain why your frames have died. Many error messages have links explaining how to work around them.
    • Restart Frames [Aborted/Dead]
      This command takes all frames in the Aborted or Dead columns and moves them back to the waiting column.
    • Abort
      This command aborts all sub-jobs, or if the job itself is processing, aborts the job. The farm charges you through the time when you abort.
    • Zip Output
      Causes a job to be submitted to the farm, which generates a .zip archive of the output folder. It is recommended that you use the “Rar Output” action instead.
    • Rar Output
      Causes a job to be submitted to the farm, which generates a series of .rar archives of the output folder. Each archive is 25 MB, so if your FTP client stalls or you lose your connection, you do not have to start at the beginning. Furthermore, Rar gives higher compression ratios than .zip.
    • Submit Copy
      Submits a copy of the job. Please note that if the job has sub-jobs, they will be copied as well.

 

Download the frames

Use your FTP client to retrieve the frames. They will be in the /renders/<job-number> folder, and named “frame_<number>.<ext>” You can compress your files prior to downloading using the “Zip Output” or “RAR Output” commands; ResPower recommends using the “RAR Output” option, since it will create a multi-part archive, with each part set to 25 MB.

Known Issues

•Output folder does not appear on FTP site
•Output folder does not contain all frames on FTP site

There are two possible causes for this:

  • The job does not have any complete frames/buckets. If frames are not complete, there is nothing for the Super/Farm to place into the Output Folder. The Output Folder is only created once at least one frame is complete.
  • Your FTP client is caching the directory listing. Many FTP clients try to save time by storing a local copy, or cache, of each FTP server’s directory listings. Because the ResPower Super/Farm™ is constantly placing files into your FTP directory, this cache can very quickly become out-of-date, which will make your client show you listings that are inaccurate. To solve this, use your client’s “Refresh” option to re-download the directory listing. For FileZilla, the hotkey for Refresh is F5.

  • •Cannot delete or overwrite files on FTP
    •”Permission denied” errors uploading to FTP

    You are not allowed to delete files from your /renders folder; these are automatically cleared out by our automated diskspace recovery tools.

    If you are trying to delete files or folders from your /source folder, or are attempting to upload into a subfolder of your /source folder, there are several likely culprits:

  • The server thinks they aren’t your files. When you unzip files, they show up on the server as belonging to a different user from you. To fix this, use the “Take Access” link available on the web site once you have logged in
  • Your FTP client is not descending into the directory. One of the reasons ResPower recommends FileZilla is that it will descend into directories to delete them.
  • Your FTP client is not deleting hidden files. One of the downsides of FileZilla is that it does not list hidden files by default, and so even if it descends into a directory, it could miss those files, preventing FileZilla from deleting the directory itself. To fix this, we recommend changing two settings. First, choose Edit->Settings from the menu. Then select Connection-> Directory Cache in the tree on the left. Make sure that “Don’t Use Cache” is checked. Select Interface Settings->Remote File List from the tree on the left. Make sure that Always show hidden files is checked. Once you have made these configuration changes, and used the “Take Access” button, you should be able to delete files in your /source/ directory at will.

  • • Gamma causes flicker

    We have received reports that the “Gamma” button on the render buttons window can cause a slight flicker when used with a render farm. The workaround is to keep this setting turned off and apply Gamma as a post-process.


    • YAFRAY does not need “xml export” turned on

    New w/ 2.41! You don’t have to turn on xml export for YAFRAY. There are apparently bugs with the xml exporter (meshes sometimes become corrupt during export) so we recommend that you not use XML export.


    Limited support for fluid simulations

    Fluids are somewhat supported – they work, but you are likely to need to play around a little bit (a few 5-10 frame tests) to get them going the way you want. You cannot pack the fluid cache into your .blend file, and you cannot generate the fluid cache on this side. We looked into supporting server-side simulation of fluids, including a days-long email conversation with Blender’s fluid code writer, Nils Thuerey. The bottom line is that you cannot simulate fluids on this side, but you can render them.

    Because you cannot pack the fluid cache into your .blend file, the normal workflow of “use relative paths, pack and upload” doesn’t work. Relative paths won’t work unless they are for a “packed file,” because your file is unpacked into a location separate from your FTP folder. Because fluid cache files are not packed inside the .blend file, Blender cannot find them with relative paths.

    For fluids, you therefore have to use a fully-qualified path that matches our file system.

    For example, if your username is super_blender, and you FTP your fluid files into “/source/fluids/”, you need to point your Blender fluid domain at “y:\super_blender\source\fluids\” Note that you must use windows-style “back” slashes, and do not place anything in front of the “y:”.

    One Response to How To: Render .blend Files on the ResPower Super/Farm™

    1. earlye says:

      We have not, yet. I’m not sure precisely when. It looks like we’re a few versions out of date (2.64a is our current latest) – we’ll have to take some time and get the latest installed.