How to create a render farm?

Not sure if Render Farm is the proper term, but i have heard it before.

But I want to be able to connect a second computer (my iMac) to do the rendering for videos i'm editing on my laptop (macbook pro).

How does one properly connect a computer to another computer to do this, and secondly, how does one tell Compressor to render the project out on the slave computer?

I have no idea how to do this with any editing system or OS, and I want to learn how to do it on Premiere on a Windows computer for my work, but first, i want to practice this and learn how to do it for myself on my Mac's using FCPX

FCP doesn't use a render farm. If you encode using Compressor that can be set up in Compressor preferences to farm put frames to other machines with Conpressor installed on a network that's at least gigabit Ethernet.

Do you know of any tutorials or explanation guides on how to do this with Compressor? Do I need to just connect one computer to another using a gigabit ethernet cable? and just have the both turned on and have compressor installed on both? or do I need to connect both computers to a router? I understand the basic concept of connecting multiple computers and using it to help render, but it's the little details that i'm fuzzy on

You should be able to do it with direct connection. Open Compressor on both machines and set the preferences to allow the selected machines on the network to process material. The other machine has to be set to allow processing on it. It pretty straight forward when you look in Compressor prefs and both machines are connected.

Hi Brady,

Rendering is not the same as encoding. Rendering is handled directly within FCPX, but if you need to encode to different formats for delivery, you can use Compressor.

Here is a summary of Apple instructions…

"Distributed processing

How do I create a cluster or group of computers for processing?

What was previously referred to as a “cluster” is now called a “group” of computers in Compressor 4.1. To configure a group, choose Compressor > Preferences, then click Shared Computers. Click the Add button at the bottom of the left pane to create a new group, then click the Add button at the bottom of the right pane to add computers to the group. Each computer in the group must have the same version of Compressor installed. Each computer must also have “Allow other computers to process batches on my computer” turned on in the My Computer tab of Compressor preferences.

How do I select a group of computers to process a batch?

When you submit a batch, select a computer or group of computers from the Process On menu at the bottom of the Compressor window. This menu appears only after you create a group of computers.

How do I set up additional instances of Compressor to process batches?

Choose Compressor > Preferences, then click Advanced. Select “Enable additional Compressor instances,” then choose a number of instances from the pop-up menu. This number specifies the additional instances of Compressor, so if the number is 1, then the total number of Compressor instances is 2. Learn more about enabling additional Compressor instances.

What computer is the controller when I submit a batch to a group of computers?

The computer you submit the batch from is always the controller. If the submitting computer is turned off or logged out, processing is suspended."

Hope this helps, Greg

Well, I'm the same boat - I get the point: you want faster export of your files by combining more Macs and a lot of people would refer to that as a renderfarm.

As stated here, the way it works in Apples world is having Compressor app to combine several Mac computers for this number crunching.

It's a topic it seems very hard to find any real life experiences with on the net and after trying it for myself a couple of times, I end up thinking that it's not a great idea.

This is what happens normally:

1. Lets say I produce an hour long show. When exporting it for YouTube upload, I send it to compressor for h.264 encoding.

2. My Mac Pro will process it and spit out the file after 45-50 minutes

This is what Apples "renderfarm" can do:

1. I take the same one hour show and send to Compressor for processing / encoding.

2. This time I tell Compressor to use my new Mac Pro, my old Mac Pro, my iMac, my Mac Mini and my MacBook Pro Retina as a combined unit for processing

3. Compressor goes "Wow, 5 computers, let's split the file in 5 pieces and send it for crunching" so it sends a part of the show to each computer via our network.

Note: I'm not sure if the process in any way takes a look at the power of the different computers and handles it accordingly, so the fastest computers in the cluster gets the most work?

4. Then each computer does it's processing and sends back to encoded part, which Compressor then stitches together.

I would say that this process would BLAST away and be done in much less time, but the sending of files over network and stiching it all together again, seems to take just as much time......

I've yet to encounter any real time differences or time savings, which I guess is the answer to why you won't find many on the net talking about how they use this.

If it WAS such a great idea, I BET you would find professionals all over the place buying stacks of 10 MacMinis to build "render farms"

This is a Compressor function, initiated in Compressor, using Compressor applications on a network. It is not a function that can be accessed by other applucations. This does not apply to Send to Compressor or use Compressor settings.

"so it sends a part of the show to each computer via our network."

What's the network?

Ok, so i followed your instructions, but when I got to 'Enable additional Compressor instances', it's grayed out on both versions of Compressor 4.1.3 (running on my iMac and Macbook Pro). Don't know if that is going to cause an issue, because i'm still not exactly sure what that does.

I was, however, able to turn on Allow other computers to process batches on my computer on both computers, and i was able to go to Shared Computers and add a new group, and was able to see my other computer show up, and i put a check next to it as well. I did this on both iMac and Macbook Pro.

Then I imported a file into compressor, selected a setting for it, left the location set as 'source' (the file is on the desktop on my Macbook), and then chanced the 'Process on" option to my new group 'Untitled' that has both my iMac and Macbook pro on it. And then I click 'Start Batch'. This is where the problem occurs...


The file goes into the Active tab bar, and the right away goes to Completed, and it has an error on it. it says...

"Failed: Write permissions are missing for the folder "/Users/Shared/Library/Application Support"


What do I do from here?

I'm not sure if you got this sorted out but I made a video tutorial here:

bradywurtz wrote: ...But I want to be able to connect a second computer (my iMac) to do the rendering for videos i'm editing on my laptop (macbook pro)...

While the terms rendering and "render farm" are sometimes loosely applied to encoding, technically you render an effect but encode to a video stream or file.

The great video Jefforig posted shows how to use Compressor for batch encoding across several machines. This would be useful if transcoding to various distribution formats from an already existing video file. It doesn't apparently work if exporting or encoding from the FCP X timeline.

To my knowledge the fastest way to export is going to single-pass H.264 from a Mac equipped with a Sandy Bridge or later CPU (but not Xeon). These have Quick Sync which is an on-chip hardware encoder. It is typically 4x or 5x faster than software-only encoding.

If you haven't tried this, select Export->Master File->Settings, and pick:

Format: Computer
Video codec: Faster Encode
Resolution: 1920 x 1080 or 1280 x 720

Using this method in some cases a MacBook Pro can export video faster than an 8-core Mac Pro.

As best I can recall, it's never been possible to use distributed processing (including Quick Clusters in prior Compressor versions) when using FCP's Send to Compressor workflow,

Russ

Nope. Once was explained to me that Sand To causes FCP to send data for one frame at a time to Compressor, so that Compressor can analyze and do a higher quality compression job. I can see technically how this would prevent smooth distributed processing, which usually send sections of a video file (a group of frames) to each CPU that is available. In the case of Long-GOP processing, each CPU is processing a full GOP at a time, and Compressor is reassembling them. Thus it would need the full file all at once, not just one frame at a time. And, I know in legacy Compressor, in the much more detailed documentation about it, some codecs can't be distributed at all. Just thinking out loud...

BenB wrote: Nope. Once was explained to me that Sand To causes FCP to send data for one frame at a time to Compressor, so that Compressor can analyze and do a higher quality compression job...In the case of Long-GOP processing, each CPU is processing a full GOP at a time, and Compressor is reassembling them. Thus it would need the full file all at once, not just one frame at a time...some codecs can't be distributed at all...

A review of the academic literature shows parallel long-GOP video encoding is a hot research topic with lots of active work, but it's fundamentally difficult.

It would appear non-GOP codecs (e.g. ProRes 422) could be more easily distributed for parallel processing, since each frame is processed independently. However so much final product is long-GOP encoded (MPEG-2, MPEG-4, H.264, etc) that's what everyone is interested in.

The task can be distributed for parallel encoding at the macroblock level, frame level, GOP level, slice level or file level. There are tradeoffs with each one. It appears Compressor uses file-level parallelism which is easy and requires no synchronization.

For real-time encoding the more parallel approaches typically incur latency which would cause lag in the stream. For video editors we mainly care about encoding to a file so that's not an issue.

However depending on the algorithm, synchronization may be required for each GOP or group of GOPs. Due to Amdahl's Law this rapidly poisons the available speedup: en.wikipedia.org/wiki/Amdahl%27s_law

For Cluster Of Workstations (COW) parallel encoding, the synchronization is at network speed which is vastly slower than inter-core synchronization, so the slightest trace of serial code will cap the speedup.

When MPEG-2, MPEG-4/H.264 were invented, the primary goal was good compression and efficient decoding, not parallelization. With H.265 the design included more built-in elements that permit parallel processing. Maybe in the future this will help.

In a sense H.264 encoding is already parallelized -- all available cores in a CPU can be leveraged. For a 12-core Mac Pro that's 24 threads, which is a lot. That code already exists so competing GPU and COW encoding approaches must be compellingly better to warrant the development. Also each new Intel CPU is further enhanced with AVX instructions, etc ( en.wikipedia.org/wiki/Advanced_Vector_Extensions ) so this further raises the bar for alternate solutions. The 4Ghz i7-4790K in a new iMac is already 1000x faster than the original Cray-1 supercomputer (117 gigaflops vs about 100 megaflops).

For the Mac Pro people lamenting the lack of Quick Sync, there is still hope further research could leverage the GPUs for encoding. The literature describes experimental GPU-based video encoders which are pretty fast. However as CPUs get faster and more integrated, it's not sufficient for the GPU to beat a single core, it must beat a 12-core or higher CPU else it's faster to just use the CPU. In some cases research projects spent lots of time developing GPU-based encoding only to have 10% speedup over current multi-thread x264 encoders.

Re parallel COW-type video encoding of long GOP material, it seems like it should be possible. Each GOP on an H.264 timeline is independent, so if you slice the material on a GOP boundary it should be possible to distribute those to multiple workstations. You'd have to tag each slice then reassemble them, which would add some overhead but the slices could be made coarse enough to minimize that.

It's possible Apple just hasn't added that to FCP X because the development, test and support cost doesn't warrant the benefit. H.264 export time on a contemporary machine is already pretty fast, especially on a higher-end Mac Pro or a new iMac or MBP with Quick Sync. The file-based encoding Compressor uses is very easy from an implementation standpoint and a good fit if you have multiple distribution formats. Parallelizing a single export would likely be a big development job.