Cloud Rendering: Rendering Final Frames [5/5] (Patreon)

The time has come to render our test project on the cloud! We've already made a PowerShell script to archive and upload the project, now we'll create a Bash script to download the archive to our VM, extract it, render it, and upload the finished frames.

Using the knowledge we've gained so far this will be a relatively painless process.

First, let's log into the VM GUI one last time to quickly write our Bash script. We can use the gedit app, a basic text editor similar to Notepad.

Here's the full script for rendering my Gaffer test scene. (All final scripts will be attached at the end of this article.)

We set up a few variables to start, including one for the Azure SAS token. Then we download the archive with azcopy and extract it using Linux's built-in unzip tool. This copies all our project files recursively to the correct folder structure, matching the updated paths we created in our scene file in Part 4.

The script takes two parameters: the project name and the number of frames to render. We then loop a Gaffer execute command over the number of frames and use azcopy to upload each rendered image. I also added a few lines to check for elapsed time and report back as each frame is finished. The \e[32m syntax changes the terminal text to green for optional extra fanciness, and \n creates a new line.

When the script is ready we can shut down the GUI. We'll also want to resize the VM from the 2 vCPUs we've been using during configuration to 96 vCPUs for fast rendering. I covered the syntax for this (and more) in the last article. On our local Windows machine, use these PowerShell commands:

$vm = Get-AzVM -ResourceGroupName "linux" -Name "render"$vm.HardwareProfile.VmSize = "Standard_D96as_v4"Update-AzVM -ResourceGroupName "linux" -VM $vm

Let's add a few extra lines to our original PS script (cloudrender.ps1 below) to start the VM and log into it via SSH. This automatically retrieves the public IP of the VM and prompts us with an SSH login. Simply type in the VM password to continue.

Start-AzVM -ResourceGroupName "linux" -Name "render"$vm_ip = Get-AzPublicIpAddress -Name render-ip -ResourceGroupName linux$vm_ip = $vm_ip.IpAddress

ssh brian@$vm_ip

We're now logged into the VM and can fully access every aspect of the system via command line. In this case all we have to do is type bash render.sh dragon 12 and it will execute the script we just created. Our local Windows SSH readout will show the progress of the job and a new frame will be uploaded as soon as it's finished.

We can keep tabs on the finished frames by either viewing our storage container in the Azure dashboard, or we can run our final frame download command from Windows:

azcopy copy "<storage account URL>/<container name>/render/$project/$token" "C:\Users\Brian\Desktop\render" --recursive --overwrite=ifSourceNewer

I wrapped that command up in a simple PowerShell script, clouddownload.ps1:

And that's all there is to it! By invoking just three scripts we archived, uploaded, downloaded, extracted, and rendered our project, then transferred the final frames to our local machine.

This job took around 25 minutes and cost only 30 cents, more or less. Working this way is by far the most cost-effective way to utilize cloud rendering.

Bonus hotness: since the rendering process is entirely controlled via SSH, you can execute it from your mobile phone too. Impress your friends! Just install an SSH app and run the script.

(Before finishing, don't forget to double check that the VM is shut down and not incurring any further charges.)

Conclusion

In this series we explored a simple but effective cloud rendering workflow that introduced every concept needed to create a more refined, advanced system adapted to your specific needs. Although this test scene was very basic, the setup should scale without any issues. As long as your DCC/renderer will process the scene then everything else will Just Work. I used Gaffer here, but you can use any app you like: Maya, Arnold, 3Delight, Blender, you name it...

Some features you might want to add:

Auto check the container for new frames and download them as they come in.

Use sshpass or a key pair to skip the SSH password input.

Upload the raw project files recursively and bypass the archive steps.

Check for existing finished frames and skip them in the event of a system crash or deallocation.

Create VMs from scratch via the terminal instead of relying on existing ones.

Split the job among multiple VMs for fast, parallel rendering.

Create a standalone job submission GUI with Python/Qt or C#, for example.

Create a job submission script for your DCC in Python.

Integrate these concepts into Deadline or other render manager.

In short, there is lot you can do to make this system even more powerful and flexible! These articles should get you off to the races and fully equipped to meet any additional challenges.

Remember that all cloud providers use very similar workflows and everything here should apply in some form to AWS or Google Cloud. The general concepts will be the same, only the terminology and scripting interfaces will be different. The azcopy tool exists only with Azure, obviously, so you would need to study the AWS/Google documentation to learn what the equivalent is for them. In theory it should be a more or less drop-in replacement in the scripts we've already created.

Good luck and have fun!