Cloud Rendering: Project Upload [4/5] (Patreon)

Now that we've got a Linux virtual machine set up and configured on Azure, we can start the asset transfer process. Our goal is to collect all the geometry, textures and scene files required by our project, archive them, and upload them to Azure storage so that they can be accessed by our VM.

In order to create a robust and flexible cloud rendering system, it's important to automate everything as much as possible. We could upload all files manually by dragging and dropping them to a service like OneDrive or Dropbox, but that would be a major time sink and prone to error, not to mention extremely annoying! To avoid all that we can use scripts to make the process almost completely hands-off.

There are many ways to tackle automation and I encourage everyone to take my solution as a general concept only. You can use it as a model to develop a system that makes sense for your workflow.

Here's the outline of my process:

Make sure all project assets are stored in a clear folder structure.

Copy assets to a temporary staging area.

Convert paths from Windows to Linux format in the scene file(s).

Archive the entire project using 7-Zip.

Upload the archive to Azure with azcopy.

Let's break this down step by step. Since we're working on Windows I'll be using PowerShell to create our automation scripts.

For demo purposes I put together a simple scene in Gaffer using geometry in Alembic format, an EXR texture, and configured to render with Cycles.

Project Structure

This comes down to personal taste, but the overall idea is you want to have every part of your project stored in a strictly-organized, consistent folder structure. In my case I use a setup like this. Every project I create has the exact same structure, aside from its unique name.

root folder for all projects/└── geo/         └── project name/└── tex/         └── project name/└── scenes/          └── project name/

Variable Declaration

Because my folder structure is the same for all my projects, I can create a universal set of variables for PowerShell to use that will locate all current project assets. The script will take one parameter, the unique name of the project. Then, if we feed the script the project name "dragon," the $geo variable below, for example, will expand to c:/onedrive/main_projects/geo/dragon/.

We'll also set up destination staging folders in Windows default temp directory ($geo_dest, and similar), plus a variable pointing to the location of the Gaffer scene file. (The entire finished script is available to download at the end of this article.)

$root = "c:/onedrive/main_projects"$root_temp = $env:TEMP + "/bhgc/" + $project

$geo = $root + "/geo/" + $project + "/"$geo_dest = $root_temp + "/geo/" + $project + "/"

$tex = $root + "/tex/" + $project + "/"$tex_dest = $root_temp + "/tex/" + $project + "/"

$scene = $root + "/scenes/gaffer/patreon/" + $project + "/"$scene_dest = $root_temp + "/scenes/gaffer/patreon/" + $project$copied_scene = $scene_dest + "/" + $project + ".gfr"

Asset Staging

This step is optional, but I like to copy all assets to a temporary folder first. There are several benefits to this:

You never have to worry about something accidentally happening to the original files.

It makes it clear exactly what files will be uploaded to the server since they are all copied to a single location first.

It allows easy path conversion without affecting the originals.

Here are the commands to copy our assets to the staging area. The -Recurse flag tells PowerShell to copy all subfolders and files as well.

Copy-Item $geo $geo_dest -RecurseCopy-Item $tex $tex_dest -RecurseCopy-Item $scene $scene_dest -Recurse

Path Conversion

A critical step when working on Windows and rendering on Linux is to update any paths in our scene files and other assets to Linux-compatible paths.

For example, a Windows path such as C:\Users\Brian\Desktop might translate on Linux to /home/brian/desktop. We'll want to mirror our project's folder structure on Linux, replacing only the root of every path. In the case of our Gaffer scene, we'll want to convert C:\onedrive\main_projects\... to /home/brian/main_projects/...

Note that Windows paths use back slashes and Linux uses forward slashes. Converting the slashes in all paths could be a bit of a pain, but thankfully most 3D apps will interpret back slashes correctly on Linux. Be sure to check with whatever app you're using just to be sure. In fact, in the case of Gaffer it automatically stores Windows paths with forward slashes, so we won't run into any problems converting to Linux.

We can use PowerShell's built-in Get-Content...Replace function to update the contents of all our text-based scene files. It's highly recommended to use only ASCII scene file formats for projects like this. For example, Gaffer GFR, Maya MA, and Arnold ASS, are all human readable, plain-text formats. Using these type of scene files makes path conversion a breeze, plus it enables a lot of flexibility to make last-minute changes without needing to use a full DCC. Changing output image resolutions, sample counts, frame ranges, and pretty anything else you can think becomes a simple search and replace operation.

Here we open the scene file pointed to by $copied_scene and update all the paths contained within:

(Get-Content $copied_scene).Replace("C:/onedrive", "/home/brian") | Set-Content $copied_scene

Archive

Archiving the scene is another optional step. You could also upload the project in its raw state, file by file. The benefit of archiving is that the overall upload size will be less and there are fewer points of failure. However, uploading files individually is more robust in the case of network failure or other technical issue with the upload. If you've uploaded 499 out of 500 files and the upload fails at the last minute all you need to do is upload the last file. But if you're uploading a 100 GB archive and it fails at 99% then you'll need to start over.

That being said, that type of failure is rare, so my preferred method is to archive first. I use 7-Zip since I've been using it for a long time and it has a simple command line interface. Windows 10 also comes with tar for archiving, plus there is PowerShell's Compress-Archive tool. It's just a matter of preference and the syntax will be similar with all these tools.

Here's how we can archive our staged project files using 7-Zip. Note that we're archiving to ZIP (-tzip) here, not 7z, simply because the native unzip tool on Linux don't support 7z. Of course you could install 7-Zip on the VM if you wanted!

Still using our "dragon" project name parameter, the following commands will add all our staged project files to an archive named dragon.zip.

$7z = $root_temp + "/" + $project + ".zip"$7z_contents = $root_temp + "/*"

7z a -tzip $7z $7z_contents

Uploading

We've now got our entire project archived and ready to upload! There is where the azcopy utility comes in. Download the 64-bit version for Windows here, then extract it to a location of your choice. It's also a good idea to add it to Windows path. That way, all you have to do is type azcopy in any terminal session to run it.

We'll also need to have an Azure storage account set up and a Shared Access Signature token.

To create a new storage account, visit the Storage account section of the Azure dashboard and click the Create button. On the page that comes up the only thing you'll need to change is the Storage account name. Everything else can be left at the defaults.

Click Review + create, then confirm by clicking Create. Next, we can set up a folder (called a container in Azure lingo) in which to store our project uploads.

Finally, in order to use azcopy to access this container we'll need a Shared Access Signature. This is a security feature that lets only people with the SAS access the storage account.

Select the Shared access signature menu item, then check Container and click the Generate SAS and connection string button.

This will give us a bunch of data strings. The important ones to note are the SAS token and the URL of the storage account, usually something like <storage account name>.blob.windows.net.

With all that set up, we can now update our script with a variable pointing to the SAS token:

$token = "?sv=2022-11-02&ss=bfqt&srt=sco&sp=rwdla..."

Then, to upload our project archive, simply run an azcopy command that includes the URL of our container and the token. (If azcopy isn't working for you, jump ahead to Part 4.5 for information about how to configure it.) Give the archive time to upload and that's all there is to it!

azcopy copy $7z "https://bhgc.blob.core.windows.net/main-projects/$token"

Checking our Azure dashboard, we can see that the archive is now in our container. Perfect!

Conclusion

We're now ready to move on to the final step in this tutorial, downloading the assets to the VM, rendering and uploading the final frames. Before we do that though, let's quickly talk about some important tools for interacting with Azure via PowerShell.