Cloud Rendering: Virtual Machine Setup [3/5] (Patreon)

The Big Three cloud computing providers are Microsoft Azure, Amazon Web Services (AWS) and Google Cloud Platform. All three offer similar products, workflow and pricing. The choice of which to use mostly comes down to personal preference. For rendering we'll be using only very basic services: virtual machine instances, storage and networking. For advanced workflows and complex projects (machine learning, web app hosting) you might find differentiating features that make one provider a clear favorite, but that's not the case for our project.

The most important considerations when choosing a provider are cost, convenience, and operating system availability:

Cost

All three offer a full breakdown of prices on their websites. The main cost for us will be virtual machine (VM) time. There are also fees for storage usage, bandwidth and more, but these will amount to only pennies and will be a small fraction of our overall costs.

Convenience

Convenience is somewhat subjective, based on things like user interface, documentation, and customer support. I find Azure the simplest to use with excellent, clear documentation. Google Cloud comes in second place for me. I find AWS to be overly complex and confusing. If you're just starting out and don't have an existing account with any of these services, I recommend Azure.

All the topics and concepts we'll cover here will apply to any provider, with minor adaptations to match their workflows and terminology.

Operating Systems

An important consideration is what operating systems are available and if there is a fee to use them. Each provider has a marketplace stocked with virtual machine images that, in theory, allow you to quickly boot up a VM and get straight to work. Note that Windows VMs will always have a license fee in addition to the compute time used. Linux machines can vary. Some preconfigured VMs from third parties have a small fee, while other are free. For example, an Ubuntu VM configured with a GUI might cost money, while the raw operating system straight from Canonical will not.

We'll want a VM image that is ready to roll with a GUI and support for a remote desktop connection. Our rendering can all be done from the terminal, but for setup purposes it will be a huge help to have a GUI. Linux gurus can of course do it all from the command line if they want!

With the big picture understood, we can focus on what we're looking for on this project. Our requirements are as follows:

A wide range of affordable general compute VMs with a variety of vCPU counts and large amounts of RAM.

Spot pricing for extra savings. Spot instances (also called "preemptible") are ones that can be "evicted" under high demand. They're considerably cheaper and you might only get evicted once a day, if that. For rendering jobs it's generally fairly easy to resume where we left off, so eviction, while inconvenient, isn't a deal-breaker.

No- or low-fee Linux VMs with an operating system that supports the apps we'll be using. CG apps usually work best on Red Hat-based OSes, such as CentOS, Rocky or Alma. Ubuntu-based distros can also be a good choice depending on the app. We won't be picking distros which historically have little to no support for CG workflows, such as SUSE or Oracle Linux, for example.

Let's start working! Our goal here to set up a Linux VM with a GUI on Azure, log in to it via Remote Desktop (RDP), and install our required apps, including Gaffer and azcopy.

VM Setup

There are an overwhelming number of choices when it comes to setting up a VM on Azure (or any other provider). It's no exaggeration to say that 95% of the available options are irrelevant in our situation, so we're going to limit ourselves to the absolute bare minimum to get up and running.

There is a pre-configured version of CentOS 8.5 with a GUI in the Azure Marketplace that runs Gaffer without a problem, so we'll use that for this tutorial. If not using Azure, a good starting point would be to search for "linux gui" in your chosen provider's marketplace.

When just starting out you'll probably have to try a few distros to make sure they work with the apps you want to run. Linux is not like Windows; the operating system itself has to be specifically configured to allow some apps to run. For example, when researching this tutorial I tried a preconfigured Rocky 8 VM and Gaffer would not launch without a lot of troubleshooting. However, it runs out of the box on this CentOS one. That's Linux for you!

Here is a rundown of the settings I chose when setting up the initial VM.

Find a Linux VM with a GUI (CentOS 8.5 from Yaseen's Market Ltd in this case) and click Create:

On the following screen, I changed a number of parameters. If you have an existing Resource Group you can create the VM in it, or choose to create a new one. I have a Resource Group called linux, so I used that. You can give the VM any name you want.

Down further, make sure to check Run with Azure spot discount. For configuration purposes we can start with a very low vCPU-count VM since we just need to run a few simple setup tasks. Here I chose a 2 vCPU instance with 8GB of RAM which only costs 4 cents per hour. We can later resize the VM or create a new one from a template.

To browse all available sizes, compare prices and see eviction rates on spot instances, click on See all sizes.

Scrolling down, I choose Password instead of SSH public key. Set a username and password to use to log in to the VM:

At the bottom of the page, click the Disks button. I switched to a Standard SSD since it's a bit cheaper than Premium and disk performance is not a concern in our case:

Next, jump ahead to the Networking screen and enable RDP. RDP is Remote Desktop Protocol and it's how we'll be able to log into the GUI on our VM.

That's all we need to do! Click the Review + create button. Azure will validate the setup and then give a summary of the total expected cost. Click the Create button.

Azure will spend a minute or two setting up and starting the VM. After it's done you can view the details of the running instance. In particular we need to copy the IP address.

Logging in to the VM

Windows comes with a very basic remote desktop app that will be fine for our needs. Open the Start menu, type "remote" and select Remote Desktop Connection.

In the app, paste the VM's IP address and click Connect. Click Yes in the warning window that pops up.

In the VM login screen type the username and password we set up. After a moment we should find ourselves in the Gnome desktop environment.

The main features we need can be accessed by clicking the Activities menu in the top left: Firefox web browser, file manager and terminal.

Software Installation

For this project we'll just need Gaffer (1.4.0.0, gcc11 version) and azcopy (Linux x86-64). Download them using Firefox and copy them to a folder of your choice. In the file manager you can double click on the downloaded archive and it will automatically extract it.

You can add Gaffer and azcopy to the system Path via .bashrc so that they can be run from any location. To do so, run gedit ~/.bashrc from a terminal and add a line similar to this:

If we try to run Gaffer it returns an error about missing libGLU. This is a simple fix: just type sudo dnf install libGLU and confirm when asked.

(Bonus Linux explainer: dnf is the "package manager" for Red Hat-based distros. You can use it to easily install a vast array of tools and apps. sudo translates to super user do and means we are giving the system permission to make changes from our user account. The install instruction is self-explanatory and libGLU is the name of the package we're installing.)

Another helpful thing we can do before continuing is updating our root and user passwords. Azure requires a long password when setting up the VM and that can be a pain to type in over and over. To change passwords in Linux, open a terminal and type sudo passwd <username>, then type in a new password and confirm.

Finally, note that running the Gaffer GUI (or any 3D app GUI for that matter) doesn't quite work via Remote Desktop. That's not what this setup is for; creating an advanced remote connection with OpenGL support is beyond the scope of this tutorial. All 3D apps will work fine when it comes to rendering from the command line, which is the only thing we care about.

Finishing Up

We've got our basic VM for rendering configured now! The last thing to do is make sure to shut down the instance to avoid further charges.

Maintaining even a shut down VM accrues very small costs over time because of the data storage on the virtual hard drive. In the long term it's best to create a snapshot of our configured VM and delete the active one, but for now it's fine to leave it as is. Just be aware that you'll see charges for a few cents every day that the VM exists, powered up or not.

Next up, we'll explore how to collect and upload our project assets in Part 4!