DreamWorks Animation Utilizes the Red Hat Portfolio to Build a Private Cloud for Financial and Creative Applications

Customer: DreamWorks Animation SKG

Industry: Entertainment

Geography: Glendale, California, and Redwood City, California

Business Challenge: Enable rapid and cost-efficient scalability of computing and resources for HPC (high-performance compute) workloads and optimal use of a multi-datacenter footprint. Enhance the reliability and availability of mission-critical services.

Solution:

• Deploy Red Hat Enterprise MRG as the grid infrastructure of DreamWorks Animation’s 30,000+ core render farm
• Deploy Red Hat Enterprise Virtualization as the platform to run mission-critical services on Red Hat Enterprise Linux

Migration Path:

• Red Hat Enterprise Linux running on HP ProLiant Blade servers
• VMware and Oracle VM to Red Hat Enterprise Linux on Red Hat Enterprise Virtualization
• Platform LSF to Red Hat Enterprise MRG (Messaging-Realtime-Grid)

Software: Red Hat Enterprise Linux, Red Hat Enterprise Virtualization, Red Hat Enterprise MRG, JBoss Enterprise Application Platform, Oracle Financials

Hardware: HP ProLiant Blade servers

Benefits: Rapid and efficient infrastructure scalability, reduced datacenter footprint and associated costs, enhanced reliability and availability

BACKGROUND

DreamWorks Animation SKG, Inc. today prides itself on being on the leading edge of both creative artistry and technology. The studio’s objective is to release two or three 3D animated feature films a year, each which carries its trademark qualities of combining compelling narratives that appeal across generations with striking computer-generated visuals. DWA is located in Glendale, California, and Redwood City, California.

OPEN SOURCE AT DREAMWORKS ANIMATION

As a company that consistently uses technology as one of its core competencies, DreamWorks Animation demonstrates the successful adoption of Red Hat products. A customer of Red Hat for more than a decade, DreamWorks Animation first deployed Red Hat Linux in its mission-critical render farm. Today, almost every DreamWorks Animation artist has Red Hat Enterprise Linux Desktop running on the latest HP workstation and nearly every DreamWorks Animation server is powered by Red Hat Enterprise Linux technology running on the latest HP servers. Lans Carstensen, Principal Engineer, DreamWorks Animation stated, “Red Hat software is a key enabling technology for our animation pipeline.”

In 2001, DreamWorks Animation adopted Red Hat Linux as the core operating system for its render farm, which today is comprised of thousands of servers used to compute and produce the final frames of DreamWorks Animation’s movies. Previously, the company had used proprietary hardware and software, until it became clear that commodity hardware powered by open source operating systems could give DreamWorks Animation the ability to scale to address the company’s growing creative ambitions. “We’ve been running Red Hat Enterprise Linux in our render farm since our move to x86 platforms,” Carstensen said. “The Red Hat distribution was stable and robust for us. We’ve worked closely with Red Hat over the years to make it even better as our use has evolved.”

Gradually, DreamWorks Animation began shifting other systems and applications over to Red Hat Enterprise Linux running on HP servers, from mission-critical rendering to corporate financial applications. “For any application we’re considering, one of the first questions is whether it runs on Red Hat Enterprise Linux,” said Carstensen.

VIRTUALIZATION CHALLENGE

Lately, those qualifying questions have been expanded to include “and can it be virtualized?” DreamWorks Animation has been using virtualization technology for many years to make the most of its hardware and software investments, and by 2009 possessed a significant number of virtualized servers. The high cost of those proprietary solutions, coupled with DreamWorks Animation’s preference for open technologies, made it relatively easy to adopt Red Hat Enterprise Virtualization technology as the preferred choice.

“We used those proprietary solutions when the primary goal was simply to consolidate servers,” said Carstensen. “Since that time, we re-assessed which virtualization platform would be best for our overall Linux server performance, scale and availability needs. We made the strategic decision to standardize on KVM [Kernel-based Virtual Machine] in Red Hat Enterprise Linux 5.4 and Red Hat Enterprise Virtualization.” Red Hat Enterprise Virtualization “has the best management capabilities for KVM, period,” said Carstensen.

Today, DreamWorks Animation uses Red Hat Enterprise Virtualization for its mission-critical service-oriented architecture (SOA) applications. For these critical business applications, the Company needed a virtualization platform that could support both high availability and live migration. “Our applications require 24×7 availability,” according to Carstensen.

GRID/CLOUD CHALLENGE

As it did in the case of Linux, DreamWorks Animation saw an additional opportunity to reduce its dependence on the proprietary and costly grid middleware it was using to run its substantial render farm. After an extensive search for a robust, scalable platform, it found its solution in Red Hat Enterprise MRG.

Today, DreamWorks Animation uses Red Hat Enterprise MRG, a next-generation infrastructure that incorporates messaging, real-time optimization, and grid functionality. “The render farm is a shared pool of resources that services all of the movies in production at any given time. It is instrumental in how we provide massively parallel computing resources and re-allocate those resources dynamically for our artists,” Carstensen said. “We run millions of jobs per week and MRG has done a fantastic job of scaling to our demand and providing the key features for our animation production pipeline.” More specifically, the grid middleware had to find the best resources for a job, and dispatch that job, all while maintaining very high utilization. “Between servers and workstations we have more than 4,000 systems, each of which has to operate at peak utilization 24x7x365,” said Carstensen.

Managing this utilization was becoming even more challenging as DreamWorks Animation increased its use of off-site private cloud facilities. Previously, DreamWorks had to run two separate server farms–one in Glendale, California, and one in Redwood City, California. DreamWorks Animation had also extended its HPC cloud rendering to datacenters in New Mexico and Las Vegas, where it could allocate or de-allocate compute and storage resources when needed. However, rebalancing supply and demand was manual and decreased utilization. “Today, we can ensure the agility demanded by the business, rendering across our geographies very fluidly. With Red Hat Enterprise MRG, we have the ability to dynamically direct workloads to the most optimal location without having to manually intervene,” said Carstensen. “We see cloud, whether infrastructure, platform, or software as a service, much the same way we saw the move to commodity hardware in 2000. It’s a way to provide cost-effective, scalable resources that we can directly align to real-time business needs.”

RED HAT SOLUTION BENEFITS

In addition to cost savings by migrating from its previous grid and virtualization platforms to Red Hat Enterprise MRG and Red Hat Enterprise Virtualization on Red Hat Enterprise Linux, Carstensen said, “We are very pleased with the technologies from a stability and performance perspective.”

DreamWorks Animation uses Red Hat Enterprise Virtualization technologies such as load balancing and live migration to keep its mission-critical applications available through planned maintenance.

With the implementation of Red Hat Enterprise MRG, DreamWorks Animation is able to maintain the performance and scalability required for its artists. Not only has the Company exceeded its utilization goals in load and stress testing, but DreamWorks Animation has also shown that Red Hat Enterprise MRG can easily scale to support the 30,000+ cores in-use and growing ambition.

“We’ve seen dramatic changes over many years in how we can provide resources to our artists by leveraging open source and partnering with Red Hat. Red Hat has impressed us with its outstanding engineering capabilities and commitment to our strategic alliance,” said Carstensen.

“We meet with Red Hat engineers on a regular basis to discuss key projects as well as initiatives we’re considering in the future,” he said. “That’s why they’re more than a vendor. That’s why their products continue to provide such significant value to our business.”