The film production process is a constantly evolving one as technologies emerge enabling new types of creativity. Today, the use of digital technology is becoming pervasive in filmmaking. Its adoption has been marked by significant tipping points, one of the first of which was the introduction of non-linear editing (NLE) systems in the late 80s. All of a sudden the craft of the editor changed radically. Editors could experiment more and be far more creative. Because their edits became virtual they could change them at any time; and so today editors begin editing from the first storyboards and keep on editing until the film is ready to print for distribution ?± sometimes even afterwards. The digital edit became the first virtual representation of a movie, albeit a rather low quality one.

Another tipping point occurred when it became feasible (and cost effective) to scan the entire movie at high resolution. This paved the way for Digital Intermediates (DI) and changed the way movies are color graded. The DI process enabled the creation of a high-quality, high-resolution, virtual representation of the movie - a digital master. Between these events use of digital visual effects had also rapidly expanded and full length CG animated movies had become highly popular.

Virtual Moviemaking is another such tipping point impacting a broader range of film production professionals from directors, art directors and cinematographers to visual effects supervisors and other production teams. But why is it so important? The answer lies in the increasing use of computer generated characters and visual effects in movies. A director wants to (and should be able to) scout locations or direct the performance of characters irrespective of whether they are real or virtual; but today much of computer graphics production is treated as a post-production process for review and approval and not as a hands on directive experience. Virtual Moviemaking processes help change that dynamic enabling filmmakers to have hands on creative input into what is an increasingly important part of the movie making experience.

Virtual production

Virtual production is an interactive, iterative and integrative process that starts from pre-production and continues till the final frame is printed to film (or packaged electronically) for distribution.

Virtual production processes leverage the proliferation of CG technology upstream and downstream of the production process, together with the latest breakthroughs in gaming technology and real-time graphics performance to provide film-makers with the instant feedback and the in the moment interaction that they have long experienced on live action shoots. Creating CG animation and visual effects shots can be experienced more like a live action shoot yet still offer the creative flexibility than is impossible on the real world set. When virtual production becomes interactive it feels much more like traditional moviemaking, hence the term Virtual Moviemaking.

Applications of virtual moviemaking

Virtual Moviemaking represents a significant metamorphosis in the use of digital technology for film production, enabling greater creative experimentation and broader collaboration between crafts. It transforms previously cumbersome, complex and alienating digital processes into more interactive and immersive ones, allowing them to be driven by the filmmakers themselves. As a result, Virtual Moviemaking is helping filmmakers to bridge the gap between the real world and the world of digital performers, props, sets and locations. It gives them the freedom to directly apply their skills in the digital world - whether for the hands-on direction of computer generated characters, or to plan a complex shoot more effectively.

Virtual Moviemaking technology provides significant benefits to live action visual effects production. The technology can be used to immediately visualize how the shot will look after post-production. Camera moves can be planned and tested in real-time and both creative and technical aspects of the shoot carefully evaluated and tested.

Motion capture camera systems

The core of any Virtual Moviemaking workflow is the camera system. The camera system consists of two key components: a virtual camera and a device to control it. The virtual camera is controlled by the operator moving a tracking controller within a tracking volume a volume of space defined by a series of remote sensing devices capable of detecting the exact orientation and movement of the tracking controller. A computer translates the data from these sensors into instructions that control (animate) the virtual camera causing it to behave exactly like the controller. The sensing devices and the tracking controllers form an integrated system known as a motion capture camera system.

The field of motion capture is rapidly evolving. Every year new systems emerge which offer greater flexibility at lower cost. The price of motion capture has dropped significantly in the past few years as the technology has improved.

Real-Time 3D animation engines

In addition to motion capture systems, the Virtual Moviemaking process also requires a robust 3D animation engine for the real-time render and display of the CG characters in the 3D scene. The animation engine provides necessary visual feedback for the director to see the action in context, make creative decisions and interact with the actors and scene. The quality of the visual feedback is determined by the quality of the animation engine.

Rendering 3D scenes with any kind of visual fidelity is computationally expensive and requires significant resources and time. Real time render engines will continue to improve with time delivering real-time images that are increasingly photo real. Current real time render engines such as the one built into Autodesk Motion Builder provides filmmakers with extensive creative tools to implement compelling art-directed representations of locations, sets, and other virtual assets.

Post production

Virtual Moviemaking ties pre-production and production more directly to post-production. By creating a virtual production environment with shared assets the different teams, from the pre-visualization team to the visual effects crew can better collaborate and share insights and expertise. When managed well the result is better planning, reduced costs and greater consistency: the virtual and practical production design elements are consistent; pre-visualization processes are consistent with production design; and pre-visualization and production design are more consistent with visual effects.

Production assets captured on the virtual sound stage can be shared with visual effects facilities and, although virtual production is not without its own problems in terms of data quality, consistency and interoperability, overall the process enables filmmakers to be more involved in directing the creation of CG content in ways that can still directly influence post-production. It is important to point out that although the real-time imagery of Virtual Moviemaking can be extremely appealing visually, the current generation of real time engines cannot produce the high image quality achieved in post-production by the visual effects teams.

Looking ahead

The movie industry is currently in the very early adoption stages of virtual production workflows. The technology is still nascent and the processes far from standard. It requires skill, talent, patience, and a taste for risk to embark upon the path of virtual production but the rewards are rich. However as with Digital Intermediates (DI) and visual effects the steady march of technological progress will make the technologies discussed in this whitepaper ever more accessible. Already today lower budget productions are using many of these techniques to improve the quality of their CG production.

As virtual production workflows become more commonplace and more and more film production professionals become comfortable with the new technology it will disseminate throughout all levels of the film production pipeline. The impact is already clear on high profile film projects such as Avatar and Tintin, not only with the Directing Team but with Cinematographers, Visual Effects Supervisors, Actors, and throughout the Art Department, as Production Designers, Art Directors, Set Designers, and Costume professionals get exposure to the benefits of the technology. Virtual Moviemaking technology allows every one of these professionals to vet out their various creative contributions and experience them in the same way that the filmmaker and ultimately the audience will see them.

Although it Is not possible to predict exactly how the technology will look five years from now, it is reasonable to expect that it will be easier to use, more affordable and produce higher quality real-time results, all key factors for ensuring broader industry adoption.

Given the importance of virtual production leading industry organizations such as the Visual Effects Society (VES), American Society of Cinematographers (ASC) and Art Directors Guild (ADG) are collaborating to ensure that, in the future, their members and the industry as a whole can benefit from both the technology and the workflow.