Lip-syncing automatically with SmartMouth in Flash Professional
Additional requirements
Sample files
Matching dialogue to a character's mouth in order to create the illusion of speech is known as lip-syncing. This effort often consumes more time than any other animation task because you need to make adjustments on nearly every frame. In fact, lip-syncing short animations (even only 1–2 minutes long) can involve hours of tedious labor.
This article explains how to make lip-syncing in Adobe Flash Professional CS5 as painless as possible by utilizing the SmartMouth extension to automatically analyze audio content and assign corresponding mouth shapes. That's right: you can sit back and relax your hands, back, neck, and eyes while SmartMouth processes the audio in the time that it takes the audio to play back, and then matches each frame using a speech algorithm. You'll also learn how use the free FrameSync extension to quickly make manual adjustments and tweaks to your character animation.
Introduction to lip-syncing in Flash
The basic unit of speech is known as a phoneme. The mouth shape and facial contortions that correspond to phonemes are known as visemes. Animators generally refer to phonemes and visemes interchangeably, even though they are technically different concepts. The standard set of about seven phonemes/visemes (mouth shapes)—not including a closed mouth—are sufficient to create the illusion of speech on an animated character. The SmartMouth Flash extension includes a sample set of those shapes (see Figure 1).
This small set of phonemes corresponds to a wide array of spoken sounds. The full range of sounds that are covered by this small set of mouth shapes are listed in Table 1.
| Abbreviated phoneme | Full letter list | Corresponding examples |
|---|---|---|
| – | none | No vocal sound (or inaudible) |
| A | A, I | ah as in "cat," a as in "say," i as in "kite" |
| O | O, U, (W), (R) | o as in "boat," u as in "clue" |
| E | E, (I) | e as in "street," eh as in "trek" |
| S | C, D, G, K, N, R, S, TH, Y,Z | s as in "stress," t as in "tent" |
| L | L | l as in "lull" |
| M | M, B, P | m as in "might," b as in "back," p as in "pass" |
| F | F, V | f and v as in "favor" |
Note: Letters in parentheses denote that some sounds made by those letters (for example, W and R) produce a viseme—specifically puckering of the lips, similar to the visemes produced by U and O sounds.
By reusing a small set of mouth shapes, you avoid reinventing the wheel on every frame. Instead, you can simply leverage the repository of mouth shapes that you've already created.
Existing methods of lip-syncing
Flash animators have developed many techniques to speed up lip-syncing in Flash. The three most common methods are swapping symbols, nesting, and nesting with labels. Each of these three methods utilize reusable mouth shapes for phonemes.
The swapping method involves creating each phoneme in its own symbol and then swapping the symbol shown on each frame to match the audio at that frame.
The nesting method involves placing all of the phonemes along the Timeline within a single graphic symbol. By manipulating the First (frame) value in the Property inspector, you can control which frame of a graphic symbol's Timeline is displayed (as shown in Figure 2). The nesting method is a bit more organized and efficient than the swapping method, but it does require that you memorize (or write down) the frame numbers that correspond with each mouth shape.
The nesting with labels method is identical to the nesting method, except that frame labels are added to the graphic symbol's Timeline to identify each distinct shape. The labels make the symbol's Timeline easier to navigate—and they can read by SmartMouth and FrameSync, eliminating the need to memorize frame numbers.
The sample project in this article focuses on using the nesting with labels method. To add a frame label, select a keyframe and enter the desired label into the Name field in the Property inspector (see Figure 3).
All manual lip-syncing in Flash relies on scrubbing. Scrubbing refers to the act of selecting the playhead in the Timeline and dragging it across a single frame. Any audio layers that have their Sync property set to Stream will play the scrubbed frame's audio (assuming Control > Mute Sounds is off), allowing you to hear a small fragment of speech. Scrubbing is a time-consuming user action because it requires a lot of precision using a mouse, track pad, or stylus. It may take a couple of scrubs on a single frame to identify the sound. Using one of the three methods described previously, you can scrub and update each frame to use the correct mouth shape symbol until the entire Timeline has been synced.
Where the SmartMouth extension fits in
SmartMouth works natively with the three methods discussed previously and edits your Timeline directly, as if you'd made the edits by hand; the Timeline and symbols remain completely editable. This flexibility allows you to make desired customizations, adding personality and subtlety to your character, just as you would manually, but in a fraction of the development time.
Setting up your file
The file setup required for SmartMouth is identical to that of a file that you lip-sync manually. The file must contain mouth shapes (in a single symbol or in several symbols) and a layer with the audio set to Stream. For SmartMouth to distinguish speech properly, your targeted audio layer should contain only one character speaking at a time with no music or sound effects. Before setting up your file, install the SmartMouth and FrameSync extensions so that they can be accessed in the Flash authoring environment.
Installing the SmartMouth and FrameSync extensions
The Adobe Extension Manager makes it really easy to install extensions. Follow these steps:
- Download the SmartMouth and FrameSync extensions.
- Unzip SmartMouth.zip.
- Double-click SmartMouth.mxp to install the SmartMouth extension. This launches Adobe Extension Manager CS5 (which is installed automatically with Flash Professional CS5). If this step does not work properly for you, or if the wrong version of Adobe Extension Manager opens, launch the Extension Manager separately. Click the Install button and browse to select the SmartMouth.mxp file.
- Read the license agreement and click Accept if you agree to the terms. You will receive an alert from Adobe Extension Manager when the extension has been successfully installed (see Figure 4).
- Repeat steps 3 and 4 to install the FrameSync.mxp file. Close Adobe Extension Manager.
- Launch (or restart) Flash Professional.
- Locate the sample files folder that you downloaded at the beginning of this article. Open the smartmouth_adc_demo_start.fla file provided in the sample files folder, or begin by creating a new ActionScript 2 or ActionScript 3 FLA file to use your own content (SmartMouth supports both versions of ActionScript).
No comments:
Post a Comment