Finding a Sane Solution In a Sea of CDs
Terabytes of data inundate entertainment. Digital asset management can control it before it controls you.

By Doug Perkins

Those whose job it is to produce television shows, radio spots, film, video games and the various other forms of consumer entertainment know that it takes a tremendous amount of audio and visual information to pull off a successful implementation. They also know that all this information is typically delivered on CDs that tend to accumulate.

It wasn't always this way. There was a time when audio was delivered on cassette, and video came on tape. Film was always on … well, that's why it's called film. Today everything is digital. Of course, then as now, the question is how to control the never-ending supply of new sound and video clips.

The answer is digital asset management.

What is Digital Asset Management?
Digital asset management is how users control the growing mass of sound and video clips necessary to produce great content for broadcast, film, advertising and other entertainment outlets. Although the concept of asset management has been around for some time, how content is stored and accessed has changed significantly over the years.

From quarter-inch tape and LP records to digital audiotape and CDs, the failure of physical media to meet the growing needs of production facilities has been apparent for some time. For starters, each of those systems allows just one person at a time to access material, and physical media is subject to wear and breakage, not to mention a tendency for entire libraries to "walk off on their own."

One main difference between this and other industries is the pure volume of material. The major audio libraries combine to deliver about 10,000 new tracks every month. And nobody throws out the old tunes when the new ones come in, so the result is seemingly unmanageable growth.

As far back as the early 1980s, several companies began creating solutions to this problem, with varying degrees of success. Leonardo offered a sound effects search program; Geffen Systems upped the ante by creating the M&E (music and effects) Library, a program that also handled production music. This was an ambitious project that included the first rudimentary searches based on categories and subcategories. In conjunction with most of the major music libraries, thousands of music clips were available through the system-and it worked well enough to spawn imitators. The concept of categories has since been embraced by the community at large and incorporated in subsequent systems.

Such programs were visionary in their scope of flexibility in searching, but the hardware on the market was not up to providing the instant access that everyone knew was coming. Control of multiple-CD consumer and prosumer level auto-changers, by companies like Denon, Sony and Pioneer, were implemented into search software in the latter part of the decade. But even though some units offered two internal CD players, these were designed for cueing up the next song from one jukebox-style playlist and, as such, only one user could control the changer at a time. The search was fast, but to play back the effect or music track you wanted to audition, it might take a few minutes for the changer's control to be released by its last user and the next CD to be loaded and cued up. Clearly, the CD changer was an interim solution at best, and many facilities adopted a "wait and see" policy before buying into this infant asset management solution.

By the 1990s, the Sonic Search system from Sonic Science offered a hybrid solution: control a CD changer to automate digitizing your CDs to a hard disc. This offered the ability to keep hard drive costs down by only having your most used libraries on the drives, keeping everything else on CD and leaving possible digitization of some libraries "for later." Geffen Systems had a similar flexible solution with its "M&E Intranet" and later "SFXNet," which included options of a CD changer, hard disk delivery, or both.

All of these solutions were limited by the types of digital audio workstations that they interfaced with and the audio libraries that were available for them. Those who used workstations that were not one of the standards were out of luck, and creating your own databases or adding existing ones to your system tended to be difficult, if not impossible. Trying to protect their data assets and tie their users to their products, software manufacturers tended toward a closed database structure. Despite all its advances, the systems available in the mid-1990s just didn't have the requisite functionality, and companies began to create new products and processes to meet those needs.

For example, Galley released mTools, which was great for facilities with limited DAWs or facilities that wanted to digitize all their audio in-house. It even integrated well with the existing network. Unfortunately, many facilities employ numerous types of DAWs that all need to share assets, and these DAWs typically use different file formats and sample rates. When mSoft introduced its ServerSound system in 1999, it became the first platform-independent system that could handle any type of audio or video file, and convert file formats as needed for each DAW, when copying files into workstation local drives.

Doing it all digitally offered several advantages, namely the ability to access everything in a library and mitigating cost concerns as the price of RAID servers dropped dramatically according to Moore's Law, which suggests that technology advances double every 18 months.

One of the benefits of advancing technology is the ability to pump more information into less space. As with previous digital asset systems, mSoft's make use of metadata, which is the text that describes content as it might be searched. Where the older systems failed-and still do in most cases-is the depth to make a search more successful.

Good metadata is required to find assets on a DAM system and, while most music and SFX libraries do provide simple search databases for their customers, the quality of the data is spotty and varies from library to library. The problem is apparent even in simple searches: search a library for "dog bark" and results are returned, but try to find a "yelp" and…nothing. For that reason, most DAM providers who deal with audio content libraries find themselves hiring full time librarians to massage the metadata into something usable in an online server system.

One problem inherent with the proprietary nature of most DAM systems is that the metadata used by someone on a Mac may be useless to his colleague trying to access the same files on a PC or Unix machine. According to industry consultant Ed Bacorn, who specializes in storage area networks like those used in production, "The metadata tells any workstation accessing the library what access levels it has and where to look for certain files," so a machine with indeterminate access levels can't access the files at all.

Therefore, a solution for any facility with multiple DAWs must allow different platforms to work from the same network and the same storage array. Overcoming the cross-platform issue the way mSoft's products do "is a huge step forward," says Bacorn.

Doug Bossi, VP of Operations, explains how much effort goes into mSoft's ServerSound and MusicCue systems. "We collect all the stock music and sound effects, re-master everything digitally, add the text metadata description of the assets, then store it all on a multi-terabyte RAID system and provide the entire package-server, database, storage, ongoing monthly music updates and our own proprietary digital asset management software that ties it all together."

He continues, "We even solve the very difficult technical problem of delivering the assets to the actual timeline of the production in all of the various digital audio and video workstations available today."

Amnon Sarig, mSoft CEO, explains that digital asset delivery is just part of the evolution of the industry. "The CD and tape cassette, as delivery mechanisms for stock assets, are going the way of the endangered species," he says. "There is not a single person in the entertainment industry that is busily investing in CD technology and the dead state-of-that-art 500-CD jukeboxes, that at the end of the long wait for the playback is ultimately still a single user machine."

Bacorn agrees. "CDs offer a non-linear approach to accessing content. But its inherent limitations are its size and speed of delivery." The best-case scenario, he says, is 700MB of data on a single CD, so you could group dozens of them as a jukebox, "but who wants to load a jukebox with 500 CDs? This simply turns a non-linear technology into linear."

CD jukeboxes are not seamless, and whoever wants to use the box is limited by the delivery rate of the CD player itself--considerably slower than high-end RAID with fiber channel or ultra 5 SCSI connectivity.

Delivering data in digital form on one drive system capable of 100MB/s of available bandwidth (200MB/s on some systems) eliminates the hassles of finding the right CDs, loading a jukebox, and waiting for the transfers to take place one system at a time. With RAID arrays, all users have simultaneous access to a non-linear disk array capable of saturation bandwidth and the array is loaded with the facility's SFX, music, videos or whatever. An important benefit to this method is that everything is protected by excess hardware RAID, so if one drive fails, it can rebuild itself while the rest of the array keeps running. Nothing is lost and the production remains on schedule.

Since a production facility never knows what it will need on a day-to-day basis, products using robotic tape drives to load a desired set of files into online drives are not a flexible enough answer. The purpose of an archive of assets for a production facility is to search everything for just the right element-not smaller subsets of the whole. Most broadcasters have twenty minutes, at the most, to cut a promo and that leaves no time for loading files from tape.

This means that a significant amount of footage or audio must be at the ready at all times. In audio post, a typical production house working in a mid-level market might have from 200-500 CDs in its library, which translates to about 150-300GB of storage at a typical 48K sample rate. Larger post facilities in New York and Los Angeles have libraries of 800-3000 CDs-a whopping 550GB to 2TB of storage. Major broadcasters are voracious for content and most have anywhere from 4,000-12,000 CDs available to producers, which uses 3-9TB of storage.

Recently, companies like Sonomic and Sound Dogs have tried to help satisfy the industry's content appetite by offering online stores for selling and delivering SFX and music over the Internet. This is a great solution for finding content that you just don't have and need expedited, but few serious operations would rely on this as their only source-production schedules are just too tight. This is precisely why sound effects and music have been delivered as a library rather than individually over the years-you never know what you will need tomorrow.

However, the possibility of downloading content from these online stores into a larger asset management system is an excellent way to manage individual effects, and mSoft, Sonomic, and other systems allow for this.

Overall, according to Sarig, "The Internet as a wholesale delivery method is still limited due to issues with bandwidth and security. The Internet pipe cannot push ten terabytes of assets to your facility unless you have a dedicated fiber optic line. We have found that most companies are still reluctant to rely on the Internet for all their various production and delivery needs; but when they are, we will be ready for that, too."

Growing Storage Needs
The worldwide market for digital delivery of stock music, sound effects and video footage is very substantial. "We identified 2,500 facilities worldwide with at least ten-terabyte production asset needs, with a conservative estimated growth rate of one terabyte per year for new music produced that year," says Bossi. "About 20,000 facilities will need at least five terabytes and 100,000 relatively small users-like corporate internal communication productions-will need slightly less than a terabyte. Production music alone will consume 200,000 terabytes by the end of this decade."

But music is just one small part of the big picture. According to Bossi, "Video is 100 times larger than sound, and HDTV needs are 700 times bigger still. We estimate the total worldwide market available for this specialized market segment-stock sound, music and footage-to be in the petabyte realm in the not-too-distant future."

Bacorn says that the natural progression of drive arrays makes digital storage the best answer for entertainment applications. Because the integrity of audio and video assets is central to productivity, "You absolutely cannot lose access to those assets," he says. That means a completely fault tolerant RAID array is critical.

Bacorn explains that disk drives have increased dramatically, just as file sizes have increased. For example, assets that used to be in the 2-4GB realm are routinely stored at the 20-30GB size now, in large part due to the conversion to high definition, which requires greater quality.

Since even CD-quality sound is relatively small, with a real time transfer of less than 200 kilobytes per second, a low cost IDE RAID-5 with SCSI-160MBPS back plane is fast enough for sound and music assets. MSoft uses RAID-5 with a hot spare to compensate for the risk of losing one of the relatively low cost drives. Coupled with a 16-bay unit from hardware provider ADTX Japan (itself an offshoot of technology giant IBM) and 250GB EIDE drives, a whopping 4TB of raw storage fits in a 4U rack space.

Bob Pomann, president of Pomann Sound in New York, says the space savings alone make the product worth the installation. "We had a room full of CDs in our facility-a penthouse in the most expensive street in America, Fifth Avenue in New York. We loaded our entire sound and music collection to an 8U rack and freed up the space to create another income producing edit bay room. The rent cost savings alone paid for the system within the first six months. The ability to pull and instantaneously audition any of our 6,000 hours of music allows us to make more artistic decisions and not decisions made out of desperation."

The number one growth driving the industry is the commoditization of storage needs. Just 18 months ago, a terabyte of RAID-5 storage cost more than $25,000. It dropped 90 percent in two years, following the meltdown of the dot-com and communication industries and the physical compaction of the storage space. When 320GB serial IDE drives become available later this year, an astonishing five terabytes will occupy only six rack units.

To answer the stock video transfer speed problem, ADTX created a dual-fiber channel back-plane to the same EIDE drives. Using 7200 RPM IDE drives, the transfer rate generated by the parallel activity of the RAID can support up to 2MB/s-fast enough even for real-time 1,500 Mb/s HDTV demands, and easily supporting the 270Mb/s of SDI.

More Assets Spells Greater Need for Technology
Even with all the assets online locally, a robust server delivery system and extensive cross-platform metadata, there was yet another problem: how to track usage for royalties reporting purposes.

Sarig demonstrates the problem. "I just don't understand how the music industry expects its users to keep up using the old 'sneaker net' methodology. In a facility like CBS, there is one person who is in charge of feeding music to the scores of producers who choose music to create their various 25,000 promotions every year. He is very good, but there is no way one person can deal with this river of incoming digital music. Fox Sports used 1,700,000 music stings last year alone. How can even a team of employees keep track of it all for the royalties usage payment?"

MSoft's answer is something called a Cue Sheet, which tracks which clips were used for which projects. Other products have begun to include a similar methodology, which means the circle is complete. There are 800,000 publicly available asset items and 200,000 more in private collections; searching these assets digitally is now a matter of seconds, not hours and days.

The continuing trend of lower costs for storage makes it possible for all but the very smallest facilities to take advantage of this kind of intelligent asset management. The recent development of using low cost IDE drives in a SCSI chain provides more than adequate speed for audio management, and it stands to reason that the market will demand that a similar solution be found for video, as well. Consumer products that have shown up recently, like HP Digital Entertainment Center, show that manufacturers are already foreseeing a future of a home with a music collection, DVD movies, home videos, and even the family photo album, all digitized and accessed from some type of hybrid A/V television and computer Intranet system. If this is the near term future for the home market, there is no doubt that the professional market should be ready to embrace this reality now. The better known facilities already are.

There's no telling what the "next big thing" in entertainment storage will be but, whatever it is, it's a long way off and won't come cheap. The natural progression of technology means more space for less money, which feeds right into digital asset management solutions and, in any case, the current digital solution should be more than adequate for some time.

There is a revolution in the making in our industry. Low cost dedicated digital asset management systems to vertical markets, in this and other industries, are naturally progressing into the realm of multi-terabyte storage. The drop-in cost, expected to continue for the foreseeable future, will continue to drive dedicated applications in the entertainment industry. The days of flipping through separate catalogs-paper or electronic-are history.

Doug Perkins is vice president of sales at mSoft Inc. (Woodland Hills, Calif.)

http://www.msoftinc.com/