An Introduction to LTFS for Digital Media

We live in an age where content is king. For the entertainment and media industry, the majority of this content is now produced in digital form, and virtually all of this content has digital distribution; that content is now digital data. Protecting that content, the lifeblood of this industry, with the right data storage system is more important than ever.

Tape is already firmly established in the media production environment whether to secure on-set content or for long-term archiving. LTFS broadens LTO technology usefulness by being easier to use and more robust. The open and self-contained LTFS format is useful if the tapes are to be sent offsite, archived or shared with a variety of recipients.

The LTFS standard was adopted by the LTO Program in April 2010. It is an open format and software specification that supports simpler and new ways to access data on tape. Although the tape model hasn’t changed dramatically over the years, the speed, storage density and features of data tape have improved significantly, ultimately providing reliable and inexpensive storage as a sequential storage medium. With LTFS, accessing files stored on the LTFS-formatted media is similar to accessing files stored on other forms of storage media, such as disks or removable USB flash drives. It has no application software dependencies, offers support for large and numerous files, and often can have a lower total cost than traditional managed tape storage.

How LTFS Works
LTFS consists of a software driver and the format specification. Drivers, some free and open source, are available for various operating and tape systems from the tape hardware vendor websites.

The format is a self-describing tape format and defines the organization of data and metadata on tape. Within the partitions, the tape contents are still stored as usual, as blocks of data and file marks. Files are mapped by LTFS to a hierarchical directory structure.

LTFS uses media partitioning, where tape is logically divided “lengthwise” into two partitions:

• Index partition: Contains file system info, index, metadata.
• Content partition: Contains the files and content bodies.

The standard LTO-5 tape cartridge is segmented into two partitions, one for the index and one for the data, so the index partition can be modified as needed without affecting the append-only data partition. This creates a self-describing tape where a user can see the tape cartridge and its contents in the operating system directory tree browser and can copy, paste or drag and drop files/folders to and from the tape. Similarly, applications can access the data on tape directly, unaware they are using tape, though there may be differences in latency inherent to tape.

When the tape is inserted and mounted in a tape drive, the information in the index partition is read and cached in the workstation’s memory. From that point on, the index is accessed and updated in the workstation memory for fast performance and so the tape head can stay positioned in the content portion of the tape to more quickly access files or write new ones.

As the tape is used, the index is updated in the workstation memory for fast performance, eliminating the need to go back to the beginning of the tape. To protect the index, LTFS periodically copies the index from the workstation’s memory to the data partition. When the tape is done being used (unmounted), the index is copied one more time at the end of the data partition, the tape is rewound and then the index is written twice to the index partition. Essentially, there are multiple copies of the index on tape for restoring in the unlikely event the index partition index is not usable or if a user wants to roll back the tape to a previous version.

For instance, you may want to restore it to the way the tape appeared last Monday by choosing an index from that date. In some tape library implementations, the tape cartridge’s indexes are cached to a server disk for fast searching without having to remount the tape cartridges. LTFS also supports extended attributes, which enable custom file metadata.


Workflow
LTFS opens up new opportunities for media and entertainment storage and distribution, providing support to media workflows. LTFS directly affects two major trends having significant impact on media and entertainment companies and digital media producers: the shift to file-based workflows and increasing storage demands.

With the advent of digital technologies, moving image (video and film) content producers and distributors are transitioning from analog/linear workflows based on film or videotape technology to digital/nonlinear workflows based on the manipulation of data files. Many organizations have already completed this transition. This has been commonly referred to as “moving to a tapeless workflow,” though it is more accurate to call it moving to a “videotapeless” workflow.

The other fact of life in media and entertainment is the demand of ever-increasing visual resolution and complexity (HD, 4K, 3-D, etc.) creating more and larger files that must be managed. Keeping hours of such media online on disk quickly becomes cost prohibitive. LTO tape, especially with LTFS, can address the challenge. LTFS can work with LTO hardware-based lossless compression, which can provide bandwidth and capacity benefits depending on the data content.


Production
LTO tape is already firmly established in the media production environment. For production, LTFS efficiently supports several important requirements.

• Camera media reuse: Digital cameras encode motion images directly to SSDs or removable disks in the camera. These media are quite expensive (three to more than 350 times the equivalent media cost on LTO). Fast transfer of their contents to tape with LTFS enables reuse of this expensive media, reducing the number of SSDs or disks that must be purchased or rented.


• Backup: Backup of daily footage to LTO tape is a common requirement as the loss of a day’s worth of production is costly. LTFS facilitates backup by enabling small portable independent systems to easily write daily content to tape.


• Transport: The density and cost of LTO-5 tapes with the self-describing capabilities of LTFS combine to create an effective transport medium. Large amounts of data can be sent more quickly and economically than network-based transmission methods. This is especially compelling for digital productions, which can produce terabytes of data for every day of shooting. The encryption features of LTO tape help secure the data in transit.


• Economic direct access to data: For any file-based production workflow, an LTFS-enabled tape drive can feed workstations or networks with content directly and relatively quickly, similar to a disk and unlike most traditional tape systems. An application via the operating system always has a direct and persistent view of a mounted LTFS tape and the files it contains. Consequently, in a workflow where access to a file is expected to be fast but not instantaneous, such as a stock footage collection or archive footage of an ongoing news story, an LTFS tape is an effective and economical choice for storage.


• Archive: LTFS-formatted tapes can be easily imported into an LTFS-compatible archive by simply reading the index and adding the file metadata to an archive manager’s catalog.

Conversely, traditional systems that use separate media for transport and archive require all the data be recopied. With LTFS, there is no need to read the much larger data partition or transfer the data to other storage media. The transport media and the archive storage media are one and the same under this scenario. The “import bandwidth” of tapes being added directly to a library en masse far exceeds any system that requires movement of the actual data.

By Rainer Richter, Broadcast Engineering