FIMS, SOA and Media Applications
An interesting white paper by David Austerberry.
Showing posts with label FIMS. Show all posts
Showing posts with label FIMS. Show all posts
SOA: The Business of Broadcast
Broadcasters have many challenges as the media business evolves, driven by new consumer devices and the increase in mobile viewing. National broadcasters are facing more competition from global operators. New entrants like YouTube and Netflix have changed the VOD landscape. Broadcasters that once aired one channel now air a multiplex of linear channels, as well provide catch-up and mobile services. Summing up, broadcasters must deliver to more platforms, linear and on-demand, in a more competitive business environment.
To meet these challenges, a business must become more agile. Many other sectors have faced similar challenges, and part of the solution for many has been to turn to new software applications, particularly Business Process Management (BPM) and the Service-Oriented Architecture (SOA). Although each can be used stand-alone, BPM and SOA are frequently used in concert as a platform to improve the performance of a business.
Operations that use videotape were constrained by the need for manual handling, but as content migrates from videotape to digital files, the way is open to use IT-based methodologies, including BPM and SOA, to aid broadcast operations.
What is SOA?
SOA is a design methodology for software systems. SOA is not a product, but an architecture to deploy loosely coupled software systems to implement the processes that deliver a business workflow. SOA provides a more viable architecture to build large and complex systems because it is a better fit to the way human activity itself is managed — by delegation. SOA has its roots in object-oriented software and component-based programming.
In the context of the media and entertainment sector, SOA can be used to implement a “media factory,” processing content from the production phase through to multi-platform delivery.
Legacy Broadcast Systems
Traditional broadcast systems comprise processing silos coupled by real-time SDI connections, file transfer and an assortment of control protocols. Such systems are optimized for a specific application and may provide a good price/performance ratio with high efficiency.
The tight coupling of legacy systems makes it difficult to upgrade or replace one or more components. These applications are typically coupled via proprietary APIs, as shown in Figure 1. If the software is upgraded to a new version, the API can change, necessitating changes to other applications that are using the API — work that is usually custom.
It makes it difficult to extend to add new functionality to the system to meet the ever-changing demands of multi-platform delivery and evolving codec standards. Storage architectures change with object-based and cloud storage to become alternates to on-premise NAS and SAN arrays.
When vendors upgrade products, the new versions often do not support legacy operating systems, leading to the need to replace underlying computer hardware platforms.
Traditional systems are just not agile enough to easily support the new demands of the media business. Often new multi-platform systems are tacked on to existing linear playout systems in an ad-hoc manner to support an immediate demand. The system grows in a way that eventually becomes difficult to maintain and operate.
Monitoring — No Overall Visibility
Traditional systems also suffer from a lack of visibility of the internal processes. Individual processes may display the status on a local user interface, but it is difficult to obtain an overall view (dashboard) of the operation of the business.
As broadcast strives for more efficiency, it is vital to have an overall view of technical operations as an aid to manage existing systems and guide future investment. Many broadcasters already have end-to-end alarm monitoring, but resource usage may only be monitored for billing purposes, and not to gain intimate knowledge of hardware and software utilization.
SOA in Media
SOA is not new; it has been in use for a decade or more in other sectors including defense, pharmaceuticals, banking and insurance. It developed from the principles of object-oriented software design and distributed processing.
If SOA is common in other sectors, why not just buy a system from a middleware provider? The problem lies with the special nature of media operations. The media sector has lagged other sectors in the adoption of such systems for a number of reasons. These include the sheer size of media objects and the duration of some processes. A query for an online airline reservation may take a minute at most; a transcode of a movie can take several hours. Conventional SOA implementations are not well suited to handling such long-running processes.
What is a Service?
A service is a mechanism to provide access to a capability. A transcoding application could expose its capability to transcode files as a transform service. Examples of services in the broadcast domain include ingest, transform, playout, file moves and file archiving. The service is defined at the business level rather than the detailed technical level.
It could be said that many broadcasters already operate service-oriented systems; They just don’t extend the methodology to the architecture of technical systems.
Services share a formal contract; now service contracts are commonplace in broadcasting and across the M&E sector, with companies calling on each other for capabilities such as playout, subtitling and effects, as examples. The service-level agreement for playout will include quality aspects such as permitted downtime (99.999 percent). Service contracts operate at the business level, and ultimately may result in monetary exchange.
Abstraction
The business management logic may call for a file to be transcoded from in-house mezzanine to YouTube delivery format, but not define the specifics of a particular make and model of transcoder or the detail of the file formats. This abstracts the business logic from the underlying technical platforms. A generic service interface for file transform can be defined, and then each transcoder is wrapped by a service adaptor that handles the complexity of the transcode process. To the business logic, the transcode is simply a job. The abstraction of the capability is a key principle of the SOA.
In a legacy system, the ingest job is delegated to an operator who configures an encoder, and then starts and stops the encoding at the appropriate times. The operator is functioning autonomously during the processes of the job. These concepts of delegation and autonomy are key to the SOA design philosophy. The encoding may well be automated as a computer process, but the principles remain the same.
Because the service is abstracted, it opens the way for broadcasters to leverage cloud services more easily. As an example, at times of peak transcode demand, a cloud transcode service could be used to supplement in-house resources. With a standard service interface for transcoding, the implementation can be an on-premise or cloud-based service. The operation of the services is orchestrated by a layer of middleware, software that manages business processes according to the needs of the business.
A transform service can be used for different business processes. For example, a transcoder could be used to transform files at ingest to the house codec or used to create multiple versions of content for multi-platform delivery. The transform services can be redeployed to different departments as the needs of the file traffic change from hour to hour.
Planning for a SOA
Migrating from traditional tightly coupled systems to use SOA principles is a big step for a media business. The efficient operation of SOA requires detailed analysis of business needs and definition of services. It also requires rigorous planning of the IT infrastructure, computers and networks for efficient operation of the services. Without the involvement of senior management down to IT services, the benefits of SOA are unlikely to be fully realized.
For broadcasters used to running departmental silos, many with real-time elements, the move to SOA will be a radical change to the way the business operates. However, the advantages of the SOA and allied systems like BPM are proving attractive propositions for the broadcaster — or service provider — running complex file-based operations for multi-platform delivery.
The problems facing a media company looking to embrace SOA and BPM include change management and the sheer problems of keeping on-air through such huge changes in the technical infrastructure supporting broadcast operations.
Many media companies have embraced the architecture, with early adopters using considerable original development of such components as service adapters — the vital link between a service like transcoding and the workflow orchestration middleware.
The use of consultants or internal software services to build a media SOA will achieve the goal, but does it make sense for all media businesses to go their own way?
FIMS
It was this issue on which the Advanced Media Workflow Association (AMWA) and the European Broadcasting Union (EBU) independently agreed when, in 2010, they decided to pool resources and set up the joint Framework for Interoperable Media Services (FIMS) Project, which would develop standards for a framework to implement a media-friendly SOA.
The road will be long, and many obstacles remain to be resolved, but the success of this project will benefit both vendors and media companies in the long run.
The FIMS solution aims to provide a flexible and cost-effective solution that is reliable and future-proof. It should allow best-of-breed content processing products to be integrated with media business systems.
The FIMS team released V1.0 in 2012 as an EBU specification, Tech 3356. Three service interfaces have been specified: transform, transfer and capture.
The FIMS Project has expanded on the conventional SOA with additional features to meet the needs of media operations. Specifically FIMS adds asynchronous operation, resource management, a media bus and security.
Asynchronous operation allows for long-running services. A transcode may take hours; conventional SOA implementations allow for processes that complete in seconds or minutes.
Although services are loosely coupled to the orchestration, jobs can still be run with time constraints. This may be simply to start a job at a certain time, but services can also be real time, like the capture and playout of streams. In these cases, the job requests for the service will also include start and stop times for the process. For playout, this concept is no different from a playlist or schedule.
SOA typically is based on an Enterprise Service Bus (ESB) that carries XML messages between service providers and consumers. A media bus provides a parallel bus to the ESB to carry the large media essence files. Many file-based operations will already have media IP networks that can be adapted to provide the platform for the media bus, as shown in Figure 2.
The Future
Software methodologies like SOA and BPM help media businesses manage file-based operations in more efficient ways and better serve the needs of multi-platform delivery. They provide a holistic approach to running business operations that provide better visibility of operations and simpler ways to leverage cloud services.
They have proved successful in other sectors and are ready to meet the unique needs of the media sector.
By David Austerberry, Broadcast Engineering
Media Services: Services Architecture Enables Functional Flexibility
Unfortunately, the term “media services” contains two overloaded terms — “media” and “services.” In this case, when we talk about media services, we are talking about small (some would say atomic), network-based applications “services” that perform simple, focused tasks.
These tasks are somehow related to either essence or metadata used by professional broadcasters, post-production facilities and the film industry.
An example of a “media service” might be a service sitting out on a network that is available to transcode content from one popular video format to another. One can imagine a host of services, including tape ingest, QC and file movement. Each of these services is available out on the network, and can be used to perform a discrete unit of work. Services perform higher functions by grouping a number of atomic services together in a logical way. But, at their core, media services are small, discrete pieces of software that can be combined in different ways to perform work.
This is a significant departure from traditional media infrastructures, where an ingest station consists of tape machines, routers, monitors and other hardware — all hard-wired together to perform a specific function. In fact, entire broadcast chains are built this way. They are highly optimized and efficient, but they can be very difficult to change. And, if one thing is certain these days, it is that change is a permanent part of our business.
A media services architecture allows discrete blocks of functionality to be combined to build complex workflows. As workflows change, blocks can be recombined into modified workflows. If new functionality is added, new services can be deployed. Additionally, discrete services may be used in multiple workflows. So, a transcoder may be deployed in a post-production scenario for one job and then redeployed in a conversion for web applications next.
Building Workflows
When using media services, it is not enough that the services are available out on the network; something must consume those services in order to perform valuable work for the organization. There are several approaches to using services, but, for this article, we are going to focus on two of them — orchestration and event-driven architecture.
Orchestration
Orchestration systems sit on top of media services and use media services to move work through a defined pipeline from start to finish. For example, an orchestration system might have a workflow that ingests a tape, transcodes the content and then saves the file on a large central server.
The orchestration system tracks the progress of the workflow, calling on various services to work on the job as it moves through the pipeline. The orchestration system is responsible for not only dealing with normal flows, but it is also responsible for dealing with error conditions such as a failed transcode. Orchestration can start out simple, but it can become complicated as engineers consider all of the various states and error conditions possible in the workflow.
Event-driven Architecture
Event-driven architecture is another way to use services to perform work. At a high level, in this architecture, something causes something else to happen (an event) that is of significance to the business. Processing engines can be set up to listen for that event, and when the event happens, they can perform actions based on the event. Other processing engines can be listening down-stream, and when one event engine finishes, others can be triggered. In an event-driven architecture, there is no central system guiding the flow of work through a pipeline. The movement of work through the facility is caused by a sequence of events and processes.
An operator finishing an ingest activity might create an “Ingest Complete” event. Event processing engines subscribe to event channels, such as the “Ingest Complete” event channel. This particular event-processing engine might have two actions: the first is to notify the QC operator that the file is available for quality control checking; and the second is to publish an “Ingest Complete” notification for other systems that might be interested in the event, such as traffic and automation systems. Both of these systems might update the status of the media based on the “Ingest Complete” event.
Note that, in this example, it would be extremely easy to add another event process engine to the event channel. This engine might be responsible for creating a number of different formats from the original ingested file format. Adding this process does not require modifying a workflow in a central system. All one has to do is to subscribe the transcoding engine to that particular event channel.
SOA and EDA
It is important to realize that orchestration and event-driven architecture are complementary, and they are frequently deployed together. For example, in the earlier referenced figure, the tape ingest function might be driven by an orchestration system that precisely controls workflow and error-handling conditions. Event-driven architecture would be used to notify other SOA processes once the ingest is complete.
Common Approach
Common service interface definitions are critical. One can imagine a whole universe of services: a content repository service, a media identification service, a publish content to ISP service, and so on. And, one can imagine that several different vendors would make such services available. If each vendor defined the interface to their service independently, the amount of software integration required to build these systems would be huge. On the other hand, if the industry would agree on the service interface definition for an ingest service, for example, then it would be possible to integrate various ingest services into a workflow with minimal additional development.
Common service interface definitions are critical, but it is also critical that we have a common overall framework within which services can be deployed. How do services communicate with orchestration systems and with each other in event-driven architectures? How do newly commissioned services make their presence known on a network? Again, having a harmonized approach to an overall media service architecture will lower costs and shorten implementation time.
Governance
One last critical element in the discussion of media services is governance. Governance brings logic and structure to media services. Areas typically in governance include: service life cycle (how services are developed, deployed, deprecated and eventually decommissioned); helping to prioritize the deployment of new services; and how the quality of deployed services can be ensured.
FIMS
There is a task force in the industry called the Framework for Interoperable Media Services (FIMS). FIMS is a collaboration between the Advanced Media Workflow Association and the European Broadcasting Union. FIMS is the first industry effort focused on developing services for the professional media industry. The FIMS group consists of a Business Board that develops business priorities for service development, and a Technical Board that oversees the development and deployment of FIMS services.
You can learn more about FIMS at http://www.fims.tv. For technical information, you can visit the FIMS wiki at http://wiki.amwa.tv/ebu. This activity has already yielded an overall framework for media services and several specific media service definitions. Work is ongoing, all the work is public, and anyone can participate.
By Brad Gilmer, Broadcast Engineering
Testing FIMS 1.0
The rapid pace of change in the media industry demands that leading media organizations implement efficient and agile digital media processing capabilities to control costs and capitalize quickly on new business opportunities. However, achieving the full promise of file-based media workflows and the efficiencies they bring requires seamless interaction among products from different vendors.
Therein lies the challenge: Getting disparate products to interoperate is like getting a group of individuals each speaking a different language to understand each other. The effort requires standards for more than just the words or the dialogue — i.e. the file formats. Seamless interaction also requires standardizing the way the dialogues are handled — i.e. the approaches to coordinating and connecting the components of media processing systems.
Most modern businesses today leverage Service-Oriented Architectures (SOAs) to assemble agile business systems. Indeed, SOAs offer tremendous advantages to processing media for playout and distribution. A key element of SOA involves exposing system components as shared network-based services. Connections with services can then be created and reconfigured quickly to respond to changing requirements and demand.
The Framework for Interoperable Media Services (FIMS) was initiated as a joint project between the Advanced Media Workflow Association (AMWA) and the European Broadcasting Union (EBU) to create a common framework for applying SOA principles to processing media.
FIMS standardizes service interface definitions for implementing media-related operations in a SOA. As one example of how it can be used, a system that ingests and prepares content for playout and nonlinear delivery can leverage FIMS to interact flexibly with media capture, transfer and transform products from multiple vendors. Using FIMS, system components can easily be added, updated and removed in response to changing business requirements and demand.
The Problem
Although workflows are commonplace for moving, processing and storing media using software-centric systems deployed on commodity IT infrastructure, many software-based media processing systems suffer from problems endemic to older hardware and physical media-centric film and video processing systems.
Whenever unique media processing requirements must be addressed — and let’s face it, what broadcaster does not have a unique requirement of some sort — there’s an expensive custom software integration project needed to delivered bespoke hardwired media processing silos.
Worse yet, many software-based systems use watch folders to hand off media between components of the system. While watch systems can allow multivendor interoperability and loose coupling, they create a whole new set of quality, reliability and management problems.
The Approach
FIMS focuses on the abstract service layer that connects applications and orchestration systems with services that perform operations on media like capture, transform and transfer. FIMS addresses media specific requirements associated with the resource-intensive nature of processing and storing media.
For example, FIMS accommodates long-running processes by incorporating an asynchronous calling pattern and resource prioritization through queuing. The FIMS 1.0 specification defines services interfaces for capturing, transforming and transferring media, with ongoing development efforts under way to define additional services.
With FIMS, a workflow that receives media, transcodes it into the required formats and then transfers the result for linear playout and nonlinear distribution can easily be automated through standards-based service orchestration.
To those unfamiliar with SOA and Web services, the breadth of products and technologies marketed by big IT vendors can be intimidating and overshadow the underlying principles. The FIMS 1.0 specification avoids much of this complexity by defining a resource model as a central component of interaction between service providers and consumers.
The resource model approach places more emphasis on what is being communicated between the service consumer and provider than on how it is being communicated. This allows a neutral position on Web services technical debates such as those concerning SOAP versus Representation State Transfer (REST).
The main resources in the FIMS 1.0 model are Services, Jobs and Media Objects. Jobs are processed by Services to perform operations on Media Objects. In the context of the resource model, the Media Objects are formally called Business Media Objects. This qualification is made because these objects only contain the metadata relevant to the business operation being performed along with information about how to access actual media content.
FIMS 1.0 also makes a clear distinction between Media Objects on the message bus and the media bus. The term bus is borrowed from computer architecture where a bus is a subsystem that transfers data between components of a system. The message bus is used to communicate information about media, including media processing instructions. The media bus is used to store, access and move master, mezzanine and finished media products. This model is analogous to the management of physical products where business systems track raw materials, components and inventory for associated real-world objects that are stored and transported using warehouse, ships, planes and trucks.
FIMS Test Harness Project
A practical approach to standardization requires more than object models and interface definitions to gain serious traction. Consequently, the FIMS Technical Board recognized the need for a tool to help FIMS implementers validate their implementations.
Signiant did as well, given the test-driven development methodology we use to develop software. Using such an approach, developers create and code tests to validate an implementation before coding it. This leads to higher quality products that can be evolved quickly.
Signiant initiated the FIMS Test Harness Project within the FIMS Technical Board and helped design and build a test harness for contribution to the FIMS effort. The ultimate goal of the test harness is to remove barriers to FIMS adoption and promote implementation of FIMS-compliant interfaces.
Through discussion among the technical board, it was agreed that the test harness should consist of a set of service consumer and service provider simulators. Doing so facilitates interaction between third-party products, which is a key benefit of FIMS.
The simulators allow FIMS implementers to independently test their service consumer or provider implementation. These independent tests give vendors confidence that their products will interact properly in real-world conditions.
Completely data-driven testing was another goal of the test harness project. As such, test cases and associated service behaviors are specified with test configuration data. Service consumer configurations specify a sequence of commands to invoke against a service provider. Service provider configuration specifies how the service responds to specific commands with mock behavior. This approach helps ensure that no coding is required to use the test harness and create new test scenarios.
In addition, the test harness project sought to leverage an existing Web services test framework. Broad availability of standards-based testing tools is a significant benefit of a standards-based SOA approach. Through a series of proposals and discussions in the technical board, the decision was made to use the soapUI test framework.
SoapUI is an open-source Web service testing application for SOAs. The functionality of soapUI covers Web service inspection, invoking, simulation and mocking, functional testing, and compliance testing. As such, the soapUI framework covers most of the service consumer behavior, and new code was only required to implement mock service provider behavior.
Despite its name, the soapUI framework supports both SOAP and RESTful interactions between service providers and consumers. SOAP is a remote procedure call (RPC) standards-based approach to Web services that references a stack of related standards. The REST approach is conceptually simpler than SOAP and uses standard HTTP operations to interact with resources.
FIMS 1.0 formally defines SOAP bindings for services and, although the resource-based approach allows for RESTful interactions, work is ongoing in the FIMS Technical Board to further specify a REST interface. The test harness design allows a REST interface to be easily plugged in on top of existing mock service implementations.
A first version of the FIMS test harness has been contributed to the FIMS initiative for free use by the FIMS community. The FIMS test harness is an easy way to gain practical experience with FIMS, including working with FIMS calling patterns and the FIMS resource model. It provides valuable feedback to both vendors implementing FIMS and users wanting to start working with FIMS. The FIMS community is encouraged to utilize and help expand the test harness through the implementation of additional test cases and behaviors.
Toward Lower Costs, Greater Agility
The full promise of file based workflows requires seamless interaction amongst products from multiple vendors. FIMS applies SOA principles to media to facilitate an important part of this interaction. The FIMS Test Harness Project provides a practical starting point for working with FIMS.
Adoption and implementation of FIMS brings the media industry a big step closer to file-based workflow’s promise of lowering costs and ultimately improving business agility.
By Ian Hamilton, Broadcast Engineering
FIMS 1.0 Jointly Published by EBU and AMWA
FIMS - the Framework for Interoperable Media Services - is an open standard for Service Oriented Architecture (SOA). An SOA-based approach replaces the tightly coupled devices and functions found in traditional systems with a set of "Media Services" that are interoperable, interchangeable and resuable. The interfaces between these different services and the centralized system that runs them are defined by FIMS.
Broadcasters that adopt FIMS will be able to overcome expensive, disruptive incompatibilities in IT-based broadcast production technologies. It will be easier for them to adapt to future delivery formats and platforms.
The FIMS 1.0 specification is now available, comprising Part 1, the General Description, and Part 2, a multi-section document describing the Base Schema and the Transfer, Transform and Capture Services. An accompanying package of schema files is also available for download.
The documentation and files are available from both the EBU (as Tech 3356) and AMWA:
Part 1: General Description
Part 2, S0: Base Schema
Part 2, S1: Transfer Service
Part 2, S2: Transform Service
Part 2, S3: Capture Service
XML Schemas
Source: EBU
FIMS and SOA
Broadcasters no longer have the monopoly on the delivery of A/V entertainment to the home. In the fiercely competitive world of media and entertainment, companies have to deliver more versions and formats, but without increasing their costs. A channel is now expected to have a Web presence, as well as mobile and tablet versions of their content.
Organizations like the EBU and Advanced Media Workflow Association (AMWA) are promoting the Service-Oriented Architecture (SOA) as a route to provide the interoperable media services that can serve the new business requirements.
For the seasoned video engineer, the world of SOA introduces terms and concepts that at first encounter seem foreign and more suited to the IT specialist. As video processing migrates to the file domain, there is no option but to become familiar with what at first sight may appear alien.
Broadcast systems have evolved around the imposed workflow of the serial processing steps of videotape operations. Over time, many processes have moved from dedicated hardware boxes with SDI in and out to software applications on a network. A typical broadcast operation is now a hybrid of SDI and IP connections.
In many cases, the workflow remains as the original tape-based flow. Over time, other applications like asset and workflow management are layered over the entire process chain. The system has grown by accident, not by design, and become a web of custom or proprietary interfaces linking the many applications.
Sure it works, it was designed that way, but when the time comes to replace a component part — say the playout automation — the inflexibility of the system rapidly comes apparent. The parts of the system are linked by a web of custom APIs, often restricted to a specific release of a specific software application. It is just not possible to swap out the automation for the latest product without attending to the web of interfaces.
To meet the demands for new services to the public, the broadcaster must add facilities for a mobile news service, a 3-D channel and interaction with a social media website. Along comes CES and some new consumer device to consume content. How do you add support for this new device? Will it mean more custom interfaces or more special workflow applications? The EBU and AMWA are developing a Framework for Interoperable Media Services (FIMS), which aims to provide a new technology platform that leverages current IT practices, like the use of the SOA, to provide business agility and to control costs.
The legacy system architectures suffer from many problems, and these scenarios serve to illustrate just two.
Scenario One: Ingest
Take the example of ingest. A VTR connects to a video noise reducer via HD-SDI. This then connects again via HD-SDI to an encoder card. The card encodes to a suitable codec for editing and also creates a low-resolution proxy.
At some point this is updated so that the encoder card ingests directly from the VTR and encodes to an uncompressed format. The noise reduction is performed in software. The uncompressed file is saved in a watch folder and picked up by transcoder software, which creates files in the wanted resolutions and codecs.
Imagine more control is needed over the clean up, and the file is passed to an operator who will apply craft skills to the clean-up and repair process. Any changes to the process flow involve wiring changes and software reconfiguration.
The system is just not flexible or agile enough for the constant changes needed to serve today's requirements.
Scenario Two: System Upgrades
A broadcaster has a transcoder that is interfaced to a DAM. The broadcaster starts with Transcoder A and DAM A. The transcoder is replaced by a later model from a different vendor, Transcoder B. This requires a new interface to the DAM. The broadcaster migrates to a new DAM B, so the interface to Transcoder B must be rewritten.
The broadcaster liked the compression quality of Transcoder A and decides to move back, but must pay for another interface to the new DAM. Next, the vendor for Transcoder A has to redesign its product to move from a 32-bit to 64-bit operating system. In the process, the API is upgraded, and a new SDK is issued. The custom interface must be similarly upgraded.
Using two transcoder vendors and two DAMs has required the development of five custom interfaces over time. This adds up to costs for professional services, and for the manufacturer, the opportunity cost of not using software resource to develop new products. It is an unsustainable business model.
Integrating an SOA
The implementation of an SOA in a media business must take into account the special needs of the processes that manage A/V files. Off-the-shelf IT products do not cater for these additional requirements without extensive customization. The work to develop FIMS will create a set of standards for vendors to create compliant products and save unnecessary professional services.
This melding of IT with a media-savvy system means that the video engineer and IT specialist can collaborate and share their skills in the creation of products and systems that meet the new demands of the media and entertainment sector.
Services
The linear nature of legacy production is implemented by serial processes acting on the content: ingest, editing, transcoding, distribution, playout and archive being typical. It may be that content is transcoded at ingest — AVC-Intra to DNxHD, for example — and then transcoded after editing to different delivery formats. This could be done by a transcoder in the edit bay and a transcoder in master control.
In a file-based environment, devices all sit on the media network. It makes sense to pool all the transcoders as a central resource and make them available as a transcode service. It could even be that cloud transcoding could be used. This offers much more efficient use of transcoder resources, but for it to work, the service should be reusable at any point in the workflow and should interoperate with all the necessary equipment in the organization, whatever make or model.
This calls for standardization so that a transcode service has a common interface, whatever make or model, and capabilities. A given transcoder may not support all codecs, but the interface can still be common. If demand increases, new transcoders can be added; the architecture scales easily.
The move to an SOA involves a change of mind-set to view processes within the organization as services. Since most post houses have a rate card of services they offer, it is not a great leap to understand this concept. The services include ingest editing, grading, dubbing, encoding, sound mixing, etc. Note that some services require extensive and lengthy use of creative staff; services are not only computer processing operations like transcoding.
Encapsulation
A post facility can take a videotape and return a copy of the tape encoded on a Blu-ray disc. All the detail of the process is hidden. You don't need to know which tape deck was used or what software was used to burn the disc. It is all abstracted from the client. The facility performs a service and bills for that service; it is a business transaction.
In an SOA, this concept of abstracting, or encapsulating, the detail of the process is key. If the post house starts using different software, or a different VTR, you don't care as long as the disc is a faithful copy of the tape.
An SOA uses loose coupling for services, possibly a foreign concept for video engineers used to real-time systems. But many services are already loosely coupled. Captioning is one example. A low-resolution proxy is sent out to the captioning facility, and at some point in the future a caption file is returned. The process is managed at a business level by a scheduling department.
Service Adaptors
Many existing products need the user to control the service via a custom API using an SDK provided by the vendor. In an SOA, services are linked to the middleware via service adaptors. These provide the abstraction from the implementation of the service — how it works — to deliver the service at the business level.
The FIMS project is working to define standard interfaces for common media services. A system with FIMS components contains two broad service categories: workflow services able to realize a given business goal (media workflow services), and infrastructure services that are essential components of the media SOA system (media infrastructure services).
The first three workflow services to be developed are:
- Transfer (moving files)
- Transform (changing essence or metadata)
- Capture (ingest stream to file)
It is envisaged further services will be added to the list as dictated by demand from vendors and users.
Building an SOA
In all but the smallest facility, some middleware is required to manage the services, scheduling resources and the workflow. To link services together to implement broadcast workflows, the SOA middleware orchestrates job requests, calling appropriate services to satisfy a request. There are many SOA middleware systems, but few are “media aware.” It is hoped that FIMS will make it easier for several manufacturers to offer compliant products.
One important term is “web services.” Web services are used for communications between the many components of the SOA implementation. SOA is just an architecture — a methodology; it is not a software application.
Web services have become a commodity, with companies like Amazon and Google offering storage, processing and many other services in the cloud. Web services are not exclusive to the Web, but can equally be used on the company LAN.
Web service standards have two interaction styles: SOAP/WSDL and RESTful. RESTful only operates over HTTP. SOAP is agnostic to transport protocol, so it can use other protocols like TCP. SOAP was designed for distributed computing, whereas RESTful is a lightweight protocol for point-to-point communications. Both can be used to bind a service provider with a service consumer. FIMS uses an object model described by XML schemas, which provides independence from the method used to bind services.
Media-Centric Issues
File-based media systems need to allow for certain issues not found in typical SOA implementations. One is the very large size of media files, terabytes in some cases. Media files cannot travel as attachments on a conventional enterprise service bus, the interconnection bus of an SOA. Instead a separate media service bus is used. This concept is common in broadcast facilities where a high-bandwidth media network carries files separately from the main network, which carries control messages, e-mails etc.
A typical SOA service, like converting a Word document to PDF, happens in seconds. Media processes can take hours or even days. Such operations must happen asynchronously from other processes. Operations such as transcoding must be scheduled to business rules to ensure resources are not used for a low-priority job, holding up a high-priority job.
A resource manager service must manage use of services according to business rules that manage the needs of broadcast operations.
The media bus is a potential bottleneck to scaling of services, so SLA monitoring is vital to ensure efficient running of the media systems. To the video engineer, used to non-blocking systems, with dedicated circuits and crosspoint routers, it is easy to keep track of bandwidth requirements. In a file-based environment, care has to be taken over the use of the media bus though continual monitoring of capacity and performance.
Summary
The AMWA and EBU have now joined with SMPTE to move forward standardization of the framework. After formalization by AMWA and EBU, the work will be submitted to SMPTE.
The standards work is not following a big-bang approach, but progressing in steps, with usable specifications released at regular intervals so that vendors and media companies can start deploying systems immediately, rather than waiting for a long, drawn-out standards process, which has been the case in the past. Businesses have to move now; there is no option to wait for an all-encompassing answer to everything in the media factory. By releasing the framework, and then key adaptors, further work will proceed as the demand arises from the community.
The move to an SOA, and to follow the FIMS route, promises many advantages to the broadcaster. The architecture is more flexible and scalable than legacy systems. The broadcaster can more easily outsource to external services or cloud provision. And finally, the system — through the dashboard — gives management a better view of key performance indicators of the business. The system agility and the better control gives management the ability to invest in new services and to improve the efficiency of existing operations.
By David Austerberry, Broadcast Engineering
AMWA/EBU FIMS Reports First Results and Roadmap
During IBC, EBU and AMWA organised a round table session around SOA and the Framework for Interoperable Media Services (FIMS) project jointly coordinated by the two organisations. During the round table, which comes after a successful meeting with respondents Amberfin, BBC, Cinegy, IBM, and Sony to its request for technology, FIMS was privileged to announce the following roadmap for developing specifications.
Phase 1 will consist of a common service definition format, after the high level architecture and framework described in the request for technology is first refined. This framework will cover all system and management requirements (service management, awareness and communication, content and time awareness, security, framework extension). The framework will be built upon IBM (SOA based Media Services Framework) and Sony (Media SOA Framework) proposals with the valued experience of service developers and users from Amberfin, BBC and Cinegy. The project will also address container issues seeking maximum compatibility with AAF and MXF.
Phase 2 will subsequently investigate the possibility of defining common services using the framework developed in phase 1. All respondents have already suggested key services. It is very encouraging to see that these contributions have been received in the framework of the AMWA licensing policy, which is compensation free. This is essential to the successful standardisation of the core technology on which the project is working.
The work will continue in a public manner via the FIMS wiki and open meetings that will allow third parties to also actively contribute to this important work. Direct participation will be subject to the signature of a ‘participation agreement' to safeguard the favourable licensing conditions under which FIMS operates.
Source: EBU
Subscribe to:
Posts (Atom)
