Declarative programming of TV applications using NCL

1.Introduction

The convergence of TV systems should be supported not only at low-level network layers (internet, broadcast, cable), but also at the application layer. It requires an application programming language able to integrate and coordinate several media objects in a simple and synchronized manner. The solution can come from languages using the declarative paradigm. Nested Context Language (NCL) [5[5] ITU 2009 H.761: Nested Context Language (NCL) and Ginga-NCL for IPTV Services. Technical Report. ITU, Geneva, Switzerland.] is a declarative XML-based language initially designed for hypermedia document specification for the web. The language’s flexibility, facilities, and mainly its intrinsic ability to easily define spatiotemporal synchronization among media assets, including viewer interactions, make it an outstanding solution.

In 2007, NCL and its player, called Ginga, was adopted in the Brazilian terrestrial DTV standard, SBTVD. At the beginning of 2009, they part of an ITU-T11.ITU Telecommunication Standardization Sector Recommendation for IPTV services [3[3] Roberto Gerson Azevedo, Eduardo Cruz Araújo, Bruno Lima, Luiz Fernando Soares, and Marcelo F. Moreno 2014 Composer: Meeting Non-Functional Aspects of Hypermedia Authoring Environment. Multimedia Tools Appl. 70, 2 (May 2014), 1199–1228. ]. NCL and Ginga were designed at the TeleMídia22. http://​www​.telemidia​.puc​-rio​.br Laboratory at the Pontifical Catholic University of Rio de Janeiro (PUC-Rio), which offer an open-source Ginga implementation.33. http://​www​.github​.com​/TeleMidia​/ginga Independent of the distribution network where they may be applied, all Ginga and NCL specifications are open and royalty-free.

Recently the Brazilian DTV system standards have been upgraded in order to address new use cases related to deeper integration between broadcast services and broadband services. Such an evolution could not disrupt the current DTV services since Brazil and many ISDB-T countries are still under the switch-off process from analog to digital. The middleware layer is the best candidate for such an incremental (yet powerful) evolution. In this way, Ginga new receptors D-profile was created and the new features include: new media players for adaptive streaming video formats (MPEG-Dash and HLS); digital copy control through DRM; support for downloaded fonts; new preparation media event; and video buffer control.

There are other declarative methods for programming TV applications from other interactive TV standards. All of them are HTML5-based technologies. Hybridcast44. http://​www​.iptvforum​.jp​/en​/hybridcast is standardized in Japan as the second generation of multimedia coding specifications that succeeds in the first one, based on BML (HTML-like). The HbbTV55. http://​www​.hbbtv​.org specification is based on web existing standards. It is a proposal that succeeds its former European MHP interactive standard and added support for some kind of media synchronization, companion devices, and new media types. ATSC66. http://​www​.atsc​.org​/standards​/atsc​-3​-0​-standards refers to digital television standards developed in the US. ATSC has been designed with HTML5 and IP as the key components of IBB services.

In this paper, we focus on introduce NCL and its recent advances addressed by the D-profile receiver. These specification efforts were carried out in the Brazilian DTV Forum under a collaborative effort led by the Academia. The paper is organized as follows. Section 2 introduce the NCL programming. In Section 3, we discuss Ginga and D-profile features. Finally, Section 4 presents our final remarks and future work.

2.NCL

NCL is today the standard declarative language for the specification of interactive applications in the Brazilian digital TV system [1[1] ABNT 2016 ABNT NBR 15606-2:2016 Televisão digital terrestre – Codificação de dados e especificações de transmissão para radio-fusão digital Parte 2: Ginga-NCL para receptores fixos e móveis – Linguagem de aplicação XML para codificação de aplicações.] and ITU-T[5[5] ITU 2009 H.761: Nested Context Language (NCL) and Ginga-NCL for IPTV Services. Technical Report. ITU, Geneva, Switzerland.]. It is used for application coding in both fixed and mobile TV receiver (Figure 1). Its main characteristics are: defining temporal synchronization among media assets and viewer interactions; layout reuse facilities; support multi-device presentation; support embed HTML code and scripts in the lightweight scripting language Lua; and an API for life-cycle controls (start, pause, resume, stop) and modifying applications on-the-fly called NCL editing command.

NCL 1.0, was developed in 2000[6[6] M. J. Antonacci, D. C. Muchaluat-Saade, R. F. Rodrigues, and L. F. G. Soares 2000 NCL: Uma Linguagem Declarativa para Especificação de Documentos Hipermídia na Web. In VI Simpósio Brasileiro de Sistemas Multimídia e Hipermídia – SBMídia 2000.], and the most recent version, NCL 3.0, was launched in 2006[1[1] ABNT 2016 ABNT NBR 15606-2:2016 Televisão digital terrestre – Codificação de dados e especificações de transmissão para radio-fusão digital Parte 2: Ginga-NCL para receptores fixos e móveis – Linguagem de aplicação XML para codificação de aplicações.]. An NCL program is an XML (eXtensible Markup Language) code that describes an interactive multimedia presentation. It can be seen as a glue language, which holds media objects together in a multimedia presentation. NCL only defines how media objects are structured and related in time and space, but does not restrict or prescribe any media object content type. It is based on NCM (Nested Context Model)[8[8] Luiz Fernando Gomes Soares, Rogério Ferreira Rodrigues, and Romualdo Monteiro de Resende Costa 2006 Nested Context Model 3.0 Part 6 – NCL (Nested Context Language) Main Profile (2006).]. This model first version dates from 1991 and consists of a conceptual model for the specification and presentation of hypermedia documents. NCM combines the ideas nodes and hypermedia links with the concept of nested contexts, which allows the scope definition and better organization of the information represented. Figure 2 presents the main entities of NCM. In an NCL application, there are a set of <media> objects and a set of synchronism <link> elements, which determine the object’s behavior during the presentation.

Figure 2.NCM main entities. Source:[7[7] Luiz Fernando Gomes Soares and Simone Diniz Junqueira Barbosa 2011 Programando em NCL 3.0 2a. Edição Versão 2.1 (1 ed.). Elsevier Campos.]

A <media> element may be image, video, audio, text, imperative, other declarative object, and so on. It is defined by type, content location, and properties. Some special types are predefined by the language: application/x-ncl-settings for global variables defined by the programmer or reserved NCL Player; application/x-time for Universal Time Coordinated (UTC); application/x-ncl-NCL for another NCL embedded application; and imperative media object types application/xncl-NCLua for code written in Lua or Java languages. It is worth mentioning that NCL treats main audiovisual streams like all other media objects it can relate. Moreover, it treats an HTML document as one of its possible media object types. Therefore, NCL does not substitute but HTML but integrates its documents.

Media objects and links are grouped in contexts, which can be nested, enabling better code organization. A <context> is a composite that contains <media>, <context>, <switch>, and <link> child elements. The <switch> element allows for the definition of alternative objects to be chosen in presentation time when testing rules. The <descriptor> element specifies temporal and spatial information needed to present each media object, including the initial position on the screen (<region>).

The <link> element defines temporal relationship between nodes. They should be based on reusable behaviors called connectors. In NCL 3.0, the main connector is <causalConnector> that defines a set of conditions that need to be satisfied to trigger a set of actions. The conditions act over nodes and may test: occurrence of media an event (presentation occurring); assessment statement to compare nodes attributes; user interactions give a specific key; or logical expressions using “and” or “or” operators through; Actions change objects presentation (start, pause, resume, stop) and attributes.

In order to illustrate an NCL use case, Listing 1 shows a simple application that presents a sequence of video, audio, and image. This sample application is organized in two parts: the <head> with reusable components (rgCenter <region>, dCenterFade <descriptor>, and onEndStart <causalConnector>) and <body> which define the application behavior (<media> and <link>). Lines 18–20 defines the three <media> video1, audio1 and img1. The visual ones use the dCenterFade descriptor to be presented on the center screen with a fade effect. video1 is stated by the entry <port>, while the first <link> (20–24) and second link (lines 24–28) define the start audio1 and img1.

2.1Authoring Tools

Given its XML syntax, there are different tools to develop NCL applications. In particular, TeleMídia lab provides some authoring tools to facilitate it, namely: NCL Eclipse, ISB Designer, and NCL Composer (see Figure 3). The NCL Eclipse is a plugin for the Eclipse IDE that provides facilities to help programmers develop NCL applications such as code suggestion (autocomplete), code navigation through hyper-links, programmatic visualization, and syntactic verification. ISB Designer [2[2] Eduardo Cruz Araújo and Luiz Fernando Gomes Soares 2014 Designing IDTV Applications through Interactive Storyboards. In Proceedings of the 20th Brazilian Symposium on Multimedia and the Web (João Pessoa, Brazil) (WebMedia ’14). Association for Computing Machinery, New York, NY, USA, 5–12. ] is a tool to support the project interactive narratives based on the concept of interactive storyboards – an extension of traditional storyboards used by designers and TV professionals for a description of narratives. The advantage of interactive storyboards is that they are easy to edit, and allow the author to build nonlinear narratives gradually (containing still incomplete artifacts) and independent of a target language-specific. ISB Designer also allows editing and viewing of strings interconnected by non-deterministic events, each of which storyboards with points of unpredictable events (for example, the interaction of the user). The NCL Composer [3[3] Roberto Gerson Azevedo, Eduardo Cruz Araújo, Bruno Lima, Luiz Fernando Soares, and Marcelo F. Moreno 2014 Composer: Meeting Non-Functional Aspects of Hypermedia Authoring Environment. Multimedia Tools Appl. 70, 2 (May 2014), 1199–1228. ] is an authoring environment for NCL applications that facilitate the use of the concepts defined by NCM and NCL by non-programmers. In addition to the textual view, which brings features similar to those of NCL Eclipse, NCL Composer also brings different graphical views, such as the structural and layout view, which allow non-programmer users to use visual abstractions for the development of multimedia applications.

3.Ginga Architecture

Ginga, the NCL Player, was initially proposed for terrestrial DTV systems, then the same architecture and facilities were applied to IPTV. Its modular architecture also allows for use with other transport systems. It was recently upgraded in order to address new use cases related to deeper integration between broadcast services and broadband services. In this way, Ginga’s new receptor D-profile was created. Figure 4 depicts the architecture this new profile. In particular, Ginga-NCL and Ginga-CC (non-dashed compponents inside the Ginga Middleware box) are mandatory in any receiver profile.

Ginga-NCL subsystem is in charge of receiving and controlling multimedia applications written in NCL. Ginga-CC (Common Core) subsystem is a reusable layer responsible for abstracting receiver vendor components: Media Players have to render contents delivered in different networks (broadcast and broadband) accessed by a receiver; SI (Service Information) processes the metadata transmitted by transport stream, such as the SI and PSI (Program Specific Information) tables inside a MPEG-2 TS77. https://​www​.iso​.org​/standard​/75928 ; DSMCC processes the data transmitted by the transport stream, such as applications and stream events. Persistence supports broadcasters to store applications and information in non-volatile memory and to install applications directly from the DSMCC to external memory (USB drive). In particular, Ginga-CC enables users to browse and launch applications.

TV manufacturers create different types of TV receivers (e.g. set-top-box, TV, SmartTV) with different hardware capabilities and prices. Therefore, the Ginga is defined by profiles of the receiver with some levels of functionality. We briefly describe the profile’s main components and changes in what follows88.For more details about the receivers profiles see[[1[1] ABNT 2016 ABNT NBR 15606-2:2016 Televisão digital terrestre – Codificação de dados e especificações de transmissão para radio-fusão digital Parte 2: Ginga-NCL para receptores fixos e móveis – Linguagem de aplicação XML para codificação de aplicações.]].

The new components proposed in D-profile receivers are illustrated as dashed elements in Figure 4. Ginga-HTML5 uses the W3C HTML5 Recommendation99. https://​www​.w3​.org​/TR​/html52/ and is compatible with the other web standards (HbbTV 2.0). The Ginga-CC-WebServices’ main goal is to promote an integrated experience of the broadcasting environment with the home and broadband environments. To do this, this component provides a set of services implemented in the form of REST APIs. Thus, as indicated in Figure 4, its functionalities can be used by: native Smart TV applications, web-based applications, and Ginga applications in the DTV receiver; devices present in the home environment, allowing, e.g., second-screen scenarios. The way in which a home device determines the existence of a DTV receiver terminal with support for Ginga-CC-WebServices and obtains the access point to the API is through the SSDP protocol.

The D-profile also upgrade Ginga-NCL with new features. It allows the definition of <font> elements for downloaded fonts, new video playback control, and new APIs for Lua scripts. The new video control includes seeking and preparation capabilities equivalent to the Java video framework JMF. The seeking use the currentTime property in an NCL <media> and the preparation uses a new related event and actions. For instance, a startPrepration refers to instantiating and preparing an appropriate media player, including the loading part of media content into the player buffer The new Lua APIS include: inserts data into a streambuf <media> to implement adaptative streaming; http/socket server; zip compression; charset conversion; and enables filtering a particular type of MPEG-2 packetized elementary stream and DSMCC.

4.Final remarks

This paper introduced the declarative programming of TV applications using NCL and presents the Ginga advances addressed by the D-profile TV receiver. Regarding NCL, we must highlight the new support for adaptative streaming and the new NCL preparation event to enable seamless switch from broadcast to broadband. For instance, by performing a startPreparation action over a streaming media, an application can switch from broadcast content and smoothly render the broadband one. This way, the viewers’ perception of source changes is reduced, giving them a seamless experience.

D-profile also better support targeted advertisement/content by tracking users viewing experience from both SmartTV and Ginga applications. For instance, after watching a broadcasted content, a Ginga application may suggest viewers watch previous or related content in the broadband channel. Moreover, while watching broadband streaming, an SmartTV application may suggest viewers watch live broadcast content related. Another possible scenario is multi-device application using Ginga-CC-WebServices. By being accessible in the home network, Ginga can interact with other devices to improve the viewer experience. For instance, second-screen experiences can be facilitated and even goes outside entertainment. It may provide information usfeul for scenarios such as healthcare, education, and wellbeing. In particular, in the context of the Internet of Things (IoT), the DTV receiver can act as an IoT entity.

The NCL and Ginga evolution efforts were carried out in the Brazilian DTV Forum and ITU/SG-16(Q13) in collaboration among academia and industry. In the future, we may cite other scenarios to be addressed by NCL and Ginga: supporting Mulsemedia [4[4] Gheorghita Ghinea, Christian Timmerer, Weisi Lin, and Stephen R. Gulliver 2014 Mulsemedia: State of the Art, Perspectives, and Challenges. 11, 1 (2014), 17:1–17:23. ] application; supporting 5G broadcasting; and supporting other new augmenting video content such as scalable video and free viewpoint.