1. Background, Vision and Ambition

Music is probably the most pervasive of the performing arts, and perhaps, the most abused (think of your recent shopping trips!). It has tremendous power to influence our emotions, often subliminally. The advent of recording in the 19th Century made it possible to enjoy music at a time, and in a place, different from the performance. Compression, broadband and the ever increasing capacity to aggregate large collections mean that the issues confronting music consumers have totally changed in nature: equally so for professionals, such as broadcasters (playlists for radio, music for documentaries, etc.) and those at the creative heart of the process: musicians, sound engineers and producers. The recorded music industry has grappled unsuccessfully with digital technology and the rate of adoption of new technologies has been slow, ironically, mostly in fear of piracy and loss of revenue. Given the social and economic importance of music, it is vital that the industry’s crisis is averted and its decline reversed. Simple semantics and metadata are already helping (for example in recommendation and sharing services) but this is just the beginning. The next generation semantic technologies that are the focus of this proposal have the power to exact the turnaround that music (and other content industries) needs but this should be established via a fundamental and principled exploration of how semantic technologies underpin music throughout the value chain.

This project brings the very latest technologies to bear on the entire industry, end-to-end, producer to consumer, making the production process more fruitful, the consumption process more engaging, and the delivery and intermediation more automated and robust. In this project we will address 3 premises: (i) that Semantic Web technologies should be deployed throughout the content value chain from producer to consumer; (ii) that advanced signal processing should be employed in the content production phases to extract “pure” features of perceptual significance and represent these in standard vocabularies; (iii) that this combination of semantic technologies and content-derived metadata leads to advantages (and new products and services) at many points in the value chain, from recording studio to end-user (listener) devices and applications.

This project is working with a range of external partners and collaborators from industry in the UK such as BBC R&D, Microsoft Research Cambridge, Omnifone, Abbey Road Studios, Universal Music and Solid State Logic, as well as internationally – the International Audio Labs, a joint initiative of the Fraunhofer Institute in Erlangan and the local university, and the Internet Archive, one of the world’s major on-line libraries. We are also engaging with other universities in the UK supported by a partnership fund and via the BBC Audio Research Partnership.

The project will foster new ways for professionals to work with music in the studio and for consumers to engage in their homes. It will support new business models that emphasise the whole experience of musical involvement, and discover ways to monetise the metadata as well as the essential content. The technologies researched support new ways of learning (about and to play) music as well as new ways of teaching and performing. And because the project encompasses vast quantities of music data and metadata, from heterogeneous sources, and stresses test emerging principles of big data, distributed intelligence and future generation web, it also addresses key questions of wide significance to EPSRC’s ICT Programme, particularly relating to Intelligent Information Systems and Working Together.

  1. Project Objectives

This five-year research project sets itself many ambitious goals that can be summarized in the following way:

“(How) Can next generation web technologies (Ontologies, Linked Data and Metadata) combined with music content analysis (particularly derived at source) bring new value and functionality to producers, creators, consumers and intermediaries of music content?

(How) Will both ends of this value chain benefit from more engaging interactions (enhanced productivity, increased enjoyment and immersion) while creating or consuming music, and can intermediaries add value with semantically enhanced services?

(What) Can other areas of science and ICT learn from end-to-end digitisation and next generation technologies adapted in the music industry?”

The project’s objectives (captured in the goals listed above) are:

* To bring together the UK’s leading experts in Audio & Music Technology, Semantic Web, e-Science and HCI;

* To move beyond fundamental research and to pursue radical innovations amidst challenges driven by the practical needs of end-users throughout the music industry from studios to sofas, with exciting proof-of-concept demonstrator projects that collectively form an intelligent music information infrastructure;

* To provide a multi-disciplinary, trans-institutional research programme to nurture the next generation of UK researchers with skills that span all the technologies and sciences relevant to next and future generation music and audio production and consumption eco-systems;

* To test the hypothesis that Semantic Web technologies combined with content analysis at the point of content creation will deliver real and lasting change to the music industry;

* To examine the benefits of treating tags, annotations and usage data as forms of crowd-sourced meta-content, with its own evolution and meaning, both dependent on and independent of the musical essence itself;

* To treat creative and production processes on a par with content, with intrinsic value, separate from but complimenting the musical essence.

  1. Research Programme and Methodology

The overall programme of work is divided into Work Threads (WTs) that constitute the major intellectual and research challenges. Woven through these are Demonstrator projects that individually integrate the challenges from the Threads, as well as affording the mechanism by which researchers employed by the different partners work together on common goals. There are seven technology focused Work Threads which will deliver world leading research. The 8th WT, on Ethnography both informs the work under the others, and evaluates their outcomes.

Diagram representing the Work-Threads and their interrelationships

Diagram representing the Work-Threads and their interrelationships

WT.1 Ontologies develops new ontologies for music applications, building on musicontology.com, integrating with other emergent ontologies, to cover an increased range of concepts important to the creation, consumption, transmission and understanding of music.

WT.2 Inference deals with derivation of high-level musical knowledge from basic musicological information, and with understanding of underlying content, using inference over RDF representations. This facilitates new search and recommendation functionality, as well as enhanced learning and immersive, contextual rendering.

WT.3 Signals studies advanced signal processing techniques to extract high quality features from audio, across a range of temporal scales from complete songs to individual notes. Areas of interest include structural segmentation, chord/harmony, rhythm/metre, intonation, vibrato and ornaments.

WT.4 Workflow takes established e-Science techniques (see researchobject.org) and applies them to semantic computational workflows in the creation and processing of music. These workflows, which underpin Music Digital Objects (MDO) in WT.7, describe the pipeline of (signal) processing that deliver new data and content, so that processes can be repeated, reused and repurposed.

WT.5 Interface researches and develops new user interfaces and interactions adapted to the new modalities for professionals and consumers. We will conduct research on context-aware sonic interfaces. This combines automatic audio mixing with psychoacoustics to render audio scenes that account for aspects like ambient noise and reproduction device properties, as well as personalisation.

WT.6 Social examines the role of social media and metadata in music creation and co-creation, and develops a concept of narrative, based on ontological principles, that can be applied to the evolution of the metadata in MDO through the content life-cycle. It also encompasses the use of social media in “citizen musicology”, combining tags with signal-derived features to harvest value from the deepest recesses of even the largest music collections. There will be research into the impact of music on physical experience, with multi-modal sensor data capture. Use of ontologies enables inference and reasoning over all value- chain metadata.

WT.7 Digital Musical Objects DMO can be shared, edited and executed by people or machines in the value chain. They can be analysed as a means to understand the creative process of music creation, form the raw material for future creative processes, and form an important element of an enhanced commercial offering to consumers.

WT.8 Ethnography & Design encompasses user-centric design, enabling the team to design appropriate demonstrators and formally evaluate them in real use scenarios, including UIs developed in WT.5. Ethnographic studies of music production and post-production processes form another important aspect, with focus on workflow, feeding into WT.4, highlighting metadata usage practices.

The demonstrators not only enhance impact, they act as a cross-cutting mechanism to drive integration across WTs, to inform one another and to foster development of each individual discipline in the context of a cross-disciplinary research agenda. They fall under 3 headings: Production – focus on the studio production process; Distribution – focus on transforming how music is distributed; Experience – focus on how semantic media will change music consumption.