Category Archives: Blog

FAST C4DM PhD student organises Sound Source Separation Workshop (HOSSS)

Written by: Delia Fano Yela 

pic 1For a second year in a row, C4DM hosted the Hands-on Sound Source Separation Workshop (HOSSS) on Saturday 11th of May, and it was a great success! It originated in 2018 out of the student frustration of not having enough space to brainstorm in conferences, when all relevant experts are in the same place.


pic 6The idea is simple: get together and spend the day actively brainstorming/hacking some source separation related topics given by some of us (so no keynotes/talks/posters). Everyone is there to engage with each other and so everyone becomes much more approachable than in a traditional conference setting. This format therefore aims to inspire early-stage researchers in the field and promote the sense of scientific community by offering a common place for discussion.

pic 3The workshop started with an ice-breaking game followed by a quick presentation of all participants.  Then, those who wanted to propose a topic for discussion did so briefly, explaining their idea on a whiteboard. As they did, we created a list of topics on the side. Once everyone who wanted had the chance to present their idea/problem/interest to the group, we proceeded to vote.

Each participant could pick 3 topics they would like to work on for the rest of the day. In doing so, and after a little pruning, work-groups were formed. We then divided into groups and started the hands-on work on the topic. Some of the topics were more broad than others; a group ended up brainstorming on the nature of singing voice, where another was full-on hacking a deep clustering model for source separation recently added to the nussl python library by one of the participants, Prem Seetharaman from the Northwestern University.

pic 2After lunch, some people changed groups. For example, after the two “Antoines” (Liutkus and Deleforge) had a productive Gaussian Processes morning with our PhD candidate Pablo Alvarado, another of our PhDs, Changhong Wang, discovered the world of median filtering source separation lead by the hand by one of the very best, A. Litukus.

In the afternoon there was a real great feeling of productivity, everyone seemed to be “in the zone”. However, one can only concentrate that much, so at some point we all gathered to discuss how far we got in our quest and share what we have learned. We left the discussion of future avenues for the pub, were we all ended up the day to celebrate with our new colleagues!

FAST’ing with Robert Thomas

Over the summer Adrian Hazzard from the Mixed Reality Lab (Nottingham) spent some time with Robert Thomas. Robert is a leading music composer and experience designer with an extensive track record in the creation of music which adapts in real-time to the listeners situation – like a soundtrack to your life.

Robert has collaborated with Massive Attack, Imogen Heap, Hans Zimmer, Ben Burtt, Richard King, Tom Holkenborg, Carl Craig, Air, Bookashade, Jimmy Edgar, Mel Wesson, Little Boots, Chiddy Bang, Console, Sophie Barker (Zero 7) and Kirsty Hawkshaw (Opus III, Orbital, Tiesto) to name a few. Robert uses a range of techniques and approaches from adaptive systems, algorithmic, generative, stochastic composition, procedural generation and machine learning / artificial intelligence.

An on-going thread of Adrian’s FAST research seeks to understand of how artists, such as Robert, approach the design, composition, and realisation of such adaptive audio experiences. To study these processes at first hand Adrian invited Robert to create a prototype adaptive audio experience.

Robert, who was supported by his colleague Franky Redente, was presented with an open design brief, and after a process of discussion and research they decided to create a locative audio ‘app’ for Nottingham City Centre, where playback of the audio responds to a listener’s movements and orientation. An interesting feature of Nottingham City Centre are the caves that sprawl under the city’s streets. Intrigued by this, Robert used the caves as the principle design feature for this audio walk: There must be all kinds of stories of people’s lives who lived in or above those caves which are lost. We don’t really have that much information about the caves’ stories in detail, so imagining people’s lives in there might be interesting”. Building on this narrative, Robert wanted listeners to “hear below them into the underbelly of Nottingham”. The finished design presents a mixture of sound design elements and tense, textural music that represents the dark hidden world beneath, accompanying users as they walk around the city’s streets. One element of this audio treatment is dripping water panned in 3D that acts as an audio marker. By following the spatial placement of these drips (as sensed by a mobile phone’s compass sensor), listeners are guided along the streets to specific locales situated above the presence of a cave. Once a cave is reached listeners are greeted with rich arrangements of music composed by Robert, which portray these ‘lost’ stories.

Robert and Franky visited Nottingham for a couple of days, walking the streets and visiting the caves, where they recorded the ambient sound within them (such as the sound of dripping water) and also capturing impulse responses of the caves natural reverberation. Using these elements to drive the composition and authoring of the completed app.

Robert and Franky

Robert and Franky recording an impulse response in a Nottingham cave

While in Nottingham, Robert also visited the Mixed Reality Lab and gave a fascinating talk about his work, the challenges of music interaction design, alongside some example of his compositional approach using Pure Data.

If you would like to find out more about Robert Thomas and his work, follow the link here to his website –



Smart contracts for fair trade of music

by Panos Kudumakis & Thomas Wilmering (Centre for Digital Music, Queen Mary University of London)

Media Value Chain Ontology (ISO/IEC 21000-19) facilitates IP rights tracking for fair and transparent royalties payment by capturing user roles and their permissible actions on a particular IP entity. However, widespread adoption of interactive music services, e.g., remixing, karaoke and collaborative music creation, thanks to Interactive Music Application Format (ISO/IEC 23000-12), raises the issue of rights monitoring when reuse of audio IP entities is involved, such as, tracks or even segments of them in new derivative works.

Audio Value Chain Ontology (ISO/IEC 21000-19 AMD1), addresses the aforementioned issue by extending MVCO functionality related to description of composite IP entities in the audio domain, whereby the components of a given IP entity can be located in time, and for the case of multi-track audio, associated with specific tracks. The introduction of an additional ‘reuse’ action enables querying and granting permissions for the reuse of existing IP entities in order to create new derivative composite IP entities.

Audio Value Chain Ontology (AVCO) conceptualisation.

Furthermore, MVCO/AVCO smart contracts by facilitating machine readable deontic expressions for permissions, obligations and prohibitions, with respect to particular users and IP entities, could be used in conjunction with distributed ledgers, e.g., blockchain, enabling both transparency and interoperability towards fair trade of music.

For further info please visit MPEG Developments.


  • ISO/IEC Information Technology – Multimedia Framework (MPEG-21) – Part 19: Media Value Chain Ontology AMENDMENT 1: Extensions on Time-Segments & Multi-Track Audio (ISO/IEC 21000-19/AMD 1:2018)
  • ISO/IEC Information Technology – Multimedia Framework (MPEG-21) – Part 8: Reference Software AMENDMENT 4: Media Value Chain Ontology Extensions on Time-Segments & Multi-Track Audio
    (ISO/IEC 21000-8/AMD 4:2018)


FAST report from the Sound Talking workshop, 3 November 2017, London Science Museum

sound-talking-logo-large-1024x363On Friday 3 November, Dr Brecht De Man (Centre for Digital Music, Queen Mary University of London) and Dr Melissa Dickson (Diseases of Modern Life, University of Oxford) organised a one-day workshop at the London Science Museum on the topic of language describing sound, and sound emulating language. Titled ‘Sound Talking‘, it brought together a diverse lineup of speakers around the common theme of sonic semantics. And with diverse we truly mean that: the programme featured a neuroscientist, a historian, an acoustician, and a Grammy-winning sound engineer, among others.

The event was born from a friendship between two academics who had for a while assumed their work could not be more different, with music technology and history of Victorian literature as their respective fields. When learning their topics were both about sound-related language, they set out to find more researchers from maximally different disciplines and make it a day of engaging talks.

After having Dr Dickson as a resident researcher earlier this year, the Science Museum generously hosted the event, providing a very appropriate and ‘neutral’ central London venue. The venue was further supported by the Diseases of Modern Life project, funded by the European Research Council, and the Centre for Digital Music at Queen Mary University of London.

The programme featured (in order of appearance):

  • Maria Chait, Professor of auditory cognitive neuroscience at UCL, on the auditory system as the brain’s early warning system
  • Jonathan Andrews, Reader in the history of psychiatry at Newcastle University, on the soundscape of the Bethlehem Hospital for Lunatics (‘Bedlam’)
  • Melissa Dickson, postdoctoral researcher in Victorian literature at University of Oxford, on the invention of the stethoscope and the development of an associated vocabulary
  • Mariana Lopez, Lecturer in sound production and post production at University of York, on making film accessible for visually impaired audiences through sound design
  • David M. Howard, Professor of Electronic Engineering at Royal Holloway University of London, on the sound of voice and the voice of sound
  • Brecht De Man, postdoctoral researcher in audio engineering at Queen Mary University of London, on defining the language of music production
  • Mandy Parnell, mastering engineer at Black Saloon Studios, on the various languages of artistic direction
  • Trevor Cox, Professor of acoustic engineering at University of Salford, on categorisation of everyday sounds

In addition to this stellar speaker lineup, Aleks Kolkowski (Recording Angels) exhibited an array of historic sound making objects, including tuning forks, listening tubes, a monochord, and a live recording of a wax cylinder. The workshop took place in a museum, after all, where Dr Kolkowski has held a research associateship, so the display was very fitting.

The full program can be found on the event’s web page. Video proceedings of the event are forthcoming.


FAST in conversation with Joshua Reiss, Centre for Digital Music, Queen Mary University of London

  1. Could you please introduce yourself?JReiss

I am Josh Reiss. My title is Professor of Audio Engineering. I am an academic with the Centre for Digital Music at Queen Mary University of London.

  1. What’s your role/work within the project?

I am new to the project, but coming in as an Investigator, mainly involved in the Production side of things, but not limited to that.

  1. Which would you say are the most exciting areas of research in music currently?

Probably algorithmic music composition. This is still very much in its infancy and has huge potential. Research in this area also can yield deep insights into creativity and high level human perception and cognition.

Other areas are also being heavily impacted by other fields. Machine learning is becoming pervasive, but metamaterials and nanotechnology are affecting all transducers (loudspeakers, microphones…), enabling amazing spatial audio systems. Object-based audio is changing the broadcast world. And the path from academic research to commercial product or real world application is rapidly accelerating.

The world of audio and music technology will look very different even just 5 years from now.

  1. What, in your opinion, makes this research project different to other research projects in the same discipline?

By this research project, I assume you mean ‘FAST IMPACt’. I like its exploratory nature, while at the same time having a strong focus on impact and dissemination. It is well-positioned to make breakthroughs.

  1. What are the research questions you find most inspiring within your area of study / field of work?

Many. But I like the questions where there is some debate. Where people have asked the question but haven’t reached a consensus. And I like to work on questions that are well-defined, where you can feel that there is an answer to be found, even if its not a single, definitive solution.

  1. What, in your opinion, is the value of the connections established between the project research and the industry?

It’s a two way street. Working with industry helps academic researchers realise the value of their work, and gain new insights into what problems and approaches really matter. For industry, working with academia gives them a means to take risks that would not necessarily make sense if attempted internally. It helps them pursue disruptive innovations and helps ensure a certain level of rigour in their R&D.

  1. What can academic research in music bring to society?

Better music, better music technology, deeper understanding of perception, cognition and creativity. Plus, it’s a field where advances can be applied in many other domains.

  1. Please tell me why do you find valuable/ exciting / inspiring to do academic research related to music.

This is probably not the answer you’re looking for, but I am inspired by the research itself, not necessarily the fact that I’m doing it with music. To quote Richard Feynman, “I’ve already got the prize. The prize is the pleasure of finding the thing out, the kick in the discovery, the observation that other people use it. Those are the real things.”

  1. Why did you choose this area of research / field of industry?

Relating to question 8, my passion is research. If you had met me as a child and asked me what I wanted to be when I grew up, I would have said ‘Scientist.’ That hasn’t really changed. If there are interesting questions and problems in an area, then I naturally become interested in it. So, I moved into this field because of a background studying nonlinear systems, which lead to audio analogue to digital converters, which lead to audio effects, which lead to lots and lots of questions. But the choice was always to do research.

  1. What are you working on at the moment?

A few things. One interesting area of research is the question of whether we can hear a difference between standard CD quality audio (44,100 samples a second, 16 bit resolution) and any higher quality. Evidence slightly suggests that the answer might be that we can. But why? It seems doubtful that we can hear any frequencies above 20 kHz, so what is the cause of any perceived difference?

Another interesting thing is that we can hear a difference between pouring hot and cold water. Again, why? Lots of people have given explanations for this, but none of them agree with some simple tests. This area of research is just for fun.

I’m also involved in a few that are meant to change the way people do things, or their workflows or the tools they use. A few of these are automatic mixing, sound synthesis (for real world sound design), and adaptive mixes for the hearing impaired.

  1. Which is the area of your practice you enjoy the most?

That moment in research where you realise you’ve got the breakthrough, solved the problem, built the killer demonstrator… And also, just when I’m able to work on a problem by myself and make some real progress… And also watching one of the researchers working with me give a great talk, successfully defend a phd, win an award or anything like that.

Maria Kallionpää: “Climb!” – A Virtuoso Piece for Live Pianist, Disklavier and Interactive System (2017)

(by Maria Kallionpää)

“Climb!” is an artistic research project carried out by composer and pianist Maria Kallionpää in collaboration with Mixed Reality Laboratory of Nottingham University and the Sound and Music Knowledge Group of Aalborg University. The composition was premiered by Kallionpää on 9th of June at Nottingham University. More performances will follow on various festivals and concert programs in 2017-2018.

What is it?

thumbnailClimb! is a musical composition that combines the ideas of a classical virtuoso piece and a computer game. Written for a Disklavier grand piano, the work engages the pianist in an unusual musical dialogue where there is no fixed musical form: the performer’s/gamer’s actions define how the story continues. As the performer plays key passages, so both the music and the instrument respond, jumping to new points in the score, transforming the sound, or independently playing along with them – physically in the case of the Disklavier. An interactive system like this resembles “an invisible chamber music partner” leading some musicians to claim that electroacoustic works can never be treated as solo compositions, but should instead be seen as “concertos in a modern guise”. Like a computer game or the World Wide Web, Climb! is non-linear, with a branching structure that guides the performer along different routes through the work. It consists of three macro-compositions called paths with some links between them. Various micro-compositions are then dispersed along each path. Within these micro-compositions musical challenges or choices are presented which determine, depending on how the performer plays, whether they continue to progress along their current path or be diverted onto another one.

Composing “Climb!”

Climb! Images_SmallAs a performing pianist I grew up with the Classical Romantic virtuoso repertoire, which would have impacted on my aesthetical development as a contemporary composer. My DPhil thesis at the University of Oxford focused on piano as a super instrument. The main purpose of my artistic research project was to explore the ways of expanding the technical capabilities of a human pianist, thus making a classical music concert performance into something more than the sum of its parts. Climb! can be seen as a direct continuation of my previous artistic goals and research interests.

On top of other kinds of compositions that I have written recently (for example, orchestral works and two operas) I have composed many acoustic and electroacoustic works for piano. Nonetheless, after writing all those pieces I still thought that it might be interesting to approach the concept of virtuosity from a more innovative perspective. I was thinking of creating a “music engine” that would provide the performing musicians such technical capabilities that would not normally be accessible (such as, for example, enabling them to play simultaneously in several octave ranges, change the tuning in the middle of the piece, or play in overwhelmingly fast tempi that would be impossible on regular concert instruments) but the question about the musical form of such a composition remained open. I and the music psychologist, composer Hans-Peter Gasselseder later got an idea of combining the concepts of a computer game and a classical virtuoso piece (The Imaginary Friend: Crossing Over Computer Scoring Techniques and Musical Expression. / Kallionpaa, Maria; Gasselseder, Hans-Peter. Electronic Visualisation and the Arts (EVA 2016)), which then gave the dramatic structure for the Climb! composition.

Because the composition was also supposed to be a game, the first step was to decide its narrative and to define the musical equivalences to each turn of the storyline. We ended up using a story about a person climbing to the top of a mountain: along the way they face different events, obstacles, weather conditions, dreams, people, and animals, to mention a few examples. The music does not attempt to reflect each dramatic turn literally but is mainly an abstract description of the overall dramaturgy. Like in any game, the narrative (and thus also the composition) changes depending on the gamer´s actions in the virtual environment: because of this each performance of “Climb!” is different.

I composed “Climb!” during 2016-2017. It was performed for the first time in June 2017 at the Djanogly Recital Hall, the Nottingham University, where the audience members were also invited to use a specifically designed smartphone app on which they could follow the progress of the game and get information about the composition. During and after the concert performance some audience feedback was collected with the help of questionnaires and interviews. The first performances have been played by the composer herself but more pianists have been invited to play “Climb!” in later concert performances during 2017-2018.


FAST in conversation with Gary Bromham, Centre for Digital Music, Queen Mary University of London


1. Could you please introduce yourself?

My name is Gary Bromham, I’m a Music Producer, Audio Engineer and Songwriter. I’m also a part-time PhD Researcher at Queen Mary University of London in the Centre for Digital Music. I’m researching the area of retro aesthetics in digital music technology and how they influence music production. Specifically, I’m focused on GUI (Graphic User Interface) design and perception of timbre connected to this area.

2. What is your role within the project?

I have a small advisory role in the project. I suppose my task is to use some of my 30 years of experience to inform on best practices in the field of record production but probably also to act as an arbiter of the usefulness and real world application of new ideas. It’s constructive criticism I suppose!

3. Which would you say are the most exciting areas of research in music currently?

The area of automatic-mixing of multi-track audio interests me greatly. The idea that data gained from observation and analysis of recording and mix engineers can be used to automate mundane and often less creative aspects of this process is very interesting. Labelling of audio tracks based on instrument recognition for example could save huge amounts of time when mixing songs. Streamlining of existing workflows which might facilitate more time for creative processes is a justifiable goal. I am also very interested in the use of semantic technologies for describing sound properties in music production and how they might be a better way for a consumer or hobbyist than a more traditional physics based method.

4. What, in your opinion, makes this research project different to other research projects in the same discipline?

Probably the length of time allotted to the project. This gives time for ideas to evolve and be carried through to a fruitful conclusion rather than merely conceptualized.

5. What are the research questions you find most inspiring within your area of study / field of work?

I’m interested in how tools we use in a more traditional analogue based recording studio can be re-evaluated and re-shaped to work in a digital context. Digital models of hardware are often based on their analogue counterparts, skeuomorphs, this method, quite often, isn’t the best one when used in a computer based music system.

6. What, in your opinion, is the value of the connections established between the project research and the industry?

The industry is very good at knowing what it doesn’t want but less good at knowing what it does want, and needs, for that matter. It is also notoriously slow at embracing new technologies and adapting new ways of working which may be beneficial. Researchers are much better placed to help identify some of the key areas that may assist with speeding up this process. A prime motivation of FAST, for example, is to look at how the production chain might be improved upon. Tracking the path from concept to consumer is a complex one and only by analysing work processes and workflows can we begin to identify what the consumer might benefit from using.

7. What can academic research in music bring to society?

It can help to predict and inform how the production chain might look in the future. Music consumption is largely about convenience not quality these days, but I believe we should aspire to provide a better user experience whist at the same time be offering high resolution audio content.

8. Please tell me why do you find valuable/ exciting / inspiring to do academic research related to music.

I’m not an academic in a traditional sense, but I do have a huge amount of hands-on knowledge and experience to draw upon. Music production is a combination of aesthetics and physics or science and the study of this area needs to be of an interdisciplinary nature if we are to understand and disseminate some of the ‘mystique’ surrounding its history. For this reason, I believe this relatively new field needs to be investigated as thoroughly as possible.

9. Why did you choose this area of research / field of industry?

I’m very interested in why we persist in looking back at retro technologies with an almost obsessional attachment to some kind of holy grail, when we could do things far better by embracing new ways of working. I think it is the duty of a conventional composer or music producer to point this out to both researchers, but maybe more importantly, to manufacturers who often rely on outdated concepts to create tools to facilitate music production processes.

10. What are you working on now?

I’m currently producing an album for a 90’s band called Kula Shaker. I’m also about to run an experiment for my PhD where I’m assessing the effects of GUI design on listeners’ perceptions of audio quality in a DAW.

11. Which is the area of your practice you enjoy the most?

If I’m honest, sitting in front of a mixing desk, but hopefully testing and embracing new hybrid methods of working rather than merely following tradition and relying on familiarity to dictate workflow.

12. What is it that inspires you?

Watching my children write music in ways that we couldn’t conceive of even 10 years ago. The ability to collaborate and share ideas with other like-minded individuals, instantaneously, inspires me greatly.

FAST in conversation with Florian Thalmann, Centre for Digital Music, Queen Mary


  1. Could you please introduce yourself?

I grew up in Switzerland in a both musical and scientific environment. During my Computer Science studies at the University of Bern, I became increasingly interested in theoretical aspects of music and music history and attended musicology classes in my free time. After my studies, I gained some experience in the industry working as a software engineer, while also frequently performing as a musician. When I was offered a PhD position that allowed me to combine my two primary passions, I decided to go back to academia. I spent the following years at the University of Minnesota to pursue a PhD in Music Theory. After my studies, I joined the Centre for Digital Music and the FAST project as a postdoctoral researcher at Queen Mary in London.

  1. What is your role/work within the project?

Within FAST I’m mainly concerned with the consumption side of things, particularly with enabling new kinds of experiences during music discovery and music listening. I’m working on new ways of automatically analyzing music and representing it in a semantically annotated and flexible way so that it can be compared, browsed, and played back nonlinearly, i.e. dynamically reorganized to adapt to the listener’s context or respond to user interaction. I’m involved in various research and artistic collaborations within the project and beyond, where these technologies are being tested and used creatively.

  1. What, in your opinion, makes this research project different to other research projects in the same discipline?

The FAST project is highly interdisciplinary and unites three research groups with very different but complementary specializations, which nurtures our equally interdisciplinary goals by bringing together signal processing, studio technology, the semantic web, artificial intelligence, workflow analysis, ethnography, and design. We are possibly the only project considering the entire value chain of the music industry, from composition, via recording, production, and distribution, to consumption. One of our main goals, using linked data to enable a musical entity to travel through the entire music value chain while accumulating more and more information that will be useful at various stages, can only be achieved through the collaborative work of people with such varied and creative backgrounds, as well as with our industrial partners.

  1. What are the research questions you find most inspiring within your area of study / field of work?

With my background in music theory, I’m thrilled by the possibilities of automatically analyzing large collections of musical material and, for instance, finding trends and long-term developments in a particular artist’s or group’s work, similarities and connections with other artists, or differences in how a work is performed on different days and/or in different environments.

From the point of view of education, I’m interested in enabling music discovery and teaching through music listening itself, e.g. by highlighting particular musical aspects directly within the audio, instead of using distracting and abstract visualizations detached from the musical content. I’m also interested in educational and creative uses of sonification in order to, for example, present results of search queries in an auditory way.

As a performer, I’m mainly interested in coming up with new ways of interacting with musical material using physical and gestural interfaces, while trying to match the versatility of traditional graphical and textual computer interfaces. From this perspective, the use of machine learning in realtime is particularly interesting, for I’ve found that such systems often behave in unpredictable, seemingly creative ways.

  1. What, in your opinion, is the value of the connections established between the project research and the industry?

It’s a great opportunity to not only discuss the interests and requirements of our industry partners first-hand, but also improve our outreach by implementing prototypes in collaboration with our partners, which could then be further developed for specific applications in the industry. Sometimes new developments in the industry can be difficult to oversee and they often remain concealed until they are launched publicly. It’s therefore very valuable to have our ideas scrutinized at an early stage and from an economic point of view by our partners.

  1. What can academic research in music bring to society?

Musical practice has always been surprisingly quick to embrace new technological developments, both in experimental circles and the mainstream. Many technological ideas emerge and are first elaborated in academic contexts, to then be applied in practice and distributed in an economic context, e.g. by startup companies. These, in turn, make it possible for new musical tools and devices to become accessible and affordable by a larger audience of musicians and consumers, who almost always use the technology in creative ways that were initially not envisioned. It’s very exciting to see this process.

Besides having a potential technological influence on the music of the future, perhaps more importantly, academic research can further the understanding of our incredibly rich musical heritage. Research not only contributes to our common knowledge about music, but it can help to organize existing musical knowledge and public music collections and make them accessible and understandable by a large community of musicians, students, researchers, and music lovers. These aspects are of course equally relevant to the music industry who may find new ways of recommending and distributing music, as well as creating new experiences.

  1. What are you working on now?

I’m currently working on pattern recognition algorithms and music similarity models that allow us to automatically recognize and analyze various kinds of musical form and structure. I’m also working on finding appropriate ways to represent such musical structures as semantically annotated hierarchical and graph-based structures so that they can be queried, compared, and navigated. These algorithms and models are being applied in several projects within FAST.

For example, we’re looking at the Grateful Dead collection in the Live Music Archive, an incredibly valuable collection of live recordings from the early days of the group in the sixties to its disbandment in the mid-nineties. We are using the algorithms on various subsets of the collection. For example, we align different tape recordings of a single concert, spatialize them, and create an immersive experience in which the listener can move around. We also compare different performances of the same song across decades and allow listeners to move through time while listening to a song.

We’re also working on reorganizing and transforming recordings of various nature. We first decompose the recordings into single sound events using source separation algorithms and then put these sounds back together dynamically. This enables us to do things such as spatialize the contents of a mono recording around the listener to highlight some musical aspects or distort a recording in time to zoom in on certain performance aspects.

As a foundation of all this, I’m building a prototype of a mobile and web-based player, the Semantic Player, which uses these algorithms and representations to find out how any provided musical material can be played back in a malleable and dynamic way. The player can receive any input from mobile device sensors and data from the web, which can then be used to modify the music. We’re currently experimenting with all kinds of experiences where, for instance, the music adapts to contextual conditions such as the weather at the listener’s current location.

  1. Which is the area of your practice you enjoy the most?

I really enjoy thinking about specific problems and trying to come up with solutions that can then be generalized to other problems or situations. I also enjoy discussing problems with other researchers and collaborating to try to bring together different parts of research in a common application. In FAST, we are experiencing this all the time when working on our joint demonstrators. Most importantly, I’m almost always driven by an idea of something I would like to use or listen to myself. Nothing is more satisfying to me than implementing a new methodology and testing it experimentally, playing around with it, or even performing with it, and then improving and optimizing it iteratively.

  1. What is it that inspires you?

A lot of my inspiration comes from other disciplines and thinking about how they connect. I love creating analogies between situations in another field of application and the one I’m currently concerned with. There is always someone that has tried similar things before, however their work can easily be overlooked if it was conducted in a different field, so bridging the gap between research in different disciplines is important. Most importantly, I get a lot of inspiration from musical practice, nature, and the arts in general.


FAST in conversation with Adrian Hazzard, University of Nottingham

in August 2016, FAST interviewed Dr Adrian Hazzard, one of the FAST partners from Nottingham about the FAST project.

Adrian photo

  1. Could you please introduce yourself?
    I’m Adrian Hazzard, a Research Fellow at the Mixed Reality Laboratory, University of Nottingham.
  2. What is your role within the project?
    I’m a full-time researcher developing and evaluating research projects within the FAST project.
  3. Which would you say are the most exciting areas of research in music currently?
    For me it is the creative possibilities that the intersection between music production / performance and digital technologies, in particular web-based and context-aware technologies offer. Web based tools are essentially pervasive across all our everyday digital interactions and this is very much the case with music consumption, you just have to think of Spotify and YouTube. However, we are now increasingly seeing web-based tools seeping into the realm of music production, they offer uniformity, access, sharing and integration with other media on a scale not previously possible. If you then add in approaches such as ‘adaptation’ and ‘augmentation’, where music can adapt or respond in real-time to contextual information, then the future of music listening seems very broad and exciting. We seem to be at a bit of tipping point between the traditional analogue recording studio paradigm, which has formed the design of digital workstations to date and new technologies that may stretch beyond into new paradigms of interaction and consumption.
  4. What, in your opinion, makes this research project different to other research projects in the same discipline?
    The FAST project is unique on several levels. As a long-term project (5yrs) it enables for a deep exploration and development of new approaches into music production and consumption that would not be possible on shorter-term research engagements. The academic team consisting of Queen Mary, University of London, University of Oxford and University of Nottingham bring together a diverse, but complimentary set of knowledge and skills and there is a broad range of industry partners who are active in shaping and guiding the projects research agenda. Collectively, these attributes enabled the project to (initially) set out with a broad remit of research: to explore and experiment into many diverse areas of music production and consumption, this has now led to a focusing on a few core areas.
  5. What are the research questions you find most inspiring within your area of study?
    I find inspiration in ‘how the composition and production of music can be enhanced and shaped by contemporary digital tools’ and then ‘what do these new forms of musical experience mean for the listener’.
  6. What, in your opinion, is the value of the connections established between the project research and the industry?
    The industry connections on the FAST project have been an essential foil for a number of reasons. This research project aims for a significant industry impact, so it is imperative that as an academic team we are sensitive to the current state of the industry, to recognise the challenges and the opportunities at play. Our industry connections give us a unique insight into many areas of the music industry, such as the world traditional recording studios (Abbey Road), and the broadcast industry (BBC). Second, our industry team provide a mentoring and advisory role, from which we can gather constructive criticism of our research directions and outputs.
  7. What can academic research in music bring to society?
    Most people engage with music in some form on a daily basis. This might be as entertainment such as listening to music on the radio, a HiFi system or a mobile device; or as part of a formal service or ceremony; or as an accompaniment to a television programme, film or computer game – music is one of the most pervasive media types and it forms an integral part of many of our everyday formal and informal experiences. In short, music is important to society and therefore it is important to understand its role and impact, as well as to explore, and push at, its boundaries. While musical artists, with support from ‘the industry’ are the creative force behind the music we listen to, academia can play a vital role in distilling how music ‘works’ and how technologies can continue to support new creative endeavours, whether in music production or in other disciplines.
  8.  Why did you choose this area of research?
    My career to date has, in some form or other, revolved around music. I’ve previously worked as a performer, composer and music teacher across a number of settings. I came into academia later on in life and this has offered me a new and exciting opportunity to play some (very small) part in capturing and charting the development of new musical approaches and experiences… what could be better than that!
  9.  What are you working on at the moment?
    Currently, I’m working on a project called Muzicodes, a web-based music feature recognition system for performers. The system enables an instrumentalist(s) to trigger performance media by playing pre-defined melodic or rhythmic fragments; essentially musical ‘codes’. The performer identifies points in a musical work where they may want to trigger a backing track, or a MIDI event, or a lighting or visual cue of some description. The musical fragment or sequence that is played at point in the music then becomes the trigger for that action. There is an interesting challenge between creating a ‘code’ that is both suitable for the musical work and consistently recognisable by the system.
  10. Which is the area of your practice you enjoy the most?
    That’s a tricky question, as I enjoy many different aspects of this practice. For instance, I particularly enjoy the generation of research ideas, bouncing them off different people or reading something that inspires and leads you to the formation of a particular idea. Having said that, the following realisation of a research idea or ‘thing’ is equally exciting, although sometimes they are accompanied by a healthy frustration of seeing what they could have been given more time, resources and skill.
  11.  What is it that inspires you?
    Any creative musical interaction, whether it is a great composition or a musical expression on an instrument performed with conviction.

FAST in conversation with David Weigl, Oxford University

dweiglin August 2016, FAST interviewed Dr David Weigl, one of the FAST partners from the Oxford e-Research Centre, Oxford University about the FAST project.

1. Could you please introduce yourself?

I’m originally from Vienna, Austria. I moved to Scotland when I was 18, where I studied at the University of Edinburgh’s School of Informatics, and played bass in Edinburgh’s finest (only!) progressive funk rock band. After a few years in industry, I decided to return to academia to combine my research interests in informatics and cognitive science with my passion for music, pursuing a PhD in Information Studies at McGill University. Montreal is an amazing place to conduct interdisciplinary music research; at the Centre for Interdisciplinary Research in Music Media and Technology (CIRMMT), I was able to collaborate directly with colleagues in the fields of musicology, music tech, cognitive psychology, library and information science, and computer science, while benefiting from even more diverse perspectives, from audio engineering to intellectual property law. For the past couple of years, I’ve had the pleasure of working at the University of Oxford’s e-Research Centre, an even more interdisciplinary place: here at the OeRC, researchers supporting Digital Humanities projects regularly exchange notes with projects down the hall in radio astronomy, bio-informatics, energy consumption, and climate prediction.

2. What is your role within the project?

My role in the FAST project revolves around the application of semantic technologies and Linked Data to music information, in various forms: from features describing the audio signal extracted from live music recordings, via symbolic music notation, to bibliographic information, including catalogue data and digitized page scans from the British Library and other sources. The projects I work on support complex interlinking between these musical resources and tie in external contextual information, supporting rich, unified explorations of the data.

3. Which would you say are the most exciting areas of research in music currently?

Research into music information retrieval (MIR), and related areas of study, has been on-going since the turn of the millennium, when digital affordances, technologies such as the MP3 format, and widespread adoption of the Internet suddenly made digital music information ubiquitously available. The music industry took a long time to respond to the corresponding challenges and opportunities, but products such as Spotify, Pandora, and YouTube are now reaching a level of maturity enabling exciting new avenues of music discovery. Yet we are only listening to the tip of the iceberg, as it were. Many powerful techniques developed by the MIR community have not yet been exploited by industry; as we see more and more of these research outcomes integrated into user-facing systems, new ways of listening and interacting with music will become available.

4. What, in your opinion, makes this research project different to other research projects in the same discipline?

The cross-disciplinary, multi-institutional nature of the project, bringing together collaborative research into digital signal processing, data modelling, incorporation of semantic context, and ethnographic study across all stages of the music lifecycle, from production to consumption – combined with a long-term outlook, seeking to predict and inform the shape of the music industry over the coming decades – make FAST a particularly exciting project to be involved in!

5. What are the research questions you find most inspiring within your area of study?

I am especially interested in the notion of relevance when searching for music. This seems like such a simple notion when searching for a particular song – either the retrieved result is relevant to the searcher, or it isn’t – but the resilience of musical identity, retained through live performances, cover versions, remixes, sampling, and quotation, swiftly makes things more complicated. Deciding what is relevant becomes even more difficult when the listener is engaging in more abstract searches, e.g. for music to fulfil a certain purpose. For example, what makes a great song to exercise to? Presumably, some aspects – liveliness, energy, a solid beat – are universal, yet the “correct” answer will vary according to the listener’s expectations, listening history, and taste profile – be it Metallica, Daft Punk, or Beethoven.

6. What, in your opinion, is the value of the connections established between the project research and the industry?

I think such connections are valuable for numerous reasons. From the academic side, access to industry perspectives allows us to validate our assumptions about requirements and use cases; application of academic outputs within industry further acts as a sort of impact multiplier, ensuring our results are applicable to the real world. For industry, the value is in exposure and access to the latest state of research, fuelled by transparent, multi-institutional collaboration.

7. What can academic research in music bring to society?

On the one hand, advances in music information research have allowed us to conquer the flood of digital music available through modern technologies, enabling efficient and usable storage and retrieval. Each of us carries the Great Jukebox in the Sky in our pockets; listeners are able to access incredibly vast quantities of music wherever and whenever they like, and are better equipped than ever to inform themselves about the latest releases, and to discover hidden gems to provide a perfect fit for their taste and situation. Musicologists are able to subject entire catalogues to levels of scholarly analysis previously reserved for individual pieces, and are able to cross-reference between works by different composers, from different time periods or cultures, with such ease that the complexity of the interlinking fades into the background, becoming barely noticed.

On the other hand, such research has democratised music making and music distribution, providing artists with immediate access to their (potential) audience. Academia and industry must collaborate to ensure that monetisation is similarly democratised, to protect and strengthen music composition and performance as a viable career path for talented and creative individuals.

8. Please tell me why do you find valuable to do academic research related to music.

I’m very fortunate in that I get to combine my academic interests with my personal passion for music. I think that every researcher in the field feels this way; everybody I talk to at conferences plays an instrument, or sings, or produces music in their spare time. Music is inherently exciting – when I’m asked about my research at a social gathering, and I answer “information retrieval,” “linked data,” or “cognitive science”, I get a polite nod of the head; when I answer that I work with music and information, people’s eyes light up!

9. What are you working on at the moment?

I’m working on a number of different projects combining music with external contextual information using Linked Data and semantic technologies.

I have developed a Semantic Alignment and Linking Tool (SALT) that supports the linking of complementary datasets from different sources; we’ve applied it to an Early Music use case, combining radio programme data from the Early Music Show on BBC Radio 3 with library catalogue metadata and beautiful digitised images of musical score from the British Library, enabling the exploration of the unified data through a simple web interface.

I’m also working on the Computational Analysis of the Live Music Archive (CALMA), a project that enables the exploration of audio feature data extracted from a massive archive of live music recordings; for instance, we have access to hundreds of recorded renditions of the same song by the same band, performed over many years and at many different venues, allowing us to track the musical evolution of the song over time and according to performance context – does the song’s average tempo increase over the years? Do the band play more energetically in-front of a home crowd?

Finally, I’ve recently presented a demonstrator on Semantic Dynamic Notation – tying external context from the web of Linked Data into musical notation expressed using the Music Encoding Initiative (MEI) framework. This technology supports scholarly use cases, such as direct referencing from musicological discussions to specific elements of the musical notation, and vice versa; as well as consumer-oriented use cases, providing a means of collaborative annotation and manipulation of musical score in a live performance context.

10. Which is the area of your practice you enjoy the most?

The best parts are when multiple, disparate data sources are made to work together in synergistic ways, and unified views across the data are possible for the first time. This provides real “aha!” moments, where the hard data modelling work pays off, and complex insight becomes available.