Siri co-founder Tom Gruber talks about his new AI-music startup LifeScore


“I think we’re looking at a new way of doing music, which is a collaboration between human expertise, human taste, and human raw talent in creating music and recording it, with the machine’s ability to combine those things and generate music all day long. And make it sound like a human made it.”

“Just imagine for a second… what you’re going to have is music that was created bespoke for you, on demand, as much as you wanted, and it sounded just as good as if a human created it. Because a human had a large amount to do with how it was created.”

Music Ally has heard plenty of ambitious pitches from AI-music startups, but this one carries some weight thanks to the pitcher: Tom Gruber. He was the co-founder of Siri, the startup that Apple acquired in 2010 to use its team and technology to launch its voice assistant (also called Siri) a year later.

Gruber retired from Apple in July 2018, reportedly to pursue “personal interests in photography and ocean conservation”. But come February 2019, he was on stage at London’s famous Abbey Road Studios, introducing AI-music startup LifeScore, where he’s co-founder, chief technology officer and VP of design.

“Let me ask you a question. What do you think would happen, or will happen, when we combine the best of AI and the best of humanity? What kind of change in the experience of creation and the experience of music is going to happen?” said Gruber, kicking off the latest demo day of the studios’ music/tech startup accelerator, Abbey Road Red – LifeScore is its newest participant.

“I’m coming from this space of being in AI for a long time, 30 years or so, and then the last 10 at Apple. In my career, I’ve seen a lot of stuff, but I’ve never seen anything happen at the pace it’s happening right now. In AI right now, the pace of innovation, the excitement, the amount of investment? Every number that can go up is going up,” said Gruber.

“It’s not just a hype curve thing. I think it’s more than that. In fact, I think it’s actually a renaissance. It’s one thing to create a product, but it’s another thing to have an entire generation being transformed by this technology.

Gruber noted that in some areas, like certain areas of medical diagnosis or mechanical design (“or in certain cases of getting you to stay online and look at things and buy things!”) AI can already demonstrably outperform humans.

“But I also see a different thing. I think we can see an opportunity where the human and the machine work together. Where the point of the computer, the point of the AI, is to augment and extend the human being. And I’m placing my bet on that path,” he said.

This is a topic that Gruber has explored before, for example in his 2017 TED Talk:

“That raises another big question: how do you make the mix? It’s not so simple, especially when it comes to something like music. How do you allocate the responsibility and skills and expertise and the power, between the human and the machine?”

Gruber drew a comparison to AI-created art, like the recent AI-painted portrait that was sold for $432.5k at auction-house Christie’s.

“It’s not the first time AI in art happened, but it’s an interesting data point. Now, if I ran a great museum in London, I wouldn’t be worried… We’ve seen this before. Remember when they had elephants that could kinda paint? It didn’t actually replace artists. It just made the ones that looked like elephants not have a job any more!” he joked.

“So there’s this thing: when AI paints, it looks like AI paintings. But when AI makes music, we don’t want it to sound like ‘AI music’. That’s a non-starter.”

Gruber thinks that LifeScore will do something better, having been approached by its CEO and founder Philip Sheppard – a respected cellist and soundtrack composer – to join his project.

The AI veteran outlined the theory behind the startup. “If you think about what makes music sound great: what has an emotional response in you? It’s really two things. It’s the raw material that goes in to the music. The actual sound of musicians playing music. And how that raw material’s composed into something bigger,” he said.

“In the case of LifeScore, the raw materials are actually recordings by the best musicians in the best recording studio in the world.” And the composition? “That magic that happens when a great composer takes the great raw materials, and creates something with emotional content and emotional meaning. And those are essentially what goes into the algorithms of LifeScore.”

Later on, Sheppard himself took the stage to explain what LifeScore is doing. He explained that the startup is an evolution of an earlier idea he had, which was writing fragments of music on acetate sheets (the kind you’d use for an overhead projector back in the day) and getting children to flip, reverse, mix and match them to compose songs.

“I stole the idea lock, stock and barrel from Bach!” he said. “Although Bach didn’t have an overhead projector and some paper lying around…”

This became a commercial game called Compose Yourself! based on its set of 60 “musical building bricks”, which could be used to create 2.5bn permutations of different melodies.

Sheppard said he realised this was “just a mere fragment of what is possible” and began to work on LifeScore, partly after realising how the concepts from Compose Yourself! could be applied to his regular walks.

“I do a lot of walking, and I don’t know what to damn-well listen to. And I’m a musician! So I thought ‘why don’t we reverse-engineer that?’. As I’m walking along, maybe the music is beat-matching me: the speed that I walk. It would be nice if it was an orchestra or band or whatever. Maybe if I change direction, it can change texture or change key…”

That’s LifeScore, which Sheppard demonstrated on-stage as an app that seamlessly stitches together the fragments of music he recorded with an orchestra at Abbey Road, while responding to his movements.

“This is a platform. It’s a language. It’s not an app: I’ve just put it in to an app to demonstrate,” he stressed. “The important thing is that when you put this in someone’s hands and they’re not a musician… suddenly they get ownership of it. ‘That’s now mine: I’m carving music in time and space’. This is something that’s endlessly adaptive from something that was recorded here [at Abbey Road] and it soundtracks your days and nights.”

(Sheppard noted that this idea of generative, adaptive music isn’t new, citing 17th-century inventor Athanasius Kircher’s ideas for automated instruments. “Although he also invented the cat piano! And thus the internet meme…” Music Ally readers with long-ish memories might think of something else: British startup RjDj, which was experimenting with ‘reactive music’ in apps between 2008 and 2013, although we can’t remember any cats being involved.)

Sheppard added that LifeScore could in the future use signals from smartwatches and other wearables.

“What would it be like if every walk you took could be soundtracked to how you’re feeling: your walk, your biometrics? Your watch can monitor that,” he said.

“What if in these world-building 360 open-roaming video games, you never got the redundancy of a track repeating? How cool would it be if, every time you listen to an album, it behaves differently? We can link music to tastes and colours and smells… We think we can do something with the driving experience…”

LifeScore’s emphasis on the human role in its technology – providing the raw music fragments – swerves one of the main points of tension around AI music-creation. Its focus on soundtracking your life chimes well with the era of mood and activity playlists on streaming services. And in Sheppard and Gruber, it has two very-persuasive storytellers.

It’s very early days to understand exactly how this technology will be best deployed (not least to find its business model) but based on the evidence of the Abbey Road Red demo day, it’s one of the startups most worth watching in the AI-music space.

Written by: Stuart Dredge