Step into the future: Sally Coleman’s project Big Sand. Working in Flinders’ motion capture studio, The Void, and pushing technological boundaries, Sally has brought her creative dream alive, merging virtual production, sci-fi and music.
Musician/Creative Sally Coleman first hit the music industry as part of the hip-hop duo Code of Conduct. Fast forward 10 years, she has an impressive career including gigs as a presenter on Triple J’s breakfast radio and at a record label. But Sally is not done yet.
“I’ve spent a lot of time in the music industry and have a lot of amazing memories from that. But I think the thing I’m doing right now is what I’m most proud of, and that’s a new science fiction band called Big Sand”, Sally tells us.
The concept behind Big Sand
Sally has big dreams with a concept that might just be what touring will look like in the future. Spurred by the support of the Adelaide arts community as well as the project’s great potential, Sally is wearing many hats: she’s the creative technical manager, the business funding development manager, releasing music and running a record label. “It’s exciting because I do feel like it’s an area where there’s just heaps of potential.”
Australia Council for the Arts has recognised this potential and awarded Sally a project grant for Big Sand to partner with Flinders University, CDW Studios and The Light Cultural Foundation which operates the Lab on Light Square in Adelaide’s CBD.
She elaborates on her vision, “Imagine it’s a Friday night, you’ve come to see some live music and walk into the space, and you are teleported into the desert. You’re surrounded by LED walls, and they’re lit up with a desert landscape. The floor has a projection of sand on it that you’re walking through, you’ve got a soundscape that’s taking you away – you really feel like you’ve walked into this world.
“The point of difference between a normal gig is that the band will be on those walls, and they’ll be cartoon characters. But instead of just watching a pre-recorded half-hour gig, a lot of the sections of the performance are going to be a live performance.
“Hopefully from The Void, I’ll then be streaming into that gig and able to see and hear the audience from cameras that are set up around the venue. So, the lead character will actually interact with the crowd the same way you would with any other performer. I want to create the feeling you’re stepping into a video game and the characters are talking to you. At the same time, you’re at a concert with your friends.”
The project’s aim
The Big Sand experiment is two-fold for Sally. On the one hand, there is the question of technical achievability. On the other hand, there is the suspense of how people will respond to the show. “Having to learn a lot very quickly to answer all these questions has been really fun, figuring out both the technical as well as the creative challenges, and then hopefully soon finding out how the audience will react.”
“I want to create an experience that makes the audience walk away with a big smile on their face and a sense of joy and curiosity – that feeling of wanting to unpack it with your friends straight away.”
What inspired Big Sand?
Like many people, Sally did a lot of soul-searching in 2020, “What do I want to do with my life? Where do I want to be in 10 years’ time? I knew that I still wanted to be making music, I loved the music industry. But I had a few things I wanted to change.
“I wanted to be able to have a lifestyle where I didn’t have to be on the road all the time – I could make music, have a successful career but also have some chickens, live a little bit out of town and grow some vegetables. I didn’t necessarily want to be on a plane every 12 hours. So that was one life goal that I drew up.
“The other thing was, I really wanted to engage more in science fiction and fantasy because that’s a genre that I love reading. My secret side passion is googling who won the Hugo and Nebula Awards and reading up on what they’re putting out.
“I was curious: I love science fiction and fantasy so how can I make a music project that incorporates my love of sci-fi? That was the genesis of the idea – bringing science fiction to pop music.”
Developing the characters
Due to the left-field nature of Big Sand, Sally wanted to work with classic sci-fi archetypes and tropes for both the setting and the characters in her virtual production. She chose outcast orphan, Tal, and her alien band members, who together navigate a mysterious desert planet.
Sally explains how the project developed, “Using the software Unreal Engine was something that came about out of necessity, it was never really part of my master plan. I had this simple question, which was, how am I going to bring this to social media when the characters don’t actually exist? I can’t go out and do press shots. I can’t just get them to do a story, how am I going to make a TikTok? So it was just this really basic question of how do I do it all?”
After talking to several animation students, Unreal Engine came up. “They all recommended building assets and using them. That was a brand-new concept for me, this idea of virtual production and creating things in a virtual space. You had to have raw footage or material to work with. I loved that idea.”
Working with Unreal Engine
As Unreal Engine is free, Sally downloaded and started playing around with the software long before she had any assets, characters, landscapes or environments. “I just watched a lot of tutorials, and read a lot of forums trying to answer my questions and I still think I don’t know many of the basics when it comes to the software. It’s so versatile, you can use it for everything from making a Pac Man game to shooting a film.
“I focused on how I can use motion capture and Sequencer, which is the in-editor filming software that lets you set up cameras around a space and record scenes. How do I animate things with my body? I’ve gradually been getting better at that and it’s been really great working with Flinders in this respect; just getting some expertise and emotional support – hearing everyone finds certain things hard, it’s not just me.”
Connecting with Flinders
The collaboration with Flinders around Big Sand kicked off when Sally met The Void’s former Business Development Manager Dan Thorsland at an AFTERS seminar in Sydney and invited her to talk to the Void’s team that is specialized in motion capture.
“He was so adamant about it on this panel that I was watching, and I thought to myself, well, I do want to do things in motion capture. Okay, I’ll send you an email. He got back to me straight away and looped in Cameron Mackness and Jason Bevan from The Void as well as James Marshall from Flinders’ VFX and entertainment design partner CDW Studios. They all jumped on a call and wanted to hear about this project. The response was great. I wasn’t expecting it from Flinders, I just was asking for some brainstorming help and some ideas.”
“My very first idea was to create a sci-fi band, but I never would have imagined that it would happen to the degree of execution that it has already. I really thought I’d be working with one character on a 2D backdrop. Now I’ve got these whole immersive spaces, sunsets and night skies with different lighting and all sorts of amazing things. It’s definitely already exceeded my expectations and it’s still very early days, I’ve only just released the second song.
“There are a lot of things about this Big Sand idea that are probably more ground-breaking than I realized at the time. A lot of this technology is really new. Even just streaming the data from A to B is a big, technical hurdle that we’ve got to cross and figure out – how do we package it? How do we unpack it at the other end? Do we need a VPN? Do we need a server? All these things, as a musician, are completely outside of my frame of reference.”
“It’s amazing how something that was just this huge pie-in-the-sky idea, that no one had ever done before and I had no idea how I was going to do, is now all happening, just one and a half years later. I’m very grateful to CDW Studios for all the assets they’ve created for me. It’s far beyond what I ever would have expected I’d have to work with.”
Different motion capture technologies
And so, the project commenced. Sally had bought motion capture suits from overseas previously but was using a motion capture technology that wasn’t ideal. She elaborates “There are two main types of motion capture. The first is inertial motion capture, in which little sensors all over your body do the same things your mobile phone can do.
“If you tilt your mobile phone sideways, the display rotates because it knows that you’ve tilted it. If you start walking it has location data. If you’re spinning around on Google Maps, it knows which way you’re facing. All the sensors have directional data, that is tracking where your body is in relation to itself. But what it can’t do is understand where the floor is, for example, where I am in relation to other people in the space, it only has the data on your body.”
The Void at Flinders uses an optical tracking system that tracks movement in relation to the physical space by using light. By bouncing light off the body, the system knows exactly where in the room a person is. “It creates this totally different way of moving around the space where your location is very precise and lets you do things that you can’t do very well in an inertial suit as I’ve got.”
Developing the concept even further
If everything works out as planned, step two of the Big Sand project is to take the show anywhere in the world, but especially to regional communities or people who usually wouldn’t have access to events like this. Sally’s vision is to have the show play in any language to anyone, have performers who speak Auslan or who are great with children – be able to adapt the production based on who the audience is.
As opposed to the screen and games industries, the music industry is not yet heavily involved in motion capture and virtual production. Sally wants to change this and generate new knowledge that she can share with others in the music community.
“COVID was obviously really difficult for the music industry in Australia, a lot of people lost their livelihoods and left their jobs as a result of that. I’d love to see the music industry in Australia get some more resources, outlets and technology that help them do things in new ways.”
Making an isolated experience collaborative
One of the things Sally enjoyed the most while working in the Void, was the collaborative nature of the space, “You can put a bunch of people in the room and see your characters interacting with each other. It’s that sort of group dynamic that makes The Void so fun.
“Obviously, the technology facilitates it, but it’s actually being able to get in the room with other students and dancers. There’s just so much more energy in the room when you’re running around with two or three other people than if you’re doing it by yourself on your laptop in your bedroom.”
As part of her project, Sally collaborated with students. Drama students that wanted to take part in the project or simply watch, were invited into the studio. Sally also did a workshop with Jason’s motion capture class, “The students split into groups. One group did some storyboarding. The second got into suits and did some blocking and workshopping. The other group got behind the screens and set the cameras up for some sweeping shots.
“Everyone was involved in different aspects that are all going on at once. There are a lot of elements that you’re juggling that have everyone working simultaneously on the day. The students were from a mix of different degrees, for example, Drama and Visual Effects and Entertainment Design.”
Sally’s advice for students interested in Creative Arts
“My grandma went to university at a later age. For a woman in the 70s, that was a big deal – it was the pinnacle of achievement in life. I think a university’s role has now changed a little bit in society, for a lot of people, it’s now about getting a job. It’s about getting the training and the skills you need to be immediately employable.
“My advice is not always the best advice for everyone, because not everyone wants the same kind of lifestyle. But for me, it shouldn’t just be about getting a job. Instead, work backwards – what do I want to do with my life? What do I enjoy doing? How do I want to spend my days? Ask yourself those questions and work backwards from there.”
“If you love writing, study writing. If you love animals, study veterinary science. I know that that’s not achievable for everyone and not everyone wants to live and breathe their work. But for me, it’s not all about immediate reward, it’s more about enjoying your degree, doing what you love and enjoying the experience that you get out of it.
“Don’t worry so much about whether there’s an immediate return on investment in terms of a career. I think it’s all about the return on your personality and personal growth that you can get out of higher education. It makes you think differently about the world and understand yourself differently.”
See Big Sand at the 2023 Adelaide Fringe
Sally has just brought out Big Sand’s second single You, Me & the Sky with Tia Gostelow for which she is currently making a Jane Fonda Workout-inspired music video in The Void.
And more exciting things are happening: Keep your eyes peeled for the Big Sand live show at the Adelaide Fringe on 9 March 2023, with tickets on sale December 6th 2022. This world premiere will showcase the creative development of the show to date, with a Q&A and audience feedback session at the end. Want more information? Email hello@bigsand.band.
Keen to work on projects like this yourself? Get into Flinders’ Master of Virtual Production.