Digital art that illustrates the land of the living and the dead

The BBC has described Stan Lai as “probably the best Chinese-language playwright and director in the world.”

So when his play Like Shadows was produced this April at China’s most prestigious theatre school, the Central Academy of Drama in Beijing, it represented the biggest stage yet for a new digital technology pioneered by Concordians Miao Song and Serguei Mokhov.

The two computer scientists expanded on Song’s doctoral research creation, the Illimitable Space System (ISS), to allow for beautiful and often emotionally charged animations and voice samples to be created, in real time, in response to the movements of performers.

Like Shadows was a hit in Beijing, and the technology it showcased will soon be on display closer to home. Song and Mokhov — both affiliate assistant professors and part-time faculty members in the Faculty of Engineering and Computer Science — are bringing their creation to Montreal’s Eureka! Festival from June 13 to 15, and to the Westmount Science Camp on Concordia’s Loyola Campus in July and August.

What makes ISS unique is that the use of a motion-sensitive Kinect camera in a theatrical context allows for the collection of data about the movements of actors, dancers or other performers; that data is then fed into a computer and translated into arresting graphic imagery.

This ostensibly means that no two shows using ISS will be exactly the same. But even more importantly, Song says, the technology allows directors and performers to express themselves more freely. With it, they no longer have to sync their movements to preprogrammed animation.

A decade ago, Song and Mokhov enrolled at Concordia as undergraduate students. Song, originally from China, had already completed a degree in theatre and worked as a television journalist in her native country. Mokhov, who hails from from St. Petersburg, Russia, started a computer science degree in his hometown and completed it at Concordia.

Since then, they’ve both completed master’s degrees and doctorates. In addition to his academic appointment, Mokhov is an employee at Academic Information Technology Services in the Faculty of Engineering and Computer Science.

The use of the expanded ISS on stage represents perhaps Song’s and Mokhov’s most audacious collaboration yet. Song sees the project as an extension of her undergraduate studies.

“Because I had a theatre background, I wanted to apply new multimedia to a traditional art form,” she says.

“I’m the person for the technical difficulties,” Mokhov adds, laughing.

Lai’s Like Shadows tells the stories of two neighbouring families. The daughter of one family dies in a car accident, but her soul does not acknowledge this death. As a result, the neighbouring daughter continues to see the soul of the deceased.

This is where Song and Mokhov’s collaboration becomes a powerful dramatic tool. In the Beijing production, ISS enables the projection of animated bodies and other images onto the theatre’s back wall, thus demarcating the land of the dead from that of the living.

But ISS is not a finished product, and much of its potential remains untapped.

“We’re still in the prototyping phase of this technology,” Mokhov says. “So far, we’ve worked in the arts world, but there could be other applications.”

Song and Mokhov have spoken extensively to Xavier-Henri Hervé (BEng 87), a mentor and coach at the District 3 Centre for Innovation and Entrepreneurship, which he co-founded. Hervé encouraged the team to explore the value of their creation among varying audiences.

At the academic end of the spectrum, Song and Mokhov have discussed the project with Peter Grogono and Sudhir Mudur, professors in the Department of Computer Science and Software Engineering.

When the ISS is on display again at the Eureka! Festival and the Westmount Science Camp, the audience will be entirely different: it will mainly consist of children. The full extent of Song and Mokhov’s creation appears almost illimitable.

The Eureka! Festival takes place from June 13 to 15 in Montreal’s Old Port.

Song and Mokhov would like to recognize the extensive support of the following organizations: Concordia University; the Faculty of Engineering and Computer Science; the Central Academy of Drama, China; the Conseil des arts et des lettres du Québec; Hexagram-Concordia; and the District 3 Centre for Innovation and Entrepreneurship.

Watch a clip from the Central Academy of Drama’s production of Like Shadows, and see Illimitable Space System in action in a production staged for Chinese New Year.