At Code Conference 2023, I tried out Google’s latest version of Project Starline, the company’s stab at making video chats far more engaging by adding depth, like 3D video. It’s a tall order for a population exhausted by years of Zoom chats over small computer screens during pandemic-imposed lockdowns, but I found it far less draining to have more humanity in my distance calls.
First introduced at Google I/O 2021, Project Starline is a supercharged video chat that uses cameras and software to simulate depth on a flat screen such that the person pops out of the screen with a faux-3D effect. CNET’s Scott Stein was impressed by the first publicly-available version of Starline he tried out last October, which took up a full booth with tech and sensors. The new, far more portable version accomplishes a similar degree of remote chatting that uses 3D to convey far more body language cues than a 2D video call. That’s more engaging for both people in the conversation, Google theorizes.
“We found, especially within Google, that you can actually use it to form a solid and new relationship — because you trust people, you get a sense of them,” Andrew Nartker, general manager of Project Starline, told me in preparation for my hands-on with the new Starline, which was introduced in May.
I walked into the Project Starline demo room, a hotel room decked out with decorative white frills lining the walls, to sit in front of the desk and a Starline unit itself. The newest version of Project Starline has been shrunk to the size of a modified flatscreen TV that’s ringed by three sets of twin cameras, one on each side and the top.
All six cameras track my exact body position and movements, not just to convey my position to who I’m talking to on the call but to see exactly where my eyes are — and showing each of my eyeballs the same image in a slightly different position to create a 3D effect. It’s kind of like a 21st-century version of the blue-and-red 3D glasses of yore.
Google found that coworkers who knew each other were able to use standard video calls just fine, but those who hadn’t met in person failed to bond and connect over Zoom calls.
Instead, Starline uses a TV-sized display and its cameras show your subject from the waist up, but with depth to show people gesturing, shifting and looking directly into their screen. There’s a little fabric arch that bends outward at the bottom of the big display, which guides the user’s eye to how far their conversation partner’s 3D image will stretch from the screen. This setup is far better for allowing people to bond over distance, the Starline team believes.
Google’s Project Starline prototype allows participants to have a remote conversation with depth effects — though difficult to see from this angle, the seated reporter is holding his hand out below where the other person is “holding” the apple.
“You read the body language and all these little things that are missing [in conventional video calls] that we’re so evolved to pick up on,” said Nartker. Nonverbal cues, hand gestures, movements and even leaning in toward your subject or away from them — all of that is held back in video calls as people stiffen up to fit in the frame of a tiny webcam.
Chatting in 3D, across the office or the world
Starline has the advantage of being limited to one-on-one calls instead of group Zoom chats as well as being on big screens, but as I settled into my demonstration, I could see how the new-and-improved Starline invited participants to bond more easily with their conversation partner.
In my case, I spoke with Nartker from a few rooms away, each with our own Starline prototype and connected over the hotel’s wired internet network. While Google didn’t share the exact speeds they were getting, it’s clear the Starline network doesn’t need a specialty network — any standard work or home network should be fine for latency-free communication, as far as I could tell. Nartker came in crisply and didn’t lag, though there was occasional jaggedness at edges of his arms and lower body, which I chalked up to the complexity of blending images for the depth effect.
As we conversed, I tried to pick up on these unspoken body language tics Nartker described. Seeing him angle toward or away from me was subtle feedback about whether he was interested or not in what I was saying — something much easier to gauge with the 3D effect on the screen, which was compelling (we’ve come a long way from the stereoscopic 3D on the Nintendo 3DS). The 3D persisted despite my wearing glasses (evidently its cameras peered through my spectacles), and it was a relief to be looking Nartker in the eye as we chatted. I found myself leaning in and gesturing more vividly while we talked, naturally matching Nartker’s level of gesturing.
Because I was participating more, I was thinking less about the emails I had to write or sessions I had to attend; in video calls, people multitask because they won’t get caught. But Nartker held out an apple and I could see it hovering out beyond the TV screen. He could tell where I was looking, and so direct the conversation spatially — which would be helpful if pointing to a corporate presentation or directing care in a telemedicine call, say.
At Code Conference 2023, Google showed off its Project Starline prototype to companies, inviting interesting and feedback.
But the Starline team noticed something else: When using their system to chat, participants remembered more than with a video call. Nartker explained that since people remember the world through spatial memory– where people stood, what they wore, how tall they were, what the room looked like — a Starline chat leaves people with a more vivid recall of what happened.
“There’s a quantified difference in how you come away from a productivity meeting or something with more thought on what happened,” Nartker said.
I remembered quite a bit from our 10-minute Starline conversation, most notably how Nartker moved and gestured, what he pointed to and his demeanor while talking. Obviously that has benefits for corporate applications where a Starline setup can bridge the gap between inexpensive video call and pricey in-person travel. While Google hasn’t revealed how much a Starline unit will cost (it’s still a prototype), it’ll conceivably be cheaper than a private jet flight for executives or travel accommodations for full teams.
The first of many Starline products?
Google introduced its new Starline via blog post back in May around Google I/O 2023, and has been taking its more mobile prototypes to select venues to gather interest. The team brought a Starline (Nartker casually referred to it in the singular, suggesting that it’s become more than just a project name, and perhaps internally a product name) to Code Conference to invite potential partners to try it out for themselves (they’ve already worked with Salesforce to refine the Starline system) as well as to brainstorm new ways to use it for feedback to Google’s team.
“It’s one of the ideas that’s come up pretty frequently with this crowd at Code Conference of really trying to eliminate travel,” Nartker said. “Where am I going today, and why am I going to these important pivotal meetings or negotiations or business deals typically in person. Could I do those instead over Starline?”
Currently, both participants in the one-to-one call need a Starline unit to use the system, which means shipping the TV, camera and speaker units to a desired location and having a Google engineer on-site to get it working. Setup takes a full day, so it’s not yet possible to just wheel in a Starline and have it immediately sync up for a corporate call.
But the new prototypes are far more mobile than older versions, and they represent a trajectory of shrinking Starline down to even smaller proportions thanks to Google’s innovations in AI, Nartker said. And it’s just one format out of several that the team is tinkering with.
“We think of [Starline] as a pretty flexible communication technology that could take a lot of different shapes and form factors,” Nartker said. “This is the one that we’ve been exploring the most, but we have versions of a Starline in kind of all the ways you can imagine people gathering.”
The team continues to make progress with recognizing objects and participant bodies and replicating them on both sides of the call. With enough development and software innovations, one day video chatters may not need a bespoke multicamera setup to get the same depth effect to feel more connected and retain more of their video chats. Nartker was coy about when we could use Starline on a laptop with a single webcam, or if it’s possible, but didn’t rule it out.
“I think in the long arc, that’s a great North Star goal at this stage,” Nartker said. “We’re just really exploring this type of prototype where, with enough cameras, we can create an effect that’s useful for people to be together. Maybe, over time, we can do that with easier systems.”
