Zoom says it has nearly 216,000 businesses using its service, with total users collectively joining meetings hundreds of millions of times each day. Otter.ai says it has more than 10 million registered users.
“A lot of busy [workers] are double or triple booked. They have to be very selective in terms of which meetings they attend live, but some of the information in other meetings may still be useful for them,” said Sam Liang, chief executive and co-founder of Otter.ai.
The Help Desk tried out the AI features during work meetings. Here’s what worked well and what didn’t.
The benefits of AI for meetings
Zoom and Otter’s AI meeting tools weren’t perfect, but they did have some benefits.
Both gave us a general sense of what was discussed, what was assigned, deadlines along with some to-do items. Zoom IQ meeting summary provided an editable summary broken down in chapters followed by next steps. OtterPilot, which shows up as a separate guest in virtual meetings on Zoom, Google Meet and Microsoft Teams, serves a live transcript alongside tabs with the live summary, topics including questions asked and follow-up steps and a chatbot that will answer inquiries or generate text for follow-up emails.
Transcription and recording capabilities on both services enabled us to playback or read back the meeting, serving as a backup. Playbacks also help when there are errors in the transcript. Otter.ai’s summary links to corresponding parts of the transcript and audio recording in case you want to hear or read more about specific items. Zoom offers its transcript, and recording separately from its summary.
“Full transcripts are still a bit useful,” said Arvind Karunakaran, assistant professor at Stanford University, who’s been studying AI as it relates to work. Colleagues “could look at the transcript and contextualize [what was said].”
It was also relatively easy to access and share transcripts, summaries and recordings with my teammates.
With Otter.ai’s new Slack integration, released this week, I posted a link to the live transcript and summary from Otter’s platform. You can do this during or after the meeting or schedule Otter to automatically post during recurring meetings. Colleagues — whether they were meeting participants — could follow along in real-time or catch up afterward. The transcript also includes screenshots of the speaker or speaker’s shared screen.
Regardless of whether they were present at the meeting, colleagues could chat with attendees during the meeting on Otter’s platform or ask the bot questions like whether anyone specified deadlines. And if you can’t — or simply don’t want to — attend a meeting, you can schedule OtterPilot to go without you.
Zoom allows you to email summaries and links to the transcript and audio and video recordings that are saved in the cloud.
Why AI for meetings isn’t perfect
AI wasn’t always precise, as that’s often the case.
Several times the services misheard words. In our case, that led to the creation of nonexistent people who were either assigned to-do items or listed as parts of a story in the AI-generated summaries. At one point, Otter misheard “roaches” as “Russians,” and Zoom heard “barky” as “Mark.” And transcripts degraded with sound quality. Noisy environments and interruptions garbled the audio or led to the services transcribing voices of people in the office who weren’t in our meeting. And sometimes, the transcription confused who was speaking.
“Technically speaking, these automated transcription services don’t always work very well, especially if there are multiple people speaking and with different accents,” said Hatim Rahman, an assistant professor at Northwestern University’s Kellogg School of Management, who studies AI’s impact on work. “There are lots of question marks on the infrastructure to capture audio.”
The AI also had trouble categorizing topics, separating ideas and accurately capturing details. When we were clear about a deadline for a specific project, the AI performed better. But when we switched a deadline or talked about two deadlines too closely together, the AI summary was inconsistent.
It also had trouble understanding what “next, next Friday” meant. Sometimes, summaries listed ideas the team rejected as items we should proceed with as next steps. And though the AI was generally decent at capturing the main idea of the meeting, it didn’t catch every follow-up item.
“For back and forth conversations, it’s a little harder,” Karunakaran said. “There’s some technical limitations around context.”
Zoom says its AI-powered features are always improving and that meeting hosts can help by giving its summaries a thumbs up or down, and share feedback. They can also edit summaries before sharing. All attendees are notified that AI-generated content may be inaccurate and are encouraged to check for accuracy.
Otter.ai said its system gets smarter as users edit transcripts. We found our second meeting’s summary to be more accurate than the first. The service also encourages people to use its chatbot to ask questions, as those answers may be more accurate. You can also correct the chatbot when it’s wrong. In our case, the bot often regurgitated errors from the transcript.
Our tests reminded us that AI is just a machine. So it may summarize your off-topic chatter. For us, it highlighted people’s weekend plans and mid-meeting comments about snacks and bugs.
Considerations before using AI for meetings
AI meeting features have the potential to be helpful in the future. But experts who study AI and work say there are several things you should consider first.
Transcription and summary software generally need a good quality audio feed. It also may have particular difficulties with people who have accents or specific tones of voices, experts said. And it may not always understand what’s most important.
Companies should also think about what consent looks like for workers, Rahman said. Is the expectation that all meetings have the potential to be recorded and summarized? Will people be able to opt out? And how will that data be used once it’s collected? Could it be used to evaluate someone’s performance? Might that create a chilling effect for people to share what they think?
AI could also lower workers’ attention in meetings if they rely too much on the technology, he added.
Another big concern: Privacy. Users should familiarize themselves with providers’ privacy policies including whether they use your data to train their AI before letting the software capture sensitive information, said Karunakaran.
After public pushback to its initial policy, Zoom said it won’t use customer content to train its AI. Otter said it de-identifies user data before training its AI models and that recordings and transcripts are not manually reviewed humans. It also encrypts its training data.
There are pros and cons to using AI meeting features. OtterPilot’s features made finding information faster, but it also had the potential to resurface errors in multiple spots. To find what you needed in Zoom’s summary, you had to look through the entire thing. But it also didn’t repeat its errors.
Experts say the technical capabilities of AI meeting features are still in the early stages and will likely advance over time. But if you do opt to use the AI features, it’s important to ensure all participants are aware and that you review all items for accuracy.
