Imagine bumping into a former colleague or friend you haven’t seen for a few years and, instead of awkward small talk, your AI-powered augmented reality (AR) glasses pull in information from their recent social media posts and career progress to help seamlessly guide you through a conversation.
That’s a future offered by Stanford University student Bryan Chiang who has, over the last few weeks, been offering a glimpse of the future by combining AR with large language models.
meet lifeOS: an operating system for your entire life 🌐
a personal AI agent delivered directly through AR smart glasses 👓
it uses computer vision to 👁️recognize👁️ your friends’ face
then brings up relevant details to talk about based on your texts with them (memory🤯) pic.twitter.com/BgCuODV4k2
— b (@bryanhpchiang) April 10, 2023
“The interplay of AI and AR will redefine personal computing and help us unlock our full potential”, Chiang said.
His videos show text appearing in a small heads-up display on the open-source Monocle from Brilliant Labs.
His phone’s microphone picks up audio from the conversation, feeds it out to an automated transcription service, sends that transcript to OpenAI which composes a response, sends it back to the phone, and displays the text on the lens.
Processing much of the information in the cloud causes slow responses (which have been edited in the videos), but it’s an interesting proof of concept that shows how we could one day use AI tools with AR devices to help guide us in social settings, or get recommendations while looking at a dinner menu, or even bluff our way through a job interview.
AI agents in AR glasses = 🤯🤯🤯
my GPT-4 JARVIS can now:
– recognize my friends faces
– understand what i'm looking at using computer vision
– respond aloud via TTShere it analyzes the Buck's menu and tells @DrakosBrown what to get based on his taste prefs + nutrition needs pic.twitter.com/azQKF4xEz6
— b (@bryanhpchiang) April 18, 2023
say goodbye to awkward dates and job interviews ☹️
we made rizzGPT — real-time Charisma as a Service (CaaS)
it listens to your conversation and tells you exactly what to say next 😱
built using GPT-4, Whisper and the Monocle AR glasses
with @C51Alix @varunshenoy_ pic.twitter.com/HycQGGXT6N
— b (@bryanhpchiang) March 26, 2023
Carli Johnston is co-founder of augmented and virtual reality company Virtual Method based out of ACS’ Harbour City Labs.
She said that while it is “ethically questionable” to cheat on a knowledge test or job interview, the ability for headsets to deliver information straight to people’s eyes in real time could be a game changer for our economy.
“In Australia right now, we have a massive, critical and growing skills shortage, especially in primary industries,” Johnston said.
“When a worker’s knowledge is augmented and upskilled in real-time by either a human remote-expert or an AI ‘expert’, we already know that it decreases downtime enormously and trains workers much faster than classroom/online pedagogies, as the knowledge they need can be utilized and synoptically embedded straightaway”.
say goodbye to awkward dates and job interviews ☹️
we made rizzGPT — real-time Charisma as a Service (CaaS)
it listens to your conversation and tells you exactly what to say next 😱
built using GPT-4, Whisper and the Monocle AR glasses
with @C51Alix @varunshenoy_ pic.twitter.com/HycQGGXT6N
— b (@bryanhpchiang) March 26, 2023
AR’s long-awaited arrival has had many false starts. Google’s Glass headset famously turned its wearers into social pariahs, Magic Leap failed to deliver on its consumer products, and Cisco’s AR video conferencing app left a lot to be desired.
But Brilliant Labs’s Monocle, on which Chiang built his demos, points a way forward for AR experimentation and research – a relatively small and self-contained unit complete with a 720p camera, a microphone, Bluetooth for connecting to your phone, a 640×400 micro OLED display, and detailed documentation.
Brilliant Labs bills its device for use by “the imaginative hacker” and its Australian $520 (US$349) price tag keeps it in range for students or independent researchers wanting to hack together the AR future.
Brilliant Labs CEO Bobak Tavangar told media that he could feel the “collective delight” now that people can see the possibility of bringing together AI and AR.
“It’s clear this is a break-out moment for AI and our device platform is the only truly open, hackable AR platform out there upon which these use cases can fully express themselves”, Tavangar said.
“This is only the beginning – stay tuned, the future is bright”.
Meta, with its re-focus onto augmented and virtual reality technology, is hoping its partnership with eyewear giant EssilorLuxottica will provide enough of a fashion incentive that potential buyers aren’t afraid of looking foolish.
Those products are still early days – the Meta Ray-Ban Stories only have a camera and microphone, no heads-up display – but they show how big tech companies are looking to AR/VR as the next evolution in computing.
Consumer technology’s worst kept secret is that Apple is poised to unveil its mixed reality headset at the Worldwide Developers Conference in June.
Speaking to GQ for a profile earlier this month, Apple CEO Tim Cook described the potential for an as-yet-unannounced headset, echoing the sentiments of Chiang and Johnston by saying that AR will change the way we interact with computers and the world around us.
“The idea that you could overlay the physical world with things from the digital world could greatly enhance people’s communication, people’s connection”, he said.
“It could empower people to achieve things they couldn’t achieve before”.
