Site navigation

Facebook Researching AI With “Egocentric” First-person Perspective

Graham Turner


Facebook Ego4D
The social media giant’s Egocentric Live 4D Perception (Ego4D) project involves AI that uses first-person video to analyse.

Facebook’s new Ego4D project imagines an AI that can see, hear and record everything you do, with scope for it to be included in products like its AR glasses.

Facebook has stressed that, at this point, this is speculative and wholly research-based, and not an AI model actively in development for commercial deployment.

The ambitious, long-term research seeks to reach five ‘benchmarks’ in AI technology.

Episodic Memory

Using recorded video, the AI will be able to respond to queries that apply to specific things the AI will have ‘seen’.

Hands and Objects

The AI can respond to motions you’re making with your hands and feet and offer insight. This could be potentially leveraged in things like learning how to play an instrument.

Audio-Visual Diarisation

A transcription tool that records everything the AI ‘hears’. Could be used to recall a conversation you’ve forgotten.

Social Interactions

This gives the AI the ability to identify individual participants in a conversation. I.e. “What did this person say at this time?”


The AI can predict your next move and potentially advise you on a course of action if it isn’t optimal.


AI with these kinds of capabilities comes with an endless amount of ethical and legal baggage, not least being the privacy concerns implied with a piece of technology that actively records and transcribes everything in its view and audio range.

The dataset from which the research finds its basis is the largest ever of its kind. Facebook partnered with 13 universities around the world to collect 3,205 hours of footage – ranging from people interacting to performing work tasks – from 855 participants living in nine different countries.

Join the Conversation: Digital Transformation 2021 Summit

The impact of digital disruption will be a key theme at the upcoming Digital Transformation Summit on 28th October.

Now in its sixth year, the Summit has established itself as Scotland’s largest annual conference focussed on digitalisation and organisational change.

For more information on how to register a free place visit:

Graham Turner

Sub Editor

Latest News

%d bloggers like this: