Technologies

Facebook’s AI research could spur smarter AR glasses and robots

Rummaging through drawers to find your keys could become a thing of the past.

Facebook envisions a future in which you’ll learn to play the drums or whip up a new recipe while wearing augmented reality glasses or other devices powered by artificial intelligence. To make that future a reality, the social network needs its AI systems to see through your eyes.

«This is the world where we’d have wearable devices that could benefit you and me in our daily life through providing information at the right moment or helping us fetch memories,» said Kristen Grauman, a lead research scientist at Facebook. The technology could eventually be used to analyze our activities, she said, to help us find misplaced items, like our keys.

That future is still a ways off, as evidenced by Facebook’s Ray-Ban branded smart glasses, which debuted in September without AR effects. Part of the challenge is training AI systems to better understand photos and videos people capture from their perspective so that the AI can help people remember important information.

Facebook said it teamed up with 13 universities and labs that recruited 750 people to capture more than 2,200 hours of first-person video over two years. The participants, who lived in the UK, Italy, India, Japan, Saudi Arabia, Singapore, the US, Rwanda and Colombia, shot videos of themselves engaging in everyday activities such as playing sports, shopping, gazing at their pets or gardening. They used a variety of wearable devices, including GoPro cameras, Vuzix Blade smart glasses and ZShades video recording sunglasses.

Starting next month, Facebook researchers will be able to request access to this trove of data, which the social network said is the world’s largest collection of first-person unscripted videos. The new project, called Ego4D, provides a glimpse into how a tech company could improve technologies like AR, virtual reality and robotics so they play a bigger role in our daily lives.

The company’s work comes during a tumultuous period for Facebook. The social network has faced scrutiny from lawmakers, advocacy groups and the public after The Wall Street Journal published a series of stories about how the company’s internal research showed it knew about the platform’s harms even as it downplayed them publicly. Frances Haugen, a former Facebook product manager turned whistleblower, testified before Congress last week about the contents of thousands of pages of confidential documents she took before leaving the company in May. She’s scheduled to testify in the UK and meet with Facebook’s semi-independent oversight board in the near future.

Even before Haugen’s revelations, Facebook’s smart glasses sparked concerns from critics who worry the device could be used to secretly record people. During its research into first-person video, the social network said it addressed privacy concerns. Camera wearers could view and delete their videos, and the company blurred the faces of bystanders and license plates that were captured.

Fueling more AI research

As part of the new project, Facebook said, it created five benchmark challenges for researchers. The benchmarks include episodic memory, so you know what happened when; forecasting, so computers know what you’re likely to do next; and hand and object manipulation, to understand what a person is doing in a video. The last two benchmarks are understanding who said what, and when, in a video, and who the partners are in the interaction.

«This sets up a bar just to get it started,» Grauman said. «This usually is quite powerful because now you’ll have a systematic way to evaluate data.»

Helping AI understand first-person video can be challenging because computers typically learn from images that are shot from the third-person perspective of a spectator. Challenges such as motion blur and footage from different angles come into play when you record yourself kicking a soccer ball or riding a roller coaster.

Facebook said it’s looking at expanding the project to other countries. The company said diversifying the video footage is important because if AR glasses are helping a person cook curry or do laundry, the AI assistant needs to understand that those activities can look different in various regions of the world.

Facebook said the video dataset includes a diverse range of activities shot in 73 locations across nine countries. The participants included people of different ages, genders and professions.

The COVID-19 pandemic also created limitations for the research. For example, more footage in the data set is of stay-at-home activities such as cooking or crafting rather than public events.

Some of the universities that partnered with Facebook include the University of Bristol in the UK, Georgia Tech in the US, the University of Tokyo in Japan and Universidad de los Andes in Colombia.

Technologies

Samsung’s iPhone Fold Killer Is Leaked

Technologies

How to Tell if Someone Else’s Apple AirTag Is Tracking You

These settings can help make sure your iPhone alerts you to unwanted location trackers.

The biggest benefit of Apple’s AirTags is that they help you find your belongings, whether you’re looking for lost keys or keeping track of your luggage while traveling. But AirTags can also be used to track you without your knowledge.

AirTags work by combining built-in sensors, wireless signals and Apple’s wide Find My network to let you keep tabs on your valuables. If you ever lose your wallet with an AirTag inside, for example, you can use the Find My app to locate it on a map, have it play a sound to help you find it nearby, or mark it as «lost,» which allows other Find My users to help you find it.

One of the biggest complaints about AirTags, however, is that someone with malicious intent could easily slip one of the tiny tags into your bag and then track your movements without your consent. Multiple people have reported AirTag-related stalking incidents where the victims didn’t know the trackers were placed on them until much later.

Apple and Google (Android users have their own choice of Bluetooth trackers, such as the Moto Tag, which works with Google’s Find Hub) have since collaborated on an industry standard that alerts the user if a device is being used to track them without their knowledge. Thanks to this collaboration, Android users will be able to know if an AirTag is being used to track them, too.

Apple, for its part, has also made some changes in the past few years that improve the ability to detect an unwanted AirTag. In the initial rollout, an AirTag would make a sound three days after it’s separated from its paired device. Now, that duration is 8 to 24 hours. If you have unwanted tracking notifications enabled (which we’ll get to below), you’ll receive an audible alert.

We should note here that the new AirTag is 50% louder than the first-generation model, and would therefore be theoretically better at alerting you to the unwanted AirTag. Apple has also said that the speaker on the second-gen AirTag is harder to remove than on the first-gen model, in case bad actors try to remove it.

Detecting unwanted trackers

To be able to detect unwanted trackers, first enable unwanted-tracking notifications. For AirTags or other Find My accessories, these pop-up notifications (e.g., «AirTag found moving with you») are available on devices with iOS 14.5 or later. For other Bluetooth tracking devices, these notifications are enabled on iOS 17.5 or later.

You should enable Location Services, Find My iPhone, Bluetooth and Allow Notifications. Here’s how:

Head to Settings, then Privacy & Security, then Location Services and toggle it on.
After that, head to Settings, then Apple Account, select Find My and turn Find My iPhone on.
To enable Bluetooth, go to Settings, then Bluetooth and turn that on.
Then go to Settings, then Notifications, scroll down to Tracking Notifications and toggle on Allow Notifications. Make sure airplane mode is off, or you won’t receive tracking notifications.

What to do when you get the tracking notification

If you do get a notification like «Unknown tracker alert» or «Item detected near you,» you can try to find the unwanted AirTag by tapping it. Tap continue and then tap Play Sound or tap Find Nearby to locate the AirTag in question.

If it doesn’t play a sound or you’re unable to find it, the item may no longer be on your person. Apple suggests checking your other belongings or the area around you, just in case. If you want to review the notification at a later time, you can open the Find My app, tap Items and then tap Items Detected With You.

Be aware that there are often «false positives,» when notifications are triggered when someone nearby has a tracker on them. If you’re traveling on a train, plane or bus, waiting in line or seated in a public space, a mistaken tracking alert could stem from glitches or high-density Bluetooth environments.

If you get an alert, though, it’s always a good idea to take it seriously and investigate what might be causing it.

If you do find an AirTag that doesn’t belong to you, hold the top of your iPhone near the tracker until you see a notification. Tap it, and this will launch a website that provides information like its serial number, the last four digits of the phone number or a blurred-out email address of its owner. If the AirTag is marked as «lost,» you may see a message with instructions on how to contact them.

If you’re concerned that the tracker is being used to monitor your movements and location, Apple advises taking a screenshot of the information above for your records. You can then disable the AirTag by pressing down on the back of the AirTag, turning it counterclockwise to remove the cover and removing the battery.

Of course, before making any of these changes, it’s important to come up with a safety plan, especially if you’re afraid you’re being tracked by a current or former abusive partner. Contact your local law enforcement if you feel like your safety is at risk, or the National Domestic Violence Hotline 800-799-SAFE (7233).