Yesterday, a blog post from developer Jeff Johnson highlighted an "Enhanced Visual Search" toggle added to the Apple Photos app in iOS 18 that gives your phone permission to send photo data to Apple. In exchange, you can identify landmarks in your photos or search for them in your picture library. It’s also turned on by default, and until now, we weren’t aware of the setting, which has largely gone unmentioned by Apple or in coverage of the new iPhone OS.
What is Enhanced Visual Search?
The feature is similar to Apple’s "Visual Look Up," which uses machine learning to identify plants or animals in your photo library. To see it in action, swipe up on a picture in your library showing a building or landmark and tap ‘Look Up Landmark.’ Photos will then ideally tell you what it is.
Here are some examples from my own library:
- That’s definitely Austin’s Cathedral of Saint Mary.
- The image on the right is not a Trappist monastery, but the Dubuque, Iowa city hall building.
How Does Enhanced Visual Search Work?
A description under the Enhanced Visual Search toggle says the feature will "privately match places in your photos," though it’s not clear from the description how that happens or that it means sending data from your photos to Apple. According to a company research blog, it involves your phone creating vector embeddings — of the part of a picture containing a landmark, and sending that and several ‘fake queries’ to Apple for analysis. Your phone then chooses the final match from a batch of possibilities that Apple sends back.
Privacy Concerns
According to Johnson, referencing Apple’s "What happens on your iPhone, stays on your iPhone" billboard from CES 2019:
…if something happens entirely on my computer, then it’s private, whereas if my computer sends data to the manufacturer of the computer, then it’s not private, or at least not entirely private. … You don’t even have to hypothesize lies, conspiracies, or malicious intentions on the part of Apple to be suspicious of their privacy claims. A software bug would be sufficient to make users vulnerable, and Apple can’t guarantee that their software includes no bugs.
Apple did not respond to our request for comment on Johnson’s concerns.
Finding Enhanced Visual Search
You can find the Enhanced Visual Search option on iOS / iPadOS under Settings > Apps > Photos, or under the ‘Search’ heading in the Photos > Settings on a Mac.
Correction and Update
An earlier version of this story misstated the location of the Enhanced Visual Search toggle. It is in iOS Settings under Apps > Phone and in Photos > Settings in the macOS Photos app. The headline was also made clearer.
Update, December 30th: Article updated for clarity.
Conclusion
The introduction of Enhanced Visual Search in Apple Photos app raises important questions about user privacy. While the feature may be useful in identifying landmarks in your photos, it also involves sending data from your phone to Apple, which could potentially compromise your personal information. As Johnson points out, even a software bug could make users vulnerable.
Recommendations
- Be aware of the Enhanced Visual Search toggle and its implications for user privacy.
- Consider turning off the feature if you are concerned about your personal data being sent to Apple.
- Be cautious when using any feature that involves sending data to a third-party company, even if it is claimed to be "private."
Further Reading
If you want to learn more about Enhanced Visual Search and its implications for user privacy, we recommend checking out the following resources:
- Apple’s Research Blog: This blog post provides more information on how Enhanced Visual Search works.
- Jeff Johnson’s Blog Post: This article raises concerns about the potential risks of sending data from your phone to Apple.
- Apple’s Privacy Policy: This page provides more information on how Apple collects and uses user data.
