How Microsoft’s HoloLens headset could help find your keys (and do much more)

This cartoon shows a user wearing a headset that’s part of a system to keep track of items such as keys (102). The headset display could highlight the location of the keys even if they end up hidden under a sheaf of papers or lost between the cushions of a couch. (Microsoft Illustration via USPTO)

Microsoft expects its HoloLens augmented-reality headset to guide you through complicated tasks in the workplace, but someday you could also use it around the house to find misplaced items, play games – and even watch movies on a virtual big screen.

The possibilities for augmented reality, or AR, are laid out in a series of patents and patent applications published over the past month or so.

Augmented reality is a cousin of virtual reality. VR creates a complete computer-generated environment, viewed through headsets ranging from the high-end Oculus Rift and HTC Vive to the smartphone-based Google Daydream and Samsung Gear systems. AR systems like Microsoft’s HoloLens goes one step further, blending computer-generated imagery with the real-life scene that’s in front of you.

HoloLens is in the midst of a gradual rollout, starting with developers and business customers. NASA has been testing the platform on the International Space Station as a tool for training and guidance from ground controllers, and it’s also using HoloLens to tweak the design for its next Mars rover.

One of the Microsoft patent applications, filed in September but published just a couple of weeks ago, envisions using augmented reality for much more down-to-earth purpose: keeping track of stuff around the house.

“Searching for misplaced car keys, wallets, mobile devices and the like may cause people to lose productive time,” the application notes. “Likewise, forgetting that the milk carton in the home refrigerator is almost empty may lead to an extra trip to the store that could have been avoided had the shopper remembered the state of the milk carton.”

Those everyday tasks get even more complicated when you add in the possibility of your spouse moving the car keys someplace else, or your thirsty teenager finishing off the carton of milk.

The patent application proposes an AR system that keeps track of the objects you want to keep track of (for example, the car keys) and can even monitor the contents of your refrigerator.

If you’re looking for your keys in the morning, just put on the AR headset, and your heads-up display will guide you to the keys’ location – for example, by guiding you with flashing arrows, or by highlighting the keys on your display.

If you’re running low on milk, a “visual overlay, an auto-generated shopping list, and/or a verbal reminder may be generated as the user walks by the milk section in the grocery store,” Microsoft’s engineers suggest.

Your inventory, and your behavior patterns, could be stored in the cloud – so that the HoloLens-enhanced system eventually knows what you should be looking for better than you do.

That vision of the future is ambitious, and more than a little scary. What’s more, the patent application doesn’t address the ultimate problem: What happens if you misplace your headset?

Here are a few more augmented-reality ideas from Microsoft:

Microsoft has filed for a string of patents that cover headset versions of mixed-reality games like Pokemon Go, last year’s gaming sensation. “A person wearing a head-mounted display device (HMD) may walk around a real-world environment and play a virtual game that is adapted to that real-world environment,” according to an application published last month. “For example, the HMD may identify environmental features within a real-world environment such as five grassy areas and two cars, and then spawn virtual monsters based on the location and type of the environmental features identified.”

This cartoon shows a game player with augmented-reality monsters popping up in the room. (Microsoft Illustration via USPTO)

Another patent application envisions a system for displaying a “virtual monitor” on your headset screen. This monitor would act like a TV screen in the real world: As you move around the room, your perspective on the virtual monitor would shift so that it appears to move relative to real-world objects. The application was filed in September and published last week.

This diagram shows three potential orientations for virtual monitors as seen in augmented reality: on a wall (208a), sitting on a tabletop (208b) and floating in space (208c). (Microsoft Illustration via USPTO)

Finnish inventor Marko Eromäki came up with a concept for fitting a smartphone into a specially designed headset with reflective surfaces, to provide VR experiences on the cheap. One option would involve making the viewing surfaces semi-transparent for augmented reality. The application was made for Microsoft in June 2015, but published last week. Eromäki was brought over from Nokia when Microsoft acquired that Finnish company in 2014. He left Microsoft this August and is now working for Huawei Technologies.

This cartoon shows a headset that lets the user insert a display device such as a smartphone (indicated by the red box) and experience VR or AR on the cheap. (Microsoft Illustration via USPTO)

Hat tip to CNet and MSPoweruser.

More from GeekWire:

• Microsoft bringing HoloLens headset to Lowe’s to visualize kitchen renovations

• Microsoft opens Windows Holographic platform to all VR and AR headset makers

• Inside the ‘world’s first HoloLens class’: UW computer science students show mixed reality projects

• Microsoft launches HoloLens developer preorders with 7 new apps for mixed reality headset