Our Best Foot Forward: Functional AR

There’s no denying that online retailers make shopping more convenient, but in some ways the digital shopping experience falls short. For one, you can’t try on clothing or fashion accessories before having them shipped to your door, which means you might discover after days of waiting that something doesn’t fit. To get around this, customers might buy several versions of an item in different sizes, which can be costly to them in the immediate term while contributing to scarcity of high-demand items—for example, sneaker releases, which have generated $20 billion in U.S. sales alone in 2017 and have a resale market valued at over $1 billion.

There has to be a way to instill purchasing confidence in consumers, which is why one of the latest projects to come out of MediaMonks Labs, our internal R&D team, is an AR tool that uses computer vision to accurately measure shoppers’ feet. While it may not sound like the most exciting use of AR (no, it won’t transport virtual monsters to your living room or render a shoe directly on your feet), “it speaks a lot to what else we do beyond the crazier stuff,” says Joe Mango, Creative Technologist at MediaMonks Labs. “We explore things where there’s need, and see how feasible—or infeasible—it is. There’s high interest in exploration and discovery.”

Sander van der Vegte, who leads the Labs team, discusses the importance for more practical uses of the tech in both its adoption and signaling value to consumers: “Everyone’s pushing AR, but why is it useful? This tool is designed purely to solve a specific problem, and is functional in doing so.”

Improving CX with a “Form Follows Function” Approach

While most people are aware of their shoe size, fits aren’t so cut and dry; shoppers often get some shoes in one size and some in another. “Shoe shopping becomes more complicated when different brands and shoes have different sizing charts, making it harder to gauge what fits,” says Mango, who brought the tool to life. It can get around this issue by automatically translating a unit measurement into a specific brand’s sizing chart, giving the perfect size recommendation as users shop online. Ideally, the tool could exist within a retailer’s app, seamlessly aligned with the browsing process.

The tool has practical value for both consumers and retailers alike: shoppers know their purchase will fit before they buy it, and retailers don’t have to foot the bill of processing returns. This also means customers don’t have to worry about being blacklisted from making frequent returns, either—a fail-safe measure some retailers have used that erodes consumer trust and confidence. But perhaps most importantly, the tool serves as example for how retailers can use emerging technology to give consumers an obvious reason to provide their data. In this case, the length of their foot improves and optimizes the customer’s journey by enabling them to make the right purchasing decision.

Improving Measurements with Computer Vision

It’s worth noting that measuring via AR isn’t an entirely new capability: Apple, for example, integrates an AR “tape measurer” into the current version of iOS. While the feature is great for measuring perfectly straight lines or edges, it’s not the most accurate tool of measuring the length of a foot: there’s a lot of user error in moving the phone and finding the right angle to measure the foot.

The tool built by MediaMonks Labs is powered by OpenCV (a programming library for computer vision) and takes a different approach. Starting with a top-down photo of the user’s foot, it uses edge detection to determine and outline its shape. It then compares the size of that shape to a reference object whose size is defined and absolute—for example, a ubiquitous coin—which results in a consistently accurate measurement. While the output is in units of inches or centimeters, the tool has the potential to translate these measurements to brands’ custom sizing charts, taking the guesswork out of determining what fits.

Taking the Next Steps in Consumer-Friendliness

While the value of the AR tool is clear, added steps can be taken to improve usability. “There is a caveat to this method of measurement,” says Mango. “You need a flat color-contrasted background for accuracy,” which helps the computer vision model detect the foot’s edges.

Mango also discovered that the tool had difficulty differentiating a foot from the rest of a leg. The solution was simple: by wearing ankle socks he could segment the two with consistent accuracy. These limitations, along with the requirement of a universal reference object, present opportunities to make the tool more user-friendly for commercial use.

Despite its current limitations, the tool shows how brands might identify common issues on both the business and consumer side of the equation, then implementing technology in new creative, new ways to solve them. In the immediate term, these unique applications can provide the retailer with brand equity through emerging technology’s novelty. More importantly, though, they can deliver a better online or offline shopping experience to customers.