Maybe, it was a coincidence…
That I happened to be traveling to Seattle for reasons other than Cloudy (Although, Cloudy is available to travel to cover future events), but nonetheless it’s our first field trip! For those of you who don’t know what AmazonGo is, it is a store without any cashiers. All you do is enter the store, scan your Amazon App, pick things off the shelf, and leave. It feels like you are stealing things. You then get charged for all of the things you picked. You can watch a video here.
While there was some initial fanfare about the launch of the store, the day I went, crowds were sparse and no line appeared outside the store.
As mentioned above, entering the store is an easy task, just download the Go app, scan it on the turnstile and away you go. The store itself is not that big, at around 1,800 square feet, but they do have a range of food, candy drinks and beer!
How do they make Go work?
Ok, enough touring, let’s get down to mechanics and figure out how this thing works. The easiest place to start is with a patent that Amazon filed on this type of technology back in 2014, which covers the following:
“an image capture device (e.g., a camera) may capture a series of images of a user’s hand before the hand crosses a plane into the inventory location and also capture a series of images of the user’s hand after it exits the inventory location. Based on a comparison of the images, it can be determined whether the user picked an item from the inventory location or placed an item into the inventory location.” And, “In addition to cameras, other input devices, such as pressure sensors, infrared sensors, a scale, load cells, a volume displacement sensor, a light curtain, etc., may be utilized…”
This is a picture of the ceiling of the store. All of those little boxes you see above, those are cameras that are most likely tracking exactly everything that was mentioned in the patent filing.
How do they know it’s you?
The biggest question to ask is how does the store know who picked up what? While these cameras can track if an object was picked up, they are most likely tracking who picked what up. My hypothesis about how this works is once you scan your app to come into the store, you are picked up by some sort of video tracking machine learning software, which follows your every move through the store. Amazon actually offers this capability through an AWS API called “Rekognition” and literally uses this example on their website about tracking people through a store.
Also, their description of how their API works is quite similar:
“When using Rekognition to analyze video, you can track people through a video even when their faces are not visible, or as they go in and out of the scene. You can also identify their movements in the frame to tell things like whether someone was entering or exiting a building.”
Is that it?
There could be more than meets the eye, but a combination of video tracking, scales and motion sensors are all you need.
After you leave, it takes about 15 mins for the app to process the transaction and eventually you get a receipt with what you purchased. The one flaw if the program is it doesn’t tell on a real time basis what you are being charged for, so you kind of hope it just works out.
One interesting thing is they offer you the ability to get a refund right inside the app. This gives almost instant feedback to how the algorithms are doing and allows the engineers to take this data and improve upon its accuracy.
How is it so accurate?
Another piece of the puzzle is how did Amazon get all of this data to make their algorithm so accurate? Amazon most likely used data from their fulfillment centers, where they are able to constantly track their workers movements. In fact, Amazon employs a team of people with machine learning expertise to do exactly this. As far back as 2013, Amazon was doing research on tracking people using image recognition algorithms and employed a Carnegie Mellon Phd to lead the effort.
The future of Whole Foods?
The big question is once Amazon starts scaling this technology, could it become prevalent in all Whole Foods? This is a difficult task given the different complexities of how a grocery store is designed (how produce is stacked, how much meat or fish weigh, etc). How will Amazon account for this? Also, what is the cost of setting up a store like this? While this store was quite small, it was stacked with different technology, most likely costing a lot of money. Will it be profitable to outfit a store 20x larger than the Go store?
The experience was interesting and certainly a glimpse into the future. Based on what the store is currently stocked with, it’s best for a quick pickup of premade food, lunch, drinks, etc. The store seemed to have a random array of products (some spices, syrup and wine) and I am guessing they want to test as many different sizes/types of products in order to collect data.
For what it’s worth, the cookie I bought was delicious