Scientific Tests
So when he wanted to know how Siri stacked up against a Google (GOOG) search, he (or his staff) asked an iPhone 1,600 questions, 800 on the busy streets of Minneapolis, 800 in a quiet room.
Change one variable, keep the rest the same - the attribute you are observing. Munster thought he’d test Siri against Google, but he failed at the first hurdle. The questions that were asked were typed into Google, rather than spoken for Siri. This invalidates the accuracy ratings immediately; many of the errors Siri made were due to recognition errors, not her own AI.
Also, Munster misses much of Siri’s appeal. The two systems were asked knowledge questions, but much of Siri is its role as a personal assistant. For example, ask Google to set a timer for three minutes and it’ll respond with irrelevant search results. This side of Siri was not tested at all in these tests.
The whole thing was a pointless expedition. You are comparing apples to oranges.