ChaCha’s Service Delivers Better Answers More Often, According to a New Report from MSearchGroove

LONDON – July 16, 2009 – Pump Up The Volume: An Assessment of Voice-Enabled Web Search on the iPhone, a performance analysis of voice-enabled mobile search services published today by MSearchGroove, found that ChaCha, the fastest growing SMS mobile search service in the industry, "proved superior" to two other voice-enabled search options for the iPhone: the Google Mobile App with Voice and Vlingo for iPhone,  a voice enabled application that allows users to direct their spoken queries to Google or Yahoo! (For the purposes of this study Vlingo provided a spoken interface to the Yahoo! search engine.)

chacha summary resultsChaCha proved to offer exceptional results, with its human guides interpreting the search query accurately in the majority of cases. According to the study, ChaCha interpreted natural language search queries, that is, queries asked as questions, accurately in 94.4% of the tests and delivered an accurate search result in 88.9% of cases. The Google voice recognition technology interpreted queries accurately in 16.7% of tests and delivered accurate search results in 22.2% of tests. The Vlingo for iPhone voice recognition technology correctly interpreted queries in 72.2 % of cases and delivered accurate results (via Yahoo!) in 27.8% of tests.

“A clear finding that emerged from this study is that the use of human agents to help interpret spoken queries and conduct searches makes a positive difference in the quality of results delivered when compared to traditional search engines that use algorithmic software to find requested documents or information on the basis of keyword matches," said Peggy Albright, the study’s lead analyst and principal author.

To evaluate the overall performance of voice-enabled mobile services offered by ChaCha, Google and Vlingo for iPhone with Yahoo!, the researchers asked a series of 18 queries representative of six typical mobile search categories (Navigational, Directions, Information Local, Information General, Social, and Long-Tail). For each query the researchers evaluated nine performance characteristics including response time, results accuracy, voice recognition accuracy, number of results received, keytaps required, relevancy of the result, location awareness, use of advertising, and presence of other value-added features. The study further took into account that a service could deliver its search results in the form of answers (as ChaCha offers) or as links to Web pages (which Google and Vlingo deliver); for each query tested, an accurate result could be achieved in either form. 

The researchers also conducted two rounds of tests, using both natural language (speaking the questions in sentences) and keyword (reducing questions to a few specific terms) queries.  ChaCha, a natural language search engine, also outperformed both Google and Vlingo with Yahoo! in its ability to deliver accurate results in response to keyword queries, a traditional strength of universal search engines such as Google and Yahoo!

Both the Google Mobile App and Vlingo for iPhone use automated voice recognition technologies, while ChaCha leverages the use of actual human agents, or Guides, to interpret and transcribe spoken search queries.  The study shows automated voice recognition programs frequently misinterpret words – often the entire question gets lost in translation.

The report also identified a key advantage ChaCha has over its competitors: Its keen focus on social search, an approach that effectively infuses human preferences and human judgments into computer algorithms to pinpoint truly relevant information and potentially better answers.

“In mobile search, where algorithmic search can deliver neither personalized search results nor peer recommendations, social search and variations that tap the wisdom of crowds to deliver the right mix of answers and entertainment have a clear competitive edge over the plain-vanilla search we know from what Google and Yahoo! offer on the PC as well as mobile,” said Peggy Anne Salz, MSearchGroove founder, chief analyst, and co-author of the report. “Beyond tipping the scales back in favor of results that are relevant rather than search- engine optimized, social search also lays the groundwork for a conversation with people on their terms, paving the way for the delivery of mobile advertising that is relevant and more likely to be appreciated.”

Another factor playing in the favor of social search on mobile devices is the progress vendors have made to deliver people a good user experience. "Speech technology has steadily improved, and mobile phones provide an incentive for its use with their small size, safety considerations, and always-there microphone," noted Bill Meisel, President, TMA Associates, and Editor, Speech Strategy News. "This adds a dimension to mobile search that broadens the battlefield as companies fight for domination in this important sector."

The complete report is available for free download from MSearchGroove. This white paper is published by MSearchGroove. It contains the findings of independent research and analysis carried out by Peggy Albright, Albright Communications, and Peggy Anne Salz, MSearchGroove in January 2009. The research methodology was developed by Peggy Albright. The research was sponsored by ChaCha. The opinions expressed in this white paper are those of Peggy Albright and Peggy Anne Salz, and do not reflect the opinions of the organizations referenced in this paper.