Really excited to interview someone who knows search inside and out. Chad Walters has been instrumental working on search at Yahoo, Powerset and Bing. Chad is now semi-retired and I’m especially appreciative of him taking the time to provide these thoughtful answers about his career in search.
1. You’ve had a number of different search engineering roles, can you tell us about a couple of favorites from your career? What was it about those roles that you liked so much?
I got to play architect and manager at all 3 of my web search stints, which was pretty great. At Yahoo, I was really doing a ton of learning but there were still some opportunities for me to have some big impacts — I’ll especially remember the run up to the launch of the Yahoo search engine, cutting over all of Yahoo’s search traffic from Google. I was working crazy hours, reviewing all the code going into the core engine, working very closely with the ops team to debug outages in the data center, interfacing with senior engineering management and even folks on the business side. It was a critical project with a ton of visibility and an insanely compressed timeline and I got to play a pretty significant role in pulling it off without major incident.
At Powerset, I got to apply everything I had learned at Yahoo! and design and build a whole web search engine basically from scratch, all while doing a ton of innovation around applying NLP at scale. We had an amazingly talented team (as evidenced by the numerous successful startups later founded by members of our team) and we were also able to really leverage the explosion of open source that was taking place at that time.
After Powerset was acquired by Microsoft, I moved up to the Seattle area to run the Bing Index serve team, which afforded me the opportunity to have a huge impact on many aspects of Bing and indeed the broader organization since Bing was leading the way in many aspects related to online services, data center management, and more. Some of my favorite projects there were onboarding Yahoo’s search traffic (deja vu!), driving the introduction of SSDs into our data centers, pushing open source at time when it was not embraced in the organization, and overseeing a full overhaul of the core search engine architecture, It was also really rewarding to reconnect with many of the great search leaders and engineers I had known at Yahoo! —Qi Lu, David Ku, Sean Suchter, Knut Magne Risvik, Jan Pedersen, and many more.
2. Your career started as an engineer not focused on search, so how did you switch over to get started working on search?
In my undergrad program (BS in Symbolic Systems at Stanford), I did a concentration on Artificial Intelligence. My early days at Connectix were largely focused on low-level work (RAM Doubler, Virtual PC) close to the machine and very performance intensive. During the dot-com era, I worked at i-drive.comwhere I got some great experience with large-scale distributed systems. After the dotcom bust, I started looking at search because it was a really interesting technical space that was also of high business value. In addition, it turned out to be an excellent way of bringing together the various aspects of my background — AI, performance work, and large-scale distributed systems. I had a friend who was working on the Inktomi search team and he referred me in for interviews right as the Yahoo! acquisition was closing — I think I was literally the first Yahoo! hire into the Inktomi search team.
3. A lot of people think Google’s early success was only because of Page Rank. Years ago you told me MapReduce was Google’s biggest innovation. What was so important about MapReduce?
Yes, I definitely think PageRank’s role in Google’s early success is somewhat over-emphasized, although looking at PageRank did also likely result in Google using anchor text before any of the other engines, and anchor text was probably the most powerful single relevance signal for web relevance. But I strongly believe that MapReduce was most responsible for Google’s sustained advantage over the competition throughout the 2000s. MapReduce provided its relevance engineers and scientists the ability to study massive data sets in a way that simply wasn’t as accessible at their competitors.
Arkady Borokovsky, one of our smartest relevance scientists at Yahoo!, presented the MapReduce paper when it was released and immediately after that presentation, I ran to some of the senior engineering management (Eric Baldeschweiler and Bharat Vemuri) and said we absolutely had to build something similar as quickly as possible; we had a bunch of large-scale infrastructure but it was way more special purpose than MapReduce and simply not amenable to general purpose work . That conversation was part of the chain of events that eventually led to Yahoo!’s investment in Hadoop — and my own interest in Open Source, but that is probably something for a different interview.
4. You contributed hugely to Yahoo search by developing query rewriting rules and technology. Can you tell us about that effort and how it contributed to overall search relevance?
That was a hugely rewarding project. It was really the brain child of Nawaaz Ahmed, who had been a key contributor on the Inktomi team for a while and was probably the single person who taught me the most about search. He had this notion that we could redesign the core of the Yahoo! web search engine to be much more flexible; rather than executing what was essentially a fixed query execution plan with minor variations, we set out to turn it into a fully programmable query execution engine. Three of us — Navaaz, me, and Bob Travis, who had been one of the long-standing members of the Alta Vista team — sat down and designed a query programming language for the core engine (QED) and a new mechanism for query parsing (QRW) that would allow for a chain of annotating and rewriting modules which would output QED programs. Then we reworked the query parser and the entire core engine to support these new mechanisms for increased flexibility without giving up any measurable performance. It was a huge technical challenge, from design to execution to deployment, that delivered some major gains immediately and laid down a strong platform for future innovation by allowing rich experimentation in query execution within the production system. It was my last project at Yahoo! and I’m told it had a major impact for quite a while after that — pretty satisfying to hear as a platform builder.
5. What was it about search that motivated you through your career, and what advice do you have for other people who want to follow a similar career path in search?
Web search is a massive technical undertaking and also has been a huge driver for innovation in a great number of areas — distributed systems, machine learning, Big Data, data center design and operations, and many more — so as a technical guy I always found it to be an amazing playground with so much to learn and explore. It’s also a space where cost and performance are critical — so it was a place where my own strong bent around performance was both highly valuable and highly impactful. In addition, web search has been a space with a ton of business value, especially once the CPC business model came into its own in the early 2000s, so the importance of the work was never in question. Finally, web search reaches a huge number of consumers which made me feel like my work was serving to help improve the lives of many people — even if in small ways.
So the combination of challenge, innovation, impact, and importance really kept me motivated and engaged. The space has changed a lot of course but I think search can still provide that magic combination to many other engineers. My advice would be to dive in, find a place where you are challenged but also provide value, and dig deep to really understand the application all the way from the user level down to the metal as much as possible.