It’s great to have James Rubinstein join us for a talk about his life in search. James currently works at LexisNexis and has previously work at eBay, Pinterest, and Apple working on search and related projects.
What is your role and what does a normal work day look like for you?
My role is “Director of Global Product Testing” which is not a super informative title. What I actually am is a data scientist, product manager, and data science team manager. I like to say that I am the lead cheerleader for A/B testing and data-driven decision making at LexisNexis. I don’t think I have a typical day … every day so far has been different. I do analysis, strategy (read: PowerPoint), and team management. These days, I’ve been doing a lot of recruiting as we are growing our Product Analytics team. I’ve found that data scientists with search, testing, and stats experience are a bit thin on the ground.
You’ve worked on search at Apple, eBay and Pinterest. Can you share some of the favorite projects you’ve worked on during your career?
There have been some really fun projects that I’ve worked on in my time. My first “baby” at eBay was our survey tool. That got used throughout the company, including for search feedback. The survey tool got me introductions to so many people at eBay. I recall working on building the evaluation pipeline for Search at eBay, and we used that to evaluate our first (and subsequent) efforts at machine learned ranking. I learned a massive amount there. The other thing I really loved at eBay was moving to a pure product role, focused on improving the Bid Layer (which is kind of important to eBay’s image, loath as they are to admit it). We used that experiment as our first betting market for experiment results!
At Apple I got to work on defining how we’d measure our point-of-interest database, and brought crowdsourcing to iTunes relevance measurement.
Pinterest was a lot of fun, with that startup mentality, we were tackling new challenges every day. Measuring query-independent Pin quality is the thing I’m most proud of. We also implemented an interleaved search AB test methodology, which was pretty cool.
At LexisNexis, growing the team, the impact that data science and experimentation can have has been the most rewarding thing. I’ve seen data go from “eh whatever” status to “first class citizen” status very quickly.
The thing I most enjoy in my job is working, collaborating, discussing, and building with great people. We don’t always agree, but we come together, try things out, and see what works. I’ve been extremely fortunate to work with smart, talented, passionate people everywhere I’ve been — not always in the places you’d expect, like UX designers often have a great feel for search and discovery problems. Don’t take the advice of others for granted, even if they aren’t “search experts”!
What are some of the challenges a company like LexisNexis faces with search? It must be a unique search space with searchers who are very sophisticated in their search behavior.
LN is really interesting. It’s an old company (by internet standards) that really led the way on electronic information retrieval back-in-the-day. But the world has moved on a bit from the 1980s. What’s so challenging is that our users want to keep searching using the old Boolean methods. Our customers are attorneys and they want super-precise results AND total control. They are trained how to search in law school. It can make it difficult for us to change how we do things. How do we make changes that improve the search experience without upsetting people’s workflow in such an important industry? It’s not easy, and we have to strike the right balance and be cautious in how we proceed when we make updates to our search algorithms.
How do you measure and improve legal search relevance? Do you get a lot of user feedback?
We use Human Relevance Testing, led by my friends Tara and Doug (https://haystackconf.com/2019/human-judgement/) and we AB test using our in-house AB testing platform, ABE. We also collect qualitative feedback from our users through the efforts of my colleagues in UX research.
How did you get started working in search and what advice do you have for other people who want to get started in the search industry?
Well, that’s a bit of a long story. When I was in grad school, we had this interesting data set that Craig Treadaway had collected as part of his Master’s thesis on individual differences and search outcomes. The dataset had been collected by an instrumented browser built by my friend, Andy Edmonds. So, while Craig had focused on outcomes, my advisor, Lee Gugerty, and I were taking the same data and looking at the process of how people searched for information online. Andy was helping us with the data and some of the analysis, so I got to know him through that project. That research also really got me interested in Search as a topic, and got me introduced to luminaries like Sue Dumais, Pete Pirolli, and Stu Card. After I graduated, Andy called me up and said he was putting together a team at eBay and wanted to know if I was interested. I was and I went.
If you really want to get started in Search, start with some of the basic IR literature, there are several good books out there for free, like Modern Information Retrieval. Start there, then go read some papers by Sue Dumais and her MS Research colleagues. Then dive deep on metrics. Metrics are the key to understanding search, I believe. There’s also a great course on Udacity that is purportedly to teach python coding, but you learn through the context of building a scraper and search engine. That is amazingly informative. Once you have a handle on that stuff, give me a call