Part 1 Skunkworks Project
Several years ago I worked with two other people to build a social media search and discovery platform called the Social News Network. SNN, as we called it, crawled and indexed outbound links from Reddit, Twitter, and other platforms. We then we applied our own ranking function to surface the most interesting and popular stories by topic.
So you could click into Sports -> MLB -> Padres and get the articles, images and videos that were the most popular of the day about the team.
To build it, our three team members were divided as such:
- Our engineer – he built the scraping and indexing functionality, and the front-end.
- Our catch all guy – he defined the product and did the branding.
- Me – I built the taxonomy and generated the heuristics and training lists we used to fine-tune things.
We were lucky because a friend of ours had built a ranking algorithm for something else he was doing and he let us use it. So we had sort of a plug and play piece, though we tweaked it to our specific needs.
We worked with the lean startup approach. We built the minimum viable product (MVP) which consisted of all the MLB teams and then a few other categories. We were not trying to capture all internet intents until we learned if it was even working. And if anyone even cared. If you’re not familiar with the lean startup approach, the idea is that startups should build as little as possible at the beginning and then see if anyone cares to use it. Rather than investing years and a lot of money before validating the idea. If no-one cares, you can pivot to something else or abandon the idea.
After we had our MVP up and running we sent it around to some friends who are baseball fans. So there was a person for the Mariners, Indians, Red Sox, Yankees, Dodgers, A’s, Giants, and me following the Padres. I asked each person to check out their team’s feed and provide any feedback. We also turned on Google Analytics so we could track user behavior.
The feedback was decent. People seemed to like it, but then again these were our friends so they would lean towards being supportive. The analytics showed moderate usage, but not as much as I had hoped. I knew this group of guys were fanatical about their teams and I knew they were checking the latest news every day. I don’t recall the exact analytics, but in general we had people come once every day or two to look at their team’s category in SNN.
But we really liked what we built. We liked how it leveraged various social media platforms to identify the best content about a topic. And while we knew we had not reinvented the wheel, we felt it had so many applications beyond sports teams and could be applied for hobbies, news, even mapping to advertising. Of course the web’s history is littered with vertical search products like ours, but we thought we were onto something because we had built everything with only three people and we felt we could scale to cover much of the web with only a couple more people on the team. All in all, we predicted we could have real-time indexing and document ranking across thousands of topics in a UI that was easy to use and easy to slice and dice for distribution. Win win.