Analysts are explorers

There’s a truism in the data community that companies always make the mistake of hiring an analyst before a data engineer. I understand why this is such a common thing to say—many companies start hiring data teams without the right tools or support systems to set them up for success, and early data hires often end up doing a lot of data engineering work whether they want to or not. Running analysis and developing metrics without well-curated datasets and data models is like hiking a rocky trail. It can certainly done, but you could go further with less effort if you just had a paved path.

That being said, I’m not convinced you should always hire a data engineer first.

Some companies have a strong idea of what they’ll want from their data teams, but most do not. Why else would we see so many postings for first data hires with lists of desired skills and expected responsibilities so long they could represent a department of 20 people? And while job postings like this are usually an indication that this company will have unrealistic expectations of what an individual working with data can accomplish, these postings do speak to a real problem: many companies don’t know what they really need.

At many companies, the data team is new. The folks who join these data teams are often disillusioned because they’re under the impression that they’re going to be moving into some established town, only to arrive and find they’re in a rough outpost or in the straight up wilderness. Their company believes there’s something valuable in there—maybe there’s even a lot that’s valuable—but they don’t know how to find it. In this case it might make sense to start building a town as a base of operations, but you probably don’t understand the surrounding landscape or what resources are available to you. Maybe you’ve got all the information you need and you can build a bustling, prosperous town, but maybe there’s a much flatter meadow just through mountain pass that would have been much better.

This is all to say that data engineers are settlers. Many companies hire analysts first because they need someone to start mapping the wilderness, and analysts are explorers.

Specifically, analysts are chartered explorers, sent on a mission by some more powerful sponsor to figure out where and how they can find a foothold in that data wilderness. This puts analysts in a difficult position—they never know what they’ll find or whether there’s even anything there in the first place, but there’s still pressure to bring something back to the city so their sponsor will think it’s worth the cost to send them back out. This is hard because you can’t plan to have amazing discoveries on a regular schedule.

In order to earn their keep, analysts must use different modes of exploration to reassure their sponsors that they’re not just goofing off in the woods. Much of their time should be spent on scouting, which is taking smaller trips into the surrounding landscape to look for water, find good places to set up camp, and make sure that the game trail they used last time is still clear. Scouting-style work represents many of the more procedural parts of analyst work, like reporting and business reviews, opportunity sizing, and small requested analyses to answer a question related to the roadmap.

Sometimes an analyst-explorers is a part of a party that thinks its on its way to hidden treasure, only to realize they’re not heading where they think they are. In this case, the analyst may asked to do some wayfinding, reading their compass and referencing perhaps-imperfectly-drawn maps to reorient the group, make a guess about where the real destination is, and monitor the route as they try to get back on track. The most dramatic form this takes is when a stakeholder tells the analyst a metric is moving in an unexpected direction and demands the analyst figure out why immediately, but it can also take the form of digging into a natural experiment or running analysis on why a product launch was more or less successful than anticipated.

Every so often, analysts get to do some trailblazing. This is the risky but rewarding stuff—delving into ruins in search of treasure or striking into unfamiliar territory to find natural beauty. Trailblazing is the deep analysis that people claim is the best part of the analyst job. But the rarely mentioned truth is that the wonders they seek may not exist, and even if they do, analysts may need to trudge through mud, get eaten alive by mosquitos, or climb a mountain to get there in the first place. Seeing is believing for many sponsors of analysts, and having to navigate rough terrain using only a handful of trail blazes means they’re not going to come along with you. They’re not explorers themselves, so they’re not going anywhere if there’s not already a road.

This is the reason so many people find analyst careers frustrating, I suspect—they imagine it will be a romantic pursuit, full of discovery and adventure, but instead most days are filled with every day sorts of maintenance tasks that yield smaller rewards. Being an explorer starts to look a lot less glamorous when you realize it how much work trekking through the wilderness actually is.

For many analysts, the day eventually comes when they realize they’d rather be a settler than an explorer, or that they’d rather move to an established town than they would scout out a place to start building one. There’s no shame in this! If you’re going to build a productive society, you’ll need structure and stability so more people can participate without it being an adventure.

The biggest problem I see for analyst careers is that they are misunderstood. The tech industry loves builders, but the work of analysts is more about motion and direction than it is about permanence. When analysts build things as a part of their work it’s often a means to an end, and in my mind the trouble comes from expecting those things to last forever. Instead of getting frustrated with data pipelines built by analysts who were mostly focused on exploration, we should instead view them as the first step in an evolving data architecture and celebrate it as progress when data engineers are brought in to tame the wilderness and building something more durable.

We also need to recognize that while exploration can be highly open-ended, that doesn’t mean there’s no technique to it. You’d prepare to travel through a desert differently than you would a rainforest, much in the same way that you’d analyze a SasS business differently than you would an e-commerce company. There are best practices out there, and we should be thinking more about how to document them so we can actually prepare our analyst-explorers for what they’ll need to do.

And most importantly, we need to help analysts appreciate the work exploration itself and not just to crave being lauded as a hero for making a discovery. It may never be fun to be swarmed by mosquitos, but many people love climbing mountains. Even if the view up top isn’t exactly what you imagine, you can revel in the peaceful feeling of exertion when you rest at the peak and you can marvel at all the wildflowers you saw along the way.