Back in April 2019, FullStory published an article about analytics theater: a term they coined to refer to the shallow and virtually meaningless analysis that underlies many of today’s web analytics platforms.
The article did a good job of explaining the phenomenon, but we think there’s a larger concern worth addressing: while some analytics platforms are mere analytics theater, there are plenty of specialists out there running analytics comedy –a farcical attempt at data science that’s closer to a slapstick bit than business insight.
Whenever a client hires a specialist, there’s an implicit understanding that the latter knows what they’re doing. But thanks to the knowledge gap that’s built into these exchanges, it can be exceedingly difficult to spot a faker until the joke has run its course.
To help you avoid getting screwed over, we’ve compiled a list of red flags you should watch for when working with a supposed expert in the field of big data.
1. Your analytics specialist insists that, “it’s tough to explain.”
If you’re a paying client of an analytics specialist, then you’re entitled to know where and how your budget is being spent. You deserve a breakdown of associated costs (ex. the cost of subscriptions to analytics tools, or the cost of gathering data), a briefing on any processes you’d like explained to you, and an explanation in the event of low ROI.
That’s why there’s rarely an excuse to be shrugged off with a dismissive, “What you’re asking for is hard to explain.” This kind of cop out answer is a red flag that you should definitely look out for when managing a consultant –not just in big data, but in any field of specialized knowledge that you’re outsourcing.
A specialist is only ever an expert on a given subject when they can teach it. Whether it’s through a crash course or a brief explanation in the style of ELI5, you can and should expect the specialists you hire to provide answers when asked.
This isn’t to say that you should take every chance to interrogate your consultants, but you’re definitely better off hiring a person or team that can talk you through the particulars of what you’re paying for.
2. You’re receiving a lot more visuals than insights.
Data visualization (or data storytelling, depending on who you’re asking) is the practice of drilling data down into a more accessible form like a chart, table, infographic, or whatever you call this.
Executed properly, these visualizations can be very useful tools in distilling and communicating information.
Executed poorly, they can give off a compelling illusion that the person you’ve hired to provide you with business insights has no idea what’s actually going on. That’s why the second red flag you should watch for is an empty fixation with visuals.
Towards Data Science published an article on why data visualization is an overrated fad, and their bottomline is salient: people need to be more critical of the difference between a graph designed to convey ideas, and a graph designed to sit still and look pretty.
The latter is a form of shock and awe that data pretenders use to give off an air of expertise. At the heart of every useless visual is the assumption that a client would believe anything you tell them if you flash some interesting media while you do the telling.
For best results, ask yourself the simple question of, “What is this chart really telling me?” when looking over a report or sitting through a presentation. If you aren’t satisfied by the conclusion you reach, you could always ask for a more useful interpretation.
3. You’re hearing a lot more numbers than sense.
The human mind wasn’t designed to think in terms of abstract numbers. For most people, having a million of anything is utterly inconceivable, and minute differences in percentages can come across as meaningless. There’s simply more work that goes into drawing insight from data than saying, “These are the numbers.”
To understand and communicate data in a way that yields results, everything needs to be contextualized: spelled out in terms that we’re naturally more adept to comprehend. The best data specialists use narratives. Spot the difference between the following two examples:
1. “Your sales are down X% in this particular region.”
2. “Your sales are down by X% in this particular region. We can examine this further, but it’s probably because of phenomenon Y for the following reasons… I recommend doing Z, because…”
Yet as anyone who’s delivered a report can tell you (after a few drinks, when they’re feeling honest), numbers can be a theatrical crutch, and dropping figures and statistics is a rhetorical device that’s been around for as long as people have needed to sound credible.
No matter how impressive it might be that your data expert has uncovered a set of numbers and trends from your data, their job isn’t done until they’ve woven those numbers and trends into a narrative or plan that yields an effect.
Moreover, every link in the causal chain has to be accounted for. Don’t settle for a number and a recommended step –demand a logical sequence of cause-and-effect if you want to be sure that your specialist isn’t taking blind guesses.
4. Your analytics specialist over-promises.
Analytics specialists are valuable because they deliver actionable insights. After handling your data, spotting trends, conducting experiments, and forecasting results, you should be left with a series of recommendations that lead you to a certain result –spend more on A to achieve B, or do X a little differently if you want to get to Y.
The expectation that often arises here is that your expert can tell you what to do to increase your revenue by a specific amount, and by a specific date.
This expectation is false. You’re hiring a business insight consultant, not summoning a genie; they’re bound by the limits of statistics and probability. As such, any consultant who promises you fixed numbers is either an absolute genius or an absolute fraud, and you can go ahead and guess which is the likelier bet.
A consultant’s track record for increasing sales, lowering costs, and delivering ROI are fair game when pitching an engagement –after all, they have to compete and make a living just like anyone else. However, once an engagement begins, the best that any data specialist can offer is a statistically sound range and a degree of certainty.
Spotting this red flag is a non-negotiable. Data science is an exercise in probability, and there are no guarantees.
Getting screwed over in an engagement with a so-called expert is no laughing matter, and you should never settle for an analyst who fails to deliver on their rate.
The list we’ve presented should give you an idea of the profile of a consultant worth hiring: they’re quick to explain, they leave you with complete and well-reasoned recommendations, and they keep away from absolutes.
As far as industries go, big data has a lot to give. Be mindful, be critical, and make sure that comedy is left to the people who can pull it off.