If I could communicate one thing to my fellow data and analytics practitioners, it would be this: The thing that feels like failure to you today actually holds the seeds of great success.
What thing am I talking about?
Questions.
When you deliver an analytics project to a real living human being–not a theoretical business user–and their mind goes to work generating a hundred new questions, that’s a success. It’s only a failure when you can’t answer those questions because it’s too hard. It’ll take too long. The project’s over. There’s no budget.
This is the situation we have with analytics pipelines fed by a traditional enterprise data warehouse (EDW).
Don’t get me wrong. The EDW is very useful. It can store a high volume of accurate and trusted data which is often organized in an easy-to-understand way. It serves as the central hub of many analytics practices and enables enterprise-wide reporting at an impressive scale.
The downside to the EDW is that the technology doesn’t allow for a whole lot of flexibility, and that has created a culture of rigidity. When you’re building a data warehouse, you need to know today what your business needs are going to be in five years. You have to get everything right at the outset, because making changes is painful–so painful that data warehouse teams try to do the impossible, which is to design the perfect requirements document that is going to capture everything the human beings who will use this data could ever want.
What happens is, you work on that data warehouse for months and months and then you build the dashboards and reports and you give them to the end users and they’re happy for two days. And then someone sends you an email saying that now that they have this, they’ve realized they actually need all this other stuff, and is there any way they could get that too?
In the data warehouse world, the answer is, get in the queue and maybe in a year or two we’ll be able to get around to it. It’s viewed as a failure, which leads to a culture that actually wants to tamp down questions. They don’t want to hear them. That’s the culture that grew up around traditional data warehousing and business intelligence. It’s a culture of “no.”
Up until recently, most analytics teams have focused solely on solving the problem of large data volumes, and a lot of progress has been made. I wouldn’t say it’s been solved, but it’s much more addressable than it was a few years ago. But that still doesn’t solve the problem of bringing the data together, and in a fast way, in order to make sense of it. We’re starting to make progress on that now, and the primary driver is an increasingly urgent need for faster and better ways to use data to find answers to business challenges.
In the past, my customers were more oriented around solving the technical challenges of big data. What I hear more of now is, “We have these specific business problems. And we’ll use whatever technology can help us solve them.”
This is a trend that I’ve seen emerging over the last few years, and it is what led us to partner with Incorta. We do data warehouse work, and we were actively looking for emerging technology that could help unravel the technical and cultural problems that I’m talking about. Incorta’s Direct Data Platform gives enterprises the means to acquire, enrich, analyze and act on business data with unmatched speed and simplicity, bypassing these cumbersome data warehouse processes. Incorta has the ability to bring data sources together very quickly. You can add any data source in the world in an afternoon. But the kicker is that once you have something in there and people start asking additional questions, it’s trivially easy to expand the world of data that’s available to them beyond what’s in the data warehouse. No longer do you have to pick and choose what you think is going to be important and hope you get it right.
The desire to use data to solve business problems fast has accelerated since the pandemic hit. Looking at the disruption that’s happened in 2020, I doubt anybody working with a traditional data warehouse has been able to meaningfully incorporate the new types of data that we find ourselves needing. They probably have to extract data out of the warehouse and mash it up with their new data sources somewhere else, because the warehouse can’t accommodate it.
Meanwhile, we’re working with a bank that is using Incorta to answer questions that have arisen out of the COVID crisis. Questions such as, what’s the difference in product mix between our existing client base and clients who came to us through the Paycheck Protection Program?
We were able to help them answer that in just a couple of weeks using Incorta, but if we had stuck with the bank’s traditional methodology, it would have been a multi-year project. We were also able to help them find answers to questions they’ve had forever and have never been able to answer, as well as a whole bunch of new questions that arose from the data. Normally to get a bank to change the way it operates and the technology they use is a very long process, but this bank was open to a different approach because of shifting attitudes and changing circumstances in the business.
When you deliver the first set of data to an end user and they come back to you with questions, you’ve generated an ‘aha’ moment for them. That is a success. And if you can get them further into that process while their mind is still in that state of curiosity, they’re going to take it and run with it. Whereas if you say, we can get you that in three weeks–which in the data warehouse world is fast turnaround—they may still be interested in the question and it will probably still be relevant to the business. But it’s what’s happening between their ears. That initial excitement is gone. You want to capture them in the moment where their head is still in that space, and that is what Incorta can do. It feels like a tool that can create a culture of yes.
Ryan Dolley is Director of Technology at PMSquare, an Incorta partner. He will be speaking on a number of analytics topics next week at BACon 2020, PMSquare’s annual Business Analytics Conference. This year the focus will be on solving organizations’ data and analytics challenges throughout the analytics life-cycle, from source data challenges to end-user presentation. Register here.