In my “3 Key Learnings from the Webinar ‘Modern Data Warehousing without the Burden of ETL” blog, I discussed what data management expert Dave Wells of the Eckerson Group believes those looking to modernize their data warehouse must acknowledge and embrace in order to succeed.
Dave became a fan of Incorta’s completely new approach to enterprise analytics after learning how our technology does indeed bypass the troublesome Extract Transform Load (ETL) process. In fact, it’s safe to say we caused Dave to rethink his stance on data warehousing. So much so that we wanted to dedicate a blog to find out exactly what it was about our technology that shifted Dave from Incorta skeptic to Incorta advocate, and what he believes our new approach means to the future of modern analytics.
Here’s what Dave had to say.
Dave, I always like to start with some background. So, in your opinion, before Incorta, what was the most innovative technology that had emerged in the data warehouse space?
Aside from the emergence of Big Data—which simply created more and bigger data silos difficult to connect together for intelligent analysis—enterprise data really had stagnated with ETL and the data warehouse, with very little innovation coming about for both of those areas since data warehouses were first introduced.
Why so little innovation since then?
I believe a confluence of several factors happened to stagnate data warehouse innovation. There were no strong drivers to do something more, and—at the same time—companies were focused on other priorities. That’s because even though data warehouses didn’t live up to their initial hype, they were providing some value, so leaders turned their attention elsewhere. Then, in 2008, tech budgets were slashed, so even those organizations that wanted something better than their existing data warehouse or warehouses couldn’t afford to invest in a new solution anyway.
That was a painful time indeed. Moving ahead to today—what do you see as the biggest business challenges for enterprise analytics and data warehousing now?
I see three main business challenges for enterprise analytics and data warehousing right now. One of the biggest challenges is for organizations to step back and rethink and re-architect their entire data management infrastructure. They’re trying to adopt modern analytics technologies, but everything they have in place to manage their infrastructure is built for legacy technologies. Massive data volumes can’t work with batch processes—old architectures don’t handle that kind of thing gracefully. And patching new technologies on top is not sustainable. So they need a completely new technology architecture and infrastructure.
Another big challenge on the business side is data literacy—getting people to be data-literate and truly capable of analyzing data, visualizing data, and reading data visualizations accurately. Data literacy is the single most important factor in creating high-impact information and preventing destructive misinformation. Schools don’t teach us to think and create visually, but, to be data-literate in the business world, we need to create and read in ways that don’t misinterpret data.
The third business challenge I see is inefficiency created by self-service analysis—redundant, inconsistent work that delivers different answers to similar questions. Now that so many people have access to data and tools like Tableau and Qlik, it’s pretty much a given that the same work is being done over and over again because analysts don’t know someone else also is doing it. Data is integrating, but analysis is disintegrating.
Thanks, those are great insights! Moving on to Incorta and what we do—I remember that when you first heard about Incorta, you were quite skeptical. Can you explain why?
Frankly, I was skeptical about Incorta’s performance claims because you set such a significantly high performance bar. You live up to your promises—you’ve done something really innovative that takes query performance to the next level—but when I first heard the numbers and ranges you published for high-speed query responses, my gut response was, “That can’t be real, that has to be exaggerated.” But it was real.
I also was a bit skeptical because Incorta’s Direct Data MappingTM and real-time data transformation sounds a lot like promises made by data virtualization. Because of that similarity, I was skeptical of Incorta’s ability to handle complex data transformations. That initial assumption, however, turns out to not be true either.
Our Direct Data Mapping technology really is something else—hats off to our founders and engineers for that! So what exactly was your “a-ha” moment with Incorta?
The really big thing was the moment I saw the value of Incorta’s technology in becoming the modern data warehouse—a total retake on data warehousing, a data warehouse without having to do ETL and all the heavy lifting integration. In my opinion, pundits who declare data warehouses dead are absolutely wrong. More than 60 percent of respondents to a survey I recently conducted state they have 2-5 data warehouses. If that many organizations operate data warehouses, then data warehouses clearly are not dead. And they’re not likely to die off anytime soon, either, because businesses need the enterprise history data warehouses provide to do time series analysis.
So that was the a-ha moment—when I realized Incorta was approaching the problem from a different perspective. You have a different take on data warehousing—Incorta doesn’t call it a data warehouse, Incorta doesn’t look like a data warehouse as we’ve known it for three decades, but it is a data warehouse—and it’s one with all of the value side of data warehousing, and less of the time and labor side.
It was at that point I realized Incorta has a very central and prominent position in the modern data management landscape.
We appreciate that. So from your perspective, how do Incorta’s approach and our Direct Data Mapping revolutionize the data analytics market?
Past processes in data management haven’t been very data-friendly, and we need to manage data differently now. Incorta’s Direct Data Mapping understands all of the relationships among the data without having to physically substantiate it—it removes all that ETL overhead of physically creating those relationships within a single, integrated database.
So in my mind, Incorta revolutionizes the data analytics market by making all the data it manages—enterprise data, Big Data, external data—available in one place, in its raw form, so we don’t have to undertake any data transformation operations that damage its analytic value. Traditional data warehousing processes actually cleanse away elements valuable to data scientists. Incorta, on the other hand, makes data quickly accessible—closer to real-time data—and better manages data relationships, without discarding things like historical snapshots or cleansing away analytic value. Those are the things that enhance the modern analytics world.
I agree. With all of that in mind, how has Incorta’s new approach changed your overall perspective on the enterprise analytics market?
I see Incorta as a new and very different design pattern from the approaches of other vendors.
Incorta shows me there are many different ways to step up to the problems we face with legacy data warehousing. I see it as a technology that stepped back and rethought the problem before trying to build the solution. So many of the data warehousing technologies we have are about further optimizing, further accelerating, further automating the things we already have. Those approaches are valid and move us ahead on the path, but Incorta took a step back and rethought all of the assumptions. As a result, you came up with a unique and very powerful approach to overcoming the data warehouse challenges.
Stepping back from the problem, and thinking about it differently, like Incorta did, is something we need more of.
Thanks, Dave, those are great points. Now, lastly, I know when we first speak with prospects, they often struggle to understand the simplicity of Incorta compared to the complexity they face day in and day out with their data warehouse-based architecture. When you introduce Incorta’s new approach to your clients, how do you describe or explain it?
I explain the need for them to rethink their data management architecture, so they can have the benefits of data warehousing without the overhead of lots and lots of ETL. No longer can they afford to transform data in such a way that it’s suitable for reporting but has lost some of its analytic value.
I also use the term “fast data movement” to describe what Incorta does: collecting time-variant data without difficult, complex data pipelines; utilizing scalable and elastic technology, so you don’t have to know all of your future data needs or future workload needs to get on board; and leveraging smart technology—like Direct Data Mapping—that simply can look at the data to determine what the relationships are.
Lastly, I reference the important role of automation as the key to scaling data engineering in the future. Modern analytics platforms continue to create more data and more consumers of data. Demand for data drives the need for more and more data engineers, and those data engineers just aren’t out there. In fact, the data engineer shortage is even greater than the data scientist shortage.
Want to hear more about Incorta from Dave Wells and the Eckerson Group? Watch the “Modern Data Warehousing without the Burden of ETL” webinar recording or download the corresponding Eckerson Group analyst report.