Opportunities for the Modern Data Stack

The modern data stack (MDS) is the defining paradigm informing how new organizations use technology to leverage data. Lots of digital ink has been spilled, and takes are growing ever spicier on all things MDS. Here’s a more benign one:

The Modern Data Stack currently focuses too much on tooling and would be more effective if it refocused on the humans using the tools. In particular, the MDS ecosystem doesn’t currently address implementation, tool adoption and certain other non-technical oriented endeavors that are critical for long term value generation. As a result, stakeholders will forever complain about the lack of value they’re deriving from their data teams.

The Modern Data Stack largely focuses on technological solutions, some process improvement, and says very little about how people actually use data. I think there are two broad reasons:

  1. Engineering problems are easier than messy, human problems
  2. The Modern Data Stack is crafted with an engineering bias, perhaps because it’s easier (per the point above) or, because of where data sits in the engineering <–> business spectrum.

As Michael Kaminsky puts it (in our Slack chats on LO):

…data is unique compared to software in that data sits right where the software meets the meat-ware in an organization. To the extent that MDS tooling has been driven by “classical” software engineers, it’s reasonable for them to only think about the left-hand-side users of the tools (other computer systems, presumably reasonable software engineers) and not the right-hand-side users (the business that works in a political environment where the “right” decision cannot be logically deduced). So companies building tools in the data space need to make sure that they’re meeting those needs instead of just the systems-engineering needs.

For example, the MDS is great at offering solutions to specific classes of problems:

These, and others, are great capabilities that I’m glad smarter folks than me are building. That said, these siloed solutions need to be integrated, not simply in a technology stack, but into an organization’s processes and culture. A metrics layer is only going to solve the debate that Marketing and Growth have about your LTV / CAC ratio when paired with people and process changes, since it’s as much (if not moreso) a human-oriented, biased decision making process, as an objective, technological one. Real change happens when people build consensus, take stock of their actions/implications and iterate, with tech setting the stage for the conversation.

That said, why else do we bother with this if not to drive value? If we want the MDS to help organizations become data-driven (or data informed or whatever), we need to address the people-oriented considerations that are dependent on, or a consequence of, what the MDS enables.

As someone who, in part, helps organizations configure and use the tools in the MDS, I’ve observed a few areas in which I believe we as a data community can bridge the gap between narrowly focused, strictly technological solutions, and human-oriented endeavors; the latter which span business functions and take many months for feedback, and is necessary for value creation.

Painless migrations

The Modern Data Stack provides an obvious and easy solution for organizations buried in manual, Excel-based workflows (caveat: sometimes Excel is the best solution for certain problems).

However as the teams and complexity of an organization grows, it becomes harder to wean folks off their favorite workflow. Take a migration from a collection of SQL scripts to dbt as an example. Some considerations include: