One Platform, All Your Data Needs…

Many of us are tempted by the idea of simplicity. No hassle, no fuss. It sounds idyllic. So, it’s no surprise that many people working in enterprise IT – including me – have at some point strived to deliver that vision. A ‘single data platform’…

For decades, Data Warehouses and Data Lakes were at the sharp end of this vision, bought and sold with the promise of storing and processing all your data. The promise of supporting all the analytics, applications and use cases we aspire to. The promise of infinite scalability and cost effectiveness. The promise of being the only data platform we would ever need to deploy.

Year after year, decade after decade, implementation after implementation these promises failed to deliver. The single data platform vision never materialized in the real-world.

Keep It Simple, Stupid

The idea of striving for simplicity is a good one, arguably the right one.

In complicated organizations there’s diverse technology, data, and needs. This inherent complexity causes problems. It becomes difficult to meet SLAs, challenging to manage costs, and expensive to deliver change. Governance becomes a bureaucratic nightmare. It’s good to focus on simplicity because it reduces complexity and the problems it brings.

Achieving simplicity often means reducing diversity: rationalizing suppliers and technologies, standardizing processes, aggregating data, etc. In many areas, this works well. Supporting complexity in commodity areas where there’s little innovation makes no sense. Over time, most technologies become commoditized and are replaced. However, this isn’t true for data.

Data isn’t technology. There’s a symbiotic relationship between the two – technology creates data and data requires technology – but they’re not the same thing and need to be treated differently, click to tweet.

The Anti-Commodity

Try as we might, data refuses to be commoditized.

The level of data diversity increases with each passing second. We’re creating more data than ever before. Structured and unstructured formats continue to proliferate. We’re realising value in more ingenious ways. Data enables digital experiences, driving intelligent automation, and fuelling algorithms. Data is increasingly recognized as a valuable business asset. But this value creates risk around ownership, value transfer, security and regulation, click to tweet.

Data has a wider scope of use than technology, but only when partnered with technology and people. It’s also persistent – many analysts and applications still use data created decades ago. When new data emerges, the old data remains; they coalesce and become more useful than before. Unlike technology, data’s created, not designed. The path to finding a use case is often slow, chaotic and experimental, rather than clearly articulated from the outset.

A False Hope

This makes it crucial to manage data and technology differently. The desire for simplicity in technology makes sense, but for data, it’s a false hope. Data will always be required across a diverse range of platforms, applications, and processes that are in a constant state of flux.

There isn’t a single data platform that meets all your needs. And that’s okay. A single data platform often means data is held captive and fails to serve the diverse range of needs, click to tweet. It also creates technical debt, increasing the time and cost of making the inevitable move to a better technology. A single data platform means trying to fit use cases to the technology, and rejecting those that don’t, rather than finding the best technology. A single data platform means waiting for your needs to be met by a single supplier with many competing demands.

Whilst a single data platform seems appealing, it fails to deliver the benefits of simplification seen in other areas and introduces new complexity. 

Manage Complexity

It sounds obvious, but the solution is to manage data complexity, not eliminate it. This means investing in dedicated infrastructure that enables the following:

  • Ownership – Source system owners must maintain control over their data. The value should be tracked and quantified. This involves treating data as a product.
  • Portability – Data must be portable across any boundary to maximize its value. Access and movement should be enabled, so data can either be moved to where it’s needed or used in-situ without the owner losing control.
  • Interoperability – Maintain multiple formats and versions of data to support interoperability and ease of use. Avoid restricting data to specific formats or schemas until it’s being used. To avoid duplication, enable customization at source and the ability to collaborate and share.
  • Governance – Implement a ‘full-lifecycle’ approach to data management, so consumers trust the data. This also helps to ensure the complete history of the data is recorded, regardless of where it was used.

A Moral Lesson?

If this fable has a moral, it’s that there’s value in complexity, but only if you can manage it. There’s no such thing as a ‘single data platform’ and there never will be. The unique characteristics and inherent complexity of data requires a different approach. It’s crucial for those seeking to maximise value to invest in infrastructure that enables data management and distribution. This must be independent of the infrastructure and applications designed to use it, which continually change.

Managing data complexity requires more than technological investment. It requires a shift in thinking about the importance of data relative to technology. It requires an acceptance that data has more permanence than technology. It requires the creation of a data-first culture focussed on delivering business value.

Reject the single data platform vision that has always failed to deliver in the real-world. Embrace a vision where data complexity is well-managed and its value can be fully realized, click to tweet.

Authored by Anthony Cosgrove (Co-founder) at Harbr