How to Drive Value in Data Management: Apply These 5 Lean Principles

September 14, 2022

If there was a way to both improve how data is managed in your organization and help drive value from that data, you'd be interested, right?

But what if that idea came from over 70 years ago?

The truth is that the Lean methodology, a concept that grew out of a Toyota Production System  developed in the 1950s and '60s (to optimize factories), continues to be highly applicable and effective in many different fields today. Data management is one of them.  

This article will open your eyes to how Lean can help your organization improve right now.

Lessons from Field of Dreams

I like to contrast the Lean methodology with what I call the “Field of Dreams” methodology. Spoiler alert, Field of Dreams was a 1989 movie where Kevin Costner played a farmer who hears a mysterious voice say, “If you build it, he will come.”

Despite risking financial ruin, the farmer plows his cornfield and builds a baseball diamond. Then ghosts of famous baseball players magically appear, hundreds of fans show up, Kevin Costner is saved from bankruptcy, and everyone lives happily ever after.

What does this have to do with data architecture? Many IT departments think the same way as the farmer in “Field of Dreams.”

They think if they just build the perfect data warehouse, users will automatically flock to it.

If that doesn’t happen, then they believe that clearly the problem was selecting the wrong technology or not building the data warehouse well enough. So they tear it all down and rebuild it with the latest cool thing.

That’s an unfortunately common strategy that you need to avoid.

The Essence of Lean

Lean takes the opposite approach. Go to your customers and then build. Lean is laser focused on providing customer value. Anything that does not contribute towards customer value is considered waste and should be removed.

The data space has two major differences from traditional Lean practitioners. The first is that our customers are usually internal to the company. The second is that building software isn’t building the same product over and over, like a factory. However, if we abstract a bit, we find that these differences don’t matter much.

So, slightly modifying the five-step process that guides the implementation of lean techniques (created by Lean Enterprise Institute, Inc. in 2016), here are five lean principles you should apply to data management.

#1: Define Value

Value is defined by the person using the product, not the person making the product. The problem is that users often struggle to articulate what they really need. Sometimes they know they have a problem, but they have no idea what the solution is. This can be a lot of work to decipher, but it is absolutely crucial.

Data does not have intrinsic value like gold. It needs to be used to create value. If no one is using your data, then all the effort is a waste.

In business intelligence, the end product isn’t reports or dashboards. It is insight and efficiency gained from the data. Reports and dashboards are just a means to that end. So this is the time to define what “value” means, or, simply stated, identify what goal you want to achieve with your data.

#2: Map the Value Stream

The value stream maps all the activities it takes to create end-user value. You need to identify every activity that must happen for you to deliver the value you defined. Each activity or action can fall in one of three categories:

1. Useful: materially contributes to user value

2. Necessary: mandatory due to legal/compliance/ethical reasons.

3. Waste: does not contribute to user value

The obvious response is to take all the waste activities and get rid of them. This might be easier said than done, as it will often require organizational change.

#3: Create Flow

Once you have removed waste, you want your development process to flow smoothly. Remediate bottlenecks or delays in your process. Communicate with others involved in the process and reach an alignment about flow. If you accomplish that, you’ll have a more efficient process and higher productivity.

# 4 Establish Pull

This step is about minimizing inventory and delivering to your customer as needed. If we were talking about products, we’d want to decrease the time it takes for customer to pull the product, i.e., time to market. With data, we want to decrease the time it takes for the customer to “pull” or access the data needed. You want to ensure that it’s available readily and as frequently as needed.

The closest equivalent in IT would be large backlogs and long delivery times. You want to decide late, minimize work in progress, and iterate quickly. If this sounds like Agile methodology, that makes sense. Lean was one of the inspirations for it

#5 Strive for Perfection

Lean is not a one-time process. It is a loop of continuous improvement that gets repeated over and over. Lean leads to collaborative thinking, efficiency, and an improvement mindset.

If you apply lean principles successfully, everyone in the organization begins to think about managing data in the same way, and then you have the buy-in to truly optimize the entire flow of data, not just optimize in a silo.

If data management is something that you need help with, Onebridge’s data and analytics experts are always here to help.

Flock of SQLS comic on why being customer-centric in data management is imperative (click to enlarge).

The original focus was on optimizing factories, but today it’s applicable to many different fields – including data management.

I like to contrast the Lean methodology with what I call the “Field of Dreams” methodology. Spoiler alert, Field of Dreams was a 1989 movie where Kevin Costner played a farmer who hears a mysterious voice say, “If you build it, he will come.” Despite risking financial ruin, the farmer plows his cornfield and builds a baseball diamond. Then ghosts of famous baseball players magically appear, hundreds of fans show up, Kevin Costner is saved from bankruptcy, and everyone lives happily ever after.

What does this have to do with data architecture? Many IT departments think the same way as the farmer in “Field of Dreams.”

They think if they just build the perfect data warehouse, users will automatically flock to it. If that doesn’t happen, then they believe that clearly the problem was selecting the wrong technology or not building the data warehouse well enough. So they tear it all down and rebuild it with the latest cool thing.

That’s an unfortunately common strategy that you need to avoid.

The Essence of Lean

Lean takes the opposite approach. Go to your customers and then build. Lean is laser focused on providing customer value. Anything that does not contribute towards customer value is considered waste and should be removed.

The data space has two major differences from traditional Lean practitioners. The first is that our customers are usually internal to the company. The second is that building software isn’t building the same product over and over, like a factory. However, if we abstract a bit, we find that these differences don’t matter much.

So, slightly modifying the five-step process that guides the implementation of lean techniques (created by Lean Enterprise Institute, Inc. in 2016), here are the five lean principles you should apply to data management.

#1: Define Value

Value is defined by the person using the product, not the person making the product. The problem is that users often struggle to articulate what they really need. Sometimes they know they have a problem, but they have no idea what the solution is. This can be a lot of work to decipher, but it is absolutely crucial.

Data does not have intrinsic value like gold. It needs to be used to create value. If no one is using your data, then all the effort is a waste.

In business intelligence, the end product isn’t reports or dashboards. It is insight and efficiency gained from the data. Reports and dashboards are just a means to that end. So this is the time to define what “value” means, or, simply stated, identify what goal you want to achieve with your data.

#2: Map the Value Stream

The value stream maps all the activities it takes to create end-user value. You need to identify every activity that must happen for you to deliver the value you defined. Each activity or action can fall in one of three categories:

1. Useful: materially contributes to user value

2. Necessary: mandatory due to legal/compliance/ethical reasons.

3. Waste: does not contribute to user value

The obvious response is to take all the waste activities and get rid of them. This might be easier said than done, as it will often require organizational change.

#3: Create Flow

Once you have removed waste, you want your development process to flow smoothly. Remediate bottlenecks or delays in your process. Communicate with others involved in the process and reach an alignment about flow. If you accomplish that, you’ll have a more efficient process and higher productivity.

# 4 Establish Pull

This step is about minimizing inventory and delivering to your customer as needed. If we were talking about products, we’d want to decrease the time it takes for customer to pull the product, i.e., time to market. With data, we want to decrease the time it takes for the customer to “pull” or access the data needed. You want to ensure that it’s available readily and as frequently as needed.

The closest equivalent in IT would be large backlogs and long delivery times. You want to decide late, minimize work in progress, and iterate quickly. If this sounds like Agile methodology, that makes sense. Lean was one of the inspirations for it

#5 Strive for Perfection

Lean is not a one-time process. It is a loop of continuous improvement that gets repeated over and over. Lean leads to collaborative thinking, efficiency, and an improvement mindset.


Summary

If you apply lean principles successfully, everyone in the organization begins to think about managing data in the same way, and then you have the buy-in to truly optimize the entire flow of data, not just optimize in a silo.

If data management is something that you need help with, Onebridge’s data and analytics experts are always here to help.

Flock of SQLS comic on why being customer-centric in data management is imperative (click to enlarge).

About the Author:

Bradley Nielsen

Senior Tech Specialist

Bradley is a well-rounded developer in the field of data science and analytics. He has been a developer and architect on a wide range of data initiatives in multiple industries. Bradley's primary specialty is in data engineering: developing, deploying, and supporting data pipelines for big data and data science. He is proficient in Python, C#, SQL Server, Apache Spark, Snowflake, Docker, and Azure.

Related Partners

Related Services

Related Technologies

Related Industries

Stay in Touch with Onebridge

* Indicates required field
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Hey there! We hope you've noticed that none of our content is "gated," meaning we don't force you to provide your information in order to read our content. We work hard to provide valuable information to serve our audience and our clients, and we're proud of it.

If you'd like to be notified of new content, events, and resources from Onebridge, sign up for our newsletter here. After signing up, you'll get a profile link where you can tell us what topics you want to hear about.With Onebridge, you control your data.

Please follow us on social media to see upcoming events and other resources, like blogs, eBooks, and more!