Have Bad Data? Here’s What You Can Do About It.
Recently, I was asked what the most common business problem that I run into is. My answer? Bad data, without a doubt. In my 25 years of consulting in a variety of industries, data integrity is the universal struggle. The good news is there are things you can do to help your data work better for you. The first step is understanding the factors that got you into a data issue to begin with.
While there are several reasons for bad data, the most common is that businesses grow very quickly and without much structure in place. Few organizations have time to slow down and clean-up nagging data issues, which are often held at bay with band-aid solutions and inefficient controls. So, the bad data problem typically persists, until one day the company dodges a major corporate risk bullet, such as paying employee stock options with a few too many zeros or accidentally handling sensitive customer information without appropriate controls. Suddenly the risk of financial, legal, or public image impacts becomes very real. For many organizations, this brush with potential disaster is the motivator needed to finally invest in data governance to get control of data quality.
Implementing effective data governance can be difficult, expensive, and time-consuming. Initial resolve can fade in the face of competing priorities, pressure to go to market, and feigning support and participation by senior leadership. Perfection is difficult to achieve. My very simple suggestion is to be realistic. Right-size the data governance effort to one you can sustain. Start small. Show results. Build from there.
If you’re looking to get moving toward improved data, here are three steps you can take.
1. STOP THE BLEEDING
One of the most important things you can do is stop the problem from propagating going forward. Implement a data governance process— put procedures in place for new data, data access, and changes to existing data. Set achievable goals and KPIs related to this effort, and limit scope to what you can handle. Maybe to start, only high-value and high-risk data must go through the review process.
For example, I once had a client that experienced a close call data breach. They quickly mobilized a data governance effort, and yet it never left the planning stage. Why? Because they were trying to bite off too much in an effort to “do it right.” It’s a respectable goal, but in this case, the program was more than the organization could support. The required involvement of high-level executives and numerous data stewards paralyzed the effort from ever getting past kick-off. The data governance initiative might have been saved if the scope had been limited to only reviewing new data or changes to data that have significant financial or legal impacts. Executives could have empowered a small group of trusted subject matter experts to conduct reviews, who only escalate high impact issues and concerns to executive attention. While this solution isn’t “perfect,” it’s a step forward.
Get leaders to define the desired data governance program state — but understand that they will achieve that vision over time. In the near term, determine what they are willing to invest in tools, staff, and their own time to scale efforts appropriately.
2. WORK THE BACKLOG
Prioritize known data integrity issues and risks and resolve them systematically, one problem at a time. Each data correction should run through the new data governance process that was put in place during your “stop the bleeding” stage.
You can start the backlog with the issues identified in the fire drill that inspired the effort. Tackle the major fires first, then look a little deeper to anticipate where additional issues exist. For example, one of my clients put out the major fire started by a new system implementation, and quickly became worried about other issues that had not yet been discovered. So, we did a quick risk assessment and created a risk and issue heat map to identify areas of concern that warranted in-depth study. This resulted in a prioritized list of risk and issues that could be picked off by level of urgency. Each fix needed to be properly vetted by our light data governance process to review system and process dependencies and impacts.
Ongoing risk assessments can be applied to any functional area to efficiently locate data integrity risks and issues to feed a backlog of improvement projects.
3. CONTINUALLY IMPROVE
It is important to continually evaluate the performance of your governance process relative to the KPIs you have put in place. Consider if the goals you set are still your highest priority, if the governance process should be refined to improve ROI, if participants are still committed to an ongoing effort, and if you’re doing well enough to move to the next level.
Adjust as needed to meet your current standards for adequate data governance outcomes. If you believe the current state is adequate, but you want to do better, develop a Data Governance Maturity Model that is a roadmap to incrementally improve until you achieve your company’s definition of where you want to be. The “to be” vision does not necessarily mean you have achieved the end state of data quality defined in our college data management textbooks or scholarly papers.
Appropriate data governance programs vary by company based on accuracy requirements, resources available, and risk tolerance. For instance, government and large global companies may have a very low tolerance for risk, while a small company that does not have significant financial exposure or potential to harm customers may have much lower standards for data governance requirements. Each company’s executives need to define the standards that meet legal requirements and achieve performance goals.
Solving for “bad data” is a big job. Take it on in doable chunks so you get started and progress continues until your corporate data integrity goals are met.