The goal of having a data intelligence strategy is to maximize the business impact of your data. With a strategy, you combine the capabilities of data catalog, data quality, data literacy and data marketplace so you can discover, govern and share high-value data across your organization. A data intelligence strategy provides your IT department, data governance teams and business users with meaningful insights into your data landscape that drive well-informed actions.
This article will detail what needs to be included in a data intelligence strategy as well as the consequences of not having one. Data governance teams will come away with a better understanding of the best practices and pitfalls to avoid when forming a strategy.
What are the consequences of not having a data intelligence strategy in place?
The advantages of a data intelligence strategy include enterprise-wide visibility of available data assets, guidance on their use, and guardrails to ensure you’re following your own data policies. But what are the potential downsides of not having a strategy?
Untrusted data
Without a data intelligence strategy, users can look at results or read reports you’ve generated, but they don’t know how accurate they are. What good are revenue numbers and cost figures if users don’t have confidence that the data behind them is accurate or has been tracked and tested? When two or more stakeholders look at the same data label and see disparate values, they naturally ask, “Which one is right: my number or your number?” Or maybe they’ve always gotten their data from the same place, and so they just assume that it’s true. The issue of trust in the data and in the source of the data arises. Users won’t trust data that has never been verified.
Duplicated efforts
In any enterprise it’s almost inevitable that multiple people are working simultaneously on the same thing. That’s common in reporting because it’s difficult to see which data sets and reports are available. Unknown to the users, the same – or nearly the same – report may exist among different groups. It’s because the groups operate in their own context and are not exposed to anything else. A need arises, so they request – or create – a report that already exists, when they could easily use the other group’s report, or at least use it as a point of departure. Duplicated effort is what comes of data silos.
Manual efforts
Does your organization have any kind of tracking system for how good your data is, how it moves and where it resides? It may have one, but in most cases it’s manual and documented in one place only, which is why it gets duplicated. But when things change, nobody remembers to update that documentation. It’s stored in spreadsheets, word processing documents or someone’s head, or it was maintained by a former employee. The larger the company, the bigger the effort is to track its data.
Penalties and fines
From an auditing perspective, it can cost you money if you can’t track your data. You can be fined for being out of compliance with regulations like the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA). Government entities expect that you’ll be able to find and produce necessary data in an audit.
Large efforts to prepare for audits
The likelihood of an audit is a compelling factor in your data intelligence strategy. There are different auditing requirements for each industry, from healthcare to financial. For compliance with privacy regulations, auditors want to know things like how you would delete a user’s data if they were to request it. Their priority is in knowing where you store your users’ data so that, if need be, you can delete it everywhere in your organization.
Auditors don’t have a prescribed procedure; they say, “I need a report that tells me this,” and you have to supply it. And, next time they could ask for something completely different. Your data needs center on what they’re after. That’s why, without any kind of tracking mechanism, preparing for an audit can be a huge effort.
Problems in information systems
Most companies have been unaccustomed to thinking about data intelligence. But with the advent of data privacy laws, they feel urgency around knowing where their data is. Regulations on data tracking started with Europe and GDPR, and they have spread to other countries, and to states (CCPA, for example) and provinces within countries.
The regulations have led to many organizations appointing a chief data officer (CDO) and forming data governance offices tasked with establishing a data intelligence strategy. And it’s no mean feat, especially if they’re overwhelmed by an information system that has been around for years. Maybe it’s a mainframe computer, or maybe it’s just an ancient Windows application. How well has it been kept up to date? How easy is it for you to run and update reports?
The biggest problem is that few legacy information systems were ever built to track and generate information about data easily. They include warehouses where data is collected from different sources, then extracted, transformed and loaded (ETL) so people can run reports off it. But they do not necessarily track the data or monitor its quality.
Getting your data under control
It’s rare that you can control your vendors. If your enterprise is big enough, you can request that your vendors send you good data, but mostly you take what you get and deal with it. There’s more effort in enterprises now to go to the source of the information under your control.
Internally, you do have some control over the quality of data going into the applications as your users enter it or as your customers provide data online. That effort is part of your data intelligence strategy, including data quality and data governance. It’s a means of getting your data under control, improving its quality and documenting how it moves through your company. That’s valuable whether your goal is to get your data in front of more users or to prepare for regulatory auditing.
The role of tools in your data intelligence strategy is to enable anybody to answer the question, “Where did this number come from?” The tool allows them to follow the number to its source without having to ask anyone, without even knowing whom to ask. It’s enabling technology to all users in the organization, not just to IT or a special group.
Data maturity and the role of data modeling
Every organization comes to its data intelligence strategy a little differently. The model of the data maturity ladder is a way for your company to develop its data intelligence. It’s an indicator that, if you already have certain elements, then you’re well placed because you can bring them together to reach a higher level of maturity. Even if you don’t yet have those elements, the model is a good way to start. You can build maturity into your organization in a different sequence, but the model gives you an idea of the baseline effort.
Data modeling in particular takes a lot of effort to change your organization’s mindset. Why? Because the data landscape in most enterprises is driven by application developers or architects. Their main goal is to answer questions like “How do I communicate quickly between my applications?” and “Which is the best landscape I can use?” whether that’s cloud computing or a programming language. Application developers are interested in moving and manipulating data, and not in the data itself or in its consistency. They’re focused on making their application work and getting their requirements in.
The role of the data architecs and the data governance teams is to enforce consistency. They use data modeling to build the underlying structures that meet the greatest number of needs and ensure that, say, the Last Update field means the same everywhere it’s used. In most companies, the application developers make those decisions, but in data-mature companies it’s the data architect who makes them. In data-mature companies, modeling comes before programming and the result is far less rework. Modeling helps drive and support the rest of that maturity model.
Best practices to create a data intelligence strategy
Evaluate your current data
Determine where your data is and how to use it
Even if you’re starting from absolute square one, some elements of a data intelligence strategy exist. Within certain reporting tools you can obtain the answers to questions like “What’s the most common object used?” and “Which columns are used most frequently in our reporting?”
If you’ve ever been audited, your compliance teams will know what the auditors have looked for, based on some sort of policy or regulation. Most of the time it’s the critical elements that everybody’s using and that drive your business. Those are hints that will put you on the path to figuring out what you need to look at.
Compliance requirements
Your industry-specific compliance requirements help identify points of integration with other systems, which are often points of continual breakage. There are problems because of some column or some data element that’s mismatched or left null when it shouldn’t be.
Potential data growth and volume
Are you tracking data growth? Over time, how big has an object – say, a database – gotten? You can examine metadata and plot how much data you are creating monthly, quarterly and yearly to determine your rate of growth. The exercise can be labor-intensive. It’s not very interesting to execs and business managers, but for data analysts and architects it’s enlightening.
Identify the pain points that a data intelligence strategy can help with
Your biggest pain points are usually the critical data elements that people look at constantly or that cause things to break. Even if they’re not the elements most commonly accessed, they can still be critical in that you want the information in them to be reliably accurate. The pain points help determine what data you want to focus on first.
Keep in mind that this is an iterative process. You don’t collect everything at once and analyze every column in every table, plus your applications. The goal is to get the pain points first because they help you identify the stakeholders who face them most often. They are the users who will be more supportive of a data intelligence strategy that will relieve their pain.
Once those stakeholders are on board, they will become cheerleaders, promoting the data intelligence strategy to other groups in the organization. That’s how you move forward with more people in the company.
See what you already have that will support data intelligence
This starts out as a process of identifying all your technologies. In the process, you also discover opportunities for data governance; for instance, finding that your enterprise uses three different ETL tools and five different database types. You soon realize what you need to get a handle on.
Do we have data quality, data catalog and/or data lineage in place?
The thing about the tools that support a data intelligence strategy is that businesses – and groups within businesses – get tired of waiting for them. They opt for homegrown because they understand their needs and can’t find anything that suits them perfectly. Data quality tools tend to gain acceptance in specific departments, like marketing. Find out how the tools are working in those corners of the enterprise and gauge the fit for wider use.
A surprising number of companies maintain data in isolated spreadsheets around the organization. Almost anything is better than that, including the moderate upgrade to, say, a homegrown data catalog in a SQL Server database. That’s an acceptable path if you at least add some data modeling to keep from painting yourself into a corner.
With data lineage, your goal is to be able to trace data through transformations. You can see the movement of data in your ETL tool, if you have access to it, the tool may also offer insight into lineage. If you do have those kinds of data intelligence tools in your organization, they are more likely scattered than centralized. And they probably extend to a variety of different technologies.
How manual are these processes currently?
Using manual techniques to track, say, data lineage is asking for things to fall quickly behind. If you’re relying on manual tasks like updating spreadsheets, you’re also relying on human memory and bandwidth. It’s a recipe for things to go out of date. Your tracking tools should enable you to automate tracking by keeping a record of every time the data is touched.
Establish common definitions of what your data means
Are there any workflows around creating or updating definitions?
What common definitions do we have? Who’s using the data and how are they defining it? You might find two different areas of the company that use the same data and call it something different. Why is that? If the definition changes or if it has gradually morphed over time, is that tracked?
Metrics are a common example of shifting definitions. You calculate a metric one way for a long time, then something happens and now you calculate it differently. Do we have a history of that change? Usually the answer is “no” because the history is in people’s heads, and some of those people have left the company. There was certainly some workflow – whether a chain of approvals or a single, arbitrary act – and a data intelligence strategy entails tracking and recording that workflow.
Is a governance program already in place?
Those common definitions generally lead into governance, some degree of which is already in place in pockets around an organization; for example, in applications and technologies. But governance is often associated with control, burdensome overhead and directives not to do certain things.
Generally, that reputation arises when the enterprise finds itself in an uncomfortable scenario precisely due to the lack of prior governance. If you’re in a bad spot, it’s often because you’ve lacked the self-control to make sure everybody’s working from a single list of technologies, applications and definitions. So there is an element of control to that, but it leads to order. Governance is not autocracy, its goal is to harvest input from everyone.
Describe how your users get access to data they need
From a high level, can you describe how your users access data? How do they get the reports they need? In particular, how do your data analysts get answers to the questions they have? They’re the ones tasked with presenting, say, quarter-over-quarter and year-over-year figures; how much work is it for them? Do they have to ask around, or is there a process already in place for them to find that data?
Mature enterprises implement data marketplaces so that users can easily shop for and select the data they need.
Develop a roadmap and plan for execution
You can’t tackle everything at once. Whether by department or by critical elements involving multiple departments, your roadmap will guide you through implementation of your data intelligence strategy.
Plan for training
How do you plan to introduce your data intelligence strategy to your users and how do you plan to train them? Your selection of a tool is part of it, but whichever solution you pick, your users will also want training in your new process. Most important is the difference between the way they used to do things and the new way.
Mistakes organizations make when implementing a data intelligence strategy
Not identifying important users
If you haven’t identified the most important places and users leveraging data intelligence, you’ll find it more difficult to promote data intelligence on the whole. You may not get the support you need for rolling out the process. Find your cheerleaders because they’ll help your implementation catch on elsewhere. You can cultivate users one area at a time, starting with a highly data-focused area. Once they see the value, they’ll help you push to the next area.
Not having clearly defined use cases
Having clearly defined use cases is how you support why you’re trying to implement a data intelligence strategy in the first place. They demonstrate the goals in your overarching goal.
Consider data lineage. Suppose your data owners get questions about lineage because your data travels widely. A big push ensues to track lineage so they can tell users where data came from and where it’s going. You have auditors and compliance regulations to respond to, so you need a better handle on lineage.
One of your initiatives can be to track from a particular starting point or use case. You get that piece down and build out from there. You could go further in either direction or track back further sources, but you need to get this first before you track more targets forward. When you have a clearly defined use case, you can illustrate how you’re going to achieve your goal.
Taking too long to get to value
Most companies make this mistake, especially at the beginning when obstacles are everywhere. They could be obstacles of architecture, getting the nod for the resources or just getting people on board with the project. The longer it takes, the more your data intelligence strategy drifts off of people’s radar. The sooner you show value and actually affect how people do their job, the more they’ll pay attention and the more they’ll stay engaged.
Even little things can show great progress to users and demonstrate that it’s something on which they can provide feedback while you’re advancing to the next step. Whether through definitions, lineage or any component of data intelligence, there’s no need for a Big Bang to show value. Sometimes it just takes a little bit of progress for people to get hooked and become keen to see more.
What are some things organizations can do to ensure a successful implementation?
Involve business resources from the start
Suppose your business users in Operations are responsible for tracking the lineage of metrics. Most likely it’s a manual effort for them: they go to multiple systems, pull numbers and drop them into spreadsheets. Or, maybe they use spreadsheet macros for the grunt work, but the macros continually break and they end up with problems like incorrect information.
You introduce them to data intelligence tools that will give them a better look at how the data comes in and where the metrics are coming from. As they start analyzing their systems with the tools, the source of the problems becomes more apparent. Maybe the same data is held in fields of different lengths between systems, or your vendor’s application labels the data differently. The process of deeply examining the metadata allows you to identify your use of fields that were never properly documented. With real data intelligence, they know now exactly what’s in that field.
The value of involving business resources is that they’re the ones who use the data. You want them to be involved in the process and understand what it takes to get the right information. Then, of course, you automate the process so that it’s hands-off.
Clearly communicate to the organization (top-down support)
C-level buy-in is essential to the successful implementation of a data intelligence strategy. When one of your execs sends an organization-wide email announcing that you’re launching the initiative or hiring a CDO, readers understand that it’s a priority. It can take time to turn the battleship, but in this case, time spent is time saved. It won’t be necessary to repeat the process of establishing credibility and value. Plus, it will help direct you to pain points so that you don’t have to dig around and find them.
Create a diagram of the technical landscape
In your efforts to involve business resources, a diagram of your technical landscape is a valuable tool. Use it to convey a high-level picture of current realities such as the technologies you’re using, the tools that are transforming your data and the reporting you use. Include details down to the level of version numbers; it’s not uncommon for business managers to discover that they’re running applications on antiquated versions of SQL Server or Oracle. They don’t upgrade unless they have to, but that takes time and effort.
Maximize the value
of your data.
Avoid elaborate customizations
As with any tool, do as much as possible with the core product. The more customizations you add, the more maintenance you’ll incur, which can lead to complications when it’s time to upgrade.
Launch, measure, iterate and publicize success
Whenever your data intelligence strategy leads to success, publicize it and get your users to talk it up to other users in the organization. Emphasize that the successful effort is part of a roadmap for implementing features like data cataloging, data literacy, data quality, data marketplace and automation capabilities. Make it known what you plan to implement and build anticipation among your users.
Conclusion
A data intelligence strategy stands the best chance of success when the organization already has a passion for data and the initiative is broadly supported within the company. Such organizations embrace that message and don’t mind investing the work that goes into doing data intelligence correctly. It’s not difficult to find IT professionals who are well skilled in what they do, but dedication to the consistency of data and the modeling piece is harder to find.
By making assets accessible to and understandable by all users, a data intelligence strategy lets you make the most of your data. Find out how much your organization can accomplish when you automate the collection and collaboration of data intelligence and put it into the hands of more of your users.
