What exactly is Data Lineage?

Metadata management is becoming a big issue (…again, finally, for the “nth” time over the years), thanks to attention to initiatives such as data governance, data quality, master data management, and others. This time it finally feels more serious. That’s a good thing. One conceptual issue that vendors and architects are pushing related to metadata management (whether home-grown or part of a packaged solution) is “data lineage.” What does that mean?

Let’s imagine that a user is confused about a value in a report…perhaps it is a numeric result labeled as “Quarterly Profit Variance”.

What do they do today to gain further awareness of this amount and trust it for making a decision? In many large enterprises, they call someone at the “help desk”. This leads to additional phone calls and emails, and then one or more analysts or developers sifting thru various systems reviewing source code and tracking down “subject matter experts.” One large bank I visited recently said that this can take _days_! …and that’s assuming they ever successfully find the answer. In too many other cases the executive cannot wait that long and makes a decision without knowing the background, with all the risks that entails.

Carefully managed metadata that supports data lineage can help.

Using the example above, quick access to a corporate glossary of terminology will enable an executive to look up the business definition for “Quarterly Profit Variance.” That may help them understand the business semantics, but may not be enough…. They or their support team may need to drill deeper. “Where did the value come from?” “How is it calculated?”

Data lineage can answer these questions, tracing the data path (it’s “lineage”) upstream from the report. Many sources and expressions may have contributed to the final value. The lineage path may run through cubes and database views, ETL processes that load a warehouse or datamart, intermediate staging tables, shells and FTP scripts, and even legacy systems on the mainframe. This lineage should be presented in a visual format, preferably with options for viewing at a summary level with an option to drill down for individual column and process details.

Knowing the original source, and understanding “what happens” to the data as it flows to a report helps boost confidence in the results and the overall business intelligence infrastructure. Ultimately this leads to better decision making. Happy reporting!

Ernie Ostic

Update (2017-03-30) to this old but still very valid discussion…..    please check out this excellent new blog entry by one of my IBM colleagues,  Distinguished Engineer and thought leader Mandy Chessell, regarding Data Lineage … https://poimnotes.blog/2017/03/19/understanding-the-origin-of-data/

Cool new Widget for Business Glossary!

Check out this cool new example of a “widget” for access to the InfoSphere Business Glossary! My IBM colleages in engineering created an example that illustrates how you can integrate the Business Glossary directly into your Portals and custom web applications using the REST API that is included with BG. This example is easy to deploy and, for the web-savvy developer, easy to customize. Enjoy!

https://www.ibm.com/developerworks/data/library/techarticle/dm-0909infosphererest/

Ernie

…another way to load Terms into InfoSphere Business Glossary

copy-of-createbgimportscsv
copyofcreatebusinesstermsandattributesxmldsx3

Here are a few other Jobs for loading new Terms and Categories into Business Glossary. Like the earlier post on Business Glossary, these DataStage Jobs read a potential source of Terms (just alter the source stage as needed) and then create a target csv file that is in the correct format for loading into Business Glossary using the new 8.1 csv import/export features available at the Information Server Web Console… Glossary tab. The Jobs are fairly well annotated and should be self explanatory. I haven’t yet set them up for Custom Attributes, nor have they been widely tested —– but they are already being implemented at a variety of locations. Please let me know if you find them useful.

Ernie

(the one with “XML” in the name is the same as the prior blog entry. Each is named .doc, but is actually a .dsx file).

Using DataStage to load new Terms into Business Glossary

There are a variety of ways to import new Terms into the InfoSphere Busines Glossary.  One of these, for initial loads, is to use an XML import.   The XML format is fairly easy to produce (a sample is provided with the Business Glossary and can be found at your Information Server Web Console)

The attached DataStage Server Job illustrates how to load new Terms and Attributes from some external structure.   In this example I use a simple sequential file, but if you look at the Job you will see that it can easily be adapted to any source, or simply write another Job to go from your source to a target that reflects the sample terms I’ve provided below. 

This was tested in 8.0, although has since been modified for use with 8.1.  I’m not sure how well it will import into 8.0.  The terms and attributes are very simplistic and use a hockey theme, just to keep things simple and allow for discussion.     The code is an example for instructional purposes only.   Please let me know if you have any questions or run into problems.

(I hope you can make use of these. Seems the blog has changed and won’t allow me to upload .txt files. I’ve tried putting all the content in “notes pages” of .ppt. You’ll need to download and then open the .ppt [one of only a few file types allowed here] and then see if you can cut and paste the sample .txt file, sample xml, and .dsx.).

Ernie

sample-terms-attributes