Avatar! What a great movie! anyone see the cool Metadata?

…I suppose I need a longer vacation if I was thinking about Metadata while watching this amazing movie. ….but did anyone else notice the detailed metadata throughout the movie, especially during Jake’s video logging? …..anyone catch the date of his final entry? ; )

Ernie

What exactly is Data Lineage?

Metadata management is becoming a big issue (…again, finally, for the “nth” time over the years), thanks to attention to initiatives such as data governance, data quality, master data management, and others. This time it finally feels more serious. That’s a good thing. One conceptual issue that vendors and architects are pushing related to metadata management (whether home-grown or part of a packaged solution) is “data lineage.” What does that mean?

Let’s imagine that a user is confused about a value in a report…perhaps it is a numeric result labeled as “Quarterly Profit Variance”.

What do they do today to gain further awareness of this amount and trust it for making a decision? In many large enterprises, they call someone at the “help desk”. This leads to additional phone calls and emails, and then one or more analysts or developers sifting thru various systems reviewing source code and tracking down “subject matter experts.” One large bank I visited recently said that this can take _days_! …and that’s assuming they ever successfully find the answer. In too many other cases the executive cannot wait that long and makes a decision without knowing the background, with all the risks that entails.

Carefully managed metadata that supports data lineage can help.

Using the example above, quick access to a corporate glossary of terminology will enable an executive to look up the business definition for “Quarterly Profit Variance.” That may help them understand the business semantics, but may not be enough…. They or their support team may need to drill deeper. “Where did the value come from?” “How is it calculated?”

Data lineage can answer these questions, tracing the data path (it’s “lineage”) upstream from the report. Many sources and expressions may have contributed to the final value. The lineage path may run through cubes and database views, ETL processes that load a warehouse or datamart, intermediate staging tables, shells and FTP scripts, and even legacy systems on the mainframe. This lineage should be presented in a visual format, preferably with options for viewing at a summary level with an option to drill down for individual column and process details.

Knowing the original source, and understanding “what happens” to the data as it flows to a report helps boost confidence in the results and the overall business intelligence infrastructure. Ultimately this leads to better decision making. Happy reporting!

Ernie Ostic

Reviewing really LARGE xml documents…

Recently I came across a tool that was very helpful in the review of large XML documents…..REALLY large xml documents…..Hundreds of megabytes and more! In the years I’ve been working with XML, every so often I come across a document that I can’t read, can’t process, or can’t otherwise easily research for a data integration application. The tools I usually employ either hit errors or take forever to open while parsing thru thousands upon thousands of elements and attributes.

XMLMax, from xponentsoftware, is a convenient alternative to using IE to open my XML, alleviating the need to wait and wait while a large document is opened (all too often failing in the process in one way or another). According to the author, XML Max uses several proprietary buffering techniques that support the immediate display of content and faster consumption of the entire document. I found that it was able to read two and three hundred megabyte documents in mere seconds when compared with other mechanisms.

XMLMax is not just an XML reader, as it also has the ability to edit and manipulate your documents. I found it intuitive to use and easy to install, and liked its features for finding the closing tags of any element, especially large ones. Another great feature is the default for ignoring DTDs. So many other tools default to looking for a DTD if specified in a document, and then blow up when trying to open them! My interests lie in particular with the study and examination of large documents for use as a source or target, and so I focused on those features. It is worth noting, however, that XMLMax also provides ways to use XSLT to break up documents into smaller pieces. At the time of this review, the product only ran on Windows, but there are apparently plans to port it to other platforms and also to provide a command line interface for programmatic operation.

Check it out at http://www.xponentsoftware.com/.

Ernie