Lately I have been spending a lot more time at our customers with the Business Glossary and Metadata Workbench. At first this was a side-effort at our major DataStage and QualityStage customer sites, but increasingly it has become a stand-alone activity with companies that are starting out with Business Glossary or implementing Business Glossary all by itself.

The key interest in Business Glossary has been its “business vocabulary” approach to managing “enterprise” terms and definitions — with focus on the word “business.” Prior solutions have revolved around “technical definitions” and almost always are related to columns and fields. Even legacy COBOL programs have comments that are sometimes captured, and every modeling tool and data oriented product on the market has a history of “explanation,” “comment”, “description” or similar property to contain explanatory text. Sharing and pushing this information all over the place has been offered in metadata plumbing for a long time.

Business Glossary is refreshing to many sites because it is “just business.” Terms are managed in a web based application that is complete with a very easy to use built-in browser and search capability, a pop-up function called Business Glossary Anywhere and a REST API that allows for complete customization. The best customers are the ones who already have sophisticated procedures and protocols for managing terms, but are tired of managing the application and delivery of those terms throughout the company.

Technically, of course, definitions in Business Glossary can be “assigned” a large variety of technical “assets”, but it is not a requirement, nor is the Business Term an “after effect.”

Anyway, it’s been fun, and eye opening, technically challenging, and combines the best of DataStage skills with years of helping sites with metadata strategies. And if you come here to read about DataStage techniques and activities, consider that Business Glossary may expand your horizons, your expertise, and if you are a consultant, your business.

Check out this cool new example of a “widget” for access to the InfoSphere Business Glossary! My IBM colleages in engineering created an example that illustrates how you can integrate the Business Glossary directly into your Portals and custom web applications using the REST API that is included with BG. This example is easy to deploy and, for the web-savvy developer, easy to customize. Enjoy!

https://www.ibm.com/developerworks/data/library/techarticle/dm-0909infosphererest/

Ernie

Just saw a great new article on DataStage and DB2 pureXML on developerWorks, written by some of my IBM colleagues who drill deep into some very cool aspects of XML and it’s integration with DB2 and DataStage. I’ve dabbled a bit myself into pureXML, but only enough to scratch the surface. This article has some good pointers and is just what I need to push me to spend some more time there…….

Enjoy!

http://www.ibm.com/developerworks/data/library/techarticle/dm-0908datastagepurexml1/index.html?S_TACT=105AGX11&S_CMP=FP

Ernie

The requirement to call Web Services from ETL (Extraction-Transformation-Load) is becoming increasingly common. Web Services are being included in ETL streams as a source or target, or for re-use of a critical calculation or function. Often they are invoked on a row-by-row basis for Transformation purposes. Web Services are defined by their WSDL (Web Services Description Language) documents, but most times that is just a “signature” that identifies the input and output parameters. What happens “inside” the Web Service? What actions are being performed on those input parameters? How useful would it be to know exactly what is being done to your data when it is sent to a remote “black box?” Services repositories, if utilized at your site, provide a rich source of metadata, and may include descriptions of the function, who owns it, when it was last updated, and provide automation for the bindings. However, many sites have not standardized on one particular repository, and more importantly, even if such information is available, it is unlikely that the repository participates in data lineage.

Data lineage is the ability to view a complete source-to-target flow, thru every process, every function, and every intermediate step. It illustrates what happens to a column, how it is used in expressions and when its name changes or when it is combined with another column to create something new. Data lineage helps decipher where a value came from, and how it was derived. Do your lineage reports have visibility into the “black boxes” that you are calling remotely? Would you like to “see” where the data flows when it goes “outside” the system? The IBM InfoSphere Metadata Workbench has a new capability called the Data Lineage Extender. This new feature lets me describe the details of any external process and include it in all of my metadata exploration and analysis.
In this blog I try to focus on real-time issues, and so will start with the benefits of defining a remote Web Service; however, the Data Lineage Extender and the “Extension Mappings” that it offers are not limited to Web Services. Extensions, as they are also called, can be used to define external scripts, Stored Procedures, Java classes, COBOL programs, third party transformation tools — anything that you believe is important for completing your lineage picture.

Extensions appear as special new objects in the Metadata Workbench, can be queried and viewed like any other object in the repository, and as noted earlier, participate fully in data lineage reports and graphs. I had the chance to start working with this new capability a few months ago in our lab, and believe that it is going to help fulfill the lineage requirement of many data integration applications that have “missing links” or “gaps” whether or not they revolve around DataStage and QualityStage.

To establish an Extension, first identify the external process you want to include in your lineage. Then decide how much detail is desired. The Extensions support specifications at many levels, including Table, Column, DataStage Stage, etc. and support one to one, one to many, and many to one relationships/mappings to other Extensions / native DataStage objects. (for example, First Name and Last name from Object A map to Name in Object B) Where possible, define the mappings of your external object at the column level.

Let’s look at an example. Here is a snippet of a Job that reaches out to a remote “Lookup” Web Service. The WSDL was brought into the Job, but there is no visibility to the Service except for the input and output parameters*:

A data lineage report without Extensions will simply show that the Web Services Transformer is invoked, but with little other detail. To “complete the lineage picture,” we will want to create an Extension that defines what happens within the Lookup. This is done inside the Metadata Workbench. The screen shot below shows the details of a new Extension Mapping. It contains the columns involved in the lookup, commentary about the type of Lookup that (below) yields the “C0_CorpExpFactor” value, and also provides the ability, via balloon help, to see the ultimate source of my Lookup Column:

Now that the Lookup Extension is defined, it will appear in a Data Lineage report. This next screen shot illustrates a data lineage report at the column level. On the left we see the lineage for the column CORPEXPFACTOR, and where it came from, and on the right we see the context for each column, whether part of a DataStage Link and Job, or within an Extension Mapping (LookupWebService).

Clicking on the “LookupWebService” icon or link on the right will take you to the details of the definition (the previous screenshot of the Extension Mapping Document) where you can review the methods used for the lookup and any commentary from the author. This first lineage view is called the “textual” view of Data Lineage. When requesting a lineage report I can alternate between this textual view and the “graphical” view:

Over the next few weeks I’ll be documenting my experiences with this feature, outlining the various use cases, and sharing the techniques and methods used to define Extension Mappings.

Ernie

* …of course, philosophically, one may wish to be entirely abstracted from a remote Web Service. “Who cares what tables it accesses, or what procedures it uses, provided I can trust it and rely on its execution” That’s one of the benefits of Service Oriented Architecture, right? This is a debate with many valid positions, however, most sites I speak to want to ensure extensive re-use of their Services, and simpler use/maintenance, but still maintain visibility into data quality and source/target details. Looking at WSDL alone doesn’t tell you very much.: If your directory strategy provides more than just a WSDL catalog, great. The Extensions described here may help complement what you are already doing, or “fill in the gaps” where there are holes.

Here’s a topic I hear less and less about these days. There are many reasons for this, the most likely being that the lines continue to blur between these technologies, especially now that various independent vendors in each space have been absorbed or morphed into other things. However, it raises its head every once in awhile, and even if classic “ETL” (extraction-transformation-load) and “EAI” (enterprise application integration) are under the same tool umbrella, it’s likely that choices will still need to be made as to the use of underlying features and capabilities.

Thought I’d share some thoughts that were discussed on a recent conference call with a site who is working thru the “ETL vs EAI for Real Time” decision tree. These are some of the major issues to consider (no particular priority attached to their order):

Protocol/medium. MQ Series? JMS? Tibco? Socket? Named Pipe? Sonic? MSMQ? other? Many of the EAI tools that still exist have more of these protocols “in the box.” ETL tools — not always. How hard or easy is it to extend your choice of tool.

The “shape” of the source. Is it relational in nature (rows and columns)? or hierarchical (tree like structure)? XML? COBOL with multiple record types, OCCURS, variable lengths? …or fixed Format? Both ETL and EAI typically handle these, consuming metadata from XML Schema, COBOL, etc. Exceptions would be things like SWIFT or EDI and EDIFACT, where EAI-style tools such as IBM WebSphere Transformation Extender typically have legacy history and metadata import capabilities. Shape of the source is important, but becomes even more significant in light of the next item.

The “shape” of the target. Is it relational (rows and columns)? Tables in an rdbms? Flat file? …or is it also XML with a complex multi-path hierarchy or COBOL with OCCURS depending on, variable length and mulitiple record types, or SWIFT, EDI, EDIFACT, HIPPA, etc.? ETL tools are best at performing relational work. Most can all do hierarchical as well — but often at the price of complexity. If the source and the target are hierarchical, how many transformations of this type are you doing? Just a few, or 98% of your project?

Data Volume.ETL tools excel here. Parallelism and grid work is a given. Being able to manage huge volumes of data, multiple terabytes, in batch, are where they are regularly exercised. EAI tools typically do not fare well in huge volume scenarios, or at least not without crafting parallelism and other configurations that are out-of-the-box with tools like DataStage.

Units of Work.Transactional paradigms. Delete from one rdbms table and write to another. Commit. If either resource fails, rollback everything. Delete from one queue and write to another, or to a relational table. Commit. If either resource fails, rollback. ETL tools can do some of this and do it well, but are often limited in the choices of queues and databases supported. In EAI tools, the semantics are usually simpler and more at the “core” of any EAI tool, richer in resource support (which databases, queuing systems, etc.) and often provide more flexibility and control to react to failures.

Skills.Beauty is in the eyes of the beholder…and ease of use/learning curve is no different. Still, ETL tools, while far from being designed for ‘end-users,’ have typically appealed to “data professionals” with technical skills but not necessarily hard core programming skills. EAI tooling is usually camped in the more technical domain. There is a lot of grey area and overlap here, but the history is clear — EAI tools have a rich development history of providing ensured delivery for binary EDI formats running 24 by 7 — and ETL tools have a rich history of providing consumable transformations for large volumes of data to support business intelligence initiatives. Enough said.

Long ago I recall a great write-up from an analyst who concluded that most companies require both technologies because the goals of each tool are so different. Years later, many companies have both technologies in their toolbox — so it may not be a purchasing decision as much as an implementation one. There are no wrong answers here — just answers that might lead to a bumpier road down the line.

Good luck as you review your variables.

Ernie

Well — I finally took the plunge and registered on Facebook. It is something I have avoided for awhile, even as Facebook has made crept into the business world. It’s certainly easy to do, and after one or two “friends” have invited you or accepted you to their “wall party,” it’s amazing to see the messages and notes fly! I’m still an infant there, not even qualified to say that my experience rates me as a “novice,” but the time spent there thus far raises several interesting thoughts:

First, I find it overwhelming to think of all the places a random thought will travel as it moves among friends walls/places/pages….and whether that’s a good thing or not. Online etiquette has grown-up over the past few years, and self-censoring is something we all do these days, even in our email. Still….Facebook may be “too” realtime in that sense. I suppose the real issue here is that I can’t imagine anyone caring at any particular moment that I’m currently updating my blog or preparing for a project out in the yard. I’m probably not a good candidate for a Twitter account! ; ) I suspect that this point of view may be altered as I spend more time on Facebook, but that’s my initial impression.

Second is how cool this technology would be for a business team, and in this discipline, and Information Server data integration team. With the right filtering, wouldn’t it be informative to have automated events and notes, driven by each user, showing up on a “Project Wall?” Such technology may exist already, or perhaps even Facebook supports APIs that would make it possible. “jsmith has just compiled DataStage Job LoadFact.” or “droberts has entered a Note about column Revenue in table Q461T711: There is an anomaly in some of the date fields….”. And from the runtime engine (again with filtering), it might be useful to know that “QualityStage Job mergeNames completed at 1:16:42 with no errors.” . We can certainly do a lot of this with instant messaging, but the broadcast capability here is inviting, as is the “no one has to think about it” aspect of event based messages, if possible, being floated throughout the team.

Guess I’m going to have to explore Facebook further…

Ernie

]]>