Data Lineage for “other” ETL tools and external processes

Hi all…

Recently I have had the pleasure of working with a couple of consulting firms who have been very creative with Metadata Workbench and the “Extension” features (Extension Mappings and Extended Data Sources) that were introduced last year in release 8.1.1. These companies have deep expertise on Information Server and DataStage as well as other ETL tools, and they have built tooling that automatically generates Extensions.

This is just a technical heads-up….. No bias here, nor am I doing their marketing, but it’s important that everyone knows that building lineage in Workbench for non DataStage processes is very do-able…all it takes is some creativity and awareness of the shell, 3GL program, or other ETL tool or process you are trying to illustrate…..we’ve had some entries in the forum on this subject, and some of you have asked me questions about it offline. Wanted to share these other resources.

Here are their sites if you are interested.

Accantec. http://www.accantec.com Cognos Data Manager. Accantec is based in Germany and has deep Cognos expertise. They have built tooling to illustrate Cognos Data Manager processes in Metadata Workbench for complete lineage.

Compact. http://www.compactBI.com Expertise in Informatica, AbI, others. They too have a solution that represents these tools within Metadata Workbench, often in combination with DataStage.

Ernie

Advertisements

2 Responses to “Data Lineage for “other” ETL tools and external processes”

  1. mayur25 Says:

    HI Ernie,

    Is it possible to view the transformation rules which is been applied in other ETL tool like SAS? If i am not wrong this thing is not doable since it is out of information server box. in case if we do populate business rules by some mapping then everytime it should be updated when there is any change in external tool like SAS.

    • dsrealtime Says:

      Absolutely! It’s not going to be “automatic” because as far as I know, no one has yet built any tooling for SAS specifically — but the infrastructure is in place to allow you to build your own objects that appear in lineage (using Extensions)…and flexible enough to include whatever degree of detail that you want. Generally it is enough to capture the critical sources and targets, meaningful high value transformations that are needed for understanding downstream results, and various bits of descriptive information about the mapping. Customers often do this themselves, and several of our partners have done this for other ETL tools, SQL scripts or other object types. A good video that illustrates this is at Compact’s site. Check it out at http://www.compactbi.eu/solutions/metadex.

      Ernie


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: