Check out this “can’t miss” recording of metadata mgmt and governance in action!

One of our IBM partners, Compact (www.compactbi.eu and www.compactbi.com ), has published a video recording on their web site that illustrates their MetaDex solution. MetaDex, in combination with Information Server Business Glossary and Metadata Workbench, enables metadata management for additional technologies that are outside of Information Server. Parsing for independent ETL tools and complex SQL scripting are just part of what MetaDex offers.

This is an excellent video that is worth eight minutes of your time. It highlights the functionality of Business Glossary and Metadata Workbench and how they work together along with MetaDex to provide a strong solution in support of your governance objectives. Enjoy!

Ernie

The recording is specifically at: http://compactbi.eu/solutions/metadex

Actional Diagnostics…great to use an “old friend” again…

Hi All…

Just a quick note. I had the pleasure today of finally getting around to installing Actional Diagonostics. This is the latest release of what used to be called “SOAPScope”…it’s been awhile since I’ve had a need for it…but it was perfect, still providing all the great things that it did in its earlier implementations.

This is something I’ve been meaning to do for a long time, but haven’t had the chance. Mindreef was acquired by Progress Software several years ago, and what was originally “SOAPScope” has been rebranded. I don’t have any specifics as to the other Actional offerings that are connected, but I can say that the experience was excellent. The download and install went very smoothly, and I was invoking a service within minutes. The screens appear to be the same, although they have probably included new functionality that I have yet to explore.

If you need an easy to use testing tool for your services, put this one on your list. There are many of them out there….all good tools….. I happen to have always liked this one because it offers a good compromise — it will appeal to users like myself who are comfortable with xml and http protocols, yet also be easily adopted by users who don’t want to be exposed to xml and simply want an easy-to-use GUI. Especially nice in Actional Diagnostics is the ability to perform load testing, where you can easily create multiple threads (thus simulating multiple users) invoking your service in concurrent fashion.

Bravo to the team who is still supporting this.

You can find the download details at http://web.progress.com/en/actional/

Ernie

New developerWorks article on DataStage and new XML Stage!

Hi all…

My esteemed colleagues on the xml development team have published a great article on the new XML Stage in 8.5….enjoy!

devWorks article on the New XML Stage!

Ernie

Reviewing the Advanced Tab in the Metadata Workbench

Hi all…

Just thought I’d throw in a quick review of the important (imho) links at the Advanced tab…..some of these factoids are buried in my other posts, but I needed to have a cheat-sheet for myself and others. Here is is:

Automated Services. This option brings up the dialog that runs the parsing or “stitching” process for the detailed metadata you have in your DataStage Jobs or Connector-imported rdbms views. It does a lot of stuff, takes time the first time you run it (if you have a ton of metadata), and should be scheduled during off-hours. After the first run against a particular project, it uses a change recognition mechanism to only pick up Jobs that have been updated. Note the “checked” DS Projects carefully. Only select those that are really critical, and once checked, don’t “uncheck” — as you will see from the warnings, this will “remove” all parsing history. Ultimately, this step is the one that reviews the Jobs, connects them via common information found in Stages, etc. See my other posts for how the connection of Jobs to each other is determined.

Stage Binding. When all else fails, you can connect two stages to each other. Use this when, for some reason, two Jobs won’t connect, or when the rules for connecting them can’t be met. I’ve needed this with some custom Stage or Operator implementations, and when I am using a technique that prevents automatic connection. Imagine having a Sequential Stage at the end of a Job that is writing out some xml content — and then I’m using the XML Stage in the next Job to read that content. There isn’t much in common between those Jobs, but I still want lineage to run directly thru them…

Data Item Binding. This provides a “manual” binding of particular Stages to Database Tables and Data Files (see other posts for what those are, how they are created, and how they are different from “DataStage Table Definitions”). Use this when you are unable to get Database Alias to work as you expect and you simply want to “bolt” a particular Database Table or Data File to a Stage in one of your Jobs to complete the lineage picture.

Data Source Identity. Use this when, for whatever reason, you want to link two identical tables for lineage purposes. Reasons? Two people might have imported the same metadata accidentally and you don’t want to delete it….or you might have the “design” information from an ERwin model and also have the “actual” table information from the rdbms catalog. There are many valid reasons. This link let’s you relate tables together. They must have the same name — the option here lets you relate the “Schemas” of two different databases. Identical tables within those schemas will become linked for lineage reporting — and therefore, also linked to whatever those individual tables connect to for lineage.

Database Alias. This option establishes the connection between an abstract string in a DataStage Stage (Server name, DSN name, etc., as defined by the relational stage) and the “Host/Database” combination that was actually imported. Database Tables in Metadata Workbench are typically “actual” tables — but in DataStage, like any well designed application, the “name” is a placeholder. This assigns the “placeholder” to the host and database. The schema.tablename used in the Stage will then be matched against the Host/Database set of Tables to create a lineage connection. The list presented at this option will be entirely empty until you perform Automated Services. Then it will be populated with each StageType and “server string” combination that it finds in your Jobs.

Hope this helps understand these options.

Ernie

2010 in review

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

The Blog-Health-o-Meter™ reads This blog is on fire!.

Crunchy numbers

A helper monkey made this abstract painting, inspired by your stats.

The average container ship can carry about 4,500 containers. This blog was viewed about 21,000 times in 2010. If each view were a shipping container, your blog would have filled about 5 fully loaded ships.

In 2010, there were 11 new posts, growing the total archive of this blog to 52 posts. There were 8 pictures uploaded, taking up a total of 3mb.

The busiest day of the year was January 5th with 152 views. The most popular post that day was Incorporating Java classes into your DataStage Jobs.

Where did they come from?

The top referring sites in 2010 were dsxchange.com, it.toolbox.com, google.co.in, google.com, and en.wordpress.com.

Some visitors came searching, mostly for cobol etl, datastage web services, real time etl, datastage real time scenarios, and complex web services.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

1

Incorporating Java classes into your DataStage Jobs March 2008
19 comments

2

Reading XML Content as a Source December 2007
10 comments

3

What is Real Time ETL anyway? November 2007
4 comments

4

Tips for using Web Services Pack with DataStage Part I: Getting Started November 2007
4 comments

5

Tips for Using Web Services Pack with DataStage: Part II – Invoking your first Service December 2007
8 comments

The new XMLPack in 8.5….generating xsd’s….

As noted in an earlier post and outlined nicely in Vincent’s blog (new xml!), the new XML Pack is here… It is very powerful, providing new features for reading and transforming hierarchical data, performs faster (and smarter) than earlier xml technologies within DataStage, and much more. It also requires that you have an XML Schema Definition (xsd) for the import of xml metadata. Most of the time this is not an issue. The xml documents you are reading and/or writing are well defined, complying with a formal xsd developed within your organization, or perhaps by a partner, yourself or by a standards body. But sometimes, there is no xsd. You may not have access to one, it might have been lost, or it never existed. The XML might be simple enough that it was just generated by another tool without the use of an xsd (or you are asked to generate it), or the xml might be old enough to pre-date the arrival of xsd’s.

There are many ways to generate an xsd. Popular tools such as Altova XMLSpy support this capability, as do many others, including xml Max, whose link I have over on the link list to the right. A quick search on the web will invite you to try a lengthy list of possibilities. One that I’ve been very successful with is called “trang”.

http://www.thaiopensource.com/relaxng/trang.html

This little tool does more than just xsd generation, although that is the functionality that I have found most useful. I’ve tried it on Windows and on Linux. It is easy to use, well documented, has references from other bloggers across the web, and does the Job. It is command line based, and requires that you have a java run time locally installed. There may be more sophisticated tools out there, but this is sufficient for what I need to be productive with the new XML Stage.

Let me know if you find any others!

Ernie

Connecting Shared Tables to your DataStage/QualityStage Jobs

At this point you should be familiar with Shared Tables, and understand how data lineage works within, and among, multiple DataStage Jobs. Stage to Stage lineage is very useful, and may be all that you require. It’s also powerful, though to go beyond this and connect your Jobs to the tables that you have imported from various places.

Why?

One of the key reasons for including Shared Tables in lineage is for “Business Lineage”. This is a high level summary of lineage that doesn’t illustrate the lower level transformation details…just the “key” sources and targets (files, tables, reports, etc.) along the way. Another is to connect your Jobs to “external, non Information Server assets.” DataStage doesn’t write directly to a (for example) Cognos report….it writes to a table somewhere, and that table is then read by the business intelligence tool. The connection thru that table is critical for accurate data lineage reporting. Here is how Metadata Workbench makes the connection between a DataStage Job and a table…

Shared Tables have a four (4) part name: Host/Database/Schema/Tablename (see other entries in the table of contents on Shared Tables for more details on the names of Shared Tables or alternatives for importing them).

Relational Stages in DataStage typically use two (2) parts to identify a particular table. “Server” or “Database” or “DSN” name, and tablename. The Server/DSN/Database name is usually in a dedicated property within the Stage. The Tablename might be in a dedicated property, or it could also be embedded in User Defined SQL. Any of them might be hard coded or established via Job Parameters.

The first thing Metadata Workbench needs to do is to “map” the abstract “Server/Database/DSN” name to a particular “Host” and “Database” combination in the list of Shared Tables. Like any application, DataStage is just pointing to some abstract “string” when trying to find a database. An ODBC DSN, for example, “might” be the name of the database, but it could also just be “myODBCdatabase”, which really points to (in the ODBC definition) a DB2 table called HRMAIN. Even if we use the string HRMAIN in the “Server” property of the Stage, we still need a way to identify the particular host that we are considering for data lineage. This is done via the “Database Alias” link in the “Advanced” tab of the Metadata Workbench.

Go to the Advanced Tab after signing into the Workbench. Perform Automated Services for one of your projects (this could take a long time if it’s the first time you are doing it). When it finishes, click on “Database Alias”. Look carefully at the values there. These are the “strings” that are used in your various Jobs to identify databases. Pick the string that is appropriate for the Stage Type that you are working with and slide your cursor over to the right. The “Add” button will allow you to select the desired host/database combination that this abstract string should connect to. In the example noted above, I might assign the string (alias) myODBCdatabase to the host/database combination of QR2H004/HRMAIN. QR2H0004 with Database HRMAIN must be something I have already imported and is viewable in the left navigation pane of the Workbench, or in the Repository Management tab of the Information Server Web Console.

Save it (button on the lower right).

The next time you perform Automated Services, whenever that Stage type with that particular string (myODBCdatabase) is found, Metadata Workbench will use QR2H004/HRMAIN, combined with the fully qualified tablename* in the Stage, to match to a particular Shared Table that has been imported previously.

[* Note... if your tablename is NOT fully qualified, Metadata Workbench will, for most RDBMS Stage Types, pick up and use the value of the Username in order to identify your table]

Are you using Job Parameters? For Design based lineage, Metadata Workbench is smart, and will use the “default values” of those Job Parameters when finding alias “strings” and also when obtaining the fully qualified (schema.tablename) tablenames to use in the linking.

Are you using $PROJDEF? Run the ProcessEnvVars shell or bat file inside of /IBM/InformationServer/ASBNode/bin to obtain the project definitions for use in this algorithm.

Operational Metadata (a whole separate blog entry is needed to discuss OMD) is used to populate the Job Parameter values from “run time”…otherwise the same rules apply. If you are just starting, get to know lineage w/o worrying about OMD. That’s an advanced topic. Understand how database alias works by performing Automated Services, reviewing the Database Alias page in the Workbench, doing the assignment and running Automated Services again. Once it is complete, go to the actual table using the navigator frame at the left (open the Host tree and find the table) and right mouse and select “data lineage”. If it is a source, select “where does this go to”…..if it is a target, select “where did this come from”…… and validate your results.

Ernie

The new XML Stage is here!

Just announced yesterday…the new XML Stage is available for 8.5! This introduces a whole new level of XML Transformation to the Information Server platform! Among its new capabilities are the ability to read single huge documents using a new streaming methodology that avoids the need to load the document into memory, support for any type of xsd, or collection of xsd’s, to define your xml metadata, and perhaps most important, a whole new hierarchical editing mode called an “Assembly”, which provides support for the creation of complex multi-node hierarchical structures! There’s much more, such as very explicit control of xml validation, a built-in test facility to ease transformation development, and support for both EE and Server Jobs. I’ve had a chance to play with the Stage over the last few months, and will share my experiences and techniques in upcoming posts.

In the meantime, I’d like to congratulate my IBM teammates in engineering for this accomplishment! This new capability will change how we approach many transformation solutions!

You can find the new XML Transformation capability at fix pack central for application to your 8.5 installation.

Ernie

Linking DataStage Jobs Together

Once you have mastered the “navigation” and asset selection options of Data Lineage reporting, it’s time to look at how DataStage Jobs are automatically linked together. By now you should be comfortable with thinking about your “starting position” for a data lineage report — your initial “perspective” if you will (what object are you standing on when you begin). You should also be comfortable with thinking about the “direction” for your Data Lineage investigation — are you looking “upstream” for “Where did this come from?” or downstream for “Where does this go to?”

If you need a refresher on the basics, please see Getting Started with Data Lineage!.

A typical production site for DataStage/QualityStage has MANY Jobs — hundreds perhaps….even thousands. All integrated and working together to transform your data and move it from one place to another. Sometimes they are written by one very hard-working developer, who might have all the lineage in his or her head, but more often it is a larger scale endeavor, with lots of team members, often scattered around the globe, and with varied skill sets and possibly working on related albeit independent solutions. They may know each other, or may not. How are the DataStage Jobs sequenced from a data flow perspective? How does data flow between a Job developed to process data received via FTP from the mainframe and then ultimately to a datamart that supports a reporting system? How does one Job connect to another? Sometimes it may be one giant Job, but its not likely. Intermediate temporary tables are often created for everything from checkpoints to Operational Data Stores to “parking lots” where data can be restructured or delivered to another application along the way. Workbench can sort this out and provide you with lineage through all these Jobs.

Intra-Job Data Lineage (Data Lineage between Jobs) is largely automatic. You simply have to pay attention to a few “good sense” leading practices and understand the pattern. Note that this has “nothing” to do with Shared Tables or Table Definitions at all… it’s entirely done by merely parsing thru your Jobs [this is a key reason why you can get immediate insight on 7.x Jobs that are imported into the 8.x environment --- even if you haven't compiled a single one or started your formal testing and QA process!]

Automated Services is the “parsing” step at the Advanced Tab….when you say “run” with your Project(s) checked, Workbench combs through your Jobs, looking for similarities that will link Jobs together end-to-end. Here’s what it looks for among Jobs:

a) Common or “like” Stages between the Target of one Job and the Source of another. Two ODBC Stages are in common, but so is ODBC and say, Oracle OCI. Or DB2Load and DB2Connector. Or two Sequential Stages.

b) At least one column in common. [Update Note... the research leading to this post was done originally in early 8.1 ... I've since discovered in 8.5 and higher that the connection is more flexible now, and one column name in common is no longer required. The default values and Stage type are the major controlling factor in bringing Jobs together.]

c) Same hard coded values (yuk…who does that?) OR….same “default” values for Job Parameters for the critical common properties. For RDBMS type Stages, it’s ServerName, Schema, and Tablename. For Sequential type Stages, it’s the filename. The Automated Services will put together multiple Job Parameter default values if needed.

If your team follows good practices of having parameter sets or common values for things like an ODBC DSN, a high degree of lineage will often occur immediately after the first Automated Services (informally known as “stitching”) occurs. Expect that the “first” time you run Automated Services, it could take a long time and be very intense. Do it during off hours if you have hundreds or thousands of Jobs. After that first time it will recognize a delta and only parse thru the Jobs that are new or have been changed.

Now when you do your lineage reporting, and you start while “standing” on the target Stage of an application down-stream Job….when you ask for “Where does this come from?”, you can expect to see Stages through many Jobs back to the ultimate source. If it dead-ends surprisingly, it’s probably because one of the three rules above didn’t apply, or there’s an odd Stage that isn’t supported for lineage (Redbrick is one of the only ones I’m aware of at this point).

If that fails, use the “Stage Binding” at the Advanced Services tab. This is one of the options for “manual” binding of metadata — a sort of “toolbox” of wrenches and bolts for when you need ‘em. The Stage Binding is designed to be used, only when absolutely necessary, to “force” two Stages together for lineage purposes. It is fairly easy to use…it prompts you first for the name of a Job, and then when it presents you with a list of Stages, slide your cursor over to the right so that you can “add” a Stage from another Job. I have used it effectively for unsupported Stages, as noted, but also when the rules above don’t apply. In one case I was sending the output of an XML stage into a Sequential File…and in the next Job, I was reading that with an “External Source” Stage. There is nothing in common between those stage types, and the columns were entirely different (the Sequential File Stage contains a column called “myXML” and the External Source merely carries the output of a unix list command (a set of filenames). I was able to establish perfect lineage however, by using a Manual “Stage Binding”, forcing the Sequential Stage of the first Job and the External Source Stage of the second Job to be “bolted” together.

Good reporting! Next topic will be “Connecting Database Tables to your Jobs.”

Ernie

Getting started with Data Lineage!

Last night I was reminded about a series of blog entries I’ve wanted to make concerning the InfoSphere Metadata Workbench and how to get the most out of its Data Lineage capabilities. The Workbench is very powerful — it illustrates relationships between processes, business concepts, people, databases, columns, data files, and much, much more. Combined with Business Glossary, it gives you reporting capabilities for the casual business user as well as the (often) more technical dedicated metadata researcher.

I’ve had a variety of entries about Workbench in the past two years (see the table of contents link in the top right, and find the metadata section), but nothing on “getting started”. As Metadata Workbench starts to support more and more objects, knowing certain skills and techniques becomes that much more important. This is especially true when trying to gain the most from Metadata Workbench when it is being used to illustrate Business Terms, Stewards, FastTrack Mappings, DataStage Jobs, Tables and Files, External ETL Tools, scripts and processes, operational metadata data and a vast list of other data integration artifacts.

Many of you who start with Metadata Workbench begin with DataStage/QualityStage Jobs.

So I will start there.

Once you have mastered lineage with DataStage, and its combination with other objects, you can then easily move on to other concepts for non-DataStage metadata, which I will also cover in this series of blog entries. If you are using Metadata Workbench and are not a DataStage user, stay tuned. As we progress I will take a tour through Extensions, Extension Mappings, Extended Data Sources and all other such concepts.

Start with your favorite, reasonably complex DataStage Job. Maybe one with a lookup or a Join, a reasonable sequence of Stages (8 to 10 or so) and preferably a single “major” target. Since you are learning about the Workbench, you should be familiar, even intimate, with this Job. That will help as you learn the various ways to navigate through the user interface, because you will know what to expect at each particular dialog, report or screen.

[this first "getting started" assumes that you have NEVER performed Automated Services against your DataStage Project....if you have, it's ok, but you might not get the same results as I am outlining below -- you may get more metadata than I am describing in this initial learning step. ...and if you don't know what I'm talking about (yet), that's ok too...]

Log into the Metadata Workbench and notice the “Engine” pull down at the left. This is the list of your DataStage Servers and their Projects. Open up the project, it’s folders, and find your Job. Click directly on it. Scroll up and down in the detailed page that appears. there is the main page with the picture of the Job (click on it and you will get an expanded view in a new window of what the Job looks like). The metadata you are viewing is up-to-date from the last moment you or a developer saved the Job in the DS Designer. Also there is a very important listing of the Stage types in the Job, along with their icon. Note below you have many “expandable” sections for things like Job Operational metadata…..investigate the options.

Now click on the “main” target Stage of this Job. This brings you to a similar looking detail page, this one for the “Stage.” Look around, but don’t click anything — when you are ready, select “Data Lineage” at the upper right. As you do so, consider “where you are standing” (you are on a “Stage”) and what sort of lineage you would like to see. As you will discover, knowing “where you are” when you start your lineage is very important.

The default option at the next dialog is “Where did this come from”. Ignore the three checked boxes for now and click “Create Report”. This will comb through ALL the possible resources for “where” data for the “stage you started on” came from. Look thru the list. Note also the highlighted line. Move it up and down. This highlight bar lets you select EXACTLY which resource you’d like to see for your actual report. The “total” collection of lineage resources is in front of you right now — you will select which one you want for a detailed source-to-target report. This is often a point of confusion because the highlight bar is not always obvious. Data lineage doesn’t show you “ALL” the sources — just the path to/from the ones that you select [we'll contrast this in a later entry with Business Lineage, which DOES provide a summary of ALL sources or ALL target from a particular resource].

Look at the bottom of the page. Find the button labeled “Display Final Assets”. Click it. The list of objects above should get much smaller. Most likely, it should just show the source stage for this Job, or maybe its ultimate source as well as a lookup source stage or a source for a Join. Pick the primary source stage for the Job and then click “Show Textual” Report.

Review the result. The textual report isn’t as pretty, but it tends to be more scalable. Scroll up and down, and note what you see on the left, and the Job details you see on the right. Everything is hyperlinked. Now find the little triangle towards the top left of this center pane where your report is (it’s called Report Selection or similar) and click on it. That should expose again the “assets” page. Now you can try “Show Graphical”. When you get there, play with it. Grab some white space around the diagram and move the whole thing around…..try the zoom bar in the upper left. Click on the various icons in the lineage and then right mouse on one of the stages and find “open details in new window”. That will bring you back to a detailed viewing page and the process starts again.

What happens if you choose the target stage of your original Job (the first stage you selected earlier) and ask for “Data Lineage” and select “Where does this go to”? If you haven’t done Automated Services as I’ve noted above, you should likely receive “No assets found” or “No data for the report”. This is because it’s the “final” target — there isn’t anything else. “Where did this come from” will yield a similar result if you happen to be “sitting” on a source when you start your lineage exercise.

If you practice this, you should become very familiar with the lineage report user interface, and will have a strong base for moving forward with more complex, and deeper, scenarios.

Next entry: Linking Jobs together……

(link to next post in this series: Linking Jobs )

Ernie