Been awhile since I’ve posted anything — been too busy researching and supporting many new things that have been added in the past year — for data lineage, for advanced governance (stewardship and workflow), and now “Open IGC”. This is the ability to create nearly “any” type of new object within the Information Governance Catalog and then connect it to other objects with a whole new lineage paradigm. If you are a user of Extensions (Extension Mapping Documents and Extended Data Sources), think of Open IGC as the “next evolution” for extending the Information Server repository. If you are a user of DataStage, think of what it would be like to create your own nested objects and hierarchies, with their own icons, and their own level of “Expand” (like zoom) capability for drilling into further detail.
This new capability is available at Fix Central for 11.3 with Roll-up 16 (RU 16) and all of its pre-requisites (FP 2 among other things) and is immediately available in 11.5.
[…and is formally documented here: http://www-01.ibm.com/support/docview.wss?uid=swg21699130 ]
[also see here for tips on the IGC API for general management of assets: http://www-01.ibm.com/support/docview.wss?uid=swg27047054&aid=1 ]
So exactly what is this “Open IGC”?
Open IGC (you may also hear or see “Open IGC for Lineage” or “Open IGC API”), is providing us with the ability to entirely define our “own” object types. This means having them exist with their own names, their own icons, and their own set of dedicated properties. They can have their own containment relationships and define just about “anything” you want. They are available via the detailed “browse” option, and appear in the query tool. They can be assigned to Terms and vice versa, and participate in Collections and be included in Extension Mappings …and then…once you have defined them, you can describe you own lineage among these objects, also via the same API, and define what you perceive as “Operational” vs “Design” based lineage (lineage without needing to use Extensions, and supporting “drill down” capabilities as we see with DataStage lineage).
Here are some use cases:
a) Represent a data integration/transformation process…or “home grown” ETL. This is the classic use case. Define what you call a “process” (like a DataStage Job)….and its component parts…the subparts like columns and transformations, and properties that are critical. Outline the internal and external flows between such processes and their connections to other existing objects (tables, etc.) in the repository.
b) Represent some objects that are “like” Extended Data Sources, but you want more definition…..such as (for example) all the parts of an MQ Series or other messaging system configuration…objects for the Servers, the Queue Managers, and individual Queues. Give them their own icons, and their own “containment” depths and relationships. Yes — you could use Extensions for this, but at some point it becomes desirable to have your own custom properties, your own object names for the user interface, and your own creative icons!
c) Overload the catalog and represent some logical “concept” that lends itself to IGCs graphical layout features, but isn’t really in the direct domain of Information Integration. One site I know of wants to show something with “ownership”…but illustrate it graphically. They are interested in having “responsibility roles” illustrated as objects…whose “lineage” is really just relationships to the objects that they control. Quite a stretch, and would need some significant justification vs using tooling more appropriate for this use case, but very do-able via this API.
It’s all done based on XML and REST, and does not require that you re-install or otherwise re-configure the repository. You design and register a “bundle” with your new assets and their properties, and then use other REST invocations to “POST” new instances of the objects you are representing.
Quite cool…….and more to come…..I will be documenting my experiences with the API and the various use cases that I encounter.
What use cases do YOU have in mind? 🙂
Next post in this series: Open IGC: a Simple Messaging Use Case