OpenIGC Accelerator

Hi Everyone…

Happy Spring! [for those of you in the northern hemisphere  ; )  ].   Great time to start “cleaning out” and “fixing up” things….whether around the house, or in the corners of our special projects.    In that latter category, I have “tidied up” a little utility I have been working on to assist everyone in building their OpenIGC prototypes or to assist in “getting to know” OpenIGC — a “form builder” for the “Publishing XML” needed to realize instances of your newly modeled and registered OpenIGC artifacts.

A lot of you have expressed the desire to get deeper into OpenIGC, but have found it difficult to get your arms around the xml aspects of it.  Either that, or cutting and pasting xml in a text editor is just not your thing.   For those reasons and others, I have been exploring various ways that a user interface could be created for OpenIGC assets — without resorting to an elegant albeit complex and lengthy GUI development effort.

Digging around, I found some open source javascript tooling to assist, and brushed off enough javascript and html skills to put it together.     At the url listed below you will find a tool that allows you to upload your bundle descriptor and generate a self-populating “form” to construct a publishing xml document for OpenIGC.   It also provides options to save the publishing xml to disk (for future use/editing) or to directly cut and paste into the igc-rest-explorer page.

It’s not “perfect” (I suspect it probably has its share of anomalies if you click on things out of order), but is hopefully a “helper” that will accelerate your efforts to implement custom assets for governance within IGC.

Please carefully READ the instructions (there is a link to instructions and a simple screen shot on the initial page).    The tool does not entirely “hide” your xml, and it REQUIRES that you understand your bundle (if you don’t know what I am talking about regarding OpenIGC and bundles, please review the blog series starting with https://dsrealtime.wordpress.com/2015/07/29/open-igc-is-here/ )! ….still, it does a few nice things for you:

• Performs all the xml tagging/formatting, ensuring that your xml remains “well-formed”
• Presents a “pull-down” select list for your classNames and attribute enumerations
• Generates the list of attributes (properties) for whatever class you select
• Automatically generates the unique “assetIDs” for the asset instances that you define
• Generates and presents a pull-down list for selecting “parent” assetIDs

As noted above, I can’t promise that it is entirely bug-free, but I can say that it has already helped me accelerate the prototyping of several bundles that I have been building recently to illustrate the power of OpenIGC for extending the repository.    Have fun, good luck, and please let me know how you make out in using this tool!       –ernie

http://www.openigcaccelerator.com

 

Accessing IGC via cURL

Hi Everyone…

This is a long overdue post …pointing to an article written by one of my IBM colleagues about accessing IGC metadata via its REST APIs — using cURL as your tooling.   He provides some excellent examples, complete with screen shots and recommendations.  Enjoy!

https://developer.ibm.com/recipes/tutorials/interact-with-your-governance-metadata-in-igc-using-rest-apis-with-curl/

Ernie

 

 

 

Tech Talk on Information Analyzer: Virtual Tables

Hi all.

Just wanted to pass along news of another Tech Talk.  This one on Information Analyzer and Virtual Tables.   Here are the details and the link to Eventbrite to register…

October 20, 2016
Time: 9:00 EST
Topic:  Virtual Table Feature in Information Analyzer

This presentation will provide a comprehensive overview of Virtual Table feature in Information Analyzer.  A Virtual Table is essentially a way to filter / limit the data from the source repository while performing IA Analysis like Column Analysis, Key Analysis, and Data Rules Analysis etc. The concept of Virtual Table is available from version 8.1 in IA workbench with quite a few limitations. A new type of Virtual Table called ‘SQL Virtual Table’ is introduced from 11.5 which eliminates all the limitations and allows users to define any complex SQL queries to filter the data during IA analysis. It also allows users to query exceptions directly from the source repository with the known queries. A SQL Virtual Table can be only defined using IA REST API / CLI at this moment. In this session, we will also see a demonstration of this feature.

Who should attend this session? – For all skill levels of current and prospective Information Analyzer users from both IT and line of business.

This topic will be presented by Suresh Tirumalasetti, Software Developer, Information Analyzer.  Suresh has extensive experience software development, customer support especially in Information Analyzer.  Suresh is located in Bangalore, India.

To attend you must register here:

https://www.eventbrite.com/e/virtual-table-feature-in-information-analyzer-tickets-28227856278

Password: Governance

Tech Talk on OpenIGC !

The session outlined below was held last week.    Marc did a fabulous job outlining how OpenIGC and its value for helping you achieve governance for ALL of your important metadata assets.     The recording can be found at https://youtu.be/0Tzz3fQYpRY

Ernie

 

Hi all…  wanted everyone to hear about the upcoming “Tech Talk” that is scheduled for next week.   Marc Haber, Offering Manager for our metadata offerings, will be presenting, while myself and others will be monitoring the chat room for questions and discussion.

Here are the details:

Event Name : Information Governance Catalog
Event Date : Wednesday, Sept 14
Event Time :  1 PM – 2 PM US (EDT) Eastern Daylight Time
Presented by : Marc Haber, Offering Manager
This presentation will provide a comprehensive overview of ability to extend the Information Governance Catalog and support governance across new and alternate Data Sources or Systems. Understand how customers satisfy their requirements for a comprehensive Governance implementation or metadata management system with Information Governance Catalog. We will explore the process for defining and structuring new Asset Types and publishing information specific to Assets. Lastly, explore the process to govern such Assets, lending meaning thru Glossary Terms, documenting requirements thru Governance Rules and mapping information to support Data Lineage and Compliance Reporting. This topic will be presented by Marc Haber, Offering Manager for Information Governance Catalog and Data Governance in general across Information Server.  Marc has extensive experience with Business Glossary, Metadata Workbench and Governance Catalog – helping customers implement governance initiatives or satisfy metadata management requirements. 

Registration –
https://www.eventbrite.com/e/ibm-tech-talk-is-open-igc-tickets-27329302680

Password:  Governance

Apache Atlas: GET-ting familiar with the REST API

Hi everyone.  Just posted the second in a series or recordings related to Apache Atlas, the Open Source initiative for metadata management and governance for hadoop.  Many of you have been asking about how to get metadata “out” of Apache Atlas so that you can load it into IGC or other repositories, or just use it for special governance reporting purposes.   In this recording we take a quick look at some of the key “GET” functions of the Apache Atlas REST API, and how you can easily do testing and prototyping of these calls using only your browser.   –ernie

https://youtu.be/6Us2zG-WvS8

 

Check out this “Recipe” for integrating Oracle ODI metadata into IGC!

Hi Everyone…

An IBM colleague has published an excellent use case on constructing an OpenIGC bundle  and publishing metadata and lineage for ETL processes represented by Oracle ODI.  She very nicely shows how to illustrate important structures and properties of a 3rd party ETL tool.   Ultimately, this leads to publishing of actual metadata instances so that IGC users can perform lineage reports and also “govern” (assign Terms, Stewards, etc.) their critical metadata.

Enjoy!

-ernie

https://developer.ibm.com/recipes/tutorials/creation-of-new-bundle-on-infosphere-information-governance-catalog/

Apache Atlas: “your first look!”

Hi Everyone.

Just finished uploading the initial video in a series of recordings concerning Apache Atlas, the evolving open source initiative for metadata management and governance in hadoop.

This recording is primarily designed for viewers who aren’t comfortable doing their own builds of open source solutions and also need some guidance on how to get started with vmware images that are available for download.  It introduces the concept and helps validate what needs to be done so that the viewer can be successful with available Apache Atlas resources on the web.  It starts with the download of existing images at the Hortonworks web site, and helps validate your environment so that you can continue with tutorials that are on the Hortonworks site, and/or start playing and exploring on your own.  This is the first in a series of recordings on Apache Atlas that share early experiences and discoveries regarding this important open source initiative for governance and metadata management in hadoop.

Recording can be found at:  https://youtu.be/C4lf_EFduqU

IBM Partners with Creative Solutions Using Open IGC !

Many of you come to these pages to understand how to extend the Information Server repository and use the various Information Governance Catalog APIs to enhance your users’ experiences and increase your governance capabilities.   But for some of you, there are too many interfaces, not enough time, not enough resources (or the right skilled resources) to complete the effort.   Please let me introduce you to various trusted IBM partners who have been trained on, and are using,  Open IGC and related techniques to help customers around the world reach their information governance goals.  Many of these partners have built formal “bridges” from various 3^rd party tools, to automate the metadata import process, and most of them also offer expert consulting on IGC and governance strategies in general.

To our partners…thank you for your efforts to spread the word about Open IGC and for helping our customers make even greater progress towards their governance objectives.

To our customers…I invite you to visit these partners’ web pages, ask them about how they can assist you with Open IGC and IGC issues in general, and challenge them to further expand their offerings to extend the repository for all your governance needs.

To our future partners…if you have built or are building a creative solution for achieving governance with the Information Governance Catalog, reach out to myself or my IBM teammates around the world so that we can introduce your efforts to the overall IGC community and ensure your listing is on this page.

Thank you!      –ernie

 

Compact Solutions  http://www.compactbi.com/solutions/data-lineage/

 

Lucid  http://www.lucidtechsol.com/get-stronger-data-governance-with-lucids-ibm-info-server-enhancements/

 

 

 

 

 

Manta  https://getmanta.com

 

Orion https://www.oriongovernance.com/

 

Prolifics  http://www.prolifics.com/solutions/information-management-analytics

 

 

 

INFORMATION-ASSET, LLC http://information-asset.com/

 

Other Vendor partners who have integrated their own direct solutions with the Information Governance Catalog via OpenIGC include:

ZALONI      www.zaloni.com

Data Migrators        www.datamigrators.com www.datamigrators.com

DiYOTTA  www.diyotta.comwww.diyotta.com

Pentaho (Hitachi Vantara)     www.hitachivantara.com

Denodo     https://www.denodo.com/enhttps://www.denodo.com/en

AxiomSL  https://www.axiomsl.com/

Evolving Atlas…

Apache Atlas is continuing to evolve, and quite quickly (see an earlier post about Atlas, including links to this open source initiative and other valuable commentary… Apache Atlas…a Common Metadata Initiative with “legs” ?).    Going beyond merely storage and process-based metadata, the Apache project is poised to introduce the ability to define a business taxonomy that increases common understanding and further defines assets across the enterprise.  The important inclusion of business vocabularies ensures that information governance incorporates the needs of ALL members of an organization, and not just IT.

As Apache Atlas takes on greater roles and open source accelerates its uptake, we can foresee a future where Atlas is called upon whenever and wherever data is accessed.  In her latest blog, Mandy Chessell floats the idea of a Connector Framework for Apache Atlas [http://www.ibmbigdatahub.com/blog/insightout-role-apache-atlas-open-metadata-ecosystem.]   Connectors of all kinds can access Atlas at the exact moment that they harvest or act upon data, with the ability to make decisions using everything that Atlas has to offer — ownership, location, data quality statistics, lineage, usage requirements and rules, and more.    This allows Apache Atlas to be more “intimate” with the data integration life-cycle and able to deliver governance rules that have real “teeth”.   –ernie.

Apache Atlas…a Common Metadata Initiative with “legs” ?

Lately there has been increasing discussions about Apache Atlas, (http://atlas.incubator.apache.org), an open source initiative for metadata and governance services.

Standards in the technology industry come and go.   Some make it and enjoy wide adoption; others do not, failing early or never really blossoming to their full potential.   Our industry is littered with examples that had promise but withered away because
vendors were unable to agree on common semantics or unwilling to let go of (or expose) proprietary intellectual material.  Meta models represent a significant investment, and often competitive leadership.  No one wants to yield hard earned territory, or potentially give away the “golden key” to their solution.   Standards like xmi, cwm, and others in the data integration and business intelligence space never fully delivered the nirvana that people hoped for.   They lacked the commitment, weren’t pushed hard enough by customers writing the checks, and capability wise, typically considered by many as nothing more than a “checkbox” requirement.  Certainly, competing vendors in niche data integration areas couldn’t stomach having their meta models shared interchangeably.

The climate for this is changing now.   Thanks to big data and open source, and trends such as the adoption of Hadoop in everyone’s sandbox (even if not in production).   Not participating, or flat-out ignoring open source, is no longer acceptable.   Being “open” is no longer a vendor liability, but a competitive advantage.   Not being open is a path to extinction. For these reasons and more, Apache Atlas is poised to be a major force in the drive for common information governance and metadata management.

Please take the time to read the blogs from two of our highly respected colleagues here at IBM (IBM Fellow Tim Vincent and Distinguished Engineer Mandy Chessell) regarding Apache Atlas and what it will mean for our industry:

insightout-metadata-and-governance

insightout-case-open-metadata-and-governance
That’s all wonderful news on the potential for Apache Atlas.  What does it mean for the InfoSphere  Information Governance Catalog (IGC)?

Along with other contributors from the vendor and user community, IBM is committed to the success of Apache Atlas.   Although still early in its incubator status at Apache, Atlas is already being implemented at customer sites for their hadoop based assets.   And while Atlas is not specifically limited to hadoop, today this is the primary domain where it plays and will mature.

In the meantime, Information Server customers using IGC want to use Apache Atlas to help federate the metadata in their hadoop distributions so that it participates in their enterprise governance ecosystem.   Atlas shows evidence of eventually supporting distributed and clustered configurations, but sites are looking to do this right now — by bringing Atlas metadata directly into the Catalog.   OpenIGC, the methodology and API for extending the IGC repository, makes this possible today.  Several customers, as well as IBM, are looking into how the two can be integrated.  Each technology supports a robust REST API, and describes similar constructs that can be illustrated in each, either directly or by extending the underlying default models.  Pulling Atlas metadata into IGC allows it to immediately participate in data lineage reporting, be assigned to subject matter experts and Stewards, related to data quality statistics, and to be connected to approved policies for data management and governance.   Sites can immediately reap the benefits of IGC in combination with their hadoop based Atlas investments, while still looking to the future and the benefits that Atlas holds for even deeper governance capabilities and participation by a vast number of vendors and technology owners.

Lots to do, and lots to keep track of!  But many things that can be done “right now” to take advantage of, and garner insight, into the future.  Stay tuned.  Atlas is moving towards becoming a standard with legs we can all stand on….

Ernie