Here is a consolidated list of all the posts, by major subject and link only. I was having trouble finding old entries myself! The Categories in the side-bar are useful, but bring up entire posts, not just a summary of the urls within that category. Let’s try this.
Just for fun!
DataStage and Minecraft!
Are your kids addicted to Minecraft?
Metadata Management
—-Apache Atlas: Thoughts and Discoveries!—-
Apache Atlas…a Common Metadata Initiative with “legs” ?
—- Governance: Open IGC! ——
[formal documentation: http://www-01.ibm.com/support/docview.wss?uid=swg21699130 ]
IBM Partners with Creative Solutions Using Open IGC !
Open IGC is here!
Open IGC: a Simple Messaging Use Case
Open IGC: Defining a new bundle!
Uploading New Assets
Defining Lineage Flows (Part 1)
Defining Lineage Flows (Part 2)
Open IGC Advanced Topics: Virtual Assets
Updating Your Bundle
Sample Bundles
Check out this “Recipe” for integrating Oracle ODI metadata into IGC!
OpenIGC Accelerator (helper tool that builds a form for your new assets!)
Building Metadata Extensions for Information Server: Why?
Methodology for Building Extensions
—- Data Lineage —-
Getting Started with Data Lineage!
Linking Jobs Together
Connecting Shared Tables to Your DataStage Jobs
Lineage for RDBMS Views
What exactly IS Data Lineage?
Data Lineage vs Business Lineage
Data Lineage for “other” ETL tools and external processes
Data Lineage and Web Services
—- Other Metadata Issues —-
Manually Creating Shared Tables and Data Files
What are Shared Table Definitions?
Business Vocabularies
ColumnNames!
Cool New Business Glossary Widget!
Anyone else Fried by Nulls?
—- General RealTime Discussions —-
ETL vs EAI Re-visited
RSS for your Data Transformations
What is Real-Time ETL Anyway?
Command Line Tools for Event Based Real-Time
Almost Real-Time Jobs
Simulating end-of-file for Real-Time ETL Transformations (what is “end of wave”)
—- XML —-
The New XML Pack is here!
Establish Meaningful Link Names when using the XML Stage!
Always use an Input Link AND and Output Link
Generating XSD’s
Online XML document validation
Reviewing REALLY LARGE xml documents!
Handling Nulls in XML Sources
Reading XML Content as a Source
another good link on using XML and Web Services
—- Java —-
Incorporating Java Classes into your DataStage Jobs
DataStage and TIBCO
—- Calling formal Web Services (DataStage as a SOAP client) —-
Tips for using Web Services with DataStage…Getting Started
Tips for using Wb Services Pack with DataStage Part II – Invoking your first Service
Tips for WSPack Part III – Best Practices and Guidelines
How to Invoke Complex Web Services
Calling Web Services that use Arrays
Great Web Services Testing Tool
—- Information Services Director (ISD) (DataStage/QualitStage as a Service Provider) —-
Using Shortcuts with Information Services Director (ISD)
Guidelines for publishing services with DataStage and QualityStage
Launching Jobs via Web Service
Data Quality and Transfomation as a Service
—- MQ —-
MQ Series – Ensuring Message Delivery from Queue to Target
—- online demo recordings —-
Real-Time Information Governance and Data Integration 
August 17, 2009 at 2:53 pm
Can we do screen scraping using datastage.We want to download some data from web by passing parmeters to web screen and then saving desired output in csv.
August 18, 2009 at 7:02 am
I haven’t tried it, but I’ve seen it done via JavaPack, where a java class using HTTP (and used within the Java Client Stage) retrieved items from a remote url, and passed them to the output link as an initial source for the Job. I recall someone trying this with Perl years ago also, but the Perl Stage is no longer generally available on the canvas. Java would likely be the best choice.
December 5, 2011 at 5:04 pm
Hi Ernie,
Are you planning to get into the world of connecting with Twitter, Facebook and Big Data from DataStage or other Information Server products any time soon?
Also, what do you think would be the best way to access data through LDAP – DataStage or something else?
December 5, 2011 at 5:50 pm
That’s a wide set of subjects….do you mean “me” personally (get into the world of connecting with…), or as a subject on this blog, or Info Server doing thing with it? 🙂
As for LDAP, best solutions I’ve seen are with JavaPack. Most accomplished Java developers are able to use well-proven libraries to tear thru LDAP easily…..incorporating such logic into a class that is designed for use in DS is then childs play….and you and the java developer’s creativity as to what you want to do with the access (send it down a link, receive it and update LDAP, etc.). If you need java examples I have some on the blog or contact me offline.
Ernie
December 5, 2011 at 6:18 pm
Well, here I meant you personally, in terms of would you be blogging about these topics. 🙂
Thanks for the suggestion, and the offer of off-line assistance.
December 5, 2011 at 8:12 pm
I’m flattered, but I don’t think anyone wants to hear my opinions about Facebook and Twitter and such… ; ) …and I’ll leave “big data” to my fellow DataStage experts who get heavily into parallelism and other framework stuff. Actually, the big data thing is interesting, and I’m keeping an eye on it, but at the moment I have my hands full with metadata and governance….but they’re coming together… it won’t be long before data lineage and governance finds its way in the hadoop world…(if it hasn’t already — just seems that the attention is on speeds and feeds and giant volume at the moment).
On Facebook…I dabble…even been experimenting with industry topic based “Groups”…but I don’t get on that often for regular Facebook activity.
Ernie
December 5, 2011 at 9:00 pm
But we would like to get some ideas about connecting to social media and big data from DataStage and related tools. Maybe Tony Curcio could blog about that. Hmm.
April 5, 2012 at 5:46 am
Hi Ernie,
Is there a way to export to a csv or excel file the datastage metadata? (where the data is sourced from and the transformation is undergoes)
March 17, 2017 at 3:25 pm
Hey Ernie,
is the problem described here: “http://www-01.ibm.com/support/docview.wss?uid=swg21678840” still only have a manual solution?
March 17, 2017 at 3:34 pm
Yes…that is still required.
Ernie