Tips for using Web Services Pack with DataStage Part I: Getting Started

Thought I’d start sharing the notes I’ve collected over the years of working with the Web Services Pack.  This is the ability for DataStage to be a SOAP Client; for a DataStage Job to reach out and invoke a Web Service located “out there in the ether.”  This is a reference to the benefits of Web Services and the fact that you don’t have to know where the Service is located, what machine its on, who wrote it, or what language it’s written in — doesn’t mean it’s not a good idea, especially if your business is relying on it, but I think you get the point.   Give me a WSDL document, and that should be enough to invoke some type of remote function [this is what you might call the opposite of WISD, or RTI, which allow you to publish a DataStage Job or other Information Server asset “as” a Web Service].

Before you get started with it, ask yourself how comfortable you are with Web Services technology.   I speaking here about SOAP over HTTP in particular, the protocol supported by WS Pack, but it would be wise to get yourself a good intro to all of SOA.    There are good books out there, but also 1000’s of great resources here on the web.   Just search in your favorite browser for something like “Web Services Introduction XML,” and poke around.  You’ll find something for everyone.

Then you need the software.  If you are a release 8 user, it’s just “there.”   If 7.5, you have to speak with your account team about downloading it, but it won’t cost you anything.   The installation is simple, although be reminded that there is a client side and server side component.

Once installed, you’ll have two new Stages, the WSClient Stage and the WSTransformer Stage.  WSClient is for Web Services that will be a source or target to your Job, and the WSTransformer is for those Services that you expect to invoke on a row-by-row basis.

I have to put in a plug for the documentation.  The WSPack documentation is very thorough.  It has some very easy to understand graphics that do their own nice job of introducing Web Services in general, as well how WSDL, SOAP, SOAP Envelopes and SOAP Bodies come into play.  What I’d like to add in these entries are some other points to consider as you prepare to include Web Services in your jobs to help you be more successful.

A stand alone Web Services testing tool is a good idea too, unless you are already comfortable using Web Services from a common IDE for Java, C#, .NET, etc.

Finally, find yourself a good Web Service to start with, just to learn the mechanics.  There are some great ones out at www.xmethods.net .  You can test them there to see how the function, and get a solid idea of what their input and output requirements are.   There’s a nice mix of publicly available services there; some are free, others are part of a business.  The free ones are sponsored by folks who are making known their expertise.  Some very solid consulting firms and companies with Web Services expertise are represented there.  I’ve seen many of the services hosted at that site continue to function and be supported for more than five years.  For your first foray into Web Services pack, pick one that has very few input and output arguments, accepts and returns one row (a perfect candidate for WSTransformer) and doesn’t require you to pay anything or come up with an access code.

This entry is getting long.  Next time I’ll walk you thru access to the Service and then discuss how to prepare for a more complex one that you might find inside your enterprise.

Advertisements

12 Responses to “Tips for using Web Services Pack with DataStage Part I: Getting Started”

  1. Johnny Chu Says:

    Hi,
    This article is very helpful in terms of getting started with DataStage Service Pack. so thank you. I am just wondering where can I find “Part II” of this article?

    • dsrealtime Says:

      Hi Johnny. Thanks.. Glad you found it helpful. I never put the hyperlinks into each post connecting them, but there is a part II and part III….check out the table of contents tab in the upper right corner and find the section on web services. –ernie

      • Johnny Chu Says:

        Thank you so much for your reply I have found Part II and III. After reading your blog and googled around. Now I have a question. DataStage Web Service Pack provides the ability to for DataStage jobs to invoke web services “out there in the ether.” Do you know if DataStage can expose its functions as web services operations and allow an outside WCF or web service to invoke, for example to start, stop or configure, DataStage jobs? I am trying to find a solution to integrate DataStage ETL functions with my .NET application. Thanks for you time.

      • dsrealtime Says:

        Interesting question. Information Server, and DataStage/QualityStage in particular, has a component called Information Services Director that allows you to publish a Job as a Service. That provides a .NET application with the ability to send and receive messages (usually SOAP, but it supports other protocols) back and forth to DataStage as the “services provider.” DataStage can then do whatever it needs to —- lookups, transforms, etc. —- before returning a response. We usually call those “real time” jobs. They are generally set up to be “always on”. I have other posts on that subject if you look in the table of contents.

        To your question more specifically, the ISD technology evolved from RTI…….when RTI first came out, it had a web service “control” component, for simply starting and stopping “regular” batch jobs. It would have been exactly what you need here. In fact, at one point 7 or 8 years ago, we had a PDA based Director for DS that did exactly this……….anyway, fast forward — there is no longer a web service for starting and stopping Jobs….. …but it was really just a “wrapper” for the C++ API that does the same thing. We call it the Job Control API. It is available as a C++ API and also in a simpler command-line format. That would be the best for controlling “regular” DS Jobs if you wanted really explicit control (as offered by the Job Control API and as you might see in the DS Director — you could start, stop, inspect, etc.). Of course, you’d have to build that “wrapper.”

        If you just want to “launch” DS Jobs, consider using a Information Services Director with a Server Job — that server Job could receive a Job name from a web service client, and then “launch” that Job from inside a Transformer, using the DSUtilityRunJob function…. It requires a license for ISD, and wouldn’t have other powers, but it would work for kicking things off…

        Ernie

  2. 2010 in review « Real-Time Data Integration Says:

    […] Tips for using Web Services Pack with DataStage Part I: Getting Started November 20074 comments 5 […]

  3. Johnny Chu Says:

    Thank you so much for your response. Your answer is helpful for the POC of DS integration for our project.

  4. Navya Says:

    I need to extract the data from ibm lotus domino database and store it in a db2 table using DataStage Web Services Pack. Could you kindly let me know how i should proceed ?

    • dsrealtime Says:

      You’ll need to spend time with the various web services threads on this page, and also do some searches on http://www.dsxchange.com. There’s no assurance that it will even work — I don’t know anything about domino’s web services, their complexity, their security requirements, or the shape of their input and output parameters.

      First — get to know WSPack in DataStage. Use some very simple web services like the temperature conversion ones at ww.xmethods.net. Then get to know the domino service(s) that you want to call. How do they work? What do they require? Are they simple, like the temperature conversion? Or are they complex, requiring an initial connection for authentication, and then secondary calls for retrieval? Read their doc. Get their WSDL. Try them from excellent testing tools like SOAP UI or Actional. If they are really complex, requring multiple calls and programming sequences, then you’ll need to go with a javaPack solution. If they can just be called as a function call, on each row, then you could go with a WSTransformer solution.

      Lots to learn and play with along the way.

      Ernie

      • Suhas Says:

        Hi Ernie,

        Thanks for the info. But we’re still encountering problems accessing data using the web service.. Also the amount of data we wish to transmit is substantially large, so is there any alternate approach that we could use to get Notes data into datastage?

        Thanks in advance 🙂

      • dsrealtime Says:

        Key to your success will be comparing your results to your successful invocation of the service with SOAP UI, Actional SOAP Tester and/or tools like them. What was your result with those?

        …and what error are you getting inside of DataStage?…..can you import the Notes WSDL?

        How much is “large”? 10meg? 20meg? 200Meg?

        Ernie

  5. Vinodh Says:

    Hi Ernie,

    Thanks for all the replys and which yoy Provided the Valuble Information.

    I have a dout…am using DS 8.5 ,In Real Time Stages how to call the ISD stsges in Web Application.
    What my requirement is “”To Publish The DS Jobs as WebApplication””…
    Can you please Suggest me..on this


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: