Launching Jobs via Web Service

Kicking off Transformation Jobs or functions via Command-line is an important feature, but it is equally important that we are able to launch such processing via Web Service or other service-oriented invocation.

Command line utilities are generally text based.  They require that you learn the vendor’s proprietary syntax.   They may require passwords and userids.   You may have to worry about clear-text or be sure there are ways to hide credential information.  You probably need to know about shell scripting to take maximum advantage of the Command language.  You may also need to know things like Job or Map names, need to understand the parameter structure, or the administrative naming conventions for the Transformation tooling (Project Names, Folders, etc.).   You probably need to know, as a developer, the hostname where the Transformation is running, or at least the engine you are connecting to for launching the process.  

Command line utitilies are powerful…. there are times, however, when a simpler invocation method is needed.  Like when the calling application doesn’t have easy access to scripts or the operating system command line.   Or when the skills of the developer(s) establishing the invocation do not include scripting or details of the Tranformation tool’s command line syntax.   Or when the ultimate location of the tranformation function being launched is unknown or moves frequently.    Service invocations using industry standards help abstract all of that….Servers can be anywhere, and underlying awareness of the Tranformation tooling is hidden from the invoking client.   Datatypes for Job Parameters are less painful to handle, and authentication and/or encryption can be handled at the transport layer.

A financial site I’ve been working with uses Web Services to kick off DataStage ETL Jobs from within a portal application.   The authors of the portal application are Visual Basic .NET experts.   They don’t know the first thing about DataStage, are under deadlines like the rest of us, and needed the ability to start an ETL Job to refresh a datamart as the result of a user pushing a button or choosing an option.   Using the RTI module for DataStage (release 7.x), the .NET development team merely consumes the automatically published WSDL definition to include this functionality.   Within minutes, they have the functionality included in their application.  No scripts to maintain, no complex awareness of DataStage.  For all the .NET developers know, the refresh process might be home-grown C++ code.    That’s the value of using the SOA industry standards.  The .NET developers don’t want to know and they don’t have to know that ultimately, it’s a DataStage Job doing the work.    Meanwhile, the skilled DataStage ETL team, who really understands the data, is maintaining the Jobs, the rules, the access to the source and target data structures, and publishing the functionality (as a Web Service, including the WSDL “contract”) for use by the rest of the enterprise. 

Command line control (in DataStage we generally call it the Job Control API or Job Control Command language, a.k.a. dsjob) in transformation tools is
great – but increasingly we need simpler methods that exploit the new standards.

Advertisements

17 Responses to “Launching Jobs via Web Service”

  1. jon Says:

    Hi,

    Just to clarify this,

    Is the RTI pack in 7.5 now the Websphere Information Services Director in v8?

  2. dsrealtime Says:

    The interface and some of the internals have changed, but yes, Information Services Director in v8 is effectively the next major release of RTI.

    Ernie

  3. jon Says:

    thx for that, currently working with IIS v8, trying to figure out how we can do kick off jobs SOAP over JMS…

    would you have any pointers to do this? 🙂

  4. dsrealtime Says:

    Hmm…. RTI supports the JMS binding, but it is not coming into v8 until 8.1 sometime later this year. One interesting alternative would be to have your own message driven bean invoked by an incoming SOAP over JMS message, and then have that bean either use SOAP over HTTP to kick off the Job via WISD, or use the EJB binding to kick off the job via WISD….

    Ernie

  5. jon Says:

    oh ok, we can’t actually do that as we don’t have WISD licenses.

    our current thinking is to use a cron job to check a JMS queue every minute… if it finds something it can kick off the DataStage job using ‘dsjob’…

    i guess we could maybe do the scheduling in DataStage somehow, but using a crontab seems more straight forward.

  6. dsrealtime Says:

    Absolutely. cron with queue checking and a call to dsjob will work fine. I tend to think WISD by default, as it’s the most elegant and doesn’t require that the client program know anything about DataStage, nor have to deal with command line authentication, project names, etc. — but if you are ok with starting things via dsjob, that will work fine.

  7. Roger Says:

    If you use dsjob to launch the Job (instead of a WS), how do you do error handling in your portal application?, ex.: what to do at the portal level if the job runs out of disk space or other run-time errors?

  8. dsrealtime Says:

    That’s a good point. In the JMS example, things are a bit “detached” (of course, one can argue that JMS provides the benefit of having things be asyncronous — it may be hours before the job finishes, or even gets launched, depending on how the queue reader is coded). In the simplest WS case, the SOAP client will be in a syncronous wait until the job completes, or will get a soap fault if the job fails.

  9. jon Says:

    we’re thinking that we’ll kick the job off using soap over jms (we have a tibco bus handy)

    then i’ll write a cron job that checks the queue

    when a message is found, it reads the parameters and does some basic sanity checks then sends a response back to the caller saying something like “initial checks ok, kicking off datastage job now”.

    then, when the datastage job has finished it will send an asynch message back to the caller saying that the datastage job has finished.

  10. dsrealtime Says:

    Excellent.

  11. Santosh Says:

    Hi,

    Can anybody provide me any link or PDF material to get start with WISD. I am not able to get any information from DataStage Documentaion.

    • dsrealtime Says:

      Hi Santosh

      WISD, or ISD as it is now called, is a part of Information Server v8. It is separate from DataStage, albeit tightly integrated and related. It allows you to publish DataStage Jobs as a service, but also supports the publication of DB2 and Oracle queries and stored procedures as services and Classic Federation, one of the offerings that allows you to define mainframe data sources.

      It has its own manual — if you are licensed for it, you should have it in your doc collection for v8. You will also see the WISD stage types in your DataStage Palette in te “real time” category.

      Ernie

  12. Milind Says:

    Hi Ernie,

    I have a requirement wherein for each row, I need to pass a set of values to the webservice. However, this set of values is more like a complex xml document(My source is hirarchical xml file which contains multiple rows.

    For eg my input has to be as follows

    Joe
    Ferrone

    23232332
    232313231
    6898089

    • dsrealtime Says:

      Hi Milind……not sure how to get the tags to stay… ; ) …but I get it…. It can be done, assuming that you are able to at least import the WSDL so that the communication bits come in correctly.

      Look carefully at the WS Transformer Stage….. on the “message” tabs for input and output you will see a pull down that says something like “user selected message”. This is a column on the input link that YOU create. This means “get the details of the SOAP body from this column”. You will need to get a sample of the xml payload “on the wire”…..you can use various web services debugging and testing tools for that, or else ask the developer of the service if they have one. Then work on getting DataStage, usually with the xmlOutput Stage, to produce that SOAP body, somewhere upstream in your Job.

  13. Milind Says:

    I am sorry but all the xml tags I had mentioned in my below post have disappeared. What do I need to do inorder to put in a sample xml document?

  14. soni Says:

    Hi all,
    Thanks for the very useful information.
    I want to know after deploy datastage job as a web service on IIS server. can we direct call web service form .net application server or we need to use any supportive tool like RAD, RSA or RDA tool.
    If yes then please tell me what is RSA or RDA tool role on .Net server.

    • dsrealtime Says:

      Once you deploy a service using ISD, it is available for ANY tooling that supports whatever industry standard you have chosen for your binding. If you chose SOAP over HTTP, just about every tool on the market nowadays can consume a WSDL document and invoke the service…that is true whether it is .NET, Java with your favorite IDE, or specifically using one of the Rational Tools. If you chose REST, then you could even test the service using your browser (REST is a protocol that uses HTTP verbs, so can be directly invoked for testing from your browser, which issues an HTTP GET in its simplest form). You don’t “need” RSA, etc. but they are certainly good choices. To get started, I always recommend using one of the many testing tools that are available, either in those development environments, or look at ones that are available from other vendors on the web (I like Actional and SOAPui, but there are many…just search for “SOAP testing tools”).

      Ernie


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: