Tips for Using Web Services Pack with DataStage: Part II – Invoking your first Service

Here’s begins a step by step set of “cheat notes” for implementing (ultimately “invoking” or calling) your first Web Service. See prior Web Services Pack entry for some thoughts on getting started in general and finding a sample service.

Assuming you have the WSDL for a nice, basic Service, and have DataStage Designer up and running, and (if on release 7.x) have WSPack successfully installed (it’s already “in” v8), then you are ready to import the WSDL based meta data. From Import…Table Definitions…Web Services Definitions, start the WSDL Meta Data Importer, when it’s up, locate your WSDL. It could be out at a URL somewhere, or use the button to find an already downloaded WSDL. When it’s opened successfully, you’ll its XML on the left, and a tree of its “Operations” on the upper right. For now, think of each “Operation” as like a ‘routine’ or ‘method’ or ‘subroutine.’ You may have one or many listed in this tree if you expand it. Select one of the Operations and then click the “Import” button on the lower right. If all goes well, you’ll see import messages for the input and output of your service.

Now take a moment and review the table definitions that were imported. In your TableDefinitions/WebServices category, find your Operation, and then in there, a set of TableDefs. Open them up. Take a look at their column names and associated meta data. These are the placeholders for the critical meta data of the service you wish to invoke. Some contain SOAP oriented information….but the key ones are named <yourOperation>_IN and <yourOperation>_OUT. They will illustrate the actual input and output arguments of your selected Operation. In most scenarios you wll never need to review these, and it’s just a curiousity for your review. But for some complex services, it will useful to review these Tables and know where they exist.

Now you are ready to build a job. The formal documentation is fairly clear on this point, but as a tip, if you’ve selected a simple service as I’ve advised, be mindful of the following:

  • Place a Web Services Transformation Stage on your canvas, and immediately preceed it and follow it by regular Transformer Stages. This is a best practice when startingthat will become clear as you get into advanced techniques.
  • When you first open the Stage, the critical button you’ll first see allows you to select your Web Service Operation. This brings up a GUI with some “gear” icons. Select your Web Service, and then click on the “Advanced” button. Note the various Web Service meta data attributes that appear. Port Address is the location of the Service, and is often a good candidate for a DataStage Job Parameter. The others are mostly informational only.
  • When “loading” metadata to your import and output links, choose the usual input and output tabs respectively, and then the “message” tab that is visible. When you “load” the message, it knows which Table Definition to bring in, automatically pulls in the namespace details, and will also automatically populate the Columns grid. The most common errors in using WSTransformer for the first time are when users try to load the Column grid first, before loading the message detail and letting the Stage populate the link grid.

If you chose a simple service (some good ones on the link in my first WSPack entry concerning Temperatures), you will be good to go at this point. Supply your input data and send your output to a Peek or Seqeuential file or whatever you like for testing and see if it works!

Advertisements

18 Responses to “Tips for Using Web Services Pack with DataStage: Part II – Invoking your first Service”

  1. Chris Says:

    I am trying to import a basic Web Service (tried several) however I enter the URL to the WSDL and the left pain shows the WSDL but there isn’t any operations in the Web Service Explorer and the Import button is disabled. I installed the WSPack for the server and client, is there anything else that needs to be enabled to use this functionality? Any help would be greatly appreciated.

  2. dsrealtime Says:

    This is one of those that deserves a long and detailed blog entry, but in short, there are a variety of reasons why you may not see anything on the right, the most common being that there is an issue with the structure of the input or output message parts. The WSTransformer is designed for single row in/out types of services, and more is needed if the service requires an array for input, and/or an array for output, or a complex structure of more than one “row” style construct. It’s very helpful at this point to have an actual payload of request/response from using the SOAP request. This can be captured in various ways, the most common being a tcp monitor. I sent you my email separately…if you can share the WSDL, and perhaps a sample payload, we can look at it and go thru the steps. There are also other reasons why you may not see Operations, but too hard to tell at this point what your symptom might be caused by…

    Ernie

  3. Santosh Says:

    Dear All,

    I am in process of building a POC using Data Stage 8.x that calls one of the web methods to verify that we can consume the web services using web service plug-ins Web services Client and Web Services Transformer.

    I am able to import the Web service Metadata using table import ‘web services WSDL definitions’ but when I am trying to fetch data from web page to a sequential file, DataStage map is throwing error as-

    ‘Unable to create Java Virtual Machine’
    ‘ASCL-DSJNI-00011`:`JNIWrapper: load library failed:’

    It seems there is some configuration required for web services at server level.
    Please let me know as soon as possible if any body have any idea about this.

    Thanks,
    -Santosh Rai

    • dsrealtime Says:

      Hi Santosh…..

      There are a variety of reasons that you’ll get errors on WSDL import….but that part looks successful…..the error on invokation looks like an environment issue. Are you using 8.0.x ? If so, there are environment variables that need to be set for DATASTAGE_JRE and DATASTAGE_JVM. What OS are you using? I don’t have my notes handy at the moment, but there are probably results for this if you search dsxchange and/or call support.

      Ernie

  4. Meera Says:

    Hi Ernie,

    That’s a wonderful blog on invoking a webservice through Datastage. highly helpful for the first timers.

    When i try invoking a webservice i face an issue with null namespace exception that’s being thrown with the webservice residing at a JBoss server.
    The exception says:
    Cannot invoke Call with null namespace URI for method null

    what could have caused this one?

    Thanks
    Meera

    • dsrealtime Says:

      Tough call…not sure… My first question would be “can you get ‘any’ web services to work from within DS”? and does “this service” work when outside of DS? …did the metadata import successfully?

      It “sounds” like a scenario when you are building your own xml soap body, and the namespace detail is left out, or the namespace details are missing on a custom header. I don’t know how far along you are in advanced scenario to advise you here, but if you are using XMLOutput prior to WSTransformer, then be sure that you have “LOADED” the namespace declarations on the Transformation Settings tab of your input link to the XMLOutput Stage.

      Ernie

  5. Derrick Says:

    I’m trying to use a user defined header, but I get a SAX Parsing error. The following is the header : I placed it in a VarBinary in hte transformer before the WS stage. ”

    The Web Service call works without the header, but the WS gives me back a exception saying I need it. I can successfully call it in SOAPUI

    This is the error
    Web_Services_Transformer_17,0: Fatal Error: Fatal: 0026`:`com.ascentialsoftware.wsclient.ServiceInvocationException: Service invocation exception: The prefix “soapenv” for element “soapenv:Header” is not bound.
    at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1602)
    at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1544)
    at com.ascentialsoftware.wspack.Transformer.process(Transformer.java:104)
    at com.ascentialsoftware.jds.StageFactory.runStage(StageFactory.java:544)
    at com.ascentialsoftware.jds.StageFactory.runTransformer(StageFactory.java:361)
    Caused by: org.xml.sax.SAXParseException: The prefix “soapenv” for element “soapenv:Header” is not bound.
    at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
    at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
    at com.ascentialsoftware.wsclient.xml.TreeBuilder.xmlStreamToDom(TreeBuilder.java:347)
    at com.ascentialsoftware.wsclient.InputRequest.buildRequestHeaderDocument(InputRequest.java:833)
    at com.ascentialsoftware.wsclient.WebService.callServiceInternal(WebService.java:1624)
    at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1591)
    at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1544)
    at com.ascentialsoftware.wspack.Transformer.process(Transformer.java:104)
    at com.ascentialsoftware.jds.StageFactory.runStage(StageFactory.java:544)

    • dsrealtime Says:

      The exact error is a new one………I usually have success when I cut/paste the exact header into an upstream transformer for initial testing, getting that header from a test tool outside of DataStage on a successful call. SOAPscope makes it very easy to capture this. I’d be interested in seeing if you can get tracing to work on your WSTransformer (Trace tab at the run dialog box….check the stage on the left and all 4 boxes on the right…in the log, if the job doesn’t abort, you will have a huge number of messages….with two xml ones in the middle….one for the outgoing message, and another for the return message…they will contain the SOAP envelopes)….. if you don’t see any (and the job doesn’t abort [in v8 the tracing option was broken]) then the error is local, and the message is not even being sent. It doesn’t like something about the soapenv namespace……

  6. Pradeep Says:

    Hi,

    I have couple of questions.

    1> When I import TD from WSDL I see multiple operations. I could not import all of them together had to do them one by one. Is there a way we can import all operations in a WSDL at the same time. If No, could you help me understand the reasons for this restriction.

    2>While importing a WSDL meta data we do it from DS client. I imported an operation and saved the TD successfully. Created a DS job, but while executing got below error. I believe the engine box has its own firewall which is preventing access. How can we test if a particular web service is accessible from the engine or not? Or am i going in the wrong direction.
    java.net.ConnectException: Connection timed out

    Thanks,
    Pradeep

  7. Nishant Says:

    Hi Erine,

    I am having the same problem as Cris. I am trying to import a basic Web Service (tried several) however I enter the URL to the WSDL and the left pain shows the WSDL but there isn’t any operations in the Web Service Explorer and the Import button is disabled. I installed the WSPack for the server and client, is there anything else that needs to be enabled to use this functionality? Any help would be greatly appreciated.

    I tried for different web-services from the list you have in the first blog. But nothing worked for me. Is there anything
    which you can suggest. Please see below for the WSDL which I am using.

    https://myteam1.medco.com/sites/mdmreports/_vti_bin/lists.asmx?WSDL

    Thanks…
    Nishant

    • dsrealtime Says:

      Hi…sorry for the delay in responding. Are you still having this issue? Could be a lot of things…quite often I find that you won’t see anything in the explorer if the wsdl has multiple “import” or “include” statements regarding its details, or something else that makes the wsdl too complex for import. Are you able to import/invoke the wsdl via any other web service testing tools?

  8. gjgoody Says:

    I am having the same problem as Nishant. We are both trying to extract data from a SharePoint list using web services. I was able to get the operations to show in the Web Service Explorer by saving the wsdl locally and adding the opening
    <?xml version ….. tag to the start of the file. But the problem is that once I go to import web service operation within the Web Services Cient stage the operations do not show up and I cannot go any further.

    Perhaps you could do a working with SharePoint web services post? I'm sure there is a lot of need for it out there with the proliferation of SharePoint within business' and the extractable data it contains.

    • dsrealtime Says:

      I would like to, if I had time and access to a Sharepoint server, but at the moment I have neither. But your issue is more primitive — the WSDL is not being read properly. I recall a thread on dsXchange where a patch was received to get past an import issue of web services? I am not in support and not familiar with every patch, but perhaps that is a possibility. Have you called your support provider? Indeed, you need that import to function properly in order to get the web services details into the Job….after that, there may be special techniques needed where I can provide pointers.

      Ernie

      • gjgoody Says:

        Many thanks Ernie. Great blog by the way, some very helpful posts. I’m new to datastage and find the lack of community and content online quite disappointing especially when compared to what is available for SSIS. Great to come across pioneers such as yourself!

  9. Thuan van Nguyen Says:

    Suddenly having issue with XML v3, dsjob,WS – all related to JVM failed to load.
    This is on v8.7 FP1

    Anyone having any clues on this?

  10. adelao Says:

    Hi can anyone help to solve one issue i have one sequential file1-> web service transformer -> sequencial file2

    I get the following error :

    Web_Services_Transformer_2,0: Failure during execution of operator logic.

    Web_Services_Transformer_2,0: Input 0 consumed 1 records.
    Web_Services_Transformer_2,0: Output 0 produced 0 records.

  11. 123456 Says:

    Can you please let me know the resolution of above error message mentioned?

    I get same error :

    Web_Services_Transformer_2,0: Failure during execution of operator logic.

    Web_Services_Transformer_2,0: Input 0 consumed 1 records.
    Web_Services_Transformer_2,0: Output 0 produced 0 records.

    Appreciate your help.

    • dsrealtime Says:

      It’s hard to say. Usually that means that the body wasn’t consumable “for some reason”. Some things to check and to try:

      a) be sure you can invoke the service from a good testing tool….such as SOAPui. Validate that it works, and that you are familiar with the return payload that it brings back.
      b) put a single column in your output link. Make it a longvarchar with some long length. Put a single “/” in the Description property. Also, on the message tab of the output link, use the pulldown for “user defined message” and select this single column. Send it to a Sequential Stage with “none” for delimiter and for quote character [assuming this is an EE job, otherwise, if Server, use 000 as value in delimeter and quote character.

      Review the payload that you receive. Build another Job that parses its xml……you may need to add that to your Job as a downstream stage.

      Ernie


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: