How to Invoke Complex Web Services

Thought it was about time for another set of entries on Web Services. A few months ago I wrote about how we can invoke or “call” a Web Service from within DataStage, with a brief overview of the process. Now it’s time to address more difficult scenarios. I’ve had some comments from some of you who have run into issues — hopefully these details will help.

What exactly is a “complex” web service? Well, the word “complex” is relative, but I guess for our purposes here, it means one that you couldn’t easily get to work the first time out-of-the-box! 😉 Seriously though, Web Services come in a lot of flavors. The industry standards are in place, but not always rigorously followed. Web Services may have been designed and written before the standards were solid, or there may simply be constructs within and issues around a particular web service and its pattern that may it difficult to implement in any client tool. In fact, understanding the pattern of the service may be the most important research you can do before trying to utilize a web service in your applications. How often is it called? What does it do? What does it deliver? Does it send back one row or many rows? Does it expect you to “give it” one row or many rows when called? Does it send back data directly or just a big chunk of xml? Does it need you to send it a big chunk of xml?

Let’s start with defining what I’d call a “simple” web service and then go on from there. A perfectly simple web service is one with single row input and single row output, with maybe just a couple of columns in/out, no out-of-the-ordinary datatypes, and located on a nearby machine behind the firewall without any security. It will be invoked for every row that flows past. For DataStage purposes, it will use the Web Services Transformer Stage. In contast, here are many of the factors I’d use to qualify a web service as “complex”:

Output or Input only. Some web services are sources or targets. Call it once and it delivers rows, or receives information at a final target. In DataStage, we use the Web Services Client Stage for these. The gui lets you choose inputs that are static, particularly important if this is a “source”. For what it’s worth though, I tend to use the Web Services Transformer stage for these also, because I prefer to send my input via a link…it allows me to be more creative with the source parameters. What’s important though is that you understand the “pattern” of your desired Web Service

Security and proxies. Can you get there from here? What do you need to get past your firewall? Is HTTPS and SSL part of the web service you need to invoke? In basic scenarios, DataStage provides properties directly for these. WS-Security on the other hand, is more difficult, and involves a lot more hand-shaking and coordination between SOAP client and the provider of the Service.

Complex SOAP Bodies. XML comes in a lot of varieties. Sometimes the SOAP body being sent or received is in a hierarchical form. If that’s the case, we’ll need to decipher that XML into its relational rows and columns after we receive it at the client.

Embedded XML. A SOAP envelope is xml, but sometimes “contains” xml. In otherwords, we might have a simple string called for in the web service “myWebServiceResponse,” but there is no detail for it — it’s simply a big giant chunk of XML that is being passed back for further deciphering by the client. This is similar to the complex SOAP body, except in this case the WSDL contract knows nothing about the structure expect that a single string is being sent back. Again, we’ll need to decipher that XML into its appropriate parts after reception.

Arrays. The request or response is looking for, or sending back, an “array,” or “list” of values. There may be multiple columns in each entry of the list, or just one. This is most easily identified by looking at the WSDL in a browser — developers usually name the data areas as “ArrayOf……” or “ListOf…..”, although not always. A weather example I saw recently had “ListOfCities,” for example.

SOAP Headers. Ah. These can be tricky. Getting SOAP Headers right usually means having to know a bit more about the Service. The authors of the service will hopefully have documented what they expect in a SOAP Header, and when. Complex Web Services APIs may have a sequence of calls…one to “login” and perhaps get an access code, and then another to get started and do real work. The access code and other details are often parts of the header. Formal SOAP Headers are one thing, and what you might call “pseudo-headers” are another. In certain cases, the authors of a Service may have chosen, for various reasons, to place userID and access details inside the Body instead. It depends on the age of the service, how mature the standards, and the SOAP clients using them were when the service was first put into production, and the creativity of the original developer. We can retrieve these from the response, or if necessary, build them for the input request.

A fellow DataStage user has shared a WSDL with some of the issues above. As we work thru it and get it operational, I’ll share the techniques required.



26 Responses to “How to Invoke Complex Web Services”

  1. venu Says:

    while im process an Embedded XML. A SOAP envelope in datastage im getting below error please adivise
    xml_job1..XML_Input_0: Xalan fatal error (publicId: , systemId: , line: 0, column: 0): An exception occurred! Type:NetAccessorException, Message:Could not connect to the socket for URL ‘{0}’. Error={1}

    xml_job1..XML_Input_0: Xalan fatal error (publicId: , systemId: , line: 1, column: 230): Fatal error encountered during schema scan

  2. dsrealtime Says:

    Venu …

    You should have received the .dsx from me with the solution. Not sure exactly what was thwarting you in your installation, but as we discussed, there are a variety of solutions. For everyone else, it appeared to be a namespace issue. In my 8.x with FP1, I was able to read the documents just fine, but if there are issues and you are reading XML to put into an rdbms or other target, there are lots of shortcuts to get you moving….the easiest is to just pass your xml string thru a Transformer and edit out the offending namespace prefixes. You can’t always do that, but very often, especially if you are just shredding the document anyway, it will be fine. DataStage has rich text functions, especially in the BASIC Transformer or Server Jobs, that can manipulate lengthy text strings, with eReplace being one of my favorites.


  3. Heartburn Home Remedy Says:

    Not that I’m totally impressed, but this is more than I expected for when I stumpled upon a link on Digg telling that the info is quite decent. Thanks.

  4. Anu Says:

    We have been reading your articles Datastage webservices and they have been very helpful. I have a doubt, was wondering if you can help us.
    We need to connect Datastage 8.0 to an application via a URL by passing username and password in the HTTP headers. This application will return the cookie and a session Id within the cookie. This session Id should be captured for further transactions.
    Now we imported the WSDL published by this application and created a Webservices Tranformer Stage but have not been able to figure out on how to pass the username and passwd in the HTTP header to connect with this app? Any help?

    • dsrealtime Says:

      It can be done, but it is not particularly easy. The WSTransformer has the ability to pass a header, but it has to be in the absolutely correct format, and namespaces have to right….. the best way to approach it would be to use some separate testing tool (Actional, SOAPscope from Mindreef, etc.) and formally capture a “working” SOAP envelope and header. Then you will be able to tell what you need to build and send. It gets more difficult than that, because you may also have to custom craft the details of the SOAP body, and then do the same with the response and any other queries/commands you may be issuing. WSTransformer is best for single row, stateless and no-session invocations that return a transformed result. If you need to do complex processing it is best to put it together in a java client and invoke via the Java Transformer. In the java client you will have all the control that you need, and can more easily unit test it outside of DS and then include in DS for integration testing once you know how Java Pack works.


  5. Viagra Says:

    Thank you for your help!


  6. Sergio De León Says:

    Hi Ernie, could you please tell me where can i get the information or example about web services with arrays, thanks.

  7. Sergio De León Says:

    Hi Ernie could you please tell me where can I get information or some example about web services with arrays, thanks.

    • dsrealtime Says:

      Hi Sergio… Dealing with arrays can be tricky, but it can be done. Here are some general guidelines:

      a) Be sure that you are 100% successful with web services that do NOT have arrays. Able to call them easily from WSTransformer or WSClient — just to make sure that everything is ok in the environment.
      b) Be sure to have a test tool, such as SOAP UI or Actional’s Soap Testing tool, so that you know EXACTLY what the behavior of the service is, and what the arrays look like when it is working, etc.
      c) Start if you can with a service that takes one row in and sends back an array (these are easier to work with initially).
      d) Import the WSDL and do as you would normally, except then, go back into the OUTPUT link of the WSTransformer, DELETE all the columns that come in automatically when you import at the “Message” Tab, and put in one giant column of your own (mySOAPresponse with longvarchar and a long length of your choice). Put a single “/” in the Description property, and then at the Message Tab, find the pull down for “user defined column” and select your new column.
      e) Send that output link to disk, with no formatting (no commas or quotes).
      f) look at it. It should be a valid chunk of xml.
      g) Now you have to parse out that xml. Do you have experience with the xml Input Stage?
      h) Put a Transformer between your WSTransformer and an XMLInput Stage, and another Sequential Stage after that.
      i) push your single column thru the Transformer and into XMLInput. Check that this is XML “content” on the xml Input Stage.
      j) on the output link for the xmlInput Stage, load the _OUT table definition that was imported from your WSDL. This will have the details for the lower level structure inside the SOAP body.

      It will take some trial and error and testing, especially if you have never used the xml stages….. good luck!

      …and if your service needs an array on input, it’s the same idea, but in reverse, but learning how to write xml with xmlOutput Stage is more difficult, so give yourself time to learn that, and send the xml you produce to a flat file and compare to the result used in one of your testing tools before trying to send it to the WS directly.


  8. kumar Says:

    Hi Ernine,

    How to send the session id while invoking the sales force web service, my sales force team is saying session id is mandatory to invoke sales force web service.


  9. kumar Says:

    Hi Ernine,

    How to send the session id while invoking the sales force web service, my sales force team is saying session id is mandatory to invoke sales force web service.

    • dsrealtime Says:

      ..I believe we covered this offline in a separate discussion… can obtain the session id in an upstream call, and then build the header needed for the next invocation.

      • Dave Says:

        Please send a link which covers the topic ” you can obtain the session id in an upstream call, and then build the header needed for the next invocation ” for salesforce webservice using datastage

      • dsrealtime Says:

        Hi Dave. I can’t think of a particular “link”, but when doing something like this, you typically have to make first an authentication call, and get back a handle or “session id”. Build a Job that just does that only. Once you have it working, you can add more. This would assume that the session id is then on an output link, going downstream to another Stage. At that point, in a Transformer, or xml stages, etc., you can build-up the header that you need for your next call to the web service provider, a call that you should also be first testing in its own Job, perhaps with dummy session id info if provided by the vendor. This can all get very tricky, especially if the service you are calling also expects you to issue repeat calls based on data retrieval, for multiple pages or batches of data. In those cases, you can build Jobs like this but it is often just simpler to use the SalesForce Pack, or else code a complex client yourself using the Java Integration Stage.


  10. nagaraju Says:

    Hi Ernie,

    I asked this question in DSXchange also but trying my luck here also i think you can only help me. Can we call web service using web service client stage with out passing any input arguments. it works fine in SoapUI but in datastage it is giving error. i strongly believe it is because of void input argument because other method in the same web service is working fine through datastage. Please help me

    • dsrealtime Says:

      Not much I can offer outside of the thoughts I put into dsxchange…. what happens with tracing, and with capture of the entire output payload?

      • nagaraju Says:

        If i change to trace from fatal ,job is successful but 0 rows. i did not understand ‘capture of the entire output payload’. I am using web services metadata to import table definitions

  11. Sri Says:

    Hello Ernie,
    I tried opening an account on DSXchange but was having issues with the registration form over there. We have to try and access a web-service using Certificate Authentication. After reviewing the logs, it seems like DS 8.5 is sending in an empty header in the SOAP message. We were able to capture the header using a .Net Test Harness so that we can build it manually in DS 8.5 but it seems like it has a lot of dynamic parts in it like the following:



    Would you be able to shed any light on how we can build the header so that the above properties are dynamically generated in DS 8.5?

    Thank you.

    • dsrealtime Says:

      SOAP headers can be created manually in the WSTransformer Stage, by building it upstream, but try to get a look directly at the xml itself…using a tool like SOAPui or Actional’s SOAP tester makes this very easy.

      It could also be (it looks like a possibility considering the string up above) that this service is using formal WS-Security elements, and may be signing and encrypting the header independent of the rest of the soap request.

      In that case, it’s much more difficult to do and you will probably end up needing a javaPack implementation. Speak to the author of the service and get the details of what EXACTLY is needed to call it.


      • Sri Says:

        Thank you for your response. When we spoke to the author, they keep saying that all we need is the certificate. We added the keystore information and reviewed debug information in DS 8.5 Director also but still have not been able to resolve this issue. We also captured the xml using SOAPUI. Is there a way for me to send this over to you?

      • dsrealtime Says:

        This is not the best forum to debug this, so I highly recommend that you contact your support provider, and to also get things worked out at dsXchange, so that your issue gets the largest audience — but in the meantime, what kinds of messages are you getting in the log? …and have you gotten any other services to work correctly?


  12. Kalai Says:


    How would add custom http headers to the transformer stage? Any help would be appreciated

  13. sa Says:

    I need an example of deploying a datastage job that gets a wsdl request for multiple operations and responds back with the requested data. I am using ISD input and ISD output stage. New to this stuff and appreciate all help.

  14. Sujatha Says:

    I have a quick question. When we are getting the WSDL file, it should have both the metadata structure and the data inside it. Am I correct?

    Because when I checked for WSDL file example, it is mentioned it should have all the elements like definitions, type, message, porttype, Binding , ports and service.

    At present, I am getting a wsdl file only with metadata structure and another url link which has only the data. Checking with you if my understanding is correct. Thanks.

    WSDL file with metadata structure-

    <xsd:element maxOccurs="1" minOccurs="0" name="approval_set" [/u]

    url with Data alone [/u]
    {"result":[{"parent":"","made_sla":"true","caused_by":"","watch_list":"","upon_reject":"cancel","sys_updated_on":"2017-06-26 21:37:51","child_incidents":"0","approval_history":"","number":"INC0014775","resolved_by":{"link":";,"value":"db876dc94fcce200902e3f828110c753"},"sys_updated_by":"KMontee","opened_by":{"link":";,"value":"db876dc94fcce200902e3f828110c753"},"user_input":"","sys_created_on":"2016-07-29 21:16:00","sys_domain":{"link":";,"value":"global"},"state":"7","u_dartnet_wf_id":"","sys_created_by":"DBourinski","knowledge":"false","order":"","calendar_stc":"7885","closed_at":"2016-07-31 00:00:00",

    Thank you,

    • dsrealtime Says:

      Sorry for the late reply here. This is a complex subject. WSDL contains the “contract” for the service, and helps establish what a SOAP client needs to send and what it can expect to receive. The DataStage Designer can import certain kinds of wsdl, although there are various things that will make it more difficult……depending on what the header and security requirements are, and the complexity (usually as a potentially multi-level hierarchy) that is required for the sending payload and the receiving one. Best practice for building a Job for calling web services is to have access to the author(s) of the web service, who deeply know exactly what is required for the payload structure, security, etc., and to have a web services testing tool, like SOAPui, so that you can fully understand the service and have it perfectly working from your platform, using a simpler tool that is only calling the service and putting the results into a window. Also, see if the authors also support REST. Nowadays that is the trend, and REST is simpler to work with, in all facets. You would then use the Hierarchical Stage to parse a JSON or XML response or request, without some of the complexity that SOAP brings along…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: