Skip navigation
Contact Us
3893 Views 9 Replies Latest reply: Nov 17, 2011 6:34 AM by Corentin CORDIER RSS
santy cv Novice 94 posts since
Jul 11, 2011
Currently Being Moderated

Nov 13, 2011 10:10 PM

Pipeline in parallel

IN our maps we need to load data onto two staging tables from 2 different source tables. We would like to load both the staging tables in parallel.  There is no dependency b/w the two staging tables. We load the staging tables as separate pipelines with in the same map. At current INFA executes these pipelines sequentially. please let me know how to make these 2 pipelines to get executed in parallel.

 

thanks.

  • sasi ramesh Master 1,110 posts since
    Oct 2, 2010
    Currently Being Moderated
    Nov 13, 2011 11:02 PM (in response to santy cv)
    Pipeline in parallel

    Hi Santy,

     

    As i am aware ,we cannot load the data as parallel if there are more than one pipeline in informatica.The target load purely depend on the target load order plan.

     

    May be the below is the silly one

     

    If there is no depedency on the tables ,then create a seperate map and  session and place these session in parallel inside a workflow .

     

    I am not sure whether the below can help you .Just gving my thoughts

    I have heard that we can run the workflow as parallel if we configured the workflow to run at Concurrent.It is avail in version 8.6.We need to pass some parameters\input to change the each instance of the workflow. Please refer help for more details.

     

    Regards

    Sasiramesh

  • Corentin CORDIER Apprentice 111 posts since
    Nov 10, 2011
    Currently Being Moderated
    Nov 15, 2011 7:16 AM (in response to santy cv)
    Pipeline in parallel

    I don't understand your approach : putting in the same mapping 2 distinct process with no relationship between them ? A mapping must include a consistent logic related to one topic (load of SALES table for instance).

    Do like Sasi suggested, make 2 mappings and run the sessions in parallel.

  • Cezary Opacki Novice 47 posts since
    Mar 6, 2011
    Currently Being Moderated
    Nov 16, 2011 2:23 AM (in response to santy cv)
    Pipeline in parallel

    Hi,

     

    You can always combine those pipelines into one by using UNION transformation. Make sure it is possible to recognize which row comes from which source (pipeline). Then use router transformation to separate the pipelines. The pipeline recognition can be done by simply marking rows using expression transformation setting port like PipeNumber to one or two,...

     

     

    Regards

     

    Cezary Opacki

    • Corentin CORDIER Apprentice 111 posts since
      Nov 10, 2011
      Currently Being Moderated
      Nov 16, 2011 5:56 AM (in response to Cezary Opacki)
      Re: Pipeline in parallel

      So, you suggest to merge the two sources with a UNION transformation, and then separate the 2 dataflows. It's so less optimized than splitting the mapping in two mappings running in parallel.

      • Cezary Opacki Novice 47 posts since
        Mar 6, 2011
        Currently Being Moderated
        Nov 16, 2011 7:01 AM (in response to Corentin CORDIER)
        Re: Pipeline in parallel

        Are you sure that, there is big performance difference?

         

        The question was to do that in one mapping.

         

        So the answer is yes, it is possible.

         

        Regards

        Cezary Opacki

        • Corentin CORDIER Apprentice 111 posts since
          Nov 10, 2011
          Currently Being Moderated
          Nov 16, 2011 7:10 AM (in response to Cezary Opacki)
          Re: Pipeline in parallel

          It depends on the volume of data transfered. If it's thousands it's fine, you won't see the difference, but with millions, that won't be the same story.

          You're right the question is how to run that in parallel inside one mapping but no, it's not possible. The solution you describe is a kind of workaround but with such a design rows are processed to one target at a time and that's not a real parallel mode. The Union will also make a bottleneck, waiting the two readers to complete before the transformation can process the rows.

          So, my answer is why does he want to put it in the same mapping...

          Please answer Santy !

          • Cezary Opacki Novice 47 posts since
            Mar 6, 2011
            Currently Being Moderated
            Nov 16, 2011 1:57 PM (in response to Corentin CORDIER)
            Pipeline in parallel

            CorentinC,

             

            You have presented your opinion. You should mention that this is just your filling or  try to provide  evidence as well, please.

             

            I am afraid, I cannot agree with you. For example, the UNION does not wait for both sources. See the "Transformation Guide":

             

            The Integration Service processes all input groups in parallel. It concurrently reads sources connected to the Union transformation and pushes blocks of data into the input groups of the transformation. The Union transformation processes the blocks of data based on the order it receives the blocks from the Integration Service.

             

            Bye

             

            Cezary Opacki

            • Corentin CORDIER Apprentice 111 posts since
              Nov 10, 2011
              Currently Being Moderated
              Nov 17, 2011 6:34 AM (in response to Cezary Opacki)
              Pipeline in parallel

              Hi Cezary,

              sorry if I was a bit rude, that was not intended and I only wanted to give my point of view. What I meant is that I don't see the advantage to develop such a design, whereas it could be so simple to split the mapping in 2. I have seen too many times this kind of development and it always lead to poor performance, unexpected results and waste of time to fix it. So, to me, and  because there is no relation between the tables, and also because I think merging the data and then separating is a bad idea, I advise to develop 2 mappings.

              One good rule to adopt when developing : KISS : Keep It Simple and Straightforward (other translations exist)

              Thanks

More Like This

  • Retrieving data ...

Bookmarked By (1)

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points