Skip navigation
Contact Us
988 Views 3 Replies Latest reply: Nov 12, 2012 8:59 PM by Corentin CORDIER RSS
Corentin CORDIER Apprentice 111 posts since
Nov 10, 2011
Currently Being Moderated

Nov 12, 2012 7:50 AM

Source reading vs. target writing

Hi all

 

I have a mapping that insert/update in different targets and for one of those, it did not get updated. The only explanation I found is that the row was skipped during the process in a joiner. It should not happened because the query should have returned something but at the moment of the execution, maybe it was not the case. Let me explain :

 

A SQ (SQ1) reads source (say S1) with a filter on a Status = ‘P’ (for Processing). The mapping sends an update on the source to ‘C’ in a specific pipeline. And another SQ (SQ2) also reads from the same table with the same filter ‘P’.

 

SQ1 -------- Tgt (update to ‘C’)

             \

              JNR ---- Tgt2

             /

SQ2 ---/

 

So, here is the story that must have happened :

1. SQ1 returned the row. Status is still ‘P’

2. Rows are processed to the different targets and the status is updated to ‘C’

3. SQ2 is still running, or even just starts, and does not find any row with status ‘P’.

4. The joiner does not find a match.

 

I fooly thought that the process would wait for all SQ to return their results but it’s clear that as soon as one gets the results from the database, those rows are processed at once.

So, is there a way to force all SQs to run before processing any rows ?

 

Likewise, I have a lookup on the same source (with again status = ‘P’) and no value was returned from it, so same issue, status was already updated to ‘C’. I could move it at the beginning for example to force the creation of the cache so it’s easy to fix. For the SQ though I can’t find any way to handle this. And I can’t move the status update to ‘C’ somewhere else since I aggregate the records (differents keys, depending on the target), so I have to update to ‘C’ just after the main SQ (SQ1).

 

Also, my session has a Target commit type and commit interval of 10,000 but only 100 rows were processed so I thought the SQs wouldn’t have seen any update to ‘C’.

 

Thanks for reading till the end and thanks in advance for any help !

    • Veeru B Master 1,242 posts since
      Sep 27, 2010
      Currently Being Moderated
      Nov 12, 2012 6:52 PM (in response to Corentin CORDIER)
      Source reading vs. target writing

      Hi Corentin,

       

      Is there any specific reason to use second source qualifier on same data set and look up?

       

      If possible try to use SQL override in SQ1 to get all the required attributes.

       

      Regards,

      Veeru

More Like This

  • Retrieving data ...

Bookmarked By (0)