Q Apply Agent Dispatch in Q Replication

Posted by Frank Fillmore on July 24, 2009 under Q-Replication. Tags: , .

Baltimore Orioles ageless sage and Hall of Fame baseball manager Earl Weaver once said “It’s what you learn after you know it all that counts.”  Truly.  I was reminded of this a few weeks ago when a customer posed a Q Replication question which had never occured to me – but should have.

The Q Apply component grabs transactions transported from a replication source via MQ Series.  MQ Series is used for assured, in sequence delivery of data from one place to another.  So far, so good.  But for greater throughput and parallelism, Q Apply can assign up to 16 DB2 agents to post the insert/update/delete transactions from a single MQ Series queue.  So now for the stumper question: With > 1 agents, how does Q Apply ensure that transactions are posted to the target database in order to ensure transaction integrity (e.g. a primary key value is gets posted before a dependant foreign key)?  More concisely, how are the DB2 agents dispatched?

For the answer, I turned to another resident sage, IBMer and Q Replication maven Donna Kelsey.  Here’s what Donna explained to me.

A Browser agent is assigned to each incoming queue (note messages at Q Apply startup for the browser).  The browser asks for the next message by message ID, not just FIFO.  Q Capture sends a message ID in each message that has a sequentially increasing portion (“Dense Number”).  Q Apply takes a ‘first look’ at the data for the transaction, and determines if there are any dependencies for it in the current mix of transactions being applied by the various apply agents.  If there is some, then this message id gets put in the “dependency graph” to wait until there are no dependencies (for instance, same row updated in an earlier transaction).  When the message ID gets to the ‘top of the graph’, only then can an Q Apply Agent take it, browse the message again from the queue, and process it.  The idea of the Browser Agent, Dense Numbering of the messages and the Dependency Graph maintained internally make sure that the transactions are always applied in order to maintain transactional integrity.

1 Comment so far

  1. Somil Kulkarni July 27, 2009 4:43 pm

    Hello

    The Browser does not read messages by message ID unless it detects a gap in the messages. The messages are normally read in FIFO order.

    Transactions read from the receive queue are compared against ALL in-memory transactions regardless of whether they are being applied or not.

    Also, its probably easier to think of the “dependency graph” as a graph of transactions rather than MQ Messages because several MQ Messages could form a single transaction.

    When a Q Apply Agent applies/processes a transaction it does not have to re-read the message from the receive queue. “Separate LOB” processing is an exception to this rule.

    Somil Kulkarni

Leave a Comment

Name

Email

Website

Comments

More