#IBM Q Replication – “Frank’s Field Notes Summer 2016” Materials @IBMIIG

Posted by Frank Fillmore on August 5, 2016 under Q-Replication. Tags: , , .

One of our best-attended webinars ever!  The recording of the webinar is found here.  There presentation materials for download: IBM Q Replication – Franks Field Notes Summer 16

Services – Available Help for the “tions”

Posted by Kim May on May 30, 2014 under Authorized Training Partner, Big Data, Data Studio, DB2 Connect, DB2 Education, DB2 for i, DB2 for Linux Unix Windows, DB2 for VSE&VM, DB2 for z/OS, DB2 Gold Consultants, DB2 Migrations, Federation, Frank Fillmore, Hadoop, IBM Champion, IBM DB2 Services, IBM Pure Systems, IDAA, InfoSphere, InfoSphere Streams, MQT's, Netezza, Optim, Oracle, pureScale, Q-Replication, SQL Tuning, TFG Blog. Tags: , , , , , , , , , , , , , , , , , .

I was on a call this morning with an IBM technical team to discuss our progress filling the seats in one of our upcoming free IDAA classes.  Not surprising, given the number of people on the call (and who they were!) we went a bit off topic and ended up chatting about services and what appears to be – to IBM customers and sellers at least – a shortage of skilled consultants available to help with the “tions” – replication, implementations, migrations and federation.

The Fillmore Group has skilled consultants available to support these efforts, at rates that are typically about half of those being charged by “the competition”.

If your team has a project pending, or one in process, where a skilled consultant, at a reasonable hourly rate, can get things back on track, please contact me.  We are ready to work!  A list of our IBM Information Management capabilities is attached here.


IBM zEnterprise Technology Summit – May 6th, Washington, DC

Posted by Kim May on March 21, 2014 under Baltimore Washington DB2 Users Group, DB2 for z/OS, IBM DB2 Services, IDAA, Optim, Q-Replication, TFG Blog. Tags: , , , , , , .

IBM has scheduled a second presentation of the 2014 zEnterprise Technology Summit due to the snowstorm that hit in January when the event ran originally.  If you are in the area and have time to attend, please do!  Information on the event is below; register on the IBM site here.  There are several other events scheduled across the US in 2014 – view them on the IBM site here.

IBM zEnterprise Technology Summit Tuesday, May 6, Washington, DC

New technology is stretching IT budgets in many directions, forcing IT to find better ways of meeting business demands in increasingly complex data centers. Read More…

Hot-off-the-presses Redbook on DB2 for z/OS High Availability #DB2forz/OS

Posted by Frank Fillmore on January 25, 2014 under DB2 for z/OS, Q-Replication. Tags: , , , .

IBM just published a new Redbook titled Understanding and Using Q Replication for High Availability Solutions on the IBM z/OS Platform.  Two of my fave Silicon Valley Lab replication developer/experts – Cecile Madsen and Jayanti Mahapatra – are co-authors.

Customer expectations in 2014 are that they can conduct business with a variety of financial, retail, and other enterprises 24x7x365.  Even with the availability characteristics of the System z platform in general and DB2 for z/OS in particular, planned and unplanned outages will occur. 

The Redbook focuses on IBM InfoSphere Data Replication (IIDR) Q Replication as the undergirding HA technology.  It details two-node and three-node active-active replication topologies.  I suspect much of the basis for the document is the massive deployment of Q Replication at a large commercial bank in Brazil.

Attending zIM Bootcamp at IBM Silicon Valley Lab Next Week @IBMDB2

Posted by Frank Fillmore on January 25, 2014 under Big Data, DB2 for z/OS, DB2 Gold Consultants, IBM Champion, IBM Information Management Software Sales, IDAA. Tags: , , .

Each year IBM assembles their Client Technical Professionals (CTPs) for System z Information Management software at Silicon Valley Lab.  For the past three years IBM has invited DB2 Gold Consultants to participate.  The 4 ½ day conference is rich with lectures and hands-on labs covering such diverse topics as DB2 for z/OS tools and utilities and IMS.  I am particularly interested in:

During the week I’ll blog and Tweet.  I look forward to your comments and questions.

Q Replication Dashboard Enhancements Requested

Posted by Frank Fillmore on October 22, 2013 under InfoSphere, Q-Replication. Tags: , , , .

If you work with IBM InfoSphere Data Replication (IIDR), you are probably familiar with the Q Replication Dashboard.  The Dashboard is a GUI which enables you to manage and monitor your Replication and Event Publishing topologies.  For example, IBM’s direction is to move away from using asnmon to set monitoring and alert thresholds and to build this functionality into the Dashboard.

Here’s an opportunity to make a good product even better.  One of my customers had a series of requests to enhance the Dashboard:

  1. Provide the ability to export/import connection and configuration settings.
  2. Enable LDAP authentication.
  3. Provide something other than e-mail alerts (e.g. SNMP or Tivoli Systems Management integration).

Take a moment to review the formal Request for Enhancemant (RFE).

Data Interoperability Webinar – Recording and Materials

Posted by Frank Fillmore on October 3, 2013 under DB2 for Linux Unix Windows, DB2 for z/OS, Federation, IBM Information Management Software Sales, IBM Mid Market Customers, Informix, InfoSphere, Optim, Oracle, Q-Replication. Tags: , , , , , , , , .

Kim and I conducted a webinar on Thursday, October 3, 2013 that focused on IBM Data Interoperability software products and their specific strengths, weaknesses, and use cases.  We highlighted the application of the InfoSphere family of products to other-vendor databases such as Oracle, SQL Server, Sybase, and Teradata.  Kudos to Kim for putting together a presentation of disparate, but complementary technologies.

The recording of the webinar can be found here.  The presentation materials can be downloaded: Data Interoperability Presentation – FINAL 10.3.13

IBM InfoSphere Data Replication (IIDR)

Posted by Frank Fillmore on April 10, 2013 under DB2 for i, DB2 for Linux Unix Windows, DB2 for z/OS, IBM Information Management Software Sales, Information on Demand Conference, InfoSphere, Q-Replication. Tags: , , .

IBM has bundled its replication technology into a single package: IBM InfoSphere Data Replication (IIDR).  IIDR combines three components:

  • SQL Replication (heritage DataPropagator- Relational)
    • Easy to set up
    • Staging tables
  • InfoSphere Replication Server (IRS – heritage Q Replication)
    • High volume, low latency
    • Native Oracle and DB2 sources and targets
    • WebSphere MQ transport layer
  • InfoSphere Change Data Capture (ICDC – heritage DataMirror)
    • Broadest set of heterogeneous sources and targets
    • TCP/IP transport layer

There are a few of important take-aways.

  1. IBM substituted an implementation decision for a buying decision.  For quite a while SQL Replication was bundled free with DB2 for LUW.  ICDC and IRS were separately purchasable technologies with a lot of functional overlap.  In 2010 my colleague, Kim May, delivered an IBM Information on Demand (IOD) presentation distinguishing between the three.  Now you purchase the IIDR bundle and determine which technology is best suited for a particular use case.  As with most IBM software on distributed platforms, the cost is based on Processor Value Units (PVUs).
  2. New feature/functionality will be built into IIDR rather than the former heritage ICDC and IRS packaging.
  3. There will be a convergence of the technologies over time.  Many of the prospective changes are still IBM Confidential, but IBM is looking to consolidate components where it makes sense to do so.  There is a long-term roadmap that I hope IBM be sharing shortly.
  4. Upgrading and migration paths are a work-in-progress.  If you currently own ICDC or IRS and want to move up to IIDR, contact The Fillmore Group for pricing and implementation assistance.

Parallel asntdiff Dramatically Improves Performance for Q Rep on z/OS

Posted by Frank Fillmore on February 11, 2013 under DB2 for z/OS, InfoSphere, Q-Replication. Tags: , , .

The asntdiff utility is used to compare the contents of two tables, primarily in the context of SQL and Q Replication.  The problem has been that very large tables can take a very long time to compare.  Moreover, in a dynamic replication environment, the data you are comparing might rapidly be changing.  There were attempts at fixing this problem, primarily by using the asntdiff -f parameter to provide a smaller range of rows to be compared.  Using the -f input file, SQL queries subset the data on a key value (e.g. account number) or transaction timestamp.  The bottom line is that many customers abandoned using asntdiff against production tables with millions or billions of rows.

Parallel asntdiff provides orders-of-magnitude improvements in performance.  The key differences between the processing of the legacy asntdiff and the parallel version are as follows:

  • Parallel processing: partitioning, difference detection, and reporting all in parallel
  • Compare multiple partitions in parallel (source and target table) and retrieve rows from each partition also in parallel from source and target
  • Minimize traffic (only send data needed)
  • Exploit computational capabilities of systems (checksum calculations, etc.)
  • Use a Stored Procedure (ASNTDSP) which runs at source and target systems
  • Avoid storing intermediate results

The results are dramatic.  A European bank killed a legacy asntdiff compare utility run after 17 hours; it might have run for 30.  Parallel asntdiff completed in 5 minutes.

There are some restrictions and advisories:

  1. The source and target tables must have the same code page and collation sequences.
  2. For optimal performance, increase the system resource limit for application threads and set NUMTHREADS to 21.  The minimal requirement is 6 threads.
  3. Since asntdiff can determine the parameter NUMBLOCKS automatically, grant the SELECT privilege on the following catalog tables: SYSIBM.SYSTABLES, SYSIBM.SYSKEYS and SYSIBM.SYSINDEXES.

There are others, but the topic is well worth exploring to speed up asntdiff – or to be able to use it at all.

Thanks to Jayanti Mahapatra and Xiao Li of IBM’s Silicon Valley Lab for some of the material presented here.