I attended IBM’s zEnterprise announcement in New York yesterday.  IBM Senior VP Steve Mills said it was the most important announcement IBM had ever made in its impact on saving customers money.  He also said IBM spent US$1.5 billion dollars on the zEnterprise research and development effort over the past several years.  So, as movie reviewers ask about the latest blockbuster: “Can you see the money up on the screen?”.  The answer comes in a few loosely coupled parts.

1. Oskar Schindler said you must have a “clever accountant”.  Mills made it clear that organizations that can accurately allocate their IT expenses will see the most benefit from zEnterprise.  zEnterprise delivers System z quality of services (QOS) across heterogeneous architectures: the aforementioned System z as well as Power 7 blade servers and (eventually) System x blades.  The problem for most organizations is that System z “mainframe” costs have been capitalized from central IT budgets for over four decades.  As the PC revolution unfolded since the early 1980’s, most of the costs for networking, systems administrator salaries, PCs themselves and the software they run have been expensed out of departmental budgets.  Organizations with the discipline to accurately accumulate these costs certainly will be able to see the benefit of deploying the zEnterprise platform.  Interestingly, the table talk at lunch indicated that some IBMers see the sweet-spot for zEnterprise in the rapidly growing economies of China and Russia.  The reason?  Tight budgetary control and hierarchical, centralized decision-making in state and quasi-state enterprises (think Gazprom) will help them “get it” immediately.  I would not be surprised to see zEnterprise adoption in emerging and growing economies exceed that of North America in the next two years.
2. IBM has been able to run Linux on System z hardware using Virtual Servers and z/VM for a decade.  And the System z has been able to dispatch workloads to specialty engines within System z such as the Integrated Facility for Linux, zIIP, and zAAP for years.  Think of zEnterprise as extending that dispatching capability out of the physical System z box to discrete blade servers.  IBM’s goal is to move away from the “you can do everything on System z” posture - which in reality was a losing, rear-guard action - to embracing disparate architectures and acknowledging that maybe a print server really should run under Linux on an x86 platform.  Yet you can benefit from the centralized management and security of the System z.  This is workload integration at the chip, firmware, hypervisor, and middleware levels.  A pretty neat trick.
3. So what can I do with the zEnterprise?  Here are two relatively simple scenarios.

The Online Travel Portal  One well-known travel reservation site front-ends their expensive Oracle transaction servers with MySQL running on cheap x86 hardware.  While you’re noodling around trying to figure out the best intinerary, all you’re seeing is data replicated from Oracle to MySQL on a near-real-time basis.  When you enter your credit card number and hit “Purchase”, you’re routed to the Oracle OLTP server.  This is called “database tiering”.  I can now architect the same topology on System z with DB2 for z/OS on the back-end and x86 blades running the DB2 Express-C freeware database.  On the zEnterprise platform, these databases will communicate over a 10Gb private, secure network with extraordinarily low latency.  Ever get the “That seat is no longer available” message?  It might be a thing of the past with zEnterprise.

The Hospitality Company  This organization runs their centralized reservation systems on DB2 for z/OS already.  In order to support their frequent-guest affinity program portal, they have WebSphere Application Server running on a separate System p AIX server.  The only problem is that sometimes transactions hang to the point that the JVMs have to be recycled.  The Java programmers say their code is tightly written and the DB2 for z/OS database administrations say that the incoming SQL requests are satisfied sub-second.  While zEnterprise alone would not resolve this problem - see pureQuery and the lyrically named Optim Performance Manager Extended Edition - the application and the database servers will be as tightly coupled as possible while each runs on the optimal platform.  Since the transfer points and the servers themselves are under unified management, an entire layer of complexity (and potential breakage) will be eliminated.

The real buzz in the announcement for me is the IBM Smart Analytics Optimizer (ISAO).  For a generation as a DB2 database administrator, I’ve told my clients that OLTP and ad hoc query workloads should not be intermingled.  The solution has been to make copies of the data using replication technologies - InfoSphere Change Data Capture and Q Replication among them.  This approach has been a boon to DBAs, and software, storage, and server salesmen everywhere.  When it achieves its full promise, ISAO will evaluate incoming database requests and dispatch them along with the data needed to satisfy the request to the appropriate platform server.  DB2 for z/OS will serve as a centralized front-end for all workloads: OLTP, OLAP, ad hoc query, etc.  ISAO will transparently run the workload on the optimal platform and return the result set to the requesting application.  Organizations will be able to dismantle the miasma of extracts, FTPs, and other artifices now necessary to keep analytic workloads from bogging down OLTP.  And they’ll reduce complexity.  And save a ton of cash.

So on whose door will IBM knock first?  Clearly the System z installed base will be getting lots of attention.  But could Facebook or some other enterprise with orders-of-magnitude scaling issues (500 million Facebook users and counting) benefit from zEnterprise?  Surprisingly, the answer is Yes!  Facebook needs to manage lots of unstructured data (pictures, videos, et al) , but they also have the need for complex analytics.  First, to target online advertising ever more precisely, but also to serve larger societal needs.  Let’s say a man declares he needs a reduction in child support because he’s nearly broke.  The local social services agency unleases a smart agent to run against social networking sites and comes up with pictures on Facebook from the man’s recent two-week vacation in Hawaii.  Too big brother-ish?  A topic for another day.

As much as I dislike the relentless repetition that’s part of the nature of the world of blogs and twitter and listserves and email blasts, here I go with a shameless pitch for a TFG special services offering I emailed to several DB2 users earlier today.  I am doing this because, at the end of the day, the rate disparity in today’s DB2 services market baffles me.  Are the ridiculously high rates being charged eroding product adoption?  I am afraid so, which is why we are offering a summer services special. 

Read More…

While there are many indications the “Great Recession” is over, the budget lag and fear of one last dip seems to be constraining IT project funding - EXCEPT - of all places, training.  While this seems perfectly logical (when there are fewer hands to do the work, making sure those hands are skilled is critical), historically, training budgets are the first to be cut when there’s a downturn.  Maybe organizations are wising up this time around…here at The Fillmore Group we are receiving more training requests than anytime in the past five years.  Read More…

IBMer David Tolleson does a terrific job of outlining the configuration options for SQL and Q Replication when the source or target DB2 database is partitioned.  This was formerly know as Data Partitioning Feature (DPF).  Today it is part of InfoSphere Warehouse.

Check out David’s presentation.

Be sure to upgrade to the latest version of the Q Replication Dashboard v9.7.1.  More flexibility is provided - including monitoring Oracle Q Replication sources.  Check the link for all of the details.

As you know, I work with IBM’s Q Replication technology - a lot.  Q Replication functionality is delivered in InfoSphere Replication Server.  The challenges are amplified when working on DB2 for z/OS with *really* large tables.  One financial institution at which I am working has tables with over 1 billion rows and hundreds of partitions.  Of course, DB2 for z/OS can manage tables of that size, but what about the tooling?

Q Replication comes with a utility called ASNTDIFF.  ASNTDIFF compares a checksum of rows in the source and target tables being replicated to validate that there are no discrepancies.  Challenge #1 is that when replicating between DB2 for z/OS subsystems, ASNTDIFF runs under Unix Systems Services (USS) that provides the Unix APIs enabled in z/OS.  There are considerations for USS applications that will form the basis for another post.

Challenge #2 is that ASNTDIFF retrieves the rows from the remote system (Application Server or AS) using a three-part-name query across a Distributed Relational Database Architecture (DRDA) connection.  For example:

SELECT * FROM <location>.<schema>.<tablename>

where location is found in the DRDA Communications Database (CDB) portion of the DB2 for z/OS Application Requestor (AR) catalog tables.  You typically run the ASNTDIFF utility on the replication target DB2 for z/OS server.  That’s because the CDB has probably already been configured to support cursor-based loading of the target tables.  Why is this a challenge?  Well, a three-part-name query across a DRDA pipe against a 1 billion row table ran for about 18 hours.  Ouch!

So the basic problem is: how can I get three-part-name queries running across a DRDA connection between DB2 for z/OS subsystems to run faster?  I asked a couple of IBMers and Jim Pickel pointed me in the direction of exploiting OPTIMZE FOR n ROWS.  There’s a good explanation of this in “Limiting the number of DRDA network transmissions”.  Right now the ASNTDIFF utility adds OPTIMIZE for 1000 ROWS and FOR READ ONLY to every ASNTDIFF query.  We’re experiementing with the recommendations in the “Limiting…” document to see if we need to override this hardcoded parameter.  I’ll keep you posted as to our findings.

Most of you are aware that DB2 9 for LUW delivers PL/SQL APIs in v9.7.  That is, you can run Oracle applications against DB2 for LUW databases with little or no modification.  What you might not have heard is that IBM’s high availability, low-latency replication product - Q Replication - will now natively capture changes natively in Oracle logs.  That is, you can replication your Oracle Financials to a InfoSphere Warehouse.  Q Replication is a feature of IBM’s InfoSphere Replication Server, so you might not have noticed the new functionality with all of the name changes.

Two links describe the enhancement.  David Tolleson and Ed Lynch of IBM have both provided good information on the specifications and details.

Just announced and (I think) just released!  Frank will deliver a webcast on the latest version of the Q Replication Dashboard.  He installed it last night at a customer site and is already deciding which new features he likes best.  Stay tuned - I am sure his prep for the webcast will fuel some blog entries.  Here’s the invitation to the webcast:

The Fillmore Group and leading experts from IBM will deliver a 45-minute webcast to introduce the Q Replication Dashboard and improvements included in the new, redesigned Dashboard scheduled for release this month (or maybe yesterday).

The Q Replication Dashboard is the free monitoring tool available from IBM that replaces the Java Q Replication Dashboard and the Data Studio Administrative Console (DSAC) with a thin-client, browser-based interface. The Dashboard provides real-time Q Replication monitoring. In the webcast presenter and DB2 Gold Consultant Frank C. Fillmore, Jr., will introduce and explain how the new redesigned Dashboard’s expanded features and better ability to customize will enhance your organization’s Q Replication monitoring capabilities. Frank will demonstrate and explain the new Dashboard’s:
•Improved performance for replication environments with large numbers of subscriptions and queues
•Greater customization and filtering capabilities
•Ability to access more data
•Better access to historical data
•Simplified security
•Improved visual appeal
•Smaller (512mb requirement) footprint for both the client and server

Two sessions of the webcast will be delivered on September 15th, one at 9:00am EDT and another at 7:00pm EDT.

To register: E-mail kim.may@thefillmoregroup.com and include your name, company name, telephone number and the session (9:00 am or 7:00 pm) you would like to attend. Read More…

Baltimore Orioles ageless sage and Hall of Fame baseball manager Earl Weaver once said “It’s what you learn after you know it all that counts.”  Truly.  I was reminded of this a few weeks ago when a customer posed a Q Replication question which had never occured to me - but should have.

The Q Apply component grabs transactions transported from a replication source via MQ Series.  MQ Series is used for assured, in sequence delivery of data from one place to another.  So far, so good.  But for greater throughput and parallelism, Q Apply can assign up to 16 DB2 agents to post the insert/update/delete transactions from a single MQ Series queue.  So now for the stumper question: With > 1 agents, how does Q Apply ensure that transactions are posted to the target database in order to ensure transaction integrity (e.g. a primary key value is gets posted before a dependant foreign key)?  More concisely, how are the DB2 agents dispatched?

For the answer, I turned to another resident sage, IBMer and Q Replication maven Donna Kelsey.  Here’s what Donna explained to me.

A Browser agent is assigned to each incoming queue (note messages at Q Apply startup for the browser).  The browser asks for the next message by message ID, not just FIFO.  Q Capture sends a message ID in each message that has a sequentially increasing portion (”Dense Number”).  Q Apply takes a ‘first look’ at the data for the transaction, and determines if there are any dependencies for it in the current mix of transactions being applied by the various apply agents.  If there is some, then this message id gets put in the “dependency graph” to wait until there are no dependencies (for instance, same row updated in an earlier transaction).  When the message ID gets to the ‘top of the graph’, only then can an Q Apply Agent take it, browse the message again from the queue, and process it.  The idea of the Browser Agent, Dense Numbering of the messages and the Dependency Graph maintained internally make sure that the transactions are always applied in order to maintain transactional integrity.

If you are using Data Studio Administration Console (DSAC) to monitor a Q Replication environment, there are a few important things you should know.

1. DSAC v1.2 Fixpack 1 is available for *free* download.  You can access both the base code and the Fixpack at the Blogroll link in the lower right hand corner of this page under “Q Replication Tools”.  There are a couple of installation “gotchas” that you need to know that I covered in and earlier post.
2. What is old is new again.  The original Q Replication Dashboard was a Java-based heavy client.  That was replaced by the current DSAC implementation: browser-based, thin-client with a lite webserver imbedded.  Q Replication monitoring is now moving away from the “Data Studio” umbrella and will be called Q Replication Dashboard again.  My guess is that the new implementation will retain a lot of the “look and feel” of DSAC, so there’s no risk in starting to use DSAC today for Q Replication monitoring.  Watch the blog for details as they become available.
3. In a follow-up to Kim’s What’s in a Name post from a while back, IBMer Anjul Bhambhri made an off-hand remark at the June meeting of the Baltimore/Washington DB2 Users Group which identified an important IBM product naming convention distinction.  Products named “Data Studio” are available for *free*; these are typically tools which provide basic functionality (e.g. DSAC).  Products named “Optim” are for-fee products offering more integration and/or robust functionality.  “Optim” is a name that derives from IBM’s acquisition of Princeton Softech.  Most folks associated “Optim” with data archiving.  Now “Optim” will be the brandlet that covers IBM cross-database tooling.  Clear?  I hope so.  At least until the next change.

If you’re not using DSAC for Q Replication real-time monitoring, how come?