Greetings after a little while.

For those of you who have been working with IBM’s data virtualization technology – as I have – since the beginning (we miss you DataJoiner, Federation Server, et al), there’s yet another new name: Watson Query a service of Cloud Pak for Data (CP4D).

DataJoiner began as a research project (code name: “Garlic”) led by then-IBMer Nelson Mattos. The goal was to provide heterogeneous data access via a Db2 for LUW “gateway”. Enterprises in the 1980s began deploying lots of discrete data repositories (primarily Oracle, Sybase, and MS SQL Server along with Db2) to serve line-of-business and departmental applications (e.g. sales, accounting, finance, manufacturing, logistics). The ability to access data from multiple disparate servers to provide a comprehensive view of business processes was a significant advance. DataJoiner was not limited to RDBMS repositories, but could also retrieve data from MS Access, MS Excel, CSV and Text files… pretty much any data on persistent storage with a definable structure.

For a while DataJoiner also provided increased performance and reduced expense benefits as well. IBM sold DataJoiner as a front-end to Oracle because (at least in some cases) a query using the cost-based Db2 optimizer embedded in DataJoiner produced a better access path than the, at that time, rules-based Oracle optimizer. Expenses were reduced due to the per-seat Oracle license charges; DataJoiner was just “one seat” which could support hundreds of concurrent users. Over time Oracle has modified both their optimizer and licensing model.

For those of you who geek-out on the roots of the technology that we apply to common data administration challenges in our day jobs – as I do – you’ll find a seminal paper on the IEEE website: DataJoiner: a practical approach to multi-database access

So why the history lesson?

• At its roots Watson Query continues to use much of the original federation “plumbing”: Wrappers, Servers, Nicknames, etc.
• Watson Query no longer relies on point-to-point connections to the back-end data repositories, but exploits a “computational mesh” which provides increased performance and resiliency. This approach leverages third-party data repository vendor advances in parallelism, caching, and compression. All of this is of enormous importance when enterprise data is distributed not just across multiple in-house geographically disparate locations, but perhaps across multiple cloud vendors as well.

• BUT (there’s usually a but), due to engineering and design within CP4D some of the capabilities in IBM’s legacy federation stack have not yet been incorporated into Watson Query. Two are of particular interest – which is why I have posted two Ideas (the successor to IBM Request for Enhancement – RFE):
  1. Formerly for data repositories with Indexes that metadata would automatically be captured for a Remote Table in, say, Oracle and propagated back to the Db2 SYSCAT.INDEXES catalog view for use by the Db2 optimizer to help build the best access path for the Remote Table represented by the Nickname. Right now that’s not being done. Hence Idea ASQL-I-16 “Propagation of Indexes for Virtualized Tables to SYSCAT.INDEXES”.
  2. Federated queries have always used Db2’s SQL dialect. There is a PASSTHRU capability that enables users to employ the native SQL dialect of the back-end data repository. For certain queries this can produce significant performance benefits. See Idea ASQL-I-17 “Federated PASSTHRU Capability in Watson Query (WQ)”.

Please review both of these Ideas and vote!

https://ibm-data-and-ai.ideas.ibm.com/ideas/ASQL-I-16

https://ibm-data-and-ai.ideas.ibm.com/ideas/ASQL-I-17

Thanks in advance.

#IBM #WatsonQuery #DataVirtualization #Federation

I want to thank everyone who helped make the inaugural Mid-Atlantic Db2 for z/OS Users Group virtual meeting such a great success:

• my colleague, Kim May, for pulling the meeting together and for filling a void of timely technical content for this topic and this region
• our presenters for crafting and delivering a comprehensive new way of considering IBM System z
  • Andrew Austin, Red Hat
  • Clark Hale, Red Hat
  • Elton (the Wizard) de Sousa, IBM

The recording of the meeting is here: https://youtu.be/Am279HW1DDY

Individual presentations are as follows:

Db2 VUG – OpenShift Introduction – Austin + Hale

Db2 VUG – OpenShift on IBM z – de Sousa

Db2 VUG – Mainframe Modernization Field Notes – Fillmore

Please share this content with your colleagues.

On Tuesday, January 13, 1987 I sat in the kitchen of my 11 foot wide rowhouse in the Highlandtown section of Baltimore and typed out on a manual typewriter the Articles of Incorporation for The Fillmore Group.  My son, Nathaniel, was five weeks old.

Some back-of-the-envelope metrics across the past 30 years:

• approximately 300 clients
• about 60 consultants have helped train these customers and implement their IBM Analytics software
• As a consultant, instructor, and/or presenter at technical conferences I have traveled to approximately 20 countries.  Prague is my favorite foreign city; I’ve been there three times and look forward to returning.

I have had two consultants, Ray and Jim, tell me that working for The Fillmore Group “has changed [their] life”.  In the beginning it was a common occurrence that an employment verification form from a mortgage underwriter would arrive shortly after a consultant started with TFG.  I know of several consultants that purchased their first home while working with us.

My favorite work environment was The World Bank.  The cosmopolitan atmosphere – working with colleagues from all over the planet – was delightful.  I helped implement a telephone billing system there using SQL/DS.

My most satisfying project was the Oracle to DB2 migration at JP Morgan Chase.  It was a tough, demanding environment with a timeline that was half of the 18 months originally estimated.  But we got it done with the help of Jim, Joe G., Joe L., John, Rebecca, and Teresa.  The customer had an equally talented, hard-working team.  Our contribution to that project was recognized by IBM SVP Steve Mills at the Insight conference that year.

As a small business owner, my favorite question from an IBMer (Scott): “Since the deadline can’t change, if money were no object what would you do?”  The answer: supply five more consultants to the project.  It was completed on time – if not under budget.

As a technician, my favorite question from an IBMer (Hunter) in the parking lot after a detailed technical presentation to a customer involving replication and federation tools: “We can do that, right?”  The answer: yes we can.  It resulted in a massive data warehouse at a large federal government healthcare agency.

I have had the privilege to learn from and share friendships with some of IBM’s original developers of relational database technology including Pat, Curt, Don and Don, and Hamid.  For IBM’s Hybrid Transaction/Analytic Platform (HTAP) implementations of today I rely on Paul, Tim, Gary, Namik, Knut, Patric, and Udo among many, many others.

I also cherish the friendships I have formed with other DB2 Gold Consultants like David, Ted, Bonnie, Jan, Juergen, Julian, Jackie, Kermit, Sheryl, Susan – and especially Gerry.

IBM has been a maddening company with which to partner, but the vast majority of the time The Fillmore Group has benefited from the technical excellence and basic decency of IBM employees.  In return, TFG consultants like Roger and Ravi have delivered innovative, cost-effective IT solutions to our mutual customers.  After more than two decades I still rely on Roger for his calm, wise counsel.

There have been tough times, too.  The Fillmore Group overcame at least three financial near-death experiences (1993, 2002, 2006).  At the risk of tempting fate: we never missed a payroll or failed to pay a debt on time.

But the best time, by far, was the day Kim May arrived to rent The Fillmore Group’s technical training classroom in downtown Baltimore in 2003.  That was the day my life changed forever.

This is reverie, not valedictory.  I am still too young to retire.  And I am as jazzed about the possibilities of HTAP, data repositories, and data interoperability (aka “plumbing”) in 2017 as I was when I first started working with SQL/DS (maximum storage capacity: 64GB) in the mid 1980s.  We’re currently working on a database rehosting project for a local government targeting DB2 for z/Linux and have an IBM DB2 Analytics Accelerator (IDAA) v6 Workshop scheduled next month.

But I wanted to take a moment to jot down a few thoughts on the past three decades.  Of course, I haven’t called out everyone by name who has contributed to our success over the years.  But I truly, sincerely appreciate your hard work, dedication, integrity, smarts, and moxie.  Thank you!

In closing, when Curt retired from IBM I asked if he was pleased with a career that had reached a pinnacle as an IBM Fellow.  His response: “It turned out better than I ever could have imagined.”  Me too.

Thanks again to everyone who joined us this morning for Frank’s presentation.  The recording and presentation files will be added in our next post.

In this post I am happy (thrilled!) to announce the Q Replication training class discussed on the webinar – the four-day, instructor-led session CE243G, “Using Queue Replication” has been posted on both the IBM and Arrow ECS websites and is therefore ready to accept registrations.  Please join us if you are interested, either in our Baltimore classroom, or via distance learning.  The class will run for four days, Tuesday – Friday, August 23-26.  Register here.

Special thanks to Megan Byrne, who helped get the class reinstated – it hasn’t run publicly for over three years – and is back on the schedule.

As a special incentive to anyone interested in attending the class in our Baltimore classroom, if you are interested in joining us for the Baltimore Orioles vs. the Washington Nationals game Tuesday night (after the first day of class), please let me know.  At The Fillmore Group we are all great O’s fans and they are, after all, in first place!

Finally, as one of today’s topics was the Oracle Native Apply now available for Q Replication, I am attaching a flyer with our Quickstart services offering to assist current Q Replication customers to transition from using Federation Server to the Oracle Native Apply.  Hope it helps!

Oracle Native Apply Quickstart 7.28.16

Thanks again for your help Megan!  Go O’s!

Frank and I attended an IBM Business Partner event in Philadelphia earlier this week where we met with several IBM’ers supporting Business Partner initiatives in 2016.  Each of the specialists presented their “best of” solutions to the partners in attendance, and then patiently answered our many questions (mostly mine – sorry team!)

The final preparations for our Woodberry Kitchen lunch event are underway:   Frank and I are polishing the presentation, IBM’s Andrew Levine and Warren Heising are tracking down answers to my Cloud deployment “what if?” questions, Megan is gathering gold coins for attendees who ask Frank good technical questions, and Jolanta at Woodberry Kitchen has me salivating with her menu recommendations.

If you will be in Baltimore April 21st and haven’t registered yet, please do!  Register HERE.

Our webinar delivered on March 11, 2016 outlining access to mainframe data specifically oriented to non-mainframers such as IT Architects and Data Scientists was a success.  The presentation materials are here: IBM z Systems – Mainframes – for IT Architects and Data Scientists

A recording of the webinar is found here.

Kim has just returned from IBM’s annual “z Bootcamp” where they prepare their System z sellers for the upcoming year with a series of product updates, announcements and education.  Her takeaway message for 2016 was loud and clear:  IBM intends to continue to enhance and extend the capabilities of System z so they can remain relevant and leading the effort to leverage the power of System z data.

Which leads, naturally, to the “how?” question.  At The Fillmore Group, as an IBM Business Partner we believe our ability to connect with Data Scientists and Data Architects and to help them understand the data stored in System z will help us all better align our interests and build a successful data and analytic ecosystem.  To kick start this effort we have planned a lunch event at the award-winning Woodberry Kitchen in Baltimore where Frank Fillmore will present an overview of System z.  Read more below – and please join us if you can!

“IBM z Systems (Mainframes!) for IT Architects and Data Scientists”

Abstract
Fifty percent or more of a large enterprise’s data – especially the lifeblood online transaction processing (OLTP) data – likely reside on IBM z Systems (aka mainframe) computers.  In many organizations, mainframes remain a black box inside the “glass house” data center.  Yet the rich transactional data they contain are crucial to predictive analytics, such as fraud detection, and real-time data warehousing applications such as point-of-sale market basket analysis.

This lunch session is intended for Architects and Data Scientists who don’t have a mainframe background, but have been tasked by lines-of-business to develop new competitive insights into customers, supply chains, insurance claims, financial transactions – and/or any of a variety of other opportunities to turn an enterprise’s OLTP data into Big Data.

DB2 Gold Consultant and IBM Champion Frank Fillmore will explain:

The types of data are found on the mainframe, including:
• DB2 for z/OS (relational database)
• IMS – Information Management System (hierarchical database)
• VSAM – Virtual Storage Access Method (indexed file)

Frank will also explain the four key approaches to unlocking the data:
1. HTAP – Hybrid Transaction and Analytics Platform
2. Replication to heterogeneous data stores
3. Virtualization of heterogeneous data stores
4. ETL – Extract Transform Load

Frank will also discuss the how these approaches apply to the types of data in the context of the application requirements such as service level agreements (SLAs).

The session will be held on Thursday, March 3rd, from 11am-2pm at the Woodberry Kitchen, 2010 Clipper Mill Road, Suite 126, Baltimore, Maryland, 21211.

Registration is required.  Register HERE.

If you are unable to join us in Baltimore, Frank will deliver similar content in a webinar on Friday, March 11, 2016.  Registration for the session, to be held between 11am and noon, EST, is HERE.

Please note:  registration is required for both sessions as space is limited.  If you are a mainframe systems programmer or database administrator this is *not* the session for you (you should already know this stuff!)

I was on a call this morning with an IBM technical team to discuss our progress filling the seats in one of our upcoming free IDAA classes.  Not surprising, given the number of people on the call (and who they were!) we went a bit off topic and ended up chatting about services and what appears to be – to IBM customers and sellers at least – a shortage of skilled consultants available to help with the “tions” – replication, implementations, migrations and federation.

The Fillmore Group has skilled consultants available to support these efforts, at rates that are typically about half of those being charged by “the competition”.

If your team has a project pending, or one in process, where a skilled consultant, at a reasonable hourly rate, can get things back on track, please contact me.  We are ready to work!  A list of our IBM Information Management capabilities is attached here.

Over the last 10 years I have become deeply immersed in three major (then) new IBM Information Management software technologies:

• IBM InfoSphere Data Replication (IIDR – Q Replication)
• InfoSphere Information Server (legacy DataStage, et al)
• PureData System for Analytics (nee Netezza) and – it’s sibling for System z – IBM DB2 Analytics Accelerator (IDAA)

If there’s a theme it’s data in motion, data transformation, and deep analytics.  In 2014 The Fillmore Group is focusing on BigData which encompasses all three.  As an IBM authorized Training Sales and Delivery Partner we offer IBM’s BigData curriculum.  As such I am gathering all of the resources I can find to dive into

1. InfoSphere BigInsights – IBM’s distribution of Hadoop (and a whole lot more)
2. InfoSphere Streams – Real Time Analytical Processing (RTAP) for data in motion like stock trades and social networking feeds
3. InfoSphere Data Explorer – federated discovery, navigation, and search in the BigData world

This week I am attending an IBM BigData Fundamentals Bootcamp.  I have four free resources that I would like to share with anyone who wants to try to understand BigData.  Two Redbooks:

• Just published: “IBM InfoSphere Streams: Accelerating Deployments with Analytic Accelerators“
• “Implementing IBM InfoSphere BigInsights on IBM System x“

In addition, there are two free VMWare images of fully-functional software available for download.  Please note: these images are to be used for education and demonstration purposes only; they cannot be used to support production workloads.

• IBM InfoSphere BigInsights Quick Start Edition
• IBM InfoSphere Streams Quick Start Edition

More to follow on my return from the Bootcamp.

Kim and I conducted a webinar on Thursday, October 3, 2013 that focused on IBM Data Interoperability software products and their specific strengths, weaknesses, and use cases.  We highlighted the application of the InfoSphere family of products to other-vendor databases such as Oracle, SQL Server, Sybase, and Teradata.  Kudos to Kim for putting together a presentation of disparate, but complementary technologies.

The recording of the webinar can be found here.  The presentation materials can be downloaded: Data Interoperability Presentation – FINAL 10.3.13