Greetings after a little while.

For those of you who have been working with IBM’s data virtualization technology – as I have – since the beginning (we miss you DataJoiner, Federation Server, et al), there’s yet another new name: Watson Query a service of Cloud Pak for Data (CP4D).

DataJoiner began as a research project (code name: “Garlic”) led by then-IBMer Nelson Mattos. The goal was to provide heterogeneous data access via a Db2 for LUW “gateway”. Enterprises in the 1980s began deploying lots of discrete data repositories (primarily Oracle, Sybase, and MS SQL Server along with Db2) to serve line-of-business and departmental applications (e.g. sales, accounting, finance, manufacturing, logistics). The ability to access data from multiple disparate servers to provide a comprehensive view of business processes was a significant advance. DataJoiner was not limited to RDBMS repositories, but could also retrieve data from MS Access, MS Excel, CSV and Text files… pretty much any data on persistent storage with a definable structure.

For a while DataJoiner also provided increased performance and reduced expense benefits as well. IBM sold DataJoiner as a front-end to Oracle because (at least in some cases) a query using the cost-based Db2 optimizer embedded in DataJoiner produced a better access path than the, at that time, rules-based Oracle optimizer. Expenses were reduced due to the per-seat Oracle license charges; DataJoiner was just “one seat” which could support hundreds of concurrent users. Over time Oracle has modified both their optimizer and licensing model.

For those of you who geek-out on the roots of the technology that we apply to common data administration challenges in our day jobs – as I do – you’ll find a seminal paper on the IEEE website: DataJoiner: a practical approach to multi-database access

So why the history lesson?

• At its roots Watson Query continues to use much of the original federation “plumbing”: Wrappers, Servers, Nicknames, etc.
• Watson Query no longer relies on point-to-point connections to the back-end data repositories, but exploits a “computational mesh” which provides increased performance and resiliency. This approach leverages third-party data repository vendor advances in parallelism, caching, and compression. All of this is of enormous importance when enterprise data is distributed not just across multiple in-house geographically disparate locations, but perhaps across multiple cloud vendors as well.

• BUT (there’s usually a but), due to engineering and design within CP4D some of the capabilities in IBM’s legacy federation stack have not yet been incorporated into Watson Query. Two are of particular interest – which is why I have posted two Ideas (the successor to IBM Request for Enhancement – RFE):
  1. Formerly for data repositories with Indexes that metadata would automatically be captured for a Remote Table in, say, Oracle and propagated back to the Db2 SYSCAT.INDEXES catalog view for use by the Db2 optimizer to help build the best access path for the Remote Table represented by the Nickname. Right now that’s not being done. Hence Idea ASQL-I-16 “Propagation of Indexes for Virtualized Tables to SYSCAT.INDEXES”.
  2. Federated queries have always used Db2’s SQL dialect. There is a PASSTHRU capability that enables users to employ the native SQL dialect of the back-end data repository. For certain queries this can produce significant performance benefits. See Idea ASQL-I-17 “Federated PASSTHRU Capability in Watson Query (WQ)”.

Please review both of these Ideas and vote!

https://ibm-data-and-ai.ideas.ibm.com/ideas/ASQL-I-16

https://ibm-data-and-ai.ideas.ibm.com/ideas/ASQL-I-17

Thanks in advance.

#IBM #WatsonQuery #DataVirtualization #Federation

Our webinar delivered on March 11, 2016 outlining access to mainframe data specifically oriented to non-mainframers such as IT Architects and Data Scientists was a success.  The presentation materials are here: IBM z Systems – Mainframes – for IT Architects and Data Scientists

A recording of the webinar is found here.

Kim has just returned from IBM’s annual “z Bootcamp” where they prepare their System z sellers for the upcoming year with a series of product updates, announcements and education.  Her takeaway message for 2016 was loud and clear:  IBM intends to continue to enhance and extend the capabilities of System z so they can remain relevant and leading the effort to leverage the power of System z data.

Which leads, naturally, to the “how?” question.  At The Fillmore Group, as an IBM Business Partner we believe our ability to connect with Data Scientists and Data Architects and to help them understand the data stored in System z will help us all better align our interests and build a successful data and analytic ecosystem.  To kick start this effort we have planned a lunch event at the award-winning Woodberry Kitchen in Baltimore where Frank Fillmore will present an overview of System z.  Read more below – and please join us if you can!

“IBM z Systems (Mainframes!) for IT Architects and Data Scientists”

Abstract
Fifty percent or more of a large enterprise’s data – especially the lifeblood online transaction processing (OLTP) data – likely reside on IBM z Systems (aka mainframe) computers.  In many organizations, mainframes remain a black box inside the “glass house” data center.  Yet the rich transactional data they contain are crucial to predictive analytics, such as fraud detection, and real-time data warehousing applications such as point-of-sale market basket analysis.

This lunch session is intended for Architects and Data Scientists who don’t have a mainframe background, but have been tasked by lines-of-business to develop new competitive insights into customers, supply chains, insurance claims, financial transactions – and/or any of a variety of other opportunities to turn an enterprise’s OLTP data into Big Data.

DB2 Gold Consultant and IBM Champion Frank Fillmore will explain:

The types of data are found on the mainframe, including:
• DB2 for z/OS (relational database)
• IMS – Information Management System (hierarchical database)
• VSAM – Virtual Storage Access Method (indexed file)

Frank will also explain the four key approaches to unlocking the data:
1. HTAP – Hybrid Transaction and Analytics Platform
2. Replication to heterogeneous data stores
3. Virtualization of heterogeneous data stores
4. ETL – Extract Transform Load

Frank will also discuss the how these approaches apply to the types of data in the context of the application requirements such as service level agreements (SLAs).

The session will be held on Thursday, March 3rd, from 11am-2pm at the Woodberry Kitchen, 2010 Clipper Mill Road, Suite 126, Baltimore, Maryland, 21211.

Registration is required.  Register HERE.

If you are unable to join us in Baltimore, Frank will deliver similar content in a webinar on Friday, March 11, 2016.  Registration for the session, to be held between 11am and noon, EST, is HERE.

Please note:  registration is required for both sessions as space is limited.  If you are a mainframe systems programmer or database administrator this is *not* the session for you (you should already know this stuff!)

Kim and I conducted a webinar on Thursday, October 3, 2013 that focused on IBM Data Interoperability software products and their specific strengths, weaknesses, and use cases.  We highlighted the application of the InfoSphere family of products to other-vendor databases such as Oracle, SQL Server, Sybase, and Teradata.  Kudos to Kim for putting together a presentation of disparate, but complementary technologies.

The recording of the webinar can be found here.  The presentation materials can be downloaded: Data Interoperability Presentation – FINAL 10.3.13

Affordable IBM Mainframe & DB2 Data Integration

Live Webinar Thursday, February 24th from 12:30pm – 1:30pm EST

Register today to learn how you can accomplish efficient and affordable IBM mainframe or DB2 data integration in a heterogeneous environment for BI, reporting, MDM, DW, and more.  The session will conclude with a live product demo and Q&A session.

Read More…

The Fillmore Group Selects Attunity to Provide Real-time Data Integration Solutions to IBM Mainframe and DB2 Customers

BURLINGTON, MA & BALTIMORE, MD – February 17, 2011 – Attunity  Ltd. (OTC BB: ATTUF.OB), a leading provider of real-time data integration and event capture software, and The Fillmore Group, a provider of IBM systems consulting, training and integration services, announced today a partnership to deliver IBM Mainframe and DB2 customers with affordable data integration solutions that leverage Attunity’s real-time data integration software and The Fillmore Group’s extensive IBM systems knowledge and expertise. Read More…