IBM’s flagship enterprise relational database Db2 has just been added to AWS Relational Database Service (RDS). Db2 joins commercial vendor relational database implementations Oracle and MS SQL Server along with open source databases MySQL, MariaDB, and PostreSQL as a fully-managed AWS offering.

Why this matters:

• Deployment I’m currently working with a large commercial vehicle manufacturer. Over two weeks ago I requested that a small Db2 database be deployed for query optimization. The clock is still ticking and the database still isn’t available. With RDS Db2 can be deployed and configured in minutes.
• Horizontal scaling Many of The Fillmore Group’s clients across several different industries have significant variability of demand – days of the week, times of the month, and seasonal variations. For large retailers purchasing ramps up in the summer, staffing in the early fall, sales in late fall and early winter, and returns in January. Managing on-prem capacity in such an environment usually means expensive compute and memory is underutilized for several months in the spring and summer. RDS provides on-demand scale up and scale down.
• Vertical scaling A large industrial client provisioned Db2 to store sensor data that will be analyzed for anomalies. Within six months one such database was at 85 Tb and continues to grow. Capacity growth is both predictable and linear, but even metronomically adding compute, memory, and storage is both budget and labor intensive.

Two more things to consider:

1. IBM invented the relational database. A Relational Model of Data for Large Shared Data Banks was published by Edgar F. Codd, IBM Research, in 1970 in the Communications of the Association of Computing Machinery (ACM). Trillions of dollars of value has been realized by organizations which have deployed IBM Db2 or other relational database variants since they became available in the 1980s.
2. Many enterprises have successfully migrated workloads from other-vendor relational database implementations. Here’s just one example: SmarterQuestions White Paper – Oracle to DB2 Migration Lessons Learned – Final

IBM hosted the inaugural TechXchange Conference in Las Vegas last week (September 11 – 15, 2023). Here’s what I liked:

• The Developer community was the primary focus of TechXchange. This is a significant change from the pre-pandemic IBM technical conferences – Think and World of Watson (sited variously in Las Vegas and San Francisco). Two of the TechXchange keynote presenters pointed out that there were no “suits” at the conference (in reference to both people and apparel).
• Artificial Intelligence (AI) was (not surprisingly) given a shout-out in almost every keynote, but I thought the most prescient comment was that in three years Quantum Computing will be where AI is today.

Lakehouse

• Since I’m a career database administrator (DBA) I am really jazzed about IBM’s Lakehouse approach to data. For decades IBM was primarily focused on IMS (hierarchical) for large-scale online transaction processing (OLTP) and Db2 (relational) for analytical workloads (and, eventually, OLTP too).
• IBM made a brief foray into Data Lake (vast amounts of unstructured or semi-structured data with varying standards of curation) with its BigInsights Hadoop distribution.
• But Lakehouse is the combination of traditional shared-nothing MPP IBM Db2 Warehouse (relatively structured with higher levels of curation, consistency, and veracity) and watsonx.data an open source stack of data repositories: Lake + House.

Key characteristics of Lakehouse:

1. Let the data decide where it should reside. Based on data attributes such as volume, structure, quality, timeliness, etc. choose the data repository that best addresses those attributes. You no longer have to force data into a repository where it’s not best-suited.
2. Seamlessly access data regardless of how and where it’s stored.
   1. Data Scientists can easily combine customer account data, transaction history, and sentiment analysis.
   2. An Auto Repair Technician can pull up vehicle history by Make/Model/Trim/Year, geography, mileage, and so forth on a tablet in a garage bay while the car is on the lift.
3. Ever increasing data storage costs have your CFO in a frenzy? Performant Object Storage will likely meet your response-time targets for most applications – at about 3% of the cost of Block Storage. Automated multi-temperature storage enables you to transition data to the storage platform most suited to its age, likelihood of being retrieved, and other factors.
4. On-prem, in the cloud, or SaaS? You decide where – and by whom – the infrastructure underpinning your OLTP and analytics workloads are provisioned.
5. Containerization enables scaling that infrastructure as required by both cyclical and black-swan events.

A much more detailed dive into Db2 Warehouse Gen 3 and watsonx.data will be in an upcoming blog post.

Greetings after a little while.

For those of you who have been working with IBM’s data virtualization technology – as I have – since the beginning (we miss you DataJoiner, Federation Server, et al), there’s yet another new name: Watson Query a service of Cloud Pak for Data (CP4D).

DataJoiner began as a research project (code name: “Garlic”) led by then-IBMer Nelson Mattos. The goal was to provide heterogeneous data access via a Db2 for LUW “gateway”. Enterprises in the 1980s began deploying lots of discrete data repositories (primarily Oracle, Sybase, and MS SQL Server along with Db2) to serve line-of-business and departmental applications (e.g. sales, accounting, finance, manufacturing, logistics). The ability to access data from multiple disparate servers to provide a comprehensive view of business processes was a significant advance. DataJoiner was not limited to RDBMS repositories, but could also retrieve data from MS Access, MS Excel, CSV and Text files… pretty much any data on persistent storage with a definable structure.

For a while DataJoiner also provided increased performance and reduced expense benefits as well. IBM sold DataJoiner as a front-end to Oracle because (at least in some cases) a query using the cost-based Db2 optimizer embedded in DataJoiner produced a better access path than the, at that time, rules-based Oracle optimizer. Expenses were reduced due to the per-seat Oracle license charges; DataJoiner was just “one seat” which could support hundreds of concurrent users. Over time Oracle has modified both their optimizer and licensing model.

For those of you who geek-out on the roots of the technology that we apply to common data administration challenges in our day jobs – as I do – you’ll find a seminal paper on the IEEE website: DataJoiner: a practical approach to multi-database access

So why the history lesson?

• At its roots Watson Query continues to use much of the original federation “plumbing”: Wrappers, Servers, Nicknames, etc.
• Watson Query no longer relies on point-to-point connections to the back-end data repositories, but exploits a “computational mesh” which provides increased performance and resiliency. This approach leverages third-party data repository vendor advances in parallelism, caching, and compression. All of this is of enormous importance when enterprise data is distributed not just across multiple in-house geographically disparate locations, but perhaps across multiple cloud vendors as well.

• BUT (there’s usually a but), due to engineering and design within CP4D some of the capabilities in IBM’s legacy federation stack have not yet been incorporated into Watson Query. Two are of particular interest – which is why I have posted two Ideas (the successor to IBM Request for Enhancement – RFE):
  1. Formerly for data repositories with Indexes that metadata would automatically be captured for a Remote Table in, say, Oracle and propagated back to the Db2 SYSCAT.INDEXES catalog view for use by the Db2 optimizer to help build the best access path for the Remote Table represented by the Nickname. Right now that’s not being done. Hence Idea ASQL-I-16 “Propagation of Indexes for Virtualized Tables to SYSCAT.INDEXES”.
  2. Federated queries have always used Db2’s SQL dialect. There is a PASSTHRU capability that enables users to employ the native SQL dialect of the back-end data repository. For certain queries this can produce significant performance benefits. See Idea ASQL-I-17 “Federated PASSTHRU Capability in Watson Query (WQ)”.

Please review both of these Ideas and vote!

https://ibm-data-and-ai.ideas.ibm.com/ideas/ASQL-I-16

https://ibm-data-and-ai.ideas.ibm.com/ideas/ASQL-I-17

Thanks in advance.

#IBM #WatsonQuery #DataVirtualization #Federation

Having participated in many board meeting debates over the years about the best place to hold meetings of the Baltimore/Washington Db2 Users Group, it’s a bit sad to learn that the 2021 IDUG North America Conference is already being planned as a virtual event, with the possibility of an in-person component.  Let’s hope we can all get together sometime in 2021!

In the meantime, it’s certainly worth noting that with so many people working remotely, many conferences and users groups have actually been quite successful adopting a virtual format.  Some changes resulting from the transition from in-person to virtual events are welcome improvements.  One of these is the more flexible session scheduling IDUG is offering.  I think it’s great, as one-size-fits-all is difficult to accommodate with technical topics.

To the many people I’ve spoken to who have said they have a topic but aren’t sure they have enough (or maybe have too much) material, IDUG 2021 may be a great option as, new this year, IDUG is offering three different presentation timeslots—30, 60, and 120 minutes. You can choose what length works best for you. You can also opt out of the live session and instead pre-record your presentation. To help inspire you, here are some of the most popular session categories:

New Db2 releases: migrating and effective usage
Analytics & Business Intelligence
New technologies: mobile applications, cloud, XaaS, etc.
Application development and data modelling
Db2 and packaged applications (ERP, etc.)
Performance, availability, & security
User experiences and best practices: what did you achieve with Db2?
Db2 and non-standard data types (JSON, etc.)

This is your chance to share your technical expertise and connect with the global community of Db2 users. If you’ve never presented at an IDUG event, this is a great opportunity. First-time speakers are always welcome, as is material geared toward beginner, intermediate, or advanced users.

To submit an abstract, visit IDUG at:  https://www.idug.org/p/cm/ld/fid=2495

I hope to “see” you at IDUG 2021!

I want to thank everyone who helped make the inaugural Mid-Atlantic Db2 for z/OS Users Group virtual meeting such a great success:

• my colleague, Kim May, for pulling the meeting together and for filling a void of timely technical content for this topic and this region
• our presenters for crafting and delivering a comprehensive new way of considering IBM System z
  • Andrew Austin, Red Hat
  • Clark Hale, Red Hat
  • Elton (the Wizard) de Sousa, IBM

The recording of the meeting is here: https://youtu.be/Am279HW1DDY

Individual presentations are as follows:

Db2 VUG – OpenShift Introduction – Austin + Hale

Db2 VUG – OpenShift on IBM z – de Sousa

Db2 VUG – Mainframe Modernization Field Notes – Fillmore

Please share this content with your colleagues.

Announcing the new Mid-Atlantic Db2 for z/OS Users Group!

With COVID-19 forcing Regional Users Groups (RUGs) to transition from in-person to virtual meetings, and with the support of the International Db2 Users Group and hearing about their regional affiliates’ success with virtual meetings, Frank and I decided to give a new virtual group a try.  We look forward to supporting the Db2 community with technical content that enables our System Z colleagues to remain active and valued contributors to the use of technology within their organizations.

Please join us for our first meeting! Register here.

September 17, 2020, from 10:00am-2:00pm EDT, will be the first virtual meeting of the System Z-focused Mid-Atlantic Db2 Users Group. The new group hopes to bring interesting, educational, and current technical presenters and sessions to mainframe users located in the Mid-Atlantic region. With a virtual format the meeting location, venue, commute time and lunch selection challenges disappear, so everyone from everywhere is welcome!

For this first meeting we will have three technical presentations and one brief interactive, bring-your-own-ideas session. The three technical sessions all focus on IBM’s efforts to use tools and technologies to support mainframe modernization. No, this is not another session on a magic code converter; these are proven tools and technologies available today that speed deployments and leverage the mainframe’s powerful resources to drive better and faster data access and new application development.

Presentations will be delivered by IBM’s Elton de Souza, Clark Hale and Andrew Austin of Red Hat, and Frank Fillmore of The Fillmore Group. The agenda, abstracts and speaker bio’s are below.

Agenda: Thursday, September 17, 2020

10:00 – 10:10 Welcome and Introductions
10:10 – 11:15 Containers, Cloud Paks, OpenShift – What Mainframers Should Know
11:15 – 11:20 Break
11:20 – 11:35 The New Replication Monitoring Tool Interactive Session – BYOI (Bring Your Own Ideas)
11:35 – 12:40 Red Hat OpenShift on System Z
12:40 – 12:45 Break
12:45 – 01:50 Mainframe Modernization – Field Notes
01:50 – 02:00 Q&A and Wrap Up

Sessions
Containers, Cloud Paks, OpenShift – What Mainframers Should Know, Elton de Souza, IBM
In this presentation IBM Chief Architect and Wizard Elton de Souza will cover the value proposition of OpenShift on IBM Z from a technical and business standpoint. Elton will also cover integration points between OpenShift and zOS services and data sources.

Red Hat OpenShift, Clark Hale & Andrew Austin, Red Hat
Red Hat OpenShift is the industry’s most secure and comprehensive enterprise-grade container platform. This presentation will explain what OpenShift is, what value it can provide to your organization, and a brief demo of its operation. Special focus will be given to the System Z release of OpenShift.

Mainframe Modernization – Field Notes, Frank Fillmore, The Fillmore Group
Frank recently served as the System Z specialist for a mainframe modernization project led by a team from Red Hat. The goal was to develop a modernization template to be shared with other mainframe customers to accelerate the effort by incorporating the best tools and technologies available today. The customer struggles with a familiar list of challenges: legacy applications developed over decades, data in VSAM and Db2, and incomplete data, the result of a recent merger. Several approach and solution options were explored. Join Frank while he discusses how IBM’s acquisition of Red Hat is transforming how mainframe modernization is being approached – and realized.

Speaker Bio’s

Elton de Souza, IBM
Elton de Souza is the Chief Architect for Cloud Native Client Success focusing on IBM Z and LinuxONE. Since OpenShift GA’d on IBM Z in Feb 2020, Elton has helped several clients across various sectors and geographies implement RHOCP on IBM Z and LinuxONE. He’s spent almost his entire career on IBM Z starting from the Java compiler for Linux and zOS to working on Open Source Solutions (MongoDB, node.js, Spark, Docker, Kubernetes, etc.) to IBM Cloud Private and now Red Hat OpenShift on IBM Z.

Clark Hale, Red Hat
Clark Hale is a Senior Consulting Architect at Red Hat. With a focus on OpenShift and “cloud native” technologies, he specializes in assisting companies adopt modern technology and development practices. Before joining Red Hat, he was a multi-language developer (COBOL, C, Java) at Lowes Companies and an OpenSource Advocate at SUN Microsystems.

Andrew Austin, Red Hat
Andrew Austin is a Senior Architect with Red Hat Consulting’s Cloud Infrastructure Practice. His focus is on delivering solutions based on Red Hat OpenShift on the IBM Z platform as well as distributed systems.

Frank Fillmore, The Fillmore Group
Frank is founder and president of The Fillmore Group, an IBM Business Partner specializing in IBM database solution sales and services. Frank is an IBM System Z Gold Consultant and IBM Champion, as well as an IBM authorized instructor and frequent presenter at users group meetings and conferences. Frank leads a team of expert consultants delivering IBM solution implementation and enablement support services.

IBM Data Replication 2020: In a Nutshell

IBM replication is based on two unique technical solutions: InfoSphere Change Data Capture, the heritage DataMirror product, and Q-Replication, the IBM-developed replication solution that uses MQ messaging.

IBM replication supports numerous sources and targets. For heterogeneous environments, that is, where either the source or target are not Db2, IBM recommends InfoSphere Change Data Capture. For Db2 to Db2 replication, IBM recommends Q Replication.

The currently available sources are targets are below. New offerings are always being added; for a current list visit the IBM replication homepage.

New offering highlight: Mainframe replication for Db2 for z and VSAM featuring “Remote Capture”, with the replication engine installed off the mainframe. The new solution delivers easier installation and administration as well as lower MIPs consumption to support replication from System z.

Coming soon: Refreshed monitoring tools built on Watson Knowledge Catalog, enabling expanded governance and integration with replication monitoring.

Implementation support, healthchecks and education:  Contact The Fillmore Group. We have several Quickstart offerings and have been delivering the IBM replication education offerings for decades.

Replication Sources: Db2 (LUW, Z/OS, iSeries), Oracle, SQL Server, PostgreSQL, VSAM, IMS, Informix, Sybase, MySQL.

Replication Targets: All above RDBMSs plus Teradata, Kafka, Hadoop, Files, Information Server (DataStage/QualityStage), Cloudant, AWS RedShift, S3, Azure SQL DB, JDBC targets (MongoDb, etc.), IBM IAS/ Db2WH/Db2WHoC, Netezza/Cloud Pak 4 Data System, and more.

Remote Capture Quickstart

Postgres Capture Quickstart

Kafka Replication Quickstart

The IBM Request for Enhancement (RFE) system has been moved from developerWorks to the IBM Data & AI Ideas portal.

Each “idea” submitted can be voted on by users who log in. The number of votes for each particular request is taken into consideration by the IBM Product Development teams when they are planning product enhancements. More votes = higher probability of getting the request fulfilled.

I am reaching out to ask IBM Data & AI solutions users to vote for an existing RFE in the Ideas system, QREP-I-216. This is a replication enhancement to IBM’s Q Replication, for customers using the Oracle QApply.

The Oracle QApply was developed several years ago when IBM was planning to merge Q Replication and InfoSphere Change Data Capture. It enabled customers to replicate from Db2 for z/OS to Oracle without adding a layer of federation (the old method of applying changes to heterogeneous databases).

The user requesting the request has had Q Replication with QApply in production for several years and is asking to have replication monitoring improved.

If you haven’t logged on the Ideas portal, it’s easy, and the homepage gives you access to the enhancement requests as well as each solution roadmap.

For this particular RFE, log in here and click on the button on the left side of the screen to vote.

Thanks for voting!!

https://ibm-data-and-ai.ideas.aha.io/ideas/QREP-I-216

The proliferation of webinars and events transitioning to virtual meetings is getting hard to keep up with, and maybe some of the event fatigue is a result of an explosion of invitations.  Some events are, however, not to be missed, and one of these is the annual North America International Db2 Users Group (IDUG) conference, scheduled for next week.

Due to the change from an in person to virtual format, IDUG is offering a generous premium membership to everyone who registers for the virtual conference (which is only $199!!) and will allow attendees to view all session recordings and content.

Imagine attending a conference where you can attend ALL sessions, and not spend hours trying to figure out how minimize schedule conflicts and get from one end of the conference center to the other (and get something to drink and hit the restroom) in 10 minutes.  Pretty sweet!

If you haven’t registered yet, consider it.  The session information and keynotes are on the IDUG homepage (www.idug.org), along with the registration link.  Also, if you’ve read this far – I understand IBM still has some remaining vouchers available so customers can attend (as a guest of IBM)…call your IBM rep if you’re interested!

From the webpage:

The event will include 60+ dynamic sessions, live Q&A with industry experts and leaders, opportunities for engagement with your favorite vendors and each other, and most importantly, cutting-edge technical education streaming straight to your home or office. Your registration also includes a complimentary premium membership so you can access exclusive IDUG content all year long.

Join the Mid-Atlantic Db2 LUW user group members – and more! – for a virtual meeting coordinated by IBM Hybrid Data Manager Technical Sales Specialist and user group supporter extraordinaire Kay Groski this coming Wednesday, June 10th.  “And more!” means, of course, that anyone from any users group (or not) is welcome to join.  Please do!

Kay has scheduled Db2 Offering Manager Karthik Gopalakrishnan to introduce Db2 v11.5.4 aka “Nebula”. Karthik’s team is also working to deliver a series of webinars detailing the new features included in Nebula. The schedule for these sessions and details are below.

For next week’s Virtual RUG:

Topic: Db2 Virtual RUG Meeting June 2020
Date: Wednesday, June 10, 2020
Time: 2:00pm – 3:15pm, Eastern Daylight Time (New York, GMT-04:00)
Meeting address: ibm.biz/June2020RUG
Program registration password: Db2RUG

The follow-on sessions will provide additional detail on specific new features.  Sessions currently scheduled are below.  For speakers, agenda and registration details go to:  http://ibm.biz/db2webinar[ibm.biz]

Overview of Db2 Editions
Overview of Db2 Editions and Entitlement migration paths from older editions to new.
June 9, 2020

Strategy and Roadmaps
Overview of Db2’s strategic direction and future Roadmap.
June 11, 2020

Container Strategy
Overview of Db2’s Containerization Strategy for Docker and Red Hat Open Shift.
June 16, 2020

Developer Features
Overview of Db2 REST Service, Python UDF and Drivers.
June 18, 2020

Unpacking 11.5.4
Deep Dive into Db2 Nebula release. We will go through exciting new features.
June 23, 2020

Graph Database
Deep Dive into Db2 Graph Capabilities.
June 25, 2020

Machine Learning
Deep Dive into Db2’s Machine Learning capabilities.
June 30, 2020

Enterprise Features
Deep Dive into Db2’s new Enterprise readiness capabilities.
June 30, 2020