IBM’s flagship enterprise relational database Db2 has just been added to AWS Relational Database Service (RDS). Db2 joins commercial vendor relational database implementations Oracle and MS SQL Server along with open source databases MySQL, MariaDB, and PostreSQL as a fully-managed AWS offering.

Why this matters:

• Deployment I’m currently working with a large commercial vehicle manufacturer. Over two weeks ago I requested that a small Db2 database be deployed for query optimization. The clock is still ticking and the database still isn’t available. With RDS Db2 can be deployed and configured in minutes.
• Horizontal scaling Many of The Fillmore Group’s clients across several different industries have significant variability of demand – days of the week, times of the month, and seasonal variations. For large retailers purchasing ramps up in the summer, staffing in the early fall, sales in late fall and early winter, and returns in January. Managing on-prem capacity in such an environment usually means expensive compute and memory is underutilized for several months in the spring and summer. RDS provides on-demand scale up and scale down.
• Vertical scaling A large industrial client provisioned Db2 to store sensor data that will be analyzed for anomalies. Within six months one such database was at 85 Tb and continues to grow. Capacity growth is both predictable and linear, but even metronomically adding compute, memory, and storage is both budget and labor intensive.

Two more things to consider:

1. IBM invented the relational database. A Relational Model of Data for Large Shared Data Banks was published by Edgar F. Codd, IBM Research, in 1970 in the Communications of the Association of Computing Machinery (ACM). Trillions of dollars of value has been realized by organizations which have deployed IBM Db2 or other relational database variants since they became available in the 1980s.
2. Many enterprises have successfully migrated workloads from other-vendor relational database implementations. Here’s just one example: SmarterQuestions White Paper – Oracle to DB2 Migration Lessons Learned – Final

IBM hosted the inaugural TechXchange Conference in Las Vegas last week (September 11 – 15, 2023). Here’s what I liked:

• The Developer community was the primary focus of TechXchange. This is a significant change from the pre-pandemic IBM technical conferences – Think and World of Watson (sited variously in Las Vegas and San Francisco). Two of the TechXchange keynote presenters pointed out that there were no “suits” at the conference (in reference to both people and apparel).
• Artificial Intelligence (AI) was (not surprisingly) given a shout-out in almost every keynote, but I thought the most prescient comment was that in three years Quantum Computing will be where AI is today.

Lakehouse

• Since I’m a career database administrator (DBA) I am really jazzed about IBM’s Lakehouse approach to data. For decades IBM was primarily focused on IMS (hierarchical) for large-scale online transaction processing (OLTP) and Db2 (relational) for analytical workloads (and, eventually, OLTP too).
• IBM made a brief foray into Data Lake (vast amounts of unstructured or semi-structured data with varying standards of curation) with its BigInsights Hadoop distribution.
• But Lakehouse is the combination of traditional shared-nothing MPP IBM Db2 Warehouse (relatively structured with higher levels of curation, consistency, and veracity) and watsonx.data an open source stack of data repositories: Lake + House.

Key characteristics of Lakehouse:

1. Let the data decide where it should reside. Based on data attributes such as volume, structure, quality, timeliness, etc. choose the data repository that best addresses those attributes. You no longer have to force data into a repository where it’s not best-suited.
2. Seamlessly access data regardless of how and where it’s stored.
   1. Data Scientists can easily combine customer account data, transaction history, and sentiment analysis.
   2. An Auto Repair Technician can pull up vehicle history by Make/Model/Trim/Year, geography, mileage, and so forth on a tablet in a garage bay while the car is on the lift.
3. Ever increasing data storage costs have your CFO in a frenzy? Performant Object Storage will likely meet your response-time targets for most applications – at about 3% of the cost of Block Storage. Automated multi-temperature storage enables you to transition data to the storage platform most suited to its age, likelihood of being retrieved, and other factors.
4. On-prem, in the cloud, or SaaS? You decide where – and by whom – the infrastructure underpinning your OLTP and analytics workloads are provisioned.
5. Containerization enables scaling that infrastructure as required by both cyclical and black-swan events.

A much more detailed dive into Db2 Warehouse Gen 3 and watsonx.data will be in an upcoming blog post.

Greetings after a little while.

For those of you who have been working with IBM’s data virtualization technology – as I have – since the beginning (we miss you DataJoiner, Federation Server, et al), there’s yet another new name: Watson Query a service of Cloud Pak for Data (CP4D).

DataJoiner began as a research project (code name: “Garlic”) led by then-IBMer Nelson Mattos. The goal was to provide heterogeneous data access via a Db2 for LUW “gateway”. Enterprises in the 1980s began deploying lots of discrete data repositories (primarily Oracle, Sybase, and MS SQL Server along with Db2) to serve line-of-business and departmental applications (e.g. sales, accounting, finance, manufacturing, logistics). The ability to access data from multiple disparate servers to provide a comprehensive view of business processes was a significant advance. DataJoiner was not limited to RDBMS repositories, but could also retrieve data from MS Access, MS Excel, CSV and Text files… pretty much any data on persistent storage with a definable structure.

For a while DataJoiner also provided increased performance and reduced expense benefits as well. IBM sold DataJoiner as a front-end to Oracle because (at least in some cases) a query using the cost-based Db2 optimizer embedded in DataJoiner produced a better access path than the, at that time, rules-based Oracle optimizer. Expenses were reduced due to the per-seat Oracle license charges; DataJoiner was just “one seat” which could support hundreds of concurrent users. Over time Oracle has modified both their optimizer and licensing model.

For those of you who geek-out on the roots of the technology that we apply to common data administration challenges in our day jobs – as I do – you’ll find a seminal paper on the IEEE website: DataJoiner: a practical approach to multi-database access

So why the history lesson?

• At its roots Watson Query continues to use much of the original federation “plumbing”: Wrappers, Servers, Nicknames, etc.
• Watson Query no longer relies on point-to-point connections to the back-end data repositories, but exploits a “computational mesh” which provides increased performance and resiliency. This approach leverages third-party data repository vendor advances in parallelism, caching, and compression. All of this is of enormous importance when enterprise data is distributed not just across multiple in-house geographically disparate locations, but perhaps across multiple cloud vendors as well.

• BUT (there’s usually a but), due to engineering and design within CP4D some of the capabilities in IBM’s legacy federation stack have not yet been incorporated into Watson Query. Two are of particular interest – which is why I have posted two Ideas (the successor to IBM Request for Enhancement – RFE):
  1. Formerly for data repositories with Indexes that metadata would automatically be captured for a Remote Table in, say, Oracle and propagated back to the Db2 SYSCAT.INDEXES catalog view for use by the Db2 optimizer to help build the best access path for the Remote Table represented by the Nickname. Right now that’s not being done. Hence Idea ASQL-I-16 “Propagation of Indexes for Virtualized Tables to SYSCAT.INDEXES”.
  2. Federated queries have always used Db2’s SQL dialect. There is a PASSTHRU capability that enables users to employ the native SQL dialect of the back-end data repository. For certain queries this can produce significant performance benefits. See Idea ASQL-I-17 “Federated PASSTHRU Capability in Watson Query (WQ)”.

Please review both of these Ideas and vote!

https://ibm-data-and-ai.ideas.ibm.com/ideas/ASQL-I-16

https://ibm-data-and-ai.ideas.ibm.com/ideas/ASQL-I-17

Thanks in advance.

#IBM #WatsonQuery #DataVirtualization #Federation

Having participated in many board meeting debates over the years about the best place to hold meetings of the Baltimore/Washington Db2 Users Group, it’s a bit sad to learn that the 2021 IDUG North America Conference is already being planned as a virtual event, with the possibility of an in-person component.  Let’s hope we can all get together sometime in 2021!

In the meantime, it’s certainly worth noting that with so many people working remotely, many conferences and users groups have actually been quite successful adopting a virtual format.  Some changes resulting from the transition from in-person to virtual events are welcome improvements.  One of these is the more flexible session scheduling IDUG is offering.  I think it’s great, as one-size-fits-all is difficult to accommodate with technical topics.

To the many people I’ve spoken to who have said they have a topic but aren’t sure they have enough (or maybe have too much) material, IDUG 2021 may be a great option as, new this year, IDUG is offering three different presentation timeslots—30, 60, and 120 minutes. You can choose what length works best for you. You can also opt out of the live session and instead pre-record your presentation. To help inspire you, here are some of the most popular session categories:

New Db2 releases: migrating and effective usage
Analytics & Business Intelligence
New technologies: mobile applications, cloud, XaaS, etc.
Application development and data modelling
Db2 and packaged applications (ERP, etc.)
Performance, availability, & security
User experiences and best practices: what did you achieve with Db2?
Db2 and non-standard data types (JSON, etc.)

This is your chance to share your technical expertise and connect with the global community of Db2 users. If you’ve never presented at an IDUG event, this is a great opportunity. First-time speakers are always welcome, as is material geared toward beginner, intermediate, or advanced users.

To submit an abstract, visit IDUG at:  https://www.idug.org/p/cm/ld/fid=2495

I hope to “see” you at IDUG 2021!

IBM Data Replication 2020: In a Nutshell

IBM replication is based on two unique technical solutions: InfoSphere Change Data Capture, the heritage DataMirror product, and Q-Replication, the IBM-developed replication solution that uses MQ messaging.

IBM replication supports numerous sources and targets. For heterogeneous environments, that is, where either the source or target are not Db2, IBM recommends InfoSphere Change Data Capture. For Db2 to Db2 replication, IBM recommends Q Replication.

The currently available sources are targets are below. New offerings are always being added; for a current list visit the IBM replication homepage.

New offering highlight: Mainframe replication for Db2 for z and VSAM featuring “Remote Capture”, with the replication engine installed off the mainframe. The new solution delivers easier installation and administration as well as lower MIPs consumption to support replication from System z.

Coming soon: Refreshed monitoring tools built on Watson Knowledge Catalog, enabling expanded governance and integration with replication monitoring.

Implementation support, healthchecks and education:  Contact The Fillmore Group. We have several Quickstart offerings and have been delivering the IBM replication education offerings for decades.

Replication Sources: Db2 (LUW, Z/OS, iSeries), Oracle, SQL Server, PostgreSQL, VSAM, IMS, Informix, Sybase, MySQL.

Replication Targets: All above RDBMSs plus Teradata, Kafka, Hadoop, Files, Information Server (DataStage/QualityStage), Cloudant, AWS RedShift, S3, Azure SQL DB, JDBC targets (MongoDb, etc.), IBM IAS/ Db2WH/Db2WHoC, Netezza/Cloud Pak 4 Data System, and more.

Remote Capture Quickstart

Postgres Capture Quickstart

Kafka Replication Quickstart

Join the Mid-Atlantic Db2 LUW user group members – and more! – for a virtual meeting coordinated by IBM Hybrid Data Manager Technical Sales Specialist and user group supporter extraordinaire Kay Groski this coming Wednesday, June 10th.  “And more!” means, of course, that anyone from any users group (or not) is welcome to join.  Please do!

Kay has scheduled Db2 Offering Manager Karthik Gopalakrishnan to introduce Db2 v11.5.4 aka “Nebula”. Karthik’s team is also working to deliver a series of webinars detailing the new features included in Nebula. The schedule for these sessions and details are below.

For next week’s Virtual RUG:

Topic: Db2 Virtual RUG Meeting June 2020
Date: Wednesday, June 10, 2020
Time: 2:00pm – 3:15pm, Eastern Daylight Time (New York, GMT-04:00)
Meeting address: ibm.biz/June2020RUG
Program registration password: Db2RUG

The follow-on sessions will provide additional detail on specific new features.  Sessions currently scheduled are below.  For speakers, agenda and registration details go to:  http://ibm.biz/db2webinar[ibm.biz]

Overview of Db2 Editions
Overview of Db2 Editions and Entitlement migration paths from older editions to new.
June 9, 2020

Strategy and Roadmaps
Overview of Db2’s strategic direction and future Roadmap.
June 11, 2020

Container Strategy
Overview of Db2’s Containerization Strategy for Docker and Red Hat Open Shift.
June 16, 2020

Developer Features
Overview of Db2 REST Service, Python UDF and Drivers.
June 18, 2020

Unpacking 11.5.4
Deep Dive into Db2 Nebula release. We will go through exciting new features.
June 23, 2020

Graph Database
Deep Dive into Db2 Graph Capabilities.
June 25, 2020

Machine Learning
Deep Dive into Db2’s Machine Learning capabilities.
June 30, 2020

Enterprise Features
Deep Dive into Db2’s new Enterprise readiness capabilities.
June 30, 2020

I can’t say I’m too disappointed to be taking a break from conference travel and post conference exhaustion, and it’s refreshing to see the creative changes facilitated by the mass transition to virtual events. Given that the primary goal of these events, whether in person or virtual, is to share technical announcements and updates, it seems either format can be successful.  And of course, each has its pros and cons.

When this is all over, however, I look forward to reconnecting – in person – with friends and colleagues to share technical updates.  Along with dinner and drinks!!

With a virtual “Cheers!” to everyone, here is a list of current and upcoming Db2 events:

1. Going on NOW – Enterprise Data & AI on IBM Z Virtual Symposium, May 19, 20, 21
This complimentary virtual event will explain how to use IBM Z data for modern engagement, analytics and AI applications to become truly insights-driven. Please note: you must register to access copies of all replays.

Register here: http://ibm.biz/BdqvcX

2. July – IDUG Virtual Db2 Tech Conference, July 20, 21, and 23
This three-day event will be followed by three additional weeks of IDUG conference content, making this the perfect way to access the cutting-edge sessions of an IDUG event directly from your home or office.

Conference registration also comes with complimentary, year-long premium membership to IDUG, enabling full access to IDUG exclusive articles, blogs, presentations, and forums.

Cost: $199
Register here: https://www.idug.org/e/in/eid=109

3. August – SHARE Boston, dates TBD
In an effort to continue to provide education, networking and influence, SHARE Boston will now be a virtual event. SHARE is looking to deliver opportunities to learn and network with peers in this new format. Stay tuned to the SHARE website for updates.

More information: https://event.share.org/

4. Db2 LUW Virtual Users Group – July, date TBD
IBM’s Db2 LUW coordinator extraordinaire Kay Groski is working to schedule speakers and topics so the groups she started in 2018 (including the Mid-Atlantic group I attend) can extend their reach by going virtual – at least for a while. As details are confirmed I will share more here.

Over 180 folks attended the “45 Minutes to Understand IBM Replication – Sources & Targets, Lower Latency, Costs” webinar on April 30, 2020 hosted by my colleague @KimMayTFG and me.  Thank you for joining us – along with the excellent questions and feedback.

These are the presentation materials IBM Data Replication Update 2020-04-30

You’ll find the recording of the webinar here.

This Thursday, April 30th, join Frank Fillmore for a fast, information-packed webinar, loaded with IBM replication solution information for both non-IBM and IBM replication customers.  Come join us and learn!  Details and the registration link are in the post below.

Please join us for a webinar Thursday, April 30th, on IBM replication:

45 Minutes to Understand IBM Replication – Sources & Targets, Lower Latency, Costs  

The webinar is intended for technical teams currently not using IBM replication solutions, although the webinar content will provide an excellent overview of IBM’s current offerings – an excellent refresher and update for IBM replication customers.  Hey, it’s a webinar – anyone can attend!!

Abstract:  Technical organizations rely on replication to deliver accurate data when its needed. IBM replication supports an extensive list of sources and targets – including legacy mainframe data repositories – as well as best of breed deployment models and specialty replication solutions.

If your organization is considering a new replication deployment or looking to replace your existing replication to extend functionality – or reduce cost – please join IBM Gold Consultant Frank C. Fillmore, Jr., for a webinar overview of IBM’s replication offerings.

The webinar will cover:

1. The sources and targets supported by IBM replication.
2. Each solution available today, and their costs.
3. Latency, deployment and administrative considerations.

Date:          Thursday, 4/30/2020

Time:          11:00am – 12:00pm EDT

Register HERE 

Presenter:  Frank Fillmore

Frank C. Fillmore, Jr. is the Founder and President of The Fillmore Group, Inc., an IBM Business Partner specializing in replication solutions in the IBM Data and Analytics portfolio. Frank has been a Db2 Gold Consultant since 1998 and is currently an IBM Champion for both Analytics and System z. Frank frequently presents on cutting-edge topics at regional and international user group meetings and is an IBM authorized instructor for delivery of replication courses including Q Replication, ICDC and pureScale.  Frank leads a team of expert consultants delivering presales, implementation and enablement support to IBM replication customers.

Want to learn more about the Gold Consultants?  Visit the IBM Gold webpage HERE.