Moving mainframe data off a legacy server, where its outdated code is comprehensible and manageable only by a select few, and onto an open-source database where it can be understood by many, makes it infinitely more valuable. Cloud developers can fluidly integrate it into new apps and tools. Non-technical users can tap it for everyday business functions and analysis. Teams looking to bring on new developer talent have a broader pool to pull from. It changes the game completely.

The very things that make the mainframe so secure and resilient can also make this migration an extremely challenging task. The proprietary data formats and unique structures specifically optimized for efficient processing and storage within the mainframe environment present a significant challenge for modern IT teams, who often lack the necessary expertise in these systems. And the inherent incompatibility of mainframe structures with modern systems makes adapting this data for use in contemporary databases is a complex task requires considerable effort for effective transformation and integration.

The mainframe’s second-to-none security is also a roadblock when it comes to efficient data migration. Through tools like z/OS Connect, DVM, or CICS Web Services, an API can be created for cloud apps to access the data in place for real-time insights. But given the complexity of the legacy code, it can be a risky undertaking that puts business-critical programs in a vulnerable spot and must be weighed against the potential rewards. In addition, the measures built around the mainframe to protect its sensitive data and ensure high availability necessitate additional time-consuming steps to get that process set up and functional.

Python is the lingua franca that bridges the gaps between these disparate technological states, facilitates the smooth transfer of information across their digital borders, and transforms their peaceful coexistence into a dynamic powerhouse partnership—a unified state known as The Modern, Cloud-Connected Mainframe. (See also, “A new way to conquer the great data divide,” Summer 2023.)

The pivotal role of Python

Thanks to its simplicity, versatility, and the broad range of applications it supports, Python has emerged as the go-to programming language for many data professionals, and an indispensable asset in creating a seamless bi-directional connection between legacy mainframe systems and modern cloud platforms. It provides a robust platform for interfacing with mainframes, handling different data formats, and performing complex data transformations.

Panda also has a rich ecosystem of libraries and tools such as Pandas and NumPy, which are particularly useful for data transformation tasks and offer a wide range of functionalities from basic data manipulation to advanced statistical analysis. And its capabilities extend beyond just data extraction. It also allows for the implementation of functions to fetch data from mainframe tables or files, functions that can be used to format and prepare data for direct access by databases or other data storage systems.

In other words, Python takes the lengthy, complicated, tedious and specialized process of data preparation and migration described above and makes it exponentially faster and easier—a task many more people have the skill to manage, in far less time, and without any degradation of data security or stability.

Streamlined, simplified, right-sized mainframe transformation in action

So, what does this process look like in the real world? Step one is to understand the client’s true business need and match it with the right approach. Real-time accessibility is often considered a must these days. And as previously mentioned, it is possible to tap mainframe data for live updates via an API, but it is also complicated, lengthy and risky. It’s more often the case that a business outcome assumed to depend on real-time information can be easily achieved with less frequent updates, say several times a day or week.

This is where that “citizen relocation” scenario comes into play—and where Python really shines.

At the core of this approach is System Management Facility (SMF) data generated by the z/OS operating system. This data is a gold mine, telling a provider everything about how the system is running, from basic operations to more detailed activities. It helps SIs understand and organize clients' system setups, manage software updates, bill clients accurately, meet service promises, plan for future needs, automate tasks and provide consistent support.

Handling all this information requires a standard way of collecting and managing it. At the heart of this strategy is the Mainframe Metrics Warehouse (MFMW), a separate storage repository for this data that uses a PostgreSQL database. This data is moved directly from the mainframe system to the MFMW, using a Python script to simplify and accelerate the process. This approach saves a lot of time, reduces complexity, and creates a standard way of supporting clients that gives them the access they need on the schedule the outcome requires, without introducing unnecessary business or operational risk.

The benefits of this simpler, more consistent data transfer process are delivered straight to clients. Staff are freed up to spend more time on data analysis and automation improvements. Support methods can be updated and improved more easily, with better service and a fully optimized modern, cloud-connected mainframe system. And Python makes it all possible.

BONUS: Best practices for a successful mainframe-cloud connection

Like all complex tasks, migrating mainframe data to an open-source database where it can be accessed by cloud platforms calls for meticulous planning and execution. Adopting the following best practices will help ensuring a smooth transition while maintaining data integrity.

1. Perform an assessment

Before initiating the migration process, it is crucial to perform a comprehensive data assessment. This assessment involves understanding the nature and structure of the data stored in the mainframe, including:

• Structure and size: Understanding the complexity and volume of the data.
• Format: Identifying the format of data, including any proprietary formats unique to mainframes.
• Quality and integrity: Evaluating the quality of data and ensuring its integrity is maintained during migration.

2. Prepare the data

Once the assessment is complete, and before commencing platform transformation, preparing the data for the next leg of its journey is critical. This multi-step process includes:

• Cleaning: Identifying and correcting errors or inconsistencies in the data.
• Formatting: Converting data into a format compatible with the target system.
• Archiving: Deciding what historical data needs to be migrated and what can be archived.

3. Preempt challenges

Several challenges can arise during the migration process related to mainframes’ complex legacy data structures, large datasets and data format incompatibility. To address these challenges and ensure a successful transformation, consider the following strategies:

• Utilize Python scripts for data cleansing: Before migration, Python scripts or tools can be used for data cleansing tasks. Python’s powerful libraries can automate and streamline data cleaning, format conversion, and even data archiving processes.
• Break down tasks into smaller increments: Managing the process in smaller, more manageable phases can enhance focus on details and allow for thorough testing and validation at each stage.

4. Validate post-migration

Ensuring data accuracy and completeness post-migration is one of the most crucial steps. This involves:

• Data reconciliation: Comparing migrated data against original datasets to ensure no data loss or corruption.
• Performance checks: Ensuring that the data operates effectively in the new environment and meets performance benchmarks.
• Security verification: Confirming that data security standards are upheld in the new system.