Audit Serve, Inc.

The Worldwide Connection for Audit, Security, Control and SOX Professionals

IBM MVS System SDLC Control Approaches

By: Mitchell H. Levine, CISA
Audit Serve, Inc.

                                                             

Low Cost & Highly Skilled 
IT Audit and SOX Consulting Resources Available Immediately
Call Mitch Levine at (203) 972-3567 or 
email levinemh@auditserve.com for additional information

Background

The review of the MVS operating system has become a major component in a Data Center audit. The main focus of many MVS Operating System reviews consists of security over the APF (i.e., Authorized Program Facility) libraries. APF libraries are subject to great exposure because it allows a user, who has update access, to create a program which could obtain the privileges (i.e., supervisor state) required to bypass security validation by the external security system. MVS also provides the following system components, which can be used to circumvent security if not controlled properly:

• Program Properties Table (PPT)
  
• I/O Appendages
  
• SVCs (Supervisor Calls)
  
• TSO APF Authorized programs and commands
  
• Exits

These system components are reviewed during the audit to ensure that all inhouse written programs were authorized and documented. However, an after the fact review designed to ensure that inhouse written system software changes were properly approved does not provide assurances that future inhouse written programs will be properly authorized. This issue is the main focus of the article. The prime objective is to discuss how all system software libraries should be restricted, the change management process for migrating system software changes and the development methodology that should be in place.

The approach that is used for installing System software changes and releases is based upon the use of IBM's SMP/E (System Management Product). SMP/4 is the original SMP product which is no longer used by most installations and is not covered within the scope of this article.

Establishing an Environment for System Software Changes

System software changes can be classified under the following three categories:

• Installing system software releases and maintenance
  
• Installing inhouse written system software (e.g., exits)
  
• Changing parameter files (e.g., members in SYS1.PARMLIB)

In a typical environment, most MVS system software resides on the System Residence Volume (i.e., SYSRES). Because of the fact that altering the running SYSRES would present an unacceptable level of risk to most installations, a typical maintenance cycle would begin by copying the running SYSRES to an alternate volume. IBM's DFDSS product and FDR are examples of products that may be used for this purpose. The copying process should also include the "cloning" of the SMP environment which represents the SYSRES. All maintenance is applied to the "cloned" environment. In this way the installation ensures the integrity of their production SYSRES and the SMP environment that describes it.

Most environments have three volumes included in a volume rotation cycle which consists of the current production (i.e., running) system, the volume in which the next release is being worked on, and a third volume which contains the previous release. Separate volumes allow an installation to IPL (i.e., boot) from a previous version when the volume containing the currently installed version fails. Some installations may only have two volumes in their volume rotation cycle and fail to have a volume representing the release. As we will discuss later in this article this is not necessarily an exposure.

MVS only requires three datasets to be installed on the SYSRES (i.e., SYS1.NUCLEUS, SYS1.SVCLIB and the OS password if it is used). However, in many environments, all MVS datasets are stored on the SYSRES in order to better manage the installation of future releases. Other IBM provided non-MVS products (i.e., CICS, NCP, and DB2), are not usually contained on the SYSRES but go through the same cloning process when they are installed. The only difference is that these products do not require an entire volume an therefore only their datasets are copied to a maintenance volume instead of performing an entire volume copy.

The concept of installing an entire release on separate pack can be used for non-SMP system software changes which includes inhouse written system software (e.g., exit) and system parameter changes (e.g., MVS's PARMLIB). However, many installations consider this to be an overkill or an unreasonable approach due to a lack of spare volumes. They would probably prefer to directly install the changes to the production SYSRES. In addition, applying MVS changes to a maintenance pack requires an IPL to define the maintenance pack as the new SYSRES, although the change would not normally require an IPL to install the change. The risk of applying changes to the currently running system is that if there is an error to a module which is critical to the IPL, a separate pack which contains the prior version will not be available to IPL from. Therefore, the system programmer would have to be available to correct the problem. This is based on the assumption that there is a process in place which allows an IPL to occur without requiring access to the currently running system resident volume (i.e., package which resides on tape to IPL off of to edit the dataset that is causing the failure).

Security Approaches for System Software Datasets

The method used to restrict access to system software datasets varies according to the approach that is used for performing system software changes. If an environment requires that any change to system software will necessitate the use of a maintenance volume (i.e., backup of the production volume), whereby the system will be IPLed from the maintenance pack (i.e., new SYSRES) to install the change, then update access to the production versions of the system software datasets is not required by the system programmer. However, since the same datasets names are used for system software datasets that reside on the SYSRES and maintenance volume, a security approach must be used to distinguish between the SYSRES and maintenance volumes.

The first approach to restricting access to system software datasets which is most commonly used, is to restrict update access to the system programmer either through their regular ID or through a maintenance ID. The maintenance ID would only be used when access is required. The disadvantage of this approach is that a system programmer would have the ability to make unauthorized changes to the production versions of the system software datasets. Using the logging facilities provided by your external security system will not provide sufficient details to disclose an unauthorized change. This is because the audit trails are provided at a dataset level. Therefore, when a system programmer is authorized to update a dataset, unauthorized changes can be made to other modules which will not be detected. A member level security package or a change control tracking system would provide an additional level of control by providing an audit trail of the modules that have been updated within a dataset.

Two products have been identified which would provide the level of control that is required. Action Software International Change Action product is an entire Change Management system which provides the ability to establish an on-line approval process for all software changes and can restrict or log update access to modules that are being migrated to any site specified library. In addition, the changes that were made to the module can be viewed on-line. Computer Associates PDSMAN (i.e., recently acquired from Legent) is another product which provides member level security and logging. However, PDSMAN's member level security and logging can be bypassed by non-conventional access methods that directly alter data (e.g., AMASPZAP).

A further enhancement to the first approach of restricting access would be to allow system programmers update access only to system datasets which are contained on a specific volume. This approach would restrict unauthorized access to the production version of the system software libraries. However, it would require that their security entitlements be changed after system maintenance is applied because most sites operate on a volume rotation approach. It is important to note that if changes to the maintenance pack are not controlled, unauthorized changes can still be applied since the maintenance pack will be the running production version after the maintenance installation has been completed. Methods which should be used for controlling changes to the maintenance pack are discussed in the next section.

Other variations to described processes can also be used which restrict access at the volume level. One such method is to allow system programmers update access to the maintenance volume only. Again, this approach would restrict unauthorized access to the running version of the system software datasets but would require the security to be changed after system maintenance is applied.

Using SMP To Install System Software

SMP assists the system programmer in installing new products, new releases of existing products, and corrective and preventive fixes. A complete audit trail is maintained about information pertaining to each product installed via SMP which includes:

• Function level (i.e. system software product)
  
• Service level (i.e., PTF level)
  
• Libraries the product is installed in

System Modification Program is used to process system modifications (SYSMOD) which are categorized as one of the following types of changes:

• Authorized Program Analysis Report (APAR) which is a program fix for a specific site which is provided as a ZAP or a module replacement)
• Program temporary fix (PTF) which is a vendor provide fix for a problem that effects all sites which is provided as a module replacement
• User modifications (USERMOD) which are inhouse written programs
• Function SYSMODs which are full product installs

SMP/E tracks all modules in a Consolidated Software Inventory (CSI) and uses ISPF panels and batch jobs to perform all of its functions.

CSI consists of the following three zones:

TARGET ZONE

The Target Zone is a group of records in a CSI data set that describes the SYSMODS, elements (e.g., Modules, Macros, source Load Modules) and their respective target libraries (e.g., Load Libraries, Macro libraries).

Target libraries are the location of the system software load libraries which will be used in the running version when the system is IPLed from the maintenance volume.

DISTRIBUTION ZONE

The Distribution Zone is a group of records in a CSI data set that describe the SYSMODS, elements (e.g., modules, macros, DDDEFs), and their respective distribution library.

The distribution libraries (i.e., DLIBs) contain master copies of all the elements in the system. These libraries are used as backups during the maintenance process.

GLOBAL ZONE

The Global Zone is a group of records in a CSI data set used to record information about SYSMODS that have been received for a particular system. The Global Zone contains information that allows SMP to access the Target and Distribution libraries. In addition, the Global Zone also contains entries that allow an installation to custom tailor the options used in SMP processing.

It is not practical to use SMP to install many third party vendor system software products. Although, the install only consists of copying load modules to predefined libraries, some products have their own maintenance process completely exclusive of SMP. For example, Top Secret (versions 4.2 and older) ships maintenance in the form of encrypted ZAPs which can not be applied using SMP.

Most vendors however are committed to using the SMP format for shipping new releases of their software as well as for maintenance. If a system software product is not shipped in the SMP format it is not practical to write the necessary code to have its installation controlled by SMP.

There are several steps that are required to receive and apply maintenance to a typical MVS system.

Preliminary Steps

As previously discussed, the initial step for an MVS install is to create a "cloned" environment to install maintenance by copying the running SYSRES to an alternate volume.

RECEIVE Phase

For maintenance, the RECEIVE phase places SYSMODs into the PTF temporary storage data set (i.e., PTS) for subsequent processing that occurs in later SMP phases. In the case of complete product installation, the RECEIVE phase stores the SMP instructions and product software in SMP REL files. These files are typically deleted after the product is installed.

During the RECEIVE phase, a job is executed to store the SYSMODs in the PTS or REL-FILES and the Global Zone is updated to reflect the receipt of the maintenance or product. The HOLDDATA, which is shipped with IBM maintenance, is also received and identifies the PTF's that were shipped on previous maintenance tapes and has been identified as the source of a system problem. SMP uses this information to ensure that the PTF (i.e., if not already applied) will not be introduced until the problem is resolved.

IBM SYSMODS are shipped with MCSs (i.e., Modification Control Statements) wrapped around the object code (i.e., source is usually not supplied) and identifies the following:

Type of modification or SYSMOD (Function, PTF, APAR, USERMOD)

Relationships to other systems and releases which are identified by ++VER and ++IF statements

Elements to be replaces through this modification which consist of ++MAC for macro replacements, ++MOD for module replacements, and ++SRC for source module replacements, ++JCLIN for Linkage characteristics.

ACCEPT Phase

Although it almost seems out of sequence, it is a good practice to ACCEPT any maintenance APPLIED in a previous maintenance cycle at this time. In this way, prior maintenance cycles are fully completed before introducing new maintenance into our environment. The ACCEPT phase is not performed after the completion of the maintenance cycle in order to allow individual changes that were applied to be backed out in the event that there is a problem after the maintenance volume is turned over for production processing.

The ACCEPT phase places the SYSMODs into Distribution libraries (i.e., DLIBs) which represent the backup or master copies of the installation's system software.

APPLY Phase

The APPLY phase places SYSMODs from the PTF Temporary Storage Dataset (i.e., for maintenance) or REL files (i.e., for complete product install) into target libraries (i.e., libraries which will represent the running system upon completion of the SMP install process). SMP applies maintenance as directed through the use of SMP control cards. Individual PTF's can be selected via the SELECT keyword, or categories of maintenance may be applied by using the SOURCEID keyword to identify a group ID assigned during the Receive Phase.

SMP ensures that the maintenance has no outstanding error HOLDS and ensures that all pre-requisites (i.e., maintenance that should have previously been applied) and co-requisites (i.e., other maintenance that must be applied at the same time) are met. SMP then processes the various module, macros, and source elements. In the case of maintenance that is applied to existing modules, SMP typically copies object code from the PTS to a temporary data set and invokes the linkage editor to link the object module into its corresponding load module in the appropriate TARGET library.

After the modules are applied, the TARGET ZONE is updated with information about the applied SYSMOD. A report from the SMP job, executed to perform the apply phase, will show whether the APPLY ran successfully, as well as the datasets and modules that were updated.

RESTORE Process (if necessary)

In the event that the SYSMODs need to be backed out, a copy of the original element (i.e. module, macro or source) will be stored in the installation-specified distribution libraries (DLIBS). The RESTORE removes SYSMODs processed by the APPLY. To perform this task, an SMP job is executed with a control card specifying the particular SYSMOD to be restored. SMP in effect will re-run the apply process, but will pull the components from the DLIB's instead of the PTS and TXLIBS.

The Restore process can only be used prior to the ACCEPT phase. This is because the Accept phase places the modification in a production status and copies it to the DLIBS. The Restore process is used only to backout changes prior to the completion of SMP install process. The Restore process cannot be used as a backout procedure once the maintenance is turned over for production use.

Controlling System Software Changes Installed Using SMP

In order to perform the various SMP phases (i.e., Receive, Accept, Apply) which are used to install a release, maintenance, USERMOD (i.e., installation written programs), JCL is executed with control cards which specify the actions to be performed. This includes moving modules to the various libraries that are used in the SMP install process. The JCL executes the SMP program, GIMSMP, which updates the target zone and distribution zone of the Consolidated Software Inventory (CSI) in order to track the location and status of a product being installed. It also performs the updates to the libraries where the modules are linked. Since the JCL updates the various libraries used in the SMP process, the system programmer would require update access to these libraries (e.g., distribution libraries and target libraries). Therefore, unauthorized changes can be made to the target libraries which will be the running production libraries upon the completion of the SMP install process.

Since many of these system libraries are APF libraries, even if the system programmer is restricted from updating the libraries once they are turned over to production, they would still have execute access. This enables them to execute APF programs which could potentially have been written to bypass security and perform unauthorized processing.

Based on the inherent weakness, which would require the system programmer to submit jobs which update the libraries used in the SMP process, a process should be in place to prevent unauthorized updates or detect their occurrence.

There is no proven control in place to prevent unauthorized updates from occurring. Having a separate librarian function to execute the JCL that performs that various functions of a product install would restrict the timely development efforts of the system programmers. However, this approach should be analyzed to determine if it can used in ones environment. Another approach consists of having a controlled library which maintains a permanent version of the JCL that is executed to perform the various SMP phases. The system programming manager would then review the JCL to ensure that all modules that were linked into load libraries used for an SMP install were pulled from the appropriate libraries.

An alternative method, which has not been considered based on survey of various MVS installations, is to program path and library path the system programmers access to the maintenance libraries through the SMP main program, GIMSMP and the GIMUTL program which is called by GIMSMP. Program and library pathing is a common control method that is utilized when a user is responsible for updating a sensitive dataset using a precoded program. Whereby the user is allowed to update the dataset only through a specific program which restricts their ability to update the dataset using any other facility. The user's access is further restricted by specifying that the program must be executed from a specific library from which the user is also restricted.

Since all the functions that are performed during the SMP install process are based on submitted JCL, which executes the GIMSMP program, this control approach may be a possible solution to the inherent weakness of the SMP install process. However, the programmer would still requires update access to the datasets used by SMP in order to allocate space for datasets. When non- SMP compatible products are installed, the system programmer would require update access to the target libraries since they are installed using IEBCOPY and ZAPs which cannot be program/library pathed.

Because in most cases it is not practical to prevent system programmers from updating the libraries used in the SMP install process, a detective control is required to identify all changes that are made at a module level. As discussed previously, the external security systems (e.g., ACF2, RACF, and Top Secret) only identifies changes at a dataset level. This is not an acceptable level of audit trails to conduct a review to determine whether only appropriate modules were changed. The reason is that there can be hundreds of modules that are changed in one dataset. Therefore, a member level security product which identifies the modules that were changed and the method that was used to make the change would be required in order to ensure that all changes are made to the target libraries and distribution libraries using the GIMSMP program.

All SMP Compatible System Software Products Were Installed Using SMP

As discussed previously in this article, all IBM and many third party system software products are delivered in a format which allows them to be installed using SMP. Asides from providing an effective mechanism for managing system software changes, SMP provides the ability to ensure that all required components of a release has been completed prior to its use in a production. Therefore, it is necessary that all system software products which are SMP compatible are actually installed using SMP. This critical control method should be verified by the auditor.

Definition of Terms

Target libraries contain all products installed by SMP. Since all shops have separately dedicated volumes for production and for applying maintenance, target libraries are usually associated with the foregoing. Therefore, the software for the currently running system are contained in target libraries.

DLIBS (i.e., Distribution libraries) are the backup versions of the target libraries which are used to restore the prior version during the maintenance cycle. Once maintenance has been applied, the DLIBS are of no significance.

FMIDs is an identifier for system software products that are installed within a particular site. FMIDs names change with every release so it is possible to see more FMIDs then products that you have installed.

Control Requirement

Global zones is the method used for tracking FMIDs and the corresponding target and DLIB (i.e., Distribution libraries) zones which they are stored in. MVS and other SMP compatible products are stored in SRELs, which are product identifiers. For example, the SREL for MVS is Z038, which remains the same regardless of the release of MVS that you are running. The SRELs identifier is checked by SMP during the RECEIVE process (i.e., load maintenance tapes to a containment library) to ensure that the SREL corresponds to the currently installed system. Different SRELs are used in order to avoid having to store all components of an MVS maintenance tape into one Global zone. Therefore, the CICS systems programmers (i.e., installs CICS), systems programmers (i.e., installs MVS), Network system programmers (i.e., installs NCP), and the database system programmers (i.e., installs DB2) can each perform a separate maintenance install cycle. However, all of the components of maintenance tape may still be installed in one global zone using separate TARGET and DLIBs zones to distinguish between the SRELs.

The first step in determining which products were installed using SMP is to execute a job to identify the target zones, DLIB zones, and FMIDs. However, you must first obtain the global zone name(s) and the SMP PROC that is used by your installation. There is no automated method available to systematically identify the global zone name. The SMP PROC is used to easily obtain the specific SMP information, instead of having to know the specific statements required for each release of SMP. All sites have their own SMP PROCs coded in order to perform this function.

JCL to determine the FMIDs installed using SMP:

//USERIDI JOB
//ACCPTCHK EXEC
// CSI=''
//SMPCNTL DD *
SET BDY()
LIST GLOBALZONE.

Based on the JCL submitted, the following output will be generated:

GLOBAL OPTIONS = OPTMVS

• ZONEINDEX = CIB271B DLIB MVSSMPE.CIB221T.CSI
  CIB271T TARGET MVSSMPE.CIB221S.CSI
  
  SREL = Z038
  FMID = CICS170 COCS001 CPC003 CPC004

Note: CIB271T is the target zone name

If Examine/MVS is available, the 2.3.3 option (INSTALLED PRODUCTS) may be used to identify the products that have been installed via SMP/E.

Ask your system programmer for the FMIDs associated with each product that is installed at the site. The FMIDs, which represent the products that have been installed using SMP, is then compared to the JCL submitted. This identifies the FMIDs installed via SMP in order to determine whether all system software products were installed using SMP.

Another critical control component within a System SDLC review is to determine whether all user written system software (e.g., exits) are installed using SMP. This control point is critical in order to verify that SMP tracks the user written code. This ensures that the user written code is applied whenever you install future releases and maintenance. You have the option of defining the object code to SMP or having SMP control the source code which it then assembled and linked to the appropriate libraries. Allowing SMP to control user written code from a source level restricts the system programmers' ability to make unauthorized changes to the code and maintains the source/load integrity. The steps to determine whether all user written programs are controlled by SMP is not discussed in this article but you can obtain the information and detailed audit steps from the ExtrAUDITnaire/MVS product that is developed and distributed by Audit Serve, Inc.

Conclusion

Based on this article, you should have an understanding about the organization of the system software environment and the controls which should be in place when installing system software. There are many other control points which should be considered when reviewing a System SDLC. This, however, goes beyond the scope of this article.

This article was written more than one year ago. Events may have changed since this article was written.

Copyright  2006, Audit Serve, Inc. All rights reserved. Reproduction, which includes links from other Web sites, is prohibited except by permission in writing.

This article appeared in a past issue of the Audit Vision E-Mail Newsletter.

Technical Articles | Conferences | Audit Programs | Audit Serve Seminars | Consulting Services | Audit Vision Newsletter | What's New | Contact US

This website has been optimized for Netscape and Internet Explorer 4.0 and above.  Your comments and suggestions are always welcome, please email webmaster@auditserve.com.
Copyright © 2000  All rights reserved.  27 Pine Street, Suite 700, New Canaan, CT 06840 USA.