IBM Software Group RDz Workbench Integration

IBM Trademarks and Copyrights 4 © Copyright IBM Corporation 2007, 2008, 2009, 2010, 2011. All rights reserved. 4 The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. 4 This information is based on current IBM product plans and strategy, which are subject to change by IBM without notice. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. 4 IBM, the IBM logo, the on-demand business logo, Rational, the Rational logo, and other IBM Rational products and services are trademarks or registered trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others. 2

Course Contributing Authors § Thanks to the following individuals, for assisting with this course: 4 Russ Courtney/IBM 4 Reginaldo Barosa/IBM-Rational 4 David Bean/IBM-Rational 4 Ed Steele/IBM-Rational 4 Dave Willsey/IBM-Rational 3

Course Overview § Audience 4 This course is designed for application developers who have learned or programmed in COBOL, and who need to do z/OS Traditional Development and Maintenance as well as build leading-edge applications using COBOL and Rational Developer for System z. § Prerequisites 4 This course assumes that the student has a basic understanding and knowledge of software computing technologies, and general data processing terms, concepts and vocabulary, as well as a working knowledge of COBOL and z/OS. 4 Knowledge of SQL (Structured Query Language) is assumed for database access is assumed as well. 4 Basic PC and mouse-driven development skills, terms and concepts are also assumed. 4

Course Topics § Course Name: Rational Developer for System z Foundation Training § Course Description: Learn how to use Rational Developer for System z to do z/OS traditional development, maintenance, support and for Enterprise Modernization of z/OS applications § Pre-requisites: Some experience developing COBOL applications using z/OS is expected. A working knowledge of SQL is also recommended. § Course Length: ~5 days – or if done in self-paced mode, at your own pace § Topics (Agenda) § § § § § Getting Started - installing and configuring RDz - and the course materials, and using Eclipse The RDz Workbench – Code analysis tools – Editing – Compiling programs – Debugging local COBOL programs The Data Perspective: – Working with relational data sources – Modifying test data – Editing and testing SQL statements Working with remote system resources: – Connecting to a mainframe – Data management – Accessing and editing files z/OS Application Development – Creating MVS Subprojects – Creating and customizing project properties Debugging z/OS Applications – Debugging Batch Applications – Setting Debug Tool for Online Applications Working with File Manager – Creating test data – Editing complex file-types Working with mainframe ABENDs using Fault Analyzer – Creating Fault History views – Analyzing and solving mainframe ABENDs Creating and modifying BMS Maps using the BMS Map Editor 5

UNIT The RDz Workbench Topics: § § Analyzing Mainframe Abends ABEND Codes and Reasons Using IBM's Fault Analyzer Appendix 6

Unit objectives After completing this unit on Production Support/Application Testing/Software Defect and IBM Mainframe COBOL ABEND Research, you should be able to: 4 Define the steps in a generalized methodology of ABEND resolution 4 List the various sources of ABEND inputs, including: § PD Tools documents § Other sysout § Dynamic trace facilities 4 List the common types of COBOL program ABENDS 7

ABEND Overview § When an application ABEND (ABnormal END-of-job) occurs, z/OS stops executing your program, closes files and buffers and generates a single high-level message in the form of a System Completion Code (Sxxx). § The System Completion Code is usually written to an output listing file through your //SYSOUT DD * JCL entry. § This completion code indicates why the system has decided to stop executing your application. § It is related to, but often only loosely related to what is really wrong with your application § Because of this the System Completion Code represents the starting point for your analysis of the problem. She won't be laughing when she gets back to her desk and finds out that last night's production jobs blew sky high! 8

Debugging Assistance § Along with the System Completion Code, using IBM’s Problem Determination tools (PD Tools) Fault Analyzer you can obtain various reports which describe What/Where and How the ABEND occurred. § Valuable information contained in the Fault Analyzer report-files includes: 4 The System Completion Code (and often a short text description of what it designates) 4 A short explanation of the cause of the ABEND 4 The COBOL instruction (statement) or line number, which contained the invalid operation causing z/OS to halt execution 4 Variables of interest – and code surrounding the instruction that halted execution 4 A "core-dump" (a hexadecimal printout) of the internal machine storage and registers relevant to the areas of your program surrounding the COBOL instruction which caused z/OS to halt execution. § This information is critical to begin understanding and researching the problem, but it is sometimes insufficient to solve the underlying application problem, which could be any combination of: 4 Incomplete, incorrect or invalid COBOL procedural logic 4 A typo such as a misplaced period, or incorrectly specified field 4 Incorrect or invalid input data 4 Batch jobs run out of sequence 4 Input files missing or corrupted (hardware errors) 4 Errors which relate to JCL problems 4 etc. 9

Getting Started There as many different ways to analyze and research COBOL ABENDs as there are individual approaches to writing procedural logic. However, if you've never done software production support work, consider starting with the following structured problem-solving approach: 1. 2. 3. 4. 5. Preparation Research Hypothesis Solution Resolution As a final note before we begin discussing the above, understand that there are often two distinct phases of application Production Support: 1. Data Center “on-call” ABEND resolution - wherein a technician receives notification that a job or transaction has ABEND’d and must be "fixed" within an extremely short timeframe (usually minutes to hours). In this case, the technician's main concern is to "patch" the problem - get the system back online, or get the batch job-stream back into production ("Patch-It"). 2. Next Day problem resolution - wherein technician(s) actually track down and solve the problem that caused the ABEND ("Fix-It"). The steps that follow represent a structured approach to "Fix-It" – in that they go well beyond the scope of the emergency measures used to "patch" the problem during an On Call emergency. 10

1. Preparation and Information Gathering – 1 of 2 Collect all necessary background information on the ABEND. WHAT happened, WHERE the ABEND occurred and HOW the ABEND occurred: Start with Fault Analyzer's reports – as they contain a broad and deep set of formatted, structured, analysis information Collect additional supporting ABEND output (SYSOUT) from the job, DISPLAY statements, etc. ) Obtain copies of the run-time: 4 4 4 § § § JCL Program source All copybooks (or expanded source listings) 11

1. Preparation and Information Gathering – 2 of 2 If batch, from the JCL learn the dataset names of input and output files accessed by the program (which you may need to browse as part of your research) RAA Batch Job Diagram showing datasets being passed from step to step § Learn the nature of the batch job from system documentation, or from an application business expert (at least at the level of module-flow and file-access) 12

Research – Analyze the Gathered ABEND Data § From the preparation step, construct a mental map (understanding) of the program's execution (HOW the ABEND occurred - which will start you down the path of research to understand "WHY" the ABEND occurred § WHY determination usually requires a combination of "Static" and "Dynamic" analysis - complementary research and investigative approaches. § These steps need not be followed in this order. Rather, in time you will develop an "intuition" as to which kind(s) of analysis will be most likely to provide the information you need to solve your problem. § To assist, you can use application research and analysis tools such as § § § IBM’s Rational Asset Analyzer, Rational Developer for System z Analyzer – Static Analysis IBM's Debug Tool or CICS I. A. – Dynamic Analysis So – if the reason "why" the ABEND occurred is not apparent at this point, perform Static and/or Dynamic Analysis on the specific areas of the application relating to the ABEND. Note: Recompiling old COB 2 programs with Enterprise COBOL can create many of these problems, even though "Nothing has changed in the program" 13

Static Analysis – Overview of Techniques 1. Structural Visualization: is the generation of an accurate mental map, understanding or mental image of the program's control structure, or logic-architecture. Using the starting point represented by the ABEND condition (the statement which caused z/OS to halt execution) and using electronic-assisted tools (such as IBM’s Rational Asset Analyzer or Rational Developer for System z), build an accurate understanding of the code invocation at: 4 The module/file level (System View) - Paragraph/Section level (Hierarchy chart) - Statement level (Flow chart) Structural Visualization can done be "top-down", by asking open-ended questions; such as learning how a particular routine "hangs-together logically", or it can be used "bottom-up", by asking specific close-ended questions about a program, such as "How does this particular paragraph get executed? " "How did this module get invoked? " 2. Data Flow Analysis: A combination of control structure analysis and data item analysis, which seeks to determine the usage of particular fields throughout a program. Data flow analysis is used to determine (from a given instance of a data item) where the next occurrences of that item exist in your program, and how the data item is used; (as a receiving field in a MOVE or mathematical operation, as the sending field in a MOVE statement, as part of a logic-branch (IF, PERFORM UNTIL/VARYING, etc. ). 3. Data Impact Analysis: An expansion of Data Flow Analysis which traces the movement of data from field-tofield throughout a program, or throughout an entire application; including I/O (screens and files). Using Data Impact Analysis, you can identify all fields that might have had an impact on the contents of a field (before the ABEND occurred). And just as importantly - you can learn the affect changing this field will have on the behavior of the application. 4. Textual or Data Item Usage: Utilized more for application maintenance and enhancement requests, this type of Static Analysis involves searching for "categories" of program-items, such as "List all fields that contain *JUL*, *GREG*, *YR*, *YEAR* (suspect date candidates for Year 2000 conversion), or list all such fields with two digits (numeric) or two-byte (alphanumeric) definitions. 5. Code Partitioning: Again, utilized more for application maintenance, enhancements and application reengineering, Code Partitioning involves mentally organizing and analyzing code by function or process, such that you understand can distinguish the usage of code by business process. For example: Find all code that relates to the calculation of premium renewal payments … or… Isolate the code that edits a particular file, with an eye towards creating a shared subroutine from the code. 14

Dynamic Analysis – Overview of Techniques 1. Tracing: Source-level interactive debugging. Watch the program execute statement-bystatement, and line-by-line. This is very useful for detailed-debugging, particularly of dense or complex instructions. Some software (for example, the Rational Developer for System z) allows you to trace the program logic, attempting to re-create the sequence of events (COBOL statements) that transpired up to and including the ABEND condition. Tracing is an invaluable method for detailed debugging. However, given the size and scope of production applications, it is generally more practical to trace specific problem areas of a program. 2. Interactive Execution: Execute (run) a program, stopping at selective Breakpoints (Pause execution each time a certain field-value changes, or when a value exceeds some threshold), and examining the contents (value) of specific fields. Interactive Execution must be done by (or with) an application analyst who understands how the system is supposed to operate. Interactive Execution is useful for observing control flow, and is often combined with line-byline tracing by setting selective breakpoints, monitoring values, "running" the application to the breakpoints, and then tracing the code line-by-line. 3. Selective Data State Collection: Execute code and establish a functional summary of specific data states that it creates. Use these states in subsequent test runs to compare results of current values to expected values. 4. Coverage: Analyze the number of times each COBOL statement is executed for a given run. Note that PD Tools/Debug Tool can run a report that shows code coverage. This technique is extremely useful for analyzing test data coverage of a given application. And it can be used effectively for debugging if it makes apparent problems such as infinite loops (S 222, S 322 and B 37 ABENDs), over-loading tables - (loading tables beyond the maximum OCCURS clause and overlaying storage, which cause things like: S 0 C 1, S 0 C 4, and S 0 C 7 ABENDs). 15

Hypothesis - Determine WHY the ABEND occurred – 1 of 3 § Your preparation and research is probably all you need to be able to describe WHAT, WHERE and HOW the ABEND occurred 4 At what point in the program the logic failed, and what sequence of COBOL statements caused the failure). § However, before modifying any logic, you must determine WHY these statements (or sequence of events) caused this particular failure 4"Why did this production input file contain spaces in a numeric field? " 4"Why did the program's logic perform the Initialization routine twice? " 4"Why did the Read routine execute past end-of-file? " 4 etc. § Only through a determination of WHY will you be able to make a change to production business logic safely, and with confidence that; 4 Your change will resolve the ABEND 4 Your change will not introduce new (additional) ABENDs 16

Hypothesis - Determine WHY the ABEND occurred – 2 of 3 § Sometimes it is relatively easy to come to an understanding of WHY certain ABEND conditions occurred. For example, perhaps a period was left off the appropriate termination point for an IF statement - which caused execution to perform an operation out of sequence. Or perhaps an IF. . NUMERIC test (which should have been coded for all numeric fields in a file) was forgotten. Or a paragraph was performed through the wrong paragraph-exit, or a production job was released before certain files were available (causing I/O errors). These types of ABEND situations can be understood (and usually resolved) fairly quickly. But, not always. 4 What if - in the case of the IF statement with the incorrect termination point - the logic that has been coded, correctly processed the first 100, 000 records in the file? § Making a change to a critical IF condition could very well affect other down-stream processing within the program, wrecking havoc with subsequent routines. 4 Or what if - in the case of the file containing blanks in the numeric fields - the input file was supposed to be "clean" (validated) by this point in the job-stream - having gone through allegedly "exhaustive" edits in prior modules. § By simply adding an IF test you may solve your program's specific ABEND, but you will not have resolved the actual problem - which exists somewhere else in the system. § In other words, localized/piecemeal approaches to resolving production ABENDs are not recommended - as they usually change the problem, instead of solving it. And sometimes they just spawn new problems. § It should be noted that, a clear understanding of the business functionality automated by this process is usually required to resolve WHY something has gone wrong. 4 Calling on business experts or "application/business" experts who understand "the big picture" - and the context in which the job executes is the rule rather than the exception to this process. 17

Hypothesis - Determine WHY the ABEND occurred – 3 of 3 § Developing a clear and accurate determination of WHY a problem that lead to an ABEND condition exists may take a considerable amount of time, depending on the: 4 Size, complexity and structure of the code 4 Your familiarity with the program's business purpose - coupled with your ability to grasp the point of each statement (assuming you didn't write the code) 4 Type of ABEND and reason for the problem (some are more diabolical than others) 4 Size of the input/output files, and capabilities of your file editor § Note that, in addition to an understanding of the reason for the ABEND, the results of your investigation should produce an understanding of the solution to the problem (the fix itself). 18

Solution - Fix the Problem and Test Your Solution Take the appropriate action to resolve any business - or system-wide issues. § Depending on how extensive the damage caused by the problem, or for how long any problems have persisted undetected: 4 4 4 § Files may have to be restored from backups from a previous point-in-time Jobs may have to be re-run from a previous point-in-time (synchronized with file generations) Files may have to be modified with "one-shot" programs, written to resolve issues that require "surgery" on the data Take the appropriate action to fix the technical (coding) problem: 4 4 4 § Edit program source - modifying the existing production logic …and/or… Modify the JCL (if the error included JCL issues) You may have to edit files using File Manager Test your solution: 4 4 Compile and Link the new version of the application Create an "image copy" of the production file system, in order to test your fix Re-Run the batch job and analyze results Run "Regression Tests" against the new code – analyze unexpected results for 19

Resolution § § Build in the test environment Migrate changes to back in to production Promote your changes to production Schedule and re-run the cycle Batch Job § http: //en. wikipedia. org/wiki/IBM_Rational_Clear. Case 20

Unit objectives After having completed this unit on Production Support/Application Testing/Software Defect and IBM Mainframe COBOL ABEND Research, you should now be able to: 4 Define the steps in a generalized methodology of ABEND resolution 4 List the various sources of ABEND inputs, including: § PD Tools documents § Fault Analyzer reports § Other SYSOUT § Static analysis § Dynamic trace facilities 4 List the common types of COBOL program ABENDS 21

UNIT The RDz Workbench Topics: § § Analyzing Mainframe Abends ABEND Codes and Reasons Using IBM's Fault Analyzer Appendix 22

ABEND Completion Codes and some typical causes § While there is a wide variety of reasons for ABEND conditions ("WHYs") in WHY production systems, it is possible (and useful) to categorize and organize HOW certain conditions often lead to certain types of ABEND completion codes - in order to expedite or streamline your analysis and research (an 80/20 approach to analysis). § The following information on a few common z/OS ABEND completion codes, and the conditions which generated them is included for you to make effective use of PD Tools/Fault Analyzer listings and the above debugging, research and analysis process. § Notes: 4 This information is available to some degree within the RDz product in the Lookup View 4 There is another source of ABEND/Debug information you might want to take a peek at: § http: //www. jsayles. com/ibm/cobol/ABEND. htm § http: //it. geocities. com/aminardi/computer/CODES. html 23

S 001: Record Length/Block Size Discrepancy Reason(s) S 001 -0: Conflict between record length specifications (program vs. JCL vs. dataset label) S 001 -2: Damaged storage media or hardware error S 001 -3: Fatal QSAM error S 001 -4: Conflict between Block specifications (program vs. JCL) S 001 -5: Attempt to read past end-of-file Instructions: OPEN, CLOSE, READ, WRITE Frequent Coding Causes: S 001 -0: Typos in FD or JCL S 001 -2: Corrupt disk or tape dataset S 001 -3: Internal z/OS problem S 001 -4: Forgot to code BLOCK CONTAINS 0 RECORDS in FD (default Block is 1) S 001 -5: Logic error (either forgot to close file, or end-of-file-switch not set, overwritten or ignored) Tools to debug/RDz equivalent return codes: S 001 -0: Cannot occur on RDz with Local ASCII/Windows (Line Sequential) files S 001 -2: Norton Utilities – if on Workstation/COBOL application S 001 -4: Cannot occur on Workstation/COBOL (no blocking for Line Sequential files) S 001 -5: Logic error: Use RDz's Perform Hierarchy or RAA's Program Flow Diagram to detect Dynamic: S 001 -0: During Debug – set a Watch Monitor on the 01 record S 001 -2: Need to have PC/IT technician investigate (may need to reformat disk) S 001 -4: Always code BLOCK CONTAINS 0 24

S 013: Conflicting DCB Parameters Reason(s) S 013 -10: Dummy data set needs buffer space; specify BLKSIZE in JCL S 013 -14: DD statement must specify a PDS S 013 -18: PDS member not found S 013 -1 C: I/O error search PDS directory S 013 -20: Block size is not a multiple of the LRECL S 013 -34: LRECL is incorrect S 013 -50: Tried to open a printer for Input of I/O S 013 -60: Block size not equal to LRECL for unblocked file S 013 -64: Attempted to Dummy out indexed or relative file S 013 -68: Block size > 32 K S 013 -A 4: SYSIN or SYSOUT not QSAM file S 013 -A 8: Invalid RECFM for SYSIN/SYSOUT S 013 -D 0: Attempted to define PDS with RECFM FBS or FS S 013 -E 4: Attempted to concatenate > 16 PDSs Instructions: OPEN, CLOSE, READ, WRITE Frequent Coding Causes: Most of these ABENDs occur running und z/OS (some may not even occur under z/OS, although older modules running OSVS or VS COBOL II code that have not been recompiled can produce them). Most are due JCL/COBOL FD inconsistencies. Tools to debug: Static S 013 -18: Open multiple windows on RAA Batch Job Diagram and program Environment Division - SELECT ASSIGN. 25

SOC 1: Invalid Instruction Reason(s) - SYSOUT DD statement missing - The value in an AFTER ADVANCING clause is < 0 or > 99 - And Index or Subscript is out of range - An I/O verb was issued against an unopened dataset Instructions: OPEN, CLOSE, READ, WRITE, Table handling routines WRITE 4 Note also that during Debug SYSOUT-DISPLAYs are written to the "console", recall how to view Frequent Coding Causes: - Incorrect logic in setting AFTER ADVANCING variable (or failure to understand 0 -99 limits) - Incorrect logic in table handling code, or number of table entries has outgrown PIC of variable (e. g. PIC 99, but 100 entries) Tools to debug: Static SYSOUT problem: Open multiple windows on RAA Batch Job Diagram and program Environment Division - SELECT ASSIGN. In RAA: Double-click on GO TO verb, or PERFORM chain, or paragraph name. In RDz: Select Paragraph name/Perform chain and select: Open Declaration Dynamic: Set Watch Breakpoint and Monitor on table index or AFTER ADVANCING variable. Set conditional advanced break point on subscript (i. e. SUB<100). 26

S 0 C 4: Protection Exception Reason(s) The program is attempting to access a memory address that is not within the applications z/OS Address Space Frequent Coding Causes: - JCL DD statement is missing or incorrectly coded - Incorrect logic in table handling code (referencing a table subscript < 1 or > max-table-size), - Number of table entries has outgrown PIC of variable (i. e. PIC 99, but 100 entries). - In IMS/TM systems, an MFS LL (length) field value is smaller than the actual input MSG length. Tools to debug: Static - DD statement problem: Open multiple windows on RAA Batch Job Diagram and program Environment Division - SELECT ASSIGN - IMS LL problem: Analyze through multiple Edit Windows (same solution as DD). - Incorrect linkage problem: - Open multiple windows on CALLing and CALLed programs - verify linkage declarations. Dynamic Incorrect linkage problem: - Set Breakpoint and Monitor on linkage declarations. - Set conditional advanced break point on subscript (i. e. IDX < 100). RAA - Utilize Run-Unit View to view all modules in Debug run-stream (both active and inactive). Incorrect logic In RDz/Debug - set a conditional break point on subscript (i. e. IDX < 100). 27

S 0 C 7: Data Exception Reason: Machine instruction expecting numeric data found invalid data Instructions: Arithmetic, IF-THEN-ELSE, MOVE (if receiving field is numeric - ) Note: RDz will also S 0 C 7 if sending field is numeric and contains non-numeric (MOVE pic 9 field TO pic. Xfield) Frequent Coding Causes: - Incorrectly initialized, or uninitialized variable - Missing or incorrect data edit - 01 to 01 level MOVE if sending field is shorter than receiving field - Move of Zeros to Group-level numeric fields - MOVE CORRESPONDING incorrect - or - MOVE field 1 to field 2 incorrect assignment Tools to debug: Static RAA report with options data selector on MOD or ALL Dynamic Set Watch points and Monitor on field. Run through to S 0 C 7. Locate the field definition, or use CSI report. Solutions: Add edit checks for all numeric fields and MOVE statements. 28

S 0 CB: Divide by zero Reason: CPU attempted to divide a number by 0. Instructions: DIVIDE, COMPUTE Frequent Coding Causes: - Incorrectly initialized, or un-initialized variable - Missing or incorrect data edits (i. e. failed to check divisor for zero value) Tools to debug: Static RAA report on all DIVIDE and COMPUTE instructions – or using RDz double-click on these verbs and select Filter from the Context Menu Dynamic Run through to the S 0 CB Locate to field definitions of the offending fields Solution: Add edit to check for zero divide: IF divisor > ZERO THEN COMPUTE. . . ELSE PERFORM error-processing routine Add ON SIZE ERROR to all arithmetic verbs. 29

S 222/S 322: Timeout/Endless loop Reason: Timeout due to program logic caught in "loop" through instruction set with no exit. Note: RDz Workstation binaries (Run-Time System) does not honor the concept of timeout. Frequent Coding Causes: - Invalid logic or fall-through logic - Invalid end-of-file logic - End-of-file switch overlaid - Subscript not large enough - Perform Thru wrong Exit - PERFORM UNTIL "End-Of-File", but not performing "READ" routine to reach EOF condition Tools to debug: Static RAA screens on logic in PERFORM chain RAA Display Perform Thru Dynamic PD Tools (mainframe) Debug to S 222 Analyze counts (color) Query and Monitor on subscript Set an Advanced Break Point - Conditional on count Solution: From within Debug, use RAAi to identify logic which could cause looping. Select and click on PERFORM THRU, PERFORM UNTIL, GO TO. Place break points on potential error lines. 30

S 806: Module Not Found Reason: CALL made to program which could not be located along normal search path (STEPLIB top-to-bottom, JOBLIB top-to-bottom, LINKPACK) Instructions: CALL, or JCL EXEC PGM=XXXX Frequent Coding Causes: - Module deleted from library, or never compiled to library - Module name spelled incorrectly - STEPLIB does not contain load library with module - I/O error occurred while z/OS searched the directory of the library Tools to debug: Static Do RSE search on module name. Dynamic Set Program Advanced Break Point (Entry) to set program break before entry to system. Solution: Spell name correctly 31

B 37/D 37/E 37 – dataset space exceeded ABENDS - B 37/D 37/E 37 (RTS-028) B 37: Disk volume out of space. D 37: Primary space exceeded, no secondary extents defined. E 37: Primary and secondary extents full. In TSO, PDS directory needs compress. E 37 -04: Disk volume table of contents (VTOC) is full. Reason: MVS could not find space for output WRITE. Note: RDz Local COBOL Run-Time System does not honor the concept of Primary/Secondary space "Out of space" condition will not occur until disk actually fills up Instructions: WRITE Frequent Coding Causes: - Not enough space initially allocated to output file(s). - (more likely) Logic error - program in (infinite) loop writing output file(s) - see S 222/S 322 reasons. Tools to debug: Static - if (unlikely as this may be) you're debugging locally (on your Workstation) you can find the out-of-space file and statistics on it as follows: . Do directory list on the file you are writing to: Go to DOS Window Type "DIR fname. ext" On the host the JCL will show the DDNAME and z/OS filespec of the dataset in question Dynamic Set an advanced conditional break point to break on a certain number on iterations See S 222/S 322 reasons and solutions Also, set break point on file WRITE statements Note: If not logic error and actually have full disk, you may have to clean up your disk and erase files: 32

Database Abends DB 2 application access rarely abends, they "die gracefully" due to return code processing. DB 2: SQLCODE 4 A unique integer which describes DB 2's reaction to your request. SQLCA 4 A large 01 block, which contains several other fields pertinent to debugging, particularly the SQLWARNs. Suggestion: Set Line Breakpoint and/or Variable Monitor on SQLCODE and other key feedback areas - or Set Line Breakpoint and Watch Monitor for /"On-Change Break" Double-click on field, Ctrl/F 3 Notes on UDB Offloading and Database Abends § UDB handles SQLCODE processing in a manner consistent with DB 2. However, it should be noted that UDB does not handle SQLWARNO-n and the SQLCA (SQLERRD, etc. ) fields compatibly. § DB 2 ABENDs 4 If DB 2 tables have not been defined, you may get a -204 return code on your SQL statement. 4 UDB may allow you to fix Database errors dynamically, without completely stopping your animator session: 33

UNIT The RDz Workbench Topics: § § Analyzing Mainframe Abends ABEND Codes and Reasons Using IBM's Fault Analyzer Appendix 34

Topic Considerations Note: In this topic you will learn how to use Fault Analyzer to debug an ABEND. The screen captures all describe connecting to a public z/OS machine that IBM makes available – during classes. If you are taking this course through standard IBM services delivery you should be able to use the properties (I/P address, port#s, etc. ), logon IDs and passwords that your instructor provides you with. But you may also be taking this course standalone – and in that case, you will need to speak to your company's Systems Programming staff to learn how to connect and logon. It goes without saying that the actual file names in the screen captures of mainframe libraries and datasets will vary. So you should focus on the process and steps and "how to" – and don't be perplexed at differences in screen captures. You also may be using your company's own Source Control Management system – to do things like builds, compiles, etc. In that case much of the remote functionality in RDz will be customized and tailored to your company's unique and idiosyncratic procedures and protocols. 35

Unit objectives After completing this unit, you should be able to: 4 Work with ABEND analysis reports created by IBM Fault Analyzer 4 Browse Report and Mini-Dump pages 4 Retrieve various Fault Analyzer view information 4 Browse and search ABEND codes 4 Use the various productivity features in the Fault Analyzer perspective Note: 4 This presentation and Lab is not a comprehensive IBM Fault Analyzer unit. It is only intended to introduce you to the RDz/Fault Analyzer interface. It is assumed that you are already working with IBM Fault Analyzer on the z/OS host. 36

Overview and Incentive! § Face facts: 4 ABEND research ("shooting a dumps") is not how you want to spend your week-nights § The good news? You don't have to. § Fault Analyzer: 4 Identifies the line where execution halted 4 Shows the points-of-interest surrounding the ABEND: § Variables and variable values § Statements § Data and buffers 4 Gives you a serious head start on the What/Where and How of ABEND debugging 37

What is Fault Analyzer? § IBM Fault Analyzer is a tool that helps you determine the cause of an application ABEND. It is used to determine: 4 What happened, how it happened, what program, what line/statement, which variables, what files, were involved, etc. variables § Fault Analyzer provides the necessary information to perform root cause analysis on an application ABEND. 4 You do not have to interpret low-level, system dumps and wade through HEX data. Information is presented in report format § IBM Fault Analyzer for z/OS® gathers information about an application and the surrounding environment at the time of an abnormal end (ABEND), providing you with the valuable information you need to work through § After analyzing information about your application and its environment, Fault Analyzer generates an analysis report (IDIREPORT) that describes the problem in IDIREPORT terms of application/program statements and variables 38

Fault Analyzer for z/OS Overview § Fault Analyzer is part of IBM’s Problem Determination Tool family of products: http: //www-01. ibm. com/software/awdtools/deployment/ § It runs in both test and production with very little overhead, and: 4 Provides support for analyzing***: § IMS and CICS® online application and system failures - with debugging facilities for all of the IBMmainstream online files and databases – IMS-DL/I, DB 2, VSAM, IDMS, etc. IDMS § Web. Sphere® Application Server for z/OS system failures § Web. Sphere MQ application failures § Batch (QSAM/VSAM/DB 2®) application failures 4 Helps you analyze failures when they occur or reanalyze them after the fact 4 Expands error messages and codes that apply to your failure with interactive reanalysis and includes a feature for using application-specific messages and codes to supplement those supplied by IBM 4 Creates a fault history file with an interactive display that helps you track and manage application failures 4 Starts automatically when an application fails, eliminating the need to recompile programs or change the job control language (JCL) 4 Integrates with Rational Developer for System z enables developers to diagnose application problems without changing user interface § For Fault Analyzer product information, see: http: //www-01. ibm. com/software/awdtools/faultanalyzer/ 39 ***See Notes

Fault Analyzer – Operational Process § The purpose of Fault Analyzer (FA) is to determine the cause of any ABENDs in an application program. § You do not have to read through application or system dumps, because the product has the ability to isolate the exact instruction that caused a particular error. 4 The Analysis engine provides automatic analysis when the application fails. 4 When an ABEND occurs, Fault Analyzer activates automatically, and then records details in a fault history file (see screen capture below) 4 Fault History files contain information about the faults analyzed by Fault Analyzer for z/OS. 4 Using Fault History files, re-analysis is available when real-time ABEND analysis isn’t enough (you can extract additional information in batch or interactive mode) ABEND happens Fault Analyzer exits are invoked Salient details (points of interest) written and stored 40

Fault Analyzer – Batch Job Outputs Fault Analyzer Is automatically invoked if your program ABENDs An IDIREPORT is produced that provides ABEND analysis information 41

Reviewing ABENDs in the Fault Analyzer Perspective § Besides the IDIREPORT, you may also wish to use RDz's Fault Analyzer perspective, to analyze and debug an ABEND situation. § To do that you'll need to: 1. Switch to the Fault Analyzer perspective in RDz 2. Specify the history file to connect with, that populates a Default ABEND view with failed online and batch job IDIREPORTs and other outputs 3. Learn how to navigate the Fault Analyzer perspective, to make use of the information contained therein § The next slides contain the step details 42

Fault Analyzer Perspective – 1 of 2 § Steps: 1. Open the Fault Analyzer Perspective 43

Fault Analyzer Perspective – 2 of 2 2. Enter: FAULTANL. <version>. HIST Ex. FAULTANL. V 11 R 1. HIST 44

Fault Analyzer Perspective – Overview Content Area shows Fault Analyzer IDIREPORT – detailed information on ABEND Fault Analyzer Explorer IDIREPORT Available in Outline view Additional Fault Analyzer views exist for other ABEND documentation including history data Can navigate using the Outline view 45

Fault Analyzer – Default List of History Files From the Default tab 4 Scroll up and down – to find a particular ABEND 4 Double-click an ABEND history file, to bring up its IDIREPORT and other stats 4 Sort the list by any of the column headings § Can also work with options of the Context Menu – with each ABEND entry 46

Fault Analyzer – Browse Report – S 0 CB Program line where the S 0 CB occurred The IDIREPORT presents a formatted, high-level summary of the points of interest necessary to debug ABEND conditions in your application. Click S 0 CB for an explanation of this ABEND Specifically, to answer the questions: • What happened? • What z/OS ABEND condition • Where did it happen? • what line or statement was executing when it happened • How did it happen? • What additional information is available for debugging purposes 47

Fault Analyzer – Browse Report – S 0 C 4 Here's an example of an IDIREPORT which shows that RPT-REC is "not addressable" …which is a euphemism for: "There's something really wrong with the file/FD/JCL DD connection" 48

Fault Analyzer – Browse Report – S 0 C 7 The IDIREPORT and supporting text varies from ABEND to ABEND depending on: • Type of ABEND • Information available at the time of the ABEND • Run-time platform Note: CUST-ACCT-BALANCE's value is shown in hex (as it represents invalid data) 49

Fault Analyzer – Browse Report – S 0 C 9 A S 0 C 9 is like a S 0 CB (divide by zero) except that a S 0 C 9 occurs because of an excessively large fixedpoint number obtained as the result of a decimal division operation 50

Fault Analyzer – Browse Report – S 0 C 1 Here is information on an IMS (DL/I) S 0 C 1 ABEND 51

Fault Analyzer – Browse Report – S 806 IDIDREPORT information on a module-not-found (S 806) ABEND Most likely SAM 2 is either a typo, or the program did not successfully compile/link into the Load Module 52

Open Source File to ABEND Instruction § Fault Analyzer can open the program source and position your cursor on the exact COBOL statement that failed. § Steps: 4 From the IDIREPORT – click the source line # 4 From the FA Invocation Options Page specify the PDS that contains the source module 53

Fault Analyzer – Lookup View for MVS and File Return Codes The Lookup view shows a great deal of background information on: • ABEND codes • DB 2 SQLCODE • IMS PCB Feedback • VSAM File Status etc. You can use the view, or double-click on the ABEND code shown in the IDIREPORT 54

Fault Analyzer Integration – Mini-Dump Reading 1 of 2 Fault Analyzer also provides for the reading/browsing of System Dump data – in Hex/Character format. Select an ABEND Scroll through the dump – Issue navigation commands: Show nnn, +nn, etc. 55

Fault Analyzer Integration – Mini-Dump Reading 2 of 2 You can assign analysis notes to the dump. 1. Right-click over the storage address 2. Add your note (click OK) 3. Your note becomes highlighted text inside the dump 56

Fault Analyzer – Organize ABEND History Views Define views which contain a set of history files. Display fault entries from all history files as if they were in one dataset. 1 3 2 57

Fault Analyzer – Fault History View The Default view shows either: • A History File • A view of a History file 58

Fault Analyzer – The TSO/ISPF Interface Available Line Commands A list of History ABENDs Batch and/or Online may be in the same list 59

Checkpoint 1. What RDz Perspective is used to view Fault Analyzer reports? 2. How does RDz obtain Fault Analyzer Information? Where does the information originate? 3. RDz Fault Analyzer interface has a Lookup View. What is it used for? 4. How can you jump to the program statement where the ABEND occurred with the RDz Fault Analyzer interface? 60

Summary Having completed this unit, you should now be able to: 4 Work with ABEND analysis reports created by IBM Fault Analyzer 4 Browse Report and Mini-Dump pages 4 Retrieve various Fault Analyzer view information 4 Browse and search ABEND codes 4 Use the various productivity features in the Fault Analyzer perspective 61

UNIT The RDz Workbench Topics: § § Analyzing Mainframe Abends ABEND Codes and Reasons Using IBM's Fault Analyzer Appendix 62

Workshop – Big Picture In this workshop you are going to: 1. Copy a several datasets from your instructor's z. Server. OS TSO ID to your ID § Details on the next slide 2. Modify JCL dataset names (and high-level qualifiers) to match your Sandbox ID 3. Compile a program named: HOSPCALC – which contains different types of COBOL ABENDs generated from invalid COBOL logic in different parts of the program 4. Run HOSPCALC (it will ABEND) – and from the Fault Analyzer IDIREPORT: § § Find the error in the COBOL source, and use the IDIREPORT ABEND analysis data to fix the error After you've solved the problem, you will save your edits, and re-compile HOSPCALC. Then run the program until you either get the next ABEND … or get a zero return code Compile Link Edit HOSPCALC Load Module Fix HOSPCALC COBOL error Analyze IDIREPORT 63

Workshop – Setup Steps – 1 of 2 Detailed steps: 1. In Remote Systems, create a new MVS Filter named: RDZFA for: DDS 0001. TEST. * 2. Expand the RDZA filter, and: § Copy DDS 0001. TEST. COBOL member: HOSPCALC to <EM 4 Zxx>. POT. COBOL PDS § Copy DDS 0001. TEST. JCL member: HOSPCALC to <EM 4 Zxx>. POT. JCL PDS § § Copy DDS 0001. TEST. JCL member: HOSPCRUN to <EM 4 Zxx>. POT. JCL PDS § 3. This is the compile JCL for HOSPCALC This is the run JCL, for HOSPCALC Right-click over DDS 0001. TEST. DATA and use Allocate Like to create a new PDS named: <HLQ>. TEST. DATA with the same dataset characteristics Example with ID: EM 4 Z 07 64

Workshop – Steps – 2 of 2 4. Expand your <HLQ>. TEST. JCL library: § Edit HOSPCALC. § § § Replace all occurrences of: DDS 0001 with your ID <HLQ> Modify the JOB card and save your edits Edit HOSPCRUN. § Replace all occurrences of: DDS 0001. TEST with § <HLQ> Modify the JOB card and save your edits 5. Submit your modified HOSPCALC JCL. When the job finishes, check the results in your JES My Jobs filter (it should compile successfully – but check that the LKED step ran) 6. Submit your modified HOSPCRUN JCL. The job should ABEND. In JES - My Jobs, view the IDIREPRT step, to see the specific system completion code and surrounding Fault Analyzer ABEND analysis information § Note that HOSPCALC contains four separate ABEND errors – that you will hit, one at a time: an 0 C 7, an 0 CB, an 0 C 4 and a combined 0 C 7/0 C 4 ABEND (Sounds strange. . . Essentially Fault Analyzer will find two different problems on one statement) § The comments and descriptions in the IDIREPRT combined with your COBOL knowledge should be sufficient to debug and fix these problems, but feel free to elicit help from your instructor if you're stuck. 7. Fix the COBOL error that caused the ABEND. Then repeat steps 5 and 6 until you find and fix all of the ABENDs in HOSPCALC. You're done when HOSPCRUN finishes with a zero return code. 65