Enhanced format file structure
- Variable Overflow that manages space within overflow blocks as containing true variable length records. This is unlike the Compatible file format which only stores in each overflow block the number of maximum length records that will fit.
- Significantly larger number of buffers per file can be allocated, offering greater I/O savings.
- Alternate Index Support
- Sysplex Record Level Sharing Support
- Single System Record Level Sharing Support
- Support of IBM hardware compression instruction
- Support of large devices (greater than 64K tracks per volume), which includes allowing up to 123 extents per volume.
- Support for EAV volumes (greater than 64k cylinders per volume)
- Support for Parallel Access Volumes
All records contained within the IAM Enhanced File Structure are treated internally as variable length records. Additionally, all files that are defined as 75 tracks or larger will be automatically eligible for the IAM data compression, which helps reduce DASD space and provides I/O performance benefits.
IAM files are self-defined, non-VSAM files, or VSAM Linear files if Pervasive Encryption is being used. The vast majority of the information about the file structure is contained within the file itself, rather than in the system catalog and VVDS, the way it is with VSAM clusters. The information kept within these system control areas, the system catalog, the VTOC, and the VVDS for DFSMS managed volumes is the same as for any non-VSAM file. Non-encrypted IAM files are treated as non-VSAM files by most DASD management products, including the FDR family of products.
An IAM file, as it appears on DASD, consists of a set of unique areas. Each area will be explained in further detail below. The basic structure is:
- File Definition and Structure Data (Two blocks)
- Prime Data Area
- Index to Prime Data Area
- Hardware Compression Dictionary (if applicable)
- Extended File Area
- Extended area index and contents description
- Extended Overflow and/or Extended PE blocks
File definition and structure data
The first block is initialized when an IAM file is defined with the basic file definition information, such as record length, key size, key offset, and so forth. With the IAM Alternate Index support, the information on related data sets (alternate indexes and paths) is also kept in the first block. After a file has been successfully loaded, the last I/O done to the file is to store the file structure data in the first block. Included is an indication that the file has been successfully loaded, and of course information about where the index begins, size of index, plus additional required data. The only updates done to this block after a file load are for statistical information that is written during file close processing, including information on number of inserts, deletes, and updates. The bulk of the information that is presented on the IAM Listcat report is taken from this control block. The statistical information and information about the extended file area that are kept in the first block may not be accurate, and do not have to be accurate as they are for informational purposes only. A listcat that is done while the file is open to any application program will not reflect the exact status of the file, in much the same way as a listcat on an open VSAM file will also not reflect the exact status and statistics for the file. The statistical information can become inaccurate if a system or address space fails while the file is opened for update processing and has not been successfully closed.
The second block contains information about the extended file area, including maximum size, and the location, by relative block, of where the information about the extended area is stored. This block is typically read during open processing and is updated immediately after a new DASD extent has been acquired, and during close processing. If the file has not been successfully closed, then open processing will detect any inconsistencies, and update this data with the correct information.
Prime data area
The Prime Data Area is built as the file is being loaded. This area contains the data records that were passed to IAM during the file load process. There are few embedded free spaces included within each block, called the Integrated Overflow area. This area is similar in concept to the VSAM CI free space, and the size is indicated by the CI% Free space parameter on the IDCAMS file DEFINE. Every loaded IAM file has a Prime Data Area, except for files that have been loaded with a single record.
Index to prime data area
This index is built during close processing for a file load. This index consists of the high key in each prime data block and may be in a compressed format. Once a file is loaded, this index structure is never changed, until the file is reorganized or reloaded.
Hardware compression dictionary
For datasets that are loaded using hardware compression, a copy of the compression dictionary is stored in the IAM file after the prime index. This is done even for dictionaries that already exist to ensure that IAM will be able to decompress the data in the file, even if the user subsequently changes the dictionary.
Extended file area
The Extended File Area consists of the data that is added to the file, either as inserted records or from updates that increased the size of data records, which could not fit into the Prime Data Area of the file. This area consists of blocks containing control information about the extended file area, and extended data blocks, which can be either Extended Overflow, or Prime Extension (PE). The control information for the Extended File area is based on the internal logical structure of the file and is not necessarily tied to actual DASD extents. This way, IAM files can have their extents merged and eliminated by products such as FDR/CPK without impacting the integrity of the IAM file structure or the data it contains.
Dynamic file expansion
The Extended File Area is acquired and formatted as needed. When there is a need to acquire an additional data block, for either Overflow or PE, a segment of the allocated and unused space is formatted. Normally, for batch processing, up to one cylinder will be formatted with empty blocks. For online processing, normally only one track will be preformatted at a time, to reduce the impact on response time. Blocks are then assigned as needed to either Extended Overflow, or PE. Once all of the allocated space is used, additional DASD space will be requested through the normal z/OS EOV service.
If an error condition occurs during the EOV processing, such as an X37 abend condition, it is captured by the IAM DCB ABEND exit to prevent the job from actually abending. The request requiring the additional DASD space is failed, with a file full logical error. The avoidance of the abend is done to be compatible with VSAM, which will not abend either. The user will see the IBM error messages relating to the error condition encountered and an IAMW13 File full error message.
When a new extent is acquired, additional Extended Area control blocks are formatted and written as necessary, and up to one track of empty blocks will also be formatted. The control information for the Extended File area consists of identifying the blocks that are assigned to Extended Overflow, and the blocks that are assigned to PE. Additionally, the high key for each assigned PE block is retained as the index for the PE area.
Extended PE blocks
As records are being added to the logical end of the file, defined as having keys higher than what have been previously loaded or added to the data set, PE blocks are assigned from the Extended File Area. Once a PE block is considered full, then the high key in that block is used as the index entry. Just as with the Prime Data blocks, Integrated Free space, or CI% free space, is left in each block. This will allow for records increasing in size, as well as for later record insertions. Once a block is assigned as PE, it will remain as a PE block until the file is reorganized or reloaded. A PE block is only able to hold records that fall into the established index key range. If records are deleted from the PE block, the free space is available for expansion of existing records within that block, or for new records added to the file within the established key range.
Extended overflow blocks
The IAM Extended Overflow area is a record-based overflow area. Extended Overflow blocks are used to handle records that are being inserted within the file when there is insufficient space within the Prime Data or Prime Extension block that the record would have been assigned to, based on the established index. Extended Overflow space will also be used when an updated data record increases in size, and there is insufficient room within the block that it currently resides in for the larger record. The format of the data in the Extended Overflow block is identical to the data in the prime blocks. With Variable Overflow enabled, IAM will fully utilize the space available within each block. When a record is deleted from Extended Overflow, the space it occupied is immediately available for reuse by any inserted or updated record, regardless of the key value. This eliminates the unusable lost space condition that can occur within VSAM files taking CI/CA splits, as records are deleted from certain key ranges, and new records are added in different key ranges.
The index to the Extended Overflow area is record based, that is each record in Extended Overflow has an entry in the index, consisting of the key and the block number of the Extended Overflow block containing the record. There will be an index to overflow for each of the prime data or extended PE blocks that have associated records in overflow. This enables IAM to use a compressed key structure for the overflow area, as well as potential reductions in CPU time to build and manage the overflow index when there are a very large number of records in the overflow area. The index is built when the file is opened, by reading all of the used Extended Overflow blocks, as indicated by the control information.