Return to list of articles

This excerpt from the Spectrum Writer Reference Manual provides some tips that you may find helpful when making reports from input records that contain arrays.

Working with Arrays

When working in a low level language, programmers often process arrays by using index variables, program loops, etc. As a non-procedural, 4GL report writer, Spectrum Writer does not have "procedural" elements such as "go to" or looping instructions. Instead, nonprocedural methods are used to process arrays with Spectrum Writer.

Using Normalization to Process Arrays

Often, the best way to process an array with Spectrum Writer is to "normalize" the array-containing records. During normalization, Spectrum Writer turns each physical input record into one or more logical records. Each logical record corresponds to one "occurrence" of the array in the physical record. These logical records are all identical to the physical record, except for the part of the record that contains the first occurrence of the array. That part of the record is overlaid, successively, by the second, third, fourth, etc. occurrence of the array.

In effect, Spectrum Writer loops through the array for you, building logical records by moving, one at a time, each occurrence of the array into the first position. The result is that instead of a single record containing multiple occurrences of an array, you end up with multiple records that each contain a single occurrence of the array in a fixed location.

Figure 78 shows an example of a file that contains an array. As you can see in the Cobol record layout, the SALES-HISTORY record contains an array named SALE-ARRAY. The array holds information for up to six sales. The NUM-SLOTS field tells how many occurrences of the array are actually used in any particular record.

Figure 79 shows the same file after it has been normalized. As you can see, the normalized file has more records than the physical file. The normalized file contains from 1 to 6 logical records for each physical record in the original file.

The normalized file in Figure 79 contains exactly the same sales information as the physical file in Figure 78. The difference is this: in the normalized file, the information for all sales appears in one location — the first occurrence of the array (where the SALE-DATE and SALE-AMT fields are defined). Also, the unused occurrences of the array are eliminated from the normalized file (that is, no logical records are created for those occurrences).

Once you have normalized a file, it is quite easy to process it with Spectrum Writer. You simply ignore the array (beyond the first occurrence). In the normalized file, there is only one occurrence of relevant data in each record, always in the same location.

For example, assume that we want a report that simply lists all of the sales over $100 in the SALES-HISTORY file. If we used the physical file, we would have to examine up to six different amount fields in each record. We might also need to check the NUM-SLOTS field to see which amount fields were actually used in a given record (unless we knew that all unused fields contained zeros). That would require six COMPUTE statements. Then, we would need six COLUMNS statements to potentially print each of the six amount fields, plus an option to suppress any zero lines, and so on. (The details of this approach can be found in "How to Print a Variable Number of Lines Per Input Record" on page 249.)

On the other hand, if we let Spectrum Writer normalize the file, we can easily produce the report using just the single SALE-DATE and SALE-AMT fields, like this:

INCLUDEIF: SALE-AMT > 100
COLUMNS: NAME SALE-DATE SALE-AMT

Remember that every sale amount in the array of the original file now exists in the SALEAMT field of some logical record. We simply include those records where that amount is over $100.

Figure 80 shows a report that uses the above statements.

Note: A normalized file exists only as a temporary, logical file. During the run, the records are built, processed by Spectrum Writer, and then discarded. They are not actually written to a physical file.

The next section explains exactly how to use the NORMALIZE parm.

The NORMALIZE Parm

To normalize an array in an input file, Spectrum Writer must know three things about that array:

the starting column of the array
the total size of each occurrence of the array
how many occurrences there are

You supply this information by adding a NORMALIZE parm to the INPUT statement. The syntax of the NORMALIZE parm is:

NORMALIZE(normalize-field, occurs-expression [, ...] )

The normalize-field tells Spectrum Writer both the starting column of the array and the length of the array occurrences. Therefore, the normalize field must be a field that defines the entire first occurrence of the array that you want to normalize. In Figure 79, the normalize field is SALE-ARRAY. SALE-ARRAY is a 13-byte character field that includes both the SALE-DATE and SALE-AMT fields. Thus, it defines the entire first occurrence of the array.

Note that in that example we could not use SALE-DATE as the normalize field. Doing so would yield incorrect results because it does not define the entire first occurrence of the array. It only defines the first six bytes of the first occurrence.

Why is the normalize field’s length important? Because it tells Spectrum Writer how many bytes to move each time it builds a new logical record. It also tells Spectrum Writer where to start taking the new bytes from (namely, from the first byte following the normalize field).

The occurs-expression in the NORMALIZE parm tells Spectrum Writer how many occurrences of the array to process. It can be a constant numeric literal, the name of a numeric field, a numeric COMPUTE field, or any valid numerical expression.

Let’s look at an example of the NORMALIZE parm in an INPUT statement:

    INPUT: SALES-HISTORY
           NORMALIZE(SALE-ARRAY,  NUM-SLOTS)

The above statement tells Spectrum Writer to normalize the input records from the SALESHISTORY file. The first occurrence of the array being normalized is defined by SALE-ARRAY. The number of occurrences to be used from the array is contained in the NUM-SLOTS field.

Figure 80 shows a report that uses this statement.

As records are read from the SALES-HISTORY file, here is what happens.

First, the unchanged physical record is processed by Spectrum Writer as usual. (That is, the INCLUDEIF tests are performed on it, and, if included, its contents are formatted into a line of the report.) This physical record is considered the first logical record.

Next, a second logical record is created by moving the 13 bytes immediately following the SALE-ARRAY field into the SALE-ARRAY area of the record. Now, SALE-DATE contains the date from the second occurrence of the array, and SALE-AMT contains the amount from the second occurrence of the array. This logical record is then processed just as if it had been read directly from the input file. (The INCLUDEIF tests are performed on it, and, if included, its contents are formatted into a line of the report.)

Then, a third logical record is created by moving the next 13 bytes into the SALE-ARRAY area. Now, SALE-DATE and SALE-AMT contain the date and the amount from the third occurrence of the array. This third logical record is then processed just as if it had been read directly from the input file, and so on

The number of logical records produced for each physical record is determined by the value of the NUM-SLOTS field (in this example). If NUM-SLOTS is 1 (or less), only the original, physical record is processed. If NUM-SLOTS contains 2, then two logical records are processed (the original physical record, plus one additional record). If NUM-SLOTS contains 3, then three logical records are processed, and so on.

Note: You can also put the NORMALIZE parm directly in the FILE statement that defines a file. This is convenient when you know that you will always want to normalize a certain file. That way, you don’t need to include the NORMALIZE parm in the INPUT statement every time you produce a report from that file. (In that case, you can still prevent normalization for individual runs by adding the NONORMALIZE parm to the INPUT statement.)

File Definition Tips for Records with Arrays

The FIELD statements that define an array will be different, depending on whether or not you will normalize that array. This section explains the differences.

Consider the SALES-HISTORY file, which contains an array with information for up to six sales. For runs where we normalize that file, we would use the file definition statements shown in Figure 79 (page 239). But for runs where we do not normalize the file, we might prefer to use the file definition statements shown in Figure 35 (page 250).

If you are not normalizing a record, and you want to access data from an array, you must define each occurrence of the array as a separate field. That is because the only way to access each occurrence of the array is to refer to it specifically by its unique field name.

Thus, in Figure 35 (page 250) each occurrence of the array in the SALES-HISTORY file is defined individually (SALE-DATE-1, SALE-AMT-1, SALE-DATE-2, SALE-AMT-2, etc.) This file definition is for use in reports where the SALES-HISTORY file is not normalized.

However, if you are using normalization to process an array, it is not necessary to define all of the occurrences of the array. It is sufficient to define just the fields in the first occurrence of the array. (Plus, you may need to define one additional field that includes the entire first occurrence of the array, for use in the NORMALIZATION parm itself.) You do not need to define the other occurrences of the array. This is because, after normalization, all array data will be located in the first occurrence of the array (of some logical record). Thus, if we know that the SALES-HISTORY file will always be normalized, we can define it as we did in Figure 79 (page 239). It would not be necessary to define SALE-DATE-2, SALE-AMT-2, SALE-DATE-3 and so on.

Remember that if the array contains more than one field in each occurrence, you must also define a "high-level" field that defines the entire first occurrence of the array. This field will be needed in the NORMALIZATION parm. In the SALES-HISTORY file, we defined SALEARRAY as a high level field which includes both SALE-DATE and SALE-AMT. You should generally define the high-level field as a character field whose length is the length of the entire first occurrence of the array. That is, its length will be the sum of the lengths of each individual field. (Of course, if the array only consists of a single field, then no additional high-level field is needed. You can use the single field itself in the NORMALIZE parm.)

Caution: Cobol sometimes adds what are called "slack bytes" at the end of each array occurrence. This is done in order to align the next occurrence on a halfword or fullword boundary. This may happen if a field in your Cobol record layout uses the SYNCHRONIZED parm. Any such slack bytes must be included in the length of the high-level field you define.

After defining the high-level field, you can use a COLUMN( * – nnn) parm on the next FIELD statement to "back up" again to the beginning of the array. You can then start "redefining" the lower level fields in that same portion of the record.

Of course, you may want to define your array so that it can be used either with or without normalization. In that case, define all of the occurrences of the array, as well as one highlevel field for the entire first occurrence.

Normalizing Nested Arrays

Some records contain "arrays within arrays." Such arrays are also called "nested arrays." Spectrum Writer is able to normalize records containing any degree of nested arrays.

Specify one NORMALIZE parm for each array level. The first NORMALIZE parm defines the outermost array. The next NORMALIZE parm defines the next deeper array, and so on.

For example, consider this Cobol record layout:

    01 RECORD.
       05 EMPL-NAME PIC X(20).
       05 SALE-ARRAY OCCURS 10 TIMES.
          10 SALE-DATE PIC 9(6).
          10 SALE-CUSTOMER PIC X(10).
          10 SALE-PRODUCT-CODE OCCURS 5 TIMES PIC X(3).
       05 RECORD-STATUS PIC X(1).

The above record contains a nested array. The outer array (SALE-ARRAY) contains an inner array (SALE-PRODUCT-CODE). We could define this record to Spectrum Writer in the following way:

FIELD: EMPL-NAME LENGTH(20)
FIELD: SALE-ARRAY LENGTH(31)
FIELD: SALE-DATE COLUMN(* – 31) TYPE(YYMMDD)
FIELD: SALE-CUSTOMER LENGTH(10)
FIELD: SALE-PRODUCT-CODE LENGTH(3) *
FIELD: RECORD-STATUS COLUMN(SALE-ARRAY + 310) LENGTH(1)

Since we will be normalizing this file, it is sufficient to define only the first occurrence of each array. (This is because, once normalized, the relevant data in each logical record will be in that first occurrence location.)

As you can see, we first defined a field that includes the whole, first 31-byte occurrence of the outer array (SALE-ARRAY). We then backed up 31 bytes in order to define the individual fields within that occurrence. First we defined the 6-byte SALE-DATE field, then the 10-byte SALE-CUSTOMER field. The last field within the 31-byte SALE-ARRAY field is the SALEPRODUCT- CODE array (a total of 15 bytes). Again, it was only necessary to define the first occurrence of that array. And, since the SALE-PRODUCT-CODE array does not have multiple fields per occurrence, it was not necessary to define a higher-level field to define its entire first occurrence. The 3-byte SALE-PRODUCT-CODE field itself defines the entire first occurrence of that inner array.

After defining the first occurrence of each array, we defined the RECORD-STATUS field. We use the COLUMN(SALE-ARRAY + 310) parm to locate this field in the correct column. This COLUMN parm tells Spectrum Writer to locate the RECORD-STATUS field 310 bytes after the start of the SALE-ARRAY field. (Ten occurrences of the 31-byte SALE-ARRAY field is a total of 310 bytes.)

After defining our file as above, we can normalize it like this:

    INPUT: OUR-FILE
           NORMALIZE(SALE-ARRAY,  10)
           NORMALIZE(SALE-PRODUCT-CODE,  5)

As Spectrum Writer normalizes the input records, it first "loops through" the most deeply nested array (the one specified in the last NORMALIZE parm). That is the SALE-PRODUCT-CODE array in this example.

Thus, the first logical record, as always, is the unchanged physical record. The second logical record will have the second product code in SALEPRODUCT- CODE. The third logical record will have the third product code in SALEPRODUCT- CODE, and so on for the fourth and fifth logical records.

Then, having fully normalized the inner array for the first occurrence of the outer array, Spectrum Writer begins normalizing the outer array (SALE-ARRAY). Thus, for the sixth logical record, Spectrum Writer moves the 31 bytes following the SALE-ARRAY field into the SALE-ARRAY location. That is, it moves the second occurrence of the outer array into the first occurrence’s location. This 31-bytes includes the second occurrence of the SALE-DATE and SALE-CUSTOMER fields. It also includes the entire 15-byte SALE-PRODUCT-CODE array from the second occurrence of SALE-ARRAY. At this point, the SALE-PRODUCT-CODE field contains the first occurrence of that array (within the second occurrence of the outer array).

Using Cobol notation, we could say it contains SALE-PRODUCT-CODE (2, 1). After processing this logical record, Spectrum Writer continues to fully normalize the inner array. Thus, for the next logical records, Spectrum Writer moves the 2nd, 3rd, 4th and 5th occurrences of the inner array to SALE-PRODUCT-CODE.

After that, Spectrum Writer moves the third occurrence of the outer array (SALE-ARRAY) into the SALE-ARRAY field, and so on.

Each time one inner level array has been fully normalized, the next higher array level is adjusted and all lower levels are normalized all over again.

Spectrum Writer continues normalization in this manner until the last occurrence of the inner array has been normalized for the last occurrence of the outer array. Thus, each physical record in this example results in 50 logical input records (the 5 occurrences of the inner array times the 10 occurrences of the outer array).

Normalizing Multiple, Non-Nested Arrays

Some records contain multiple arrays that are not nested. That is, there may be two or more independent arrays in a record. For example, consider this Cobol record layout:

    01 RECORD.
       05 EMPL-NAME PIC X(20).
       05 SALE-DATE OCCURS 10 TIMES PIC 9(6).
       05 HIRE-DATE PIC 9(6).
       05 SALE-CUSTOMER OCCURS 10 TIMES PIC X(10).
       05 RECORD-STATUS PIC X(1).

In this record layout, both SALE-DATE and SALE-CUSTOMER are arrays. But they are not nested. We could define this record to Spectrum Writer in the following way:

FIELD: EMPL-NAME LENGTH(20)
FIELD: SALE-DATE TYPE(YYMMDD)
FIELD: HIRE-DATE COLUMN(*+54) TYPE(YYMMDD)
FIELD: SALE-CUSTOMER LENGTH(10)
FIELD: RECORD-STATUS COLUMN(*+90) LENGTH(1)

Note again that when normalizing a file, we only need to define the first occurrence of each array. Thus we defined just the first SALE-DATE field at the beginning of that array. Since this is an array of just a single field, we did not need to define a separate "high level" field. The SALE-DATE field itself defines the entire first occurrence of the array.

We used the COLUMN(*+54) parm on the HIRE-DATE field to skip over the other 9 undefined occurrences of the 6-byte sales date field. (We could also have specified COLUMN(SALEDATE + 60). Either method will properly locate the HIRE-DATE field.)

Similarly, we defined only the first occurrence of the SALE-CUSTOMER array. We used the COLUMN(*+90) parm on the RECORD-STATUS field to skip over the other nine undefined occurrences of the 10-byte customer field.

Now that we have defined the file, how do we normalize it? There are two different ways to normalize records that contain non-nested arrays. The method you choose will depend on the logical relationship between the data in the two arrays.

Often, the data in the two arrays will have a one-to-one relationship. That is, the first date in the SALE-DATE array will correspond to the first customer in the SALE-CUSTOMER array. The second date in the SALE-DATE array will correspond to the second customer in the SALECUSTOMER array, and so on.

If this is the case, you want to normalize the two arrays in parallel. You can think of it as "stepping through" both arrays in sync. To normalize two or more arrays in parallel, specify all of the arrays in a single NORMALIZE parm:

    INPUT: OUR-FILE
           NORMALIZE(SALE-DATE,  10,  SALE-CUSTOMER,  10)

When performing this normalization, the first logical record, as always, will be the unchanged physical record. (The first date is in SALE-DATE and the first customer is in SALE-CUSTOMER.) The second logical record will have the second date in SALE-DATE and the second customer in SALE-CUSTOMER. The third logical record will have the third date in SALE-DATE and the third customer in SALE-CUSTOMER, and so on. If you normalize the above file in this manner, each physical record will result in 10 logical records.

On the other hand, you may have two separate arrays in a record whose data is not related in the manner described above. In other words, all occurrences of the first array may apply to all occurrences of the second array. In that case, you would normalize them as if they were nested (even though they are not physically nested). That will cause one logical record to be created for every possible combination of items from the two arrays. To normalize in this manner, use two separate NORMALIZE parms (just as for true nested arrays):

     INPUT: OUR-FILE
            NORMALIZE(SALE-DATE,  10)
            NORMALIZE(SALE-CUSTOMER,  10)

When performing this normalization, the first logical record is, as always, the unchanged physical record. (The first date is in SALE-DATE and the first customer is in SALECUSTOMER.) The second logical record will retain the first date in SALE-DATE, but move the second customer to SALE-CUSTOMER. The third logical record again retains the first date in SALE-DATE and now has the third customer in SALE-CUSTOMER. Thus, the first ten logical records all have the first date in SALE-DATE, while the SALE-CUSTOMER array is normalized. Next (for the eleventh logical record), the second date is moved to SALE-DATE and SALECUSTOMER is re-initialized to contain the first customer. The next logical record has the second date and the second customer. The next one has the second date and the third customer, and so on.

When normalized in this way, each physical record results in 100 logical records. (Ten occurrences of the first array times the ten occurrences of the second array.)

You can specify the two NORMALIZE parms in either order (since the arrays are not physically nested). The only difference will be the order in which the logical records are built. Spectrum Writer always normalizes the last NORMALIZE parm first.

Normalizing only Certain Records

Some files contain more than one type of record. For example, a file may contain a combination of header records and detail records. Perhaps the detail records contain an array that must be normalized, while the header records do not contain that array. Or, the header records might contain an entirely different array in a different location.

For such files, you need conditional normalization. Spectrum Writer provides the NORMWHEN parm to perform conditional normalization.

When a NORMALIZE parm is present in an INPUT or READ statement, the default is for Spectrum Writer to normalize all of the records from that file. You can, however, put a NORMWHEN parm ahead of the NORMALIZE parm(s). In that case, the normalization is done only on records where the condition specified in the NORMWHEN parm is true. For example:

    INPUT: BATCH-FILE
           NORMWHEN(RECORD-TYPE = ’HDR’)
              NORMALIZE(STATUS-ARRAY,  5)
           NORMWHEN(RECORD-TYPE = ’DET’)
              NORMALIZE(CUSTOMER-ARRAY,  8)

The above statements tell Spectrum Writer to normalize the STATUS-ARRAY only for those physical records where the RECORD-TYPE field contains "HDR". And the CUSTOMER-ARRAY will be normalized only for those physical records where the RECORD-TYPE field contains "DET". Records with any other value in the RECORD-TYPE field will not be normalized at all. (That is, only the physical records themselves will be processed.)

Each NORMWHEN parm governs the NORMALIZE parm(s) that follow it (until the next NORMWHEN parm, if any). Spectrum Writer first tests the condition in the first NORMWHEN parm. If true, it performs the complete normalization specified in the following NORMALIZE parm(s). After that normalization, Spectrum Writer then tests the condition in the next NORMWHEN parm. If true, Spectrum Writer then performs the normalization specified by the following NORMALIZE parm(s), and so on.

If the conditions in multiple NORMWHEN parms are true, each of the corresponding normalizations will be performed on that record. When a record is normalized multiple times, the unchanged physical record is processed only one time (not one time per normalization).

Any valid conditional expression is allowed within the NORMWHEN parm.

If any NORMALIZE parms precede the first NORMWHEN parm, their normalization will be performed on every record.

Normalizing an Auxiliary Input File

If the records read from an auxiliary input file contain an array, you may want to normalize those records as well. Do that by adding the necessary NORMWHEN and/or NORMALIZE parms to your READ statement.

Remember that, by default, a READ statement only returns a single record (the first record whose key matches the READKEY value). Therefore to successfully normalize an auxiliary input file, you must also specify the MULTI parm in the READ statement. The MULTI parm tells Spectrum Writer to use all of the records from the file whose key matches the READKEY value. The MULTI parm allows all of the logical records created during the normalization process to be used in the report.

Normalization Errors

If Spectrum Writer detects erroneous normalization information in a record, it does not normalize the field in question for that record. (If normalization was requested for more than one field, Spectrum Writer will still attempt to perform the other normalization(s).)

Examples of normalization errors are:

an error occurs while trying to compute the "occurs" value. For example, a numeric field involved in the computation might contain a non-numeric value.
the occurs expression results in a value that is negative or zero.
the occurs expression results in a value that is too big. That is, the last occurrence of the array would be beyond the end of the record area.

When Spectrum Writer encounters any of these normalization errors, it prints a message in the control listing, along with a "dump" (hex listing) of the record in question. By default, only the first ten such normalization errors are printed. You can use the MAXNORMDUMP option (in an OPTIONS statement) to print more (or fewer) such messages. The syntax of the MAXNORMDUMP option is:

OPTIONS: MAXNORMDUMP(nnnnn/10)

By default, normalization error messages are treated as informational messages only. When a normalization error occurs, Spectrum Writer processes the physical record, and then skips the normalization in question for that record. If you want normalization errors to be treated as more serious errors, use the ONNORMERROR option (in an INPUT, READ or OPTIONS statement). The syntax of the ONNORMERROR option is:

ONNORMERROR(DEFAULT/WARNING/ERROR/STOP)

For example, to stop the entire run (with a "user ABEND") if a normalization error occurs, specify:

OPTION: ONNORMERROR(STOP)