Support moved compressed rows in SAS data files

[hpoettker]

Introduction

This PR adds support for "moved" compressed rows in SAS data files.

I'm not aware of a public description of this feature but I've investigated the hex dumps of SAS files that ReadStat currently cannot read, reverse-engineered the logic, and then validated the read data against exports of the data files as produced by SAS.

I'm not aware what the exact conditions are that trigger the "moving" of rows but these conditions seem to be necessary:

• the data files are written with compression enabled
• the data files are both input and output to a SAS procedure like e.g. a data step

The technical term "moved row" is something that I've made up on the basis of what I've observed. The naming is up to discussion but I'll drop the quotation marks in the text below.

Compression Types

The most widely known compression types that can be read in subheader pointers of SAS data files are

• 0x00, indicating no compression of the linked content
• 0x01, indicating that the linked content can be skipped
• 0x04, indicating that the linked content contains a compressed row

With the feature of moved rows, there are three additional compression types:

• 0x03, indicating the logical position of a row that is actually on a different page of the data file
• 0x06, indicating the physical position of a row that is referred to by a 0x03 compression type subheader pointer
• 0x0d, indicating subheader pointers that can be skipped similarly to 0x01 for currently unknown reasons

A speculative interpretation of this list of compression types is that the compression type in subheader pointers is actually a bitmap with the following meanings of the bits:

• 0x01 - any data that the subheader pointer may directly refer to shall be ignored
• 0x02 - the subheader pointer is related to rows that have been moved
• 0x04 - the subheader pointer directly refers to a compressed row
• 0x08 - unknown (but I've only observed it as part of the type 0x0d, which also matches the bit 0x01 and can thus be ignored)

Compression type 0x03

The typical subheader pointer in a SAS data file contains the following pieces of information:

• an offset that refers to a position within the same page
• a length that in combination with the offset delineates an area of the same page
• the compression type
• an additional flag

With compression type 0x03, the byte positions and lengths within the subheader pointer are the same but the meaning of the values is different:

• the first value represents a page number (with the numbers starting at 1)
• the second value represents a subheader pointer number (with numbers starting at 1)
• the compression type field contains the value 0x03
• the additional flag, whose meaning is not relevant here

The order of rows in a SAS data file is normally defined by the order in which a pass of the file encounters them. But when a subheader pointer with compression type 0x03 is encountered, this only defines the logical position of the row in the order of encounter while the actual data is on a different (and as far as I can tell later to be encountered) page.

For subheader pointers of compression type 0x03, the previous and the next pointer will usually refer to neighboring areas within the same page.

Compression type 0x06

The reference from a subheader pointer with compression type 0x03 is always to a subheader pointer on another page that has the compression type 0x06.

A subheader pointer with compression type 0x06 does not represent any logical position of a row in the usual order of encounter. But the compressed row that is at the phyical position that the pointer refers to can be read exactly like rows of compression type 0x04.

Compression type 0x0d

I don't have a good explanation for this compression type. But just skipping subheaders with this type as one does with compression type 0x01 leads to the correct result when comparing the exports of ReadStat with those of SAS itself.

Implementation alternative

The implementation proposed in this PR respects the difference between the logical and the physical order of rows in a SAS data file, and replicates the order in which SAS itself presents the rows of a data file.

An alternative implementation that would be more efficient but would loose the faithfulness to the logical order of rows would be to

• treat compression type 0x06 the same as 0x04
• ignore compression types 0x03 and 0x0d just like 0x01

For many use cases, this would be good enough.

Validation

The change proposed in this PR works as expected on the SAS data files that I have access to. But I can share neither the data files themselves nor the SAS code that produces them.

As stated in the introduction, I'm not aware of what the precise trigger for SAS is to move rows to another page. So I'm also unable to provide generic SAS code that would create a toy example of a SAS data file with moved rows.

I've implemented the proposed change with the goal to introduce minimal risk. It should not affect affect any SAS data file that ReadStat currently reads successfully and contains validations against all constraints that I've observed.

I'm opening this PR in the hope of either someone of the community stepping forward with supporting information on moved rows in SAS data files or a leap of faith on part of the maintainer.

If there is any kind of follow-up question on this proposed change, I'd be happy to engage.

[hpoettker]

Building a Reproducer

I've now managed to create a generic reproducing SAS data file with a moved row for which I can post the file itself and the SAS code to produce it.

The file features the compression codes 0x03 and 0x06 in its subheader pointers but unfortunately not 0x0d, which is SAS_COMPRESSION_MYSTERY in the PR. My current guess is that you could get 0x0d when already moved rows are moved again in some way but this would need to be investigated further.

The idea of the reproducer is to

• first produce a multi-page file with rather short string values in compressed rows,
• then perform a single-row update that adds a long hard-to-compress string value.

With the SAS code below on a Linux x64 host with SAS 9.0401M9, I consistently see the effect that the 2000th of 3000 rows is physically moved from page 5 to 7 where the file has 8 pages in total and enumerations are starting from 1. Logically, the row remains at position 2000.

The index on id created in the first step could be a necessary condition for this behavior as no moved row is created without it. A plausable explanation is that moving rows for SAS is the more efficient thing to do on single-row updates when an index allows SAS to only read/update the data file starting from page 5 and an index rewrite can be avoided.

options compress=binary reuse=no;

/* STEP 1: Create a large enough initial dataset with well-compressable strings. */
data mylib.mcve_moved_rows (
    label="Heavy master table for direct modification"
    index=(id /unique)
);
    length id 8;
    length category $20;
    length description $200;
    length payload_to_update $2000;

    do id = 1 to 3000;
        category = 'CATEGORY_' || left(put(id, 8.));
        description = 'This is the detailed description for record number ' ||
                      left(put(id, 8.)) || ', created on ' || put(today(), yymmdd10.) ||
                      '. We need this to add weight to the file.';

        /* Initialize our target column with highly compressible blanks */
        payload_to_update = ' ';

        output;
    end;

    keep id category description payload_to_update;
run;

/* STEP 2: Perform a direct, in-place update of row 2000 with non-compressable string. */
data mylib.mcve_moved_rows;
    length non_compressible_string $2000;

    if _N_ = 1 then do;
        non_compressible_string = '';
        do _i = 1 to 56;
            non_compressible_string = catx('', non_compressible_string, uuidgen());
        end;
    end;
    retain non_compressible_string;

    id = 2000;
    modify mylib.mcve_moved_rows key=id;
    payload_to_update = non_compressible_string;
    replace;

    stop;
run;

Hexdump Investigation

The header and page size of the reproducer zipped in mcve_moved_rows.zip are both 0x18000 (98,304).

When trying to read the reproducer with the current version of ReadStat an error is shown for page 5 (using the enumeration starting with 1 as SAS data files use internally):

$ ./readstat mcve_moved_rows.sas7bdat
Format: SAS data file (SAS7BDAT)
ReadStat: Error parsing page 4, bytes 491520-589823
Error processing mcve_moved_rows.sas7bdat: Invalid file, or file has unsupported features

The hexdump of the unsupported 24-bit subheader pointer on page 5 with compression type 0x03 together with the previous and subsequent pointer is:

$ hexdump -C -s 0x798b8 -n 72 mcve_moved_rows.sas7bdat
000798b8  0e eb 00 00 00 00 00 00  90 00 00 00 00 00 00 00
000798c8  04 01 00 00 00 00 00 00  07 00 00 00 00 00 00 00
000798d8  67 00 00 00 00 00 00 00  03 01 00 00 00 00 00 00
000798e8  7d ea 00 00 00 00 00 00  91 00 00 00 00 00 00 00
000798f8  04 01 00 00 00 00 00 00

In the middle, we see the subheader pointer with compression type 0x03 and the "offset" 0x07 and "length" 0x67, which SAS here re-purposes as a reference to subheader id 0x67 on page 7. It's also notable that the previous and subsequent pointers reference contiguous subheaders as the distance of the offsets 0xea7d and 0xeb0e matches the length of the subsequent subheader, i.e 0x91.

On page 7 we can see the subheader pointer with compression type 0x06 that is being referred to by the pointer above. Again with previous and subsequent pointer:

$ hexdump -C -s 0xa89a0 -n 72 mcve_moved_rows.sas7bdat
000a89a0  3d 46 01 00 00 00 00 00  90 00 00 00 00 00 00 00
000a89b0  04 01 00 00 00 00 00 00  08 3e 01 00 00 00 00 00
000a89c0  35 08 00 00 00 00 00 00  06 01 00 00 00 00 00 00
000a89d0  e8 09 00 00 00 00 00 00  20 34 01 00 00 00 00 00
000a89e0  01 01 00 00 00 00 00 00

Here all three pointers reference contiguous subheaders as the distance between the offets 0x13e08 and 0x1463d is exactly 0x835, and the distance between the offsets 0x9e8 and 0x13e08 is exactly 0x13420.

With the changes from this PR, the reproducer is being read correctly.

Next steps

Adding moved rows to the write feature of ReadStat doesn't seem useful to me. So an automated test should rather rely on a SAS data file committed to the repository as test resource.

The current reproducer is bit large for that with 800 KB. But I'm optimistic that a smaller reproducer can be found by tweaking parameters like the page size.

The PR #366 to support soft-deleted rows also contains the suggestion to use fixed resources for tests. If we find a way to merge that PR, I can further investigate how to add a test with a reasonably sized fixed resource for this one.