pvlib.iotools.read_panond for PAN files with Borland Pascal format?

[kurt-rhee]

Is your feature request related to a problem? Please describe.

• PAN files come in either a utf-8 or Borland Pascal format
• Currently read_panond works for utf-8 format but not Borland Pascal
• Is there any interest in a Real48 extension to the read_panond function that is in iotools?

Describe the solution you'd like

• A function which would read the Real48 format PAN file into a python dictionary

Describe alternatives you've considered

• We could not support the Borland Pascal format

Additional context

• I don't think that there are any papers that we could reference, but there is this CASSYS implementation: https://github.com/CanadianSolar/CASSYS/blob/b5487bb4e9e77174c805d64e3c960c46d357b7e2/CASSYS%20Interface/DatabaseImportModule.vba#L4

[mikofski]

It it’s Borland pascal. Also doesn’t Frederic Rivollier repo (https://github.com/frivollier/pvsyst_tools) also convert these, tho might be similar or same as CASSYS since he was also Canadian solar / recurrent.

[kurt-rhee]

I also took a look at that library and couldn't find any mention of the Borland Pascal format, editing the title/description! Thanks Mark

[cwhanse]

Are the Pascal files text or binary?

[kurt-rhee]

They are binary

[kandersolar]

Is the binary format still relevant? I had the impression that it was phased out in favor of the text format a while ago, but maybe that's wrong.

From https://www.pvsyst.com/help-pvsyst7/format_of_pvsyst_files.htm:

Up to PVsyst version 6.39, all PVsyst files (project, variant, meteo, etc.) were saved in binary format and it was not possible to read or edit those files.
Starting with PVsyst version 6.40, the format of all PVsyst files has changed to text. This new format will simplify PVsyst upgrades.
Starting with PVsyst version 6.60, the format of PVsyst project version (VCi) has been improved.
Starting with PVsyst version 6.80, the encoding of all PVsyst files (*.PRJ, *.VCi, *.PAN, *.OND, etc.) has changed to UTF-8 to support international characters.
For each of these evolutions, the new formats are incompatible with older versions. This is the reason why the corresponding workspaces have different names (PVsyst6_Data, PVsyst640_Data, PVsyst660_Data, PVsyst680_Data, PVsyst7.0_Data).

For context, PVsyst 6.40 was released Jan 2016.

[cwhanse]

They are binary

Ugh. I'm guessing you would have to know the data structure and types in advance, to be able to read the file with python.

[kurt-rhee]

@kandersolar They are definitely less relevant than the newer text format. My use case is for modeling operating plants that were installed a long time ago.

I'm attempting to write a translation for Proximal now and would be happy to contribute it to pvlib if you all think it would be interesting/useful to have for the community.

[echedey-ls]

I'm attempting to write a translation for Proximal now and would be happy to contribute it to pvlib if you all think it would be interesting/useful to have for the community.

Was about to state just that. There surely is the possibility of reverse engineering them. Does PVsyst provide a translator for them? That trims the logic behind the file format.

[cwhanse]

My concern is about maintaining it

[echedey-ls]

My concern is about maintaining it

Fair enough. I guess it depends on how many versions of PAN files there are. And even if it turns out that it's overcomplicated, a public gist may be of help for other people in the same situation as Kurt. So I wouldn't stop that until I see a PR ;)

May be of interest, back in 2014 they were open to release the format! https://forum.pvsyst.com/topic/430-pan-file-format/ (4th comment)

[kurt-rhee]

@echedey-ls I'm happy to post a gist of it on my personal github instead if that reduces the maintenance burden on pvlib!

[echedey-ls]

@kurt-rhee I'd go ahead with a PR, and if it seems too unmaintanable, then switch to the Gist solution. Whatever is the case, I find valuable that it gets some visibility in pvlib. There can be more people in your same situation.

[kurt-rhee]

@echedey-ls Here is how it looks in Proximal's codebase, if it is still of interest i'll remove the type annotations and open a PR

# --- Constants ---
SEMICOLON_MARKER = 0x3B
DOT_MARKER = 0x09
DOUBLE_DOT_MARKER = 0x0A
FORWARD_SLASH_MARKER = 0x2F
CR_MARKER = 0x0D  # Carriage Return
VERTICAL_BAR_MARKER = 0xA6


# --- Supporting Functions ---
def _read48_to_float(*, real48: bytes) -> float:
    """
    Converts a 6-byte Delphi Real48 encoded value to a standard Python float.

    The format consists of:
    - 1 byte: Exponent (offset by 129)
    - 5 bytes: Mantissa, with the last bit of the 5th byte as the sign bit.
    """
    if not real48 or len(real48) != 6 or real48[0] == 0:
        return 0.0

    # The exponent is the first byte, with an offset of 129
    exponent = float(real48[0] - 129)

    mantissa = 0.0

    # Process the first 4 bytes of the mantissa
    # The division by 256 (or multiplication by 0.00390625) shifts the bytes
    for i in range(4, 0, -1):
        mantissa += real48[i]
        mantissa /= 256.0

    # Process the 5th byte of the mantissa
    mantissa += real48[5] & 0x7F  # Use only the first 7 bits
    mantissa /= 128.0  # equivalent to * 0.0078125
    mantissa += 1.0

    # Check the sign bit (the last bit of the 6th byte)
    if (real48[5] & 0x80) == 0x80:
        mantissa = -mantissa

    # Final calculation using the exponent
    return mantissa * (2.0**exponent)


def _find_marker_index(*, marker: int, start_index: int, byte_array: bytes) -> int:
    """
    Finds the index of the first occurrence of a hex marker after a start index.
    Returns the index right after the marker.
    """
    # bytearray.find is more efficient than a manual loop
    found_index = byte_array.find(bytes([marker]), start_index)
    if found_index != -1:
        return found_index + 1
    if found_index is None:
        raise ValueError(f"Marker {marker} not found in byte array")
    return found_index


def _get_param_index(*, start_index: int, offset_num: int) -> int:
    """Calculates the start index of a Real48 parameter."""
    return start_index + 6 * offset_num


def _extract_byte_parameters(
    *, byte_array: bytes, start_index: int, num_bytes: int
) -> bytes:
    """
    This function extracts bytes that form a single parameter from the original byte array
    (contains the bytes from the whole file) into a smaller byte array that it returns.
    """
    # Check bounds to avoid index errors
    if start_index + num_bytes > len(byte_array):
        raise IndexError(
            f"Not enough bytes: need {num_bytes} bytes starting at {start_index}"
        )

    # Extract the specified number of bytes starting at start_index
    param_byte_sequence = byte_array[start_index : start_index + num_bytes]

    return param_byte_sequence



def read_pan_binary(*, file_content: bytes) -> dict:
    """
    Parses a binary .PAN file and returns its contents as a dictionary.

    Args:
        pan_file_path: The full path to the .PAN file.

    Returns:
        A dictionary containing the parsed data from the PAN file.
    """
    data: dict[str, Any] = {}
    byte_array = file_content
    if not byte_array:
        raise ValueError("File is empty")

    # --- Find start indices for string parameters ---
    try:
        manu_start_index = _find_marker_index(
            marker=SEMICOLON_MARKER, start_index=0, byte_array=byte_array
        )
        panel_start_index = _find_marker_index(
            marker=DOT_MARKER, start_index=0, byte_array=byte_array
        )
        source_start_index = _find_marker_index(
            marker=DOT_MARKER, start_index=panel_start_index, byte_array=byte_array
        )
        version_start_index = _find_marker_index(
            marker=DOUBLE_DOT_MARKER,
            start_index=source_start_index,
            byte_array=byte_array,
        )
        version_end_index = _find_marker_index(
            marker=SEMICOLON_MARKER,
            start_index=version_start_index,
            byte_array=byte_array,
        )
        year_start_index = _find_marker_index(
            marker=SEMICOLON_MARKER,
            start_index=version_end_index,
            byte_array=byte_array,
        )
        technology_start_index = _find_marker_index(
            marker=DOUBLE_DOT_MARKER,
            start_index=year_start_index,
            byte_array=byte_array,
        )
        cells_in_series_start_index = _find_marker_index(
            marker=SEMICOLON_MARKER,
            start_index=technology_start_index,
            byte_array=byte_array,
        )
        cells_in_parallel_start_index = _find_marker_index(
            marker=SEMICOLON_MARKER,
            start_index=cells_in_series_start_index,
            byte_array=byte_array,
        )
        bypass_diodes_start_index = _find_marker_index(
            marker=SEMICOLON_MARKER,
            start_index=cells_in_parallel_start_index,
            byte_array=byte_array,
        )

        # --- Find start of Real48 encoded data ---
        cr_counter = 0
        real48_start_index = 0
        for i, byte in enumerate(byte_array):
            if byte == CR_MARKER:
                cr_counter += 1
            if cr_counter == 3:
                real48_start_index = i + 2  # Skip <CR><LF>
                break

        if real48_start_index == 0:
            return {"error": "Could not find start of Real48 data block."}

        # --- Extract string parameters ---
        # Note: latin-1 is used as it can decode any byte value without error
        data["Manufacturer"] = (
            byte_array[manu_start_index : panel_start_index - 1]
            .decode("latin-1")
            .strip()
        )
        data["Model"] = (
            byte_array[panel_start_index : source_start_index - 1]
            .decode("latin-1")
            .strip()
        )
        data["Source"] = (
            byte_array[source_start_index : version_start_index - 4]
            .decode("latin-1")
            .strip()
        )
        data["Version"] = (
            byte_array[version_start_index : version_end_index - 2]
            .decode("latin-1")
            .replace("Version", "PVsyst")
            .strip()
        )
        data["Year"] = (
            byte_array[year_start_index : year_start_index + 4]
            .decode("latin-1")
            .strip()
        )
        data["Technology"] = (
            byte_array[technology_start_index : cells_in_series_start_index - 1]
            .decode("latin-1")
            .strip()
        )
        data["Cells_In_Series"] = (
            byte_array[cells_in_series_start_index : cells_in_parallel_start_index - 1]
            .decode("latin-1")
            .strip()
        )
        data["Cells_In_Parallel"] = (
            byte_array[cells_in_parallel_start_index : bypass_diodes_start_index - 1]
            .decode("latin-1")
            .strip()
        )

        # --- Parse Real48 encoded parameters ---
        param_map = {
            "PNom": 0,
            "VMax": 1,
            "Tolerance": 2,
            "AreaM": 3,
            "CellArea": 4,
            "GRef": 5,
            "TRef": 6,
            "Isc": 8,
            "muISC": 9,
            "Voc": 10,
            "muVocSpec": 11,
            "Imp": 12,
            "Vmp": 13,
            "BypassDiodeVoltage": 14,
            "RShunt": 17,
            "RSerie": 18,
            "RShunt_0": 23,
            "RShunt_exp": 24,
            "muPmp": 25,
        }

        for name, offset in param_map.items():
            start = _get_param_index(start_index=real48_start_index, offset_num=offset)
            end = start + 6
            param_bytes = byte_array[start:end]
            value = _read48_to_float(real48=param_bytes)
            if name == "Tolerance":
                value *= 100  # Convert to percentage
                if value > 100:
                    value = 0.0
            data[name] = value

        # --- Check for and Parse IAM Profile ---
        dot_counter = 0
        iam_start_index = 0
        dot_position = data["Version"].find(".")
        major_version = int(data["Version"][dot_position - 1 : dot_position])
        if major_version < 6:
            for i in range(real48_start_index + 170, len(byte_array)):
                if byte_array[i] == DOT_MARKER:
                    dot_counter += 1
                if dot_counter == 2:
                    iam_start_index = i + 4
                    break

        if iam_start_index > 0:
            data["IAMProfile"] = _extract_iam_profile(
                start_index=iam_start_index, byte_array=byte_array
            )

    except (IndexError, TypeError, struct.error) as e:
        return {"error": f"Failed to parse binary PAN file: {e}"}

    return data

[cwhanse]

@kurt-rhee do you have a sample binary file that will serve for testing?