Added sample fixture and multiline parser

[agstephens]

This fix includes two things:

1. Multi-line parser.

Where there are attributes (variable or global) that include arrays, then the lines end in a comma. The parser now ensures these are parsed correctly into a tuple rather than raising an exception.

A test has been added for this in: tests/test_readers.py

2. Sample fixtures - for regression testing

One danger is that new changes to the cdl.readers classes will modify how the content is parsed into checksit. If that happens unintentionally then the integrity of the whole process is damaged.

I have captured some example parsed content into a python shelve, within tests/sample-fixture. This was created from the example files in: tests/sample-fixture/sample-ncs/ using the script: tests/sample-fixture/make-sample-fixture.py.

When we make changes to the readers, the tests in tests/test_sample_data_matches.py will help to ensure that the readers have not been compromised.

[joshua-hampton]

I'm getting an error running pytest on test_readers.py

(checksit_dev_env) (base) [earjham@rsg1-202206011551 checksit]$ pytest tests/test_readers.py 
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.7.4, pytest-4.6.5, py-1.11.0, pluggy-0.13.1
rootdir: /home/users/earjham/checksit, inifile: setup.cfg
collected 2 items                                                                                                                                                                                                 

tests/test_readers.py .F                                                                                                                                                                                    [100%]

==================================================================================================== FAILURES =====================================================================================================
_______________________________________________________________________________________ test_cdl_reader_multiline_parser_2 ________________________________________________________________________________________

self = <checksit.readers.cdl.CDLParser object at 0x7fbfc0ef06d0>, value = '"0B, 1B'

    def _safe_parse_value(self, value):
        if value in ("NaN", "NaNf", "UNLIMITED"):
            value = f'"{value}"'
    
        try:
>           return eval(value)
E             File "<string>", line 1
E               "0B, 1B
E                     ^
E           SyntaxError: EOL while scanning string literal

checksit/readers/cdl.py:107: SyntaxError

During handling of the above exception, another exception occurred:

    def test_cdl_reader_multiline_parser_2():
        cci_file = os.path.join(TESTDATA_DIR, "esacci/ESACCI-GHG-L2-CO2-GOSAT2-SRFP-20191231-fv2.cdl")
>       resp = read_cdl(cci_file)

tests/test_readers.py:32: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
checksit/readers/cdl.py:195: in read
    return CDLParser(fpath, verbose=verbose)
checksit/readers/cdl.py:20: in __init__
    self._parse(inpt)
checksit/readers/cdl.py:49: in _parse
    self.variables, self.global_attrs = self._split_vars_globals(sections[1])
checksit/readers/cdl.py:92: in _split_vars_globals
    return self._construct_variables(variables), self._ordered_dict(global_attrs)
checksit/readers/cdl.py:137: in _construct_variables
    current[key] = self._safe_parse_value(value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <checksit.readers.cdl.CDLParser object at 0x7fbfc0ef06d0>, value = '"0B, 1B'

    def _safe_parse_value(self, value):
        if value in ("NaN", "NaNf", "UNLIMITED"):
            value = f'"{value}"'
    
        try:
            return eval(value)
        except:
            # Remove datatype suffixes and parse as list if commas are in value
>           return eval(", ".join([part.strip().rstrip("bBcCfFiIlLsS") for part in value.split(",")]))
E             File "<string>", line 1
E               "0, 1
E                   ^
E           SyntaxError: EOL while scanning string literal

checksit/readers/cdl.py:110: SyntaxError
======================================================================================= 1 failed, 1 passed in 0.09 seconds ========================================================================================

[agstephens]

Hi @joshua-hampton, that's a good point. I actually made the second test but it does correctly fail at the moment. I'll mark it to skip for now.

[agstephens]

@joshua-hampton please re-pull the change and check it is now set to accept the failure for that test.

[joshua-hampton]

That test now successfully fails with grace, happy to approve.