TG2-AMENDMENT_EVENTDATE_FROM_VERBATIM

[iDigBioBot]

TestField	Value
GUID	6d0a0c10-5e4a-4759-b448-88932f399812
Label	AMENDMENT_EVENTDATE_FROM_VERBATIM
Description	Proposes an amendment to the value of dwc:eventDate from the content of dwc:verbatimEventDate.
TestType	Amendment
Darwin Core Class	dwc:Event
Information Elements ActedUpon	dwc:eventDate
Information Elements Consulted	dwc:verbatimEventDate
Expected Response	INTERNAL_PREREQUISITES_NOT_MET if dwc:eventDate is bdq:NotEmpty or the value of dwc:verbatimEventDate is bdq:Empty; FILLED_IN the value of dwc:eventDate if an unambiguous ISO 8601 date is interpreted from dwc:verbatimEventDate; otherwise NOT_AMENDED
Data Quality Dimension	Completeness
Term-Actions	EVENTDATE_FROM_VERBATIM
Parameter(s)
Source Authority
Specification Last Updated	2024-09-16
Examples	[dwc:eventDate="", dwc:verbatimEventDate="Friday 29th Oct. 2021": Response.status=FILLED_IN, Response.result=dwc:eventDate="2021-10-29", Response.comment="dwc:verbatimEventDate contains an interpretable value (assuming some external lookup thesauri)"]
	[dwc:eventDate="", dwc:verbatimEventDate="03/04/2020": Response.status=NOT_AMENDED, Response.result="", Response.comment="dwc:verbatimEventDate is ambiguous - could be either 3rd April or 4th March"]
Source	VertNet, FP, Kurator
References	ISO (2019) ISO 8601-1:2019(en) Date and time — Representations for information interchange — Part 1: Basic rules. https://www.iso.org/obp/ui/ Wikipedia (2020) ISO 8601. https://en.wikipedia.org/wiki/ISO_8601 Library of Congress (2019) Extended Date/Time Format (EDTF). https://www.loc.gov/standards/datetime/
Example Implementations (Mechanisms)	Kurator/FilteredPush event_date_qc Library DOI: 10.5281/zenodo.596795
Link to Specification Source Code	https://github.com/FilteredPush/event_date_qc/blob/v3.0.5/src/main/java/org/filteredpush/qc/date/DwCEventDQ.java#L334 For a minimum set of unit tests see: DwcEventDQTest, see also unit tests for underlying implementation in DateUtilsTest and DateUtilsTest
Notes	If the proposed eventDate is prior to 1918-02-14, the Response.comment will include a note that the "verbatimDate was assumed to be in the Gregorian calendar". When running the test, the original precision, e.g. year=1980, month=1 should be retained, e.g. dwc:eventDate should become 1980-01, not 1980-01-01/1980-01-31.

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet:
If dwc:eventTime has a value, this should also be included in the filled in dwc:eventDate

[tucotuco]

An issue needs to be resolved about what to do with ambiguous verbatimEventDate. Should we a) populate eventDate with a range that encompasses all possibilities in verbatimEventDate or b) not interpret the ambiguous verbatimEventDate.

[ArthurChapman]

An alternative @chicoreus is to change the Test Preqrequisites to add "unambiguously" and to read "The field dwc:eventDate is EMPTY and the field dwc:verbatimEventDate is not EMPTY and is unambiguously interpretable as an ISO 8601:2004(E) date"

[Tasilee]

Thanks @chicoreus and @ArthurChapman. I like Arthur's idea. What we have not formalized is the type of ASSERTIONs. I'm presuming we have the equivalent of one of for VALIDATION: COMPLIANT (PASS), NOT COMPLIANT (FAIL) and PREREQUISITES_NOTMET, and for AMENDMENT: RUN, PREREQUISITES_NOTMET...?

[chicoreus]

In Kurator (coming out of experiences in FilteredPush, we've been using the following as a result.status values for all forms of tests: NOT_RUN, AMBIGUOUS, INTERNAL_PREREQUISITES_NOT_MET, EXTERNAL_PREREQUISITES_NOT_MET,
RUN_HAS_RESULT. And then for Amendments: the following extensions for values of result.state: TRANSPOSED, CHANGED, FILLED_IN, NO_CHANGE. An important distinction for consumers of data quality reports is between FILLED_IN, where no value was present, but one has been proposed, and CHANGED, where a change has been proposed to an existing value.

Measures, as defined in the framework have a result.value of {some number}, COMPLETE, or NOT_COMPLETE.

Validations, as defined in the framework have result.value of COMPLIANT or NOT_COMPLIANT

The concept of a Problem that TG2 has described to TG1 for formalization in the framework would be the one that would have result.value in some set equivalent to PASS and FAIL, this has yet to be defined.

For an implementation, see: https://github.com/kurator-org/ffdq-api/tree/master/src/main/java/org/datakurator/ffdq/api

The verbatim values "1880", "1880s", "February 1880" are all unambiguously interpretable to the ISO format, though not to specific dates, and different implementers could easily interpret the meaning of "unambiguously interpretable" in different ways. One sense of ambiguity I am concerned with is "2/5/1925" where which value is month and which value is day has ambiguity to it. This value can (and I would argue should) be converted to an eventDate which makes the data fit for many of the core uses: "1925-02-05/1925-05-02", where the ISO date format isn't able to represent just the pair of dates 1925-02-05 or 1925-05-02, but can represent the possible interval into which the date falls. For consumers who bin data by year or decade, providing this value of eventDate makes the data fit for their purpose. For consumers who need data with a resolution to one day, then the data are not fit for use in either case. By casting the wider net of interpretation, we can make limited data fit for use by more consumers.

Some form of result.status marking ambiguity in the interpretation feels superior to not making assertions about potentially ambiguous data, where the assertion captures the range of ambiguity.

In addition, filtering data quality reports on a result.status of AMBIGUOUS is a good way to find approachable units of work (data cleanup projects) for curators of databases of record - if I filter a data quality report by AMENDMENT_EVENTDATE_FROM_VERBATIM where result.state = AMBIGUOUS, then I've located a set of records with a common problem to examine. Treating all of these as records in the report as uninterpretable would leave them grouped with other classes of problems, and wouldn't isolate an useful data cleanup project.

From both the perspective of research consumers of data and curators of databases of record, it is more advantageous to provide interpretations of ambiguous verbatim data, so long as the presence of ambiguity is marked in the data quality reports.

[Tasilee]

Thanks @chicoreus. The way I visualize the results of running the tests (in my usual simple way), is (currently) 94 additional values for each record that take an atomised form of one of TG1/TG2/etc terms

COMPLIANT
NOT COMPLIANT
PREREQUISITES NOT MET
FILLED IN
CHANGED
Value (MEASURE)
...
AMBIGUOUS?
...?

This means, like Darwin Core terms in the record, we have a test name (with link) and test result/value/status. The tests do have a suite of descriptive parameters that should adequately document the test and the meaning of the result (and dependencies etc).

Regarding your ambiguous dates, I would feel far more comfortable in flagging the ambiguity rather than complicating things by generating an amendment which suggests a possible date range.