Public dataset for testing

[eightysteele]

@tucotuco, @laurarussell: CSV file with Darwin Core data for public testing that's representative of real world data.

[tucotuco]

@eightysteele Laura will give you some "unclean" data for this. Do you want to start loading up real data in real collections as well? Hold off? What are the immediate goals on the data store side?

[eightysteele]

For now let's just upload our data sets (public and internal) to the VertNet collection on Google Docs: http://goo.gl/xSMka

[tucotuco]

Clarification. @laurarussell Please put the uncleaned data set as a csv with a Darwin Core header into the VertNet/Dev/Data collection on Google Docs, located at the link in the previous comment. Call it Test_[InstitutionCode]_[CollectionCode].csv, substituting the actual InstitutionCode and CollectionCode into the filename and omitting the square brackets. We will not assemble any other data sets at this time.

[laurarussell]

Added test_NYSM_ZM_data.csv and test_NYSM_ZM_README.txt to the data folder on Google Docs. I'll continue to add more.

[eightysteele]

+1

[eightysteele]

@laurarussell: Actually, for test data sets right now, can we add a column for occurrenceid?

[laurarussell]

@tucotuco is it okay to just rename the CatalogNumberNumeric field to occurrenceid? Otherwise I can try and add it via Excel, but I've been very careful not to mess with the structure of this file so as not to lose the line feeds.

[tucotuco]

In this case yes, we can. If they are not unique, that's actually a good part of the test. @eightysteele This is actually a mis-use of the Darwin Core term occurrenceID, which is supposed to be a persistent resolvable GUID, and the reason GBIF called it the sourceID in the IPT. We may want to add similar logic. That is, if there is an occurrenceID in the source, then use it not only as the dwc:occurrenceID, but also as a unique identifier within the collection, or sourceID.

[eightysteele]

@tucotuco: See issue #1 for background on our thinking about sourceid. Basically any column in the CSV can be specified by the publisher to be the sourceid. Of course, if sourceid is a column they don't have to specify it. Using occurrenceid is just a short term hack until #1 is complete.

[laurarussell]

@eightysteele @tucotuco - uploaded a new version with CatalogNumberNumerica now as occurrenceID. (Didn't check yet to see if there are any dups in there though.)

[laurarussell]

whoops, didn't mean to close issue. I thought comment and close meant it would close my tab. Sorry :-(

[eightysteele]

:)