Handle case when database doesn't have exposure_id

[paulobarrera14]

In ZTF database there were no exposure ids so we turned that column into a empty column of strings. This allowed the precovery to index the hdf5 file but caused other issues when running the precovery search. When we looked for a specific asteroid all the "mjd_utc" fell on the same night. We need a solution for the datasets that don't have exposure ids.

df5["exposure_id"] = df5["mjd_utc"].apply(lambda x:str(x))
df5.to_hdf('ztf_observations_610_624.h5', key = 'data', mode='w', format='table', encoding = 'utf-8')

This is what we ended up doing so that "exposure_id" could have values based on "mjd.utc"

[ntellis]

To clarify, how much data was indexed for this? Just one night? Were there unique values of mjd_utc for separate exposures, they were just all falling on the same night?

Otherwise I don't see any problem with using the midpoint time of the exposure as a unique identifier for that exposure. I can't see a situation where there would be overlap there, unless there's some truncation on the mjd_utc field you are referring to.

[paulobarrera14]

This data was indexed over two weeks. Yes there were unique values of mjd_utc for that two week period (In the .hdf5 but not unique values in the indexed file).

Having the exposure_id column as an empty string caused some error in the indexing so that when I did a search with an extremely high tolerance all the mjd_utcs were 58364.1304861.

Joachim fixed this by adding the code I put above and re-indexing the database and wanted me to file this issue so that its documented/fixed later.

[moeyensj]

The issue is in this function:

precovery/precovery/sourcecatalog.py

Line 78 in c7ac7d5

def iterate_exposures(

The code assumes each exposure has a unique ID, so when we pass an empty string for all exposures then the assumption is that we only loaded in one exposure. The first observation time read is then the only observation time mapped to all indexed observations.

We need to support the LSST use case where each observation in each exposure will have a slightly varying time of observation, implementing that solution could conceivably also fix this issue. Regardless, we will need to come up with an in-house schema to assign exposure IDs when the dataset has none. Or we delegate it to the user.