Merge pull request NOAA-ORR-ERD#55 from kthyng/agg_fix

Agg fix

Author: kthyng

docs/aggregations.md

@@ -34,7 +34,7 @@ As of October 2022, developmental versions of LSOFS and LOOFS have started to re
 
 ##### General
 
-Usually, nowcast and forecast files are created four times a day, and output is hourly in individual files. So, each update generates 6 nowcast files and 48 forecast files (for a 48 hour forecast; the forecast length varies by model). The update cycle time will be the last model output timestep in the nowcast files and the first timestep in the forecast files.
+Usually, nowcast and forecast files are created four times a day, and output is hourly in individual files. (WCOFS is updated once a day which changes these details for that model.) So, each update generates 6 nowcast files and 48 forecast files (for a 48 hour forecast; the forecast length varies by model). The update cycle time will be the last model output timestep in the nowcast files and the first timestep in the forecast files.
 
 Example filenames from one update cycle (`20141027.t15z`):
 
@@ -103,11 +103,11 @@ The datetimes associated with a given NOAA OFS file is not obvious from the file
 
 #### General
 
-The ``n006`` file for timing cycle ``t00z`` is at midnight of the day listed in the filename. Files ``n000`` to ``n005`` for timing cycle ``t00z`` count backward in time from there. Forecast files do not have the 6 hour shift backward. The hour in the timing cycle should be added to this convention. Datetime translations are given in the following table for sample files.
+For most models, the ``n006`` file for timing cycle ``t00z`` is at midnight of the day listed in the filename. Files ``n000`` to ``n005`` for timing cycle ``t00z`` count backward in time from there. Forecast files do not have the 6 hour shift backward. The hour in the timing cycle should be added to this convention. Datetime translations are given in the following table for sample files. These are for a filename with the pattern `nos.MODELNAME.fields.[n|f]HHH.YYYYMMDD.tCCz.nc`. Note that the WCOFS model is distinct in that it updates only once a day and so has nowcast files up to ``n024`` and you need to subtract 24 hours from the formula instead of 6.
 
 The formula are:
 
-- Nowcast files: time shift from midnight on date listed = CC + HHH - 6
+- Nowcast files: time shift from midnight on date listed = CC + HHH - update period in hours (6 for most, 24 for WCOFS)
 - Forecast files: time shift from midnight on date listed = CC + HHH
 
 <details>

docs/whats_new.rst

@@ -1,6 +1,11 @@
 :mod:`What's New`
 ----------------------------
 
+v0.7.0 (March 17, 2023)
+=======================
+* The sorting in filedates2df was slightly wrong and would not consistently return the desired nowcast file over the forecast file. Seems to be correct now.
+* WCOFS has a different update frequency which leads to a different conversion from filename to datetime. This is now included in file2dt.
+
 v0.6.0 (February 17, 2023)
 ==========================
 * Updated docs.

model_catalogs/tests/test_catalogs.py

@@ -535,7 +535,7 @@ def test_filedates2df():
         "nos.creofs.fields.n003.20220920.t15z.nc",
         "nos.creofs.fields.n004.20220920.t15z.nc",
         "nos.creofs.fields.n005.20220920.t15z.nc",
-        "nos.creofs.fields.f000.20220920.t15z.nc",
+        "nos.creofs.fields.n006.20220920.t15z.nc",
         "nos.creofs.fields.f001.20220920.t15z.nc",
         "nos.creofs.fields.f002.20220920.t15z.nc",
         "nos.creofs.fields.f003.20220920.t15z.nc",

model_catalogs/utils.py

@@ -103,16 +103,22 @@ def file2dt(filename):
     # Main style of NOAA OFS files, 1 file per time step
     elif fnmatch.fnmatch(filename, "*.n???.*") or fnmatch.fnmatch(filename, "*.f???.*"):
 
+        # number of hours in repeat cycle for forecast
+        if "wcofs" in filename:
+            tshift = 24
+        else:
+            tshift = 6
+
         # pull hours from filename
         regex = re.compile(".[n,f][0-9]{3}.")
         hour = int(regex.findall(filename)[0][2:-1])
 
         # calculate hours
         dt = cycle + hour
 
-        # if nowcast file, subtract 6 hours
+        # if nowcast file, subtract dt hours
         if fnmatch.fnmatch(filename, "*.n???.*"):
-            dt -= 6
+            dt -= tshift
 
         # construct datetime. dt might be negative.
         date += pd.Timedelta(f"{dt} hours")
@@ -323,10 +329,12 @@ def filedates2df(filelocs):
     df = pd.DataFrame(index=filedates, data={"filenames": filenames})
 
     # Sort resulting df by filenames and then by index which is the datetime of each file
-    df = df.sort_values(axis="index", by="filenames").sort_index()
+    df = df.reset_index().sort_values(by=["index", "filenames"]).set_index("index")
+
+    # df = df.sort_values(axis="index", by="filenames").sort_index()
 
     # remove rows if index is duplicated, sorting makes it so nowcast files are kept
-    df = df[~df.index.duplicated(keep="first")]
+    df = df[~df.index.duplicated(keep="last")]
 
     return df