Skip to content

Issue #1292 mete grids from imod5#1293

Merged
JoerivanEngelen merged 22 commits into
issue_#1260_from_imod5_data_metaswapfrom
issue_#1292_mete_grids_from_imod5
Nov 27, 2024
Merged

Issue #1292 mete grids from imod5#1293
JoerivanEngelen merged 22 commits into
issue_#1260_from_imod5_data_metaswapfrom
issue_#1292_mete_grids_from_imod5

Conversation

@JoerivanEngelen

@JoerivanEngelen JoerivanEngelen commented Nov 15, 2024

Copy link
Copy Markdown
Contributor

Fixes #1292

Description

Context: MetaSWAP can map meteorological grids, which have a coarser grid but a very fine time resolution, to its svats.
The mete_grid.inp file contains paths to meteorological information, stored in ASCII grids, and there are separate mapping files "svat2precgrid.inp" and "svat2etrefgrid.inp". Nota bene: MetaSWAP has hardcoded filenames for packages, hence why I can search for "mete_grid.inp".

Adds the following:

  • A MeteoGridCopy class to copy "mete_grid.inp" files, to avoid having to read and write the crazy amount of meteo files that exist in existing databases.
  • A MeteoMapping.from_imod5_data method which looks for the first ascii grid in the mete_grid.inp file, and reads that to derive mappings from meteorological cell to metaswap grid cell.
  • Add Imod5DataDict type alias, which can be implemented for other from_imod5_data methods in the future.

Only thing I doubt now is that I naively copy mete_grid.inp, which works when the paths to ASCII grids are absolute, but not when they are relative. I don't think iMOD5 does this as well, so present databases all contain absolute paths. @HendrikKok do you foresee a usecase to support relative paths in MeteoGridCopy?

Checklist

  • Links to correct issue
  • Update changelog, if changes affect users
  • PR title starts with Issue #nr, e.g. Issue #737
  • Unit tests were added
  • If feature added: Added/extended example

@JoerivanEngelen JoerivanEngelen changed the base branch from master to issue_#1260_from_imod5_data_metaswap November 15, 2024 16:34
@JoerivanEngelen JoerivanEngelen marked this pull request as ready for review November 22, 2024 14:28

def find_in_file_list(filename: str, paths: list[str]) -> str:
for file in paths:
if filename == Path(file[0]).name.lower():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the zeroth index of a file return?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be frank, I copy+pasted that from the prototype drafted by @HendrikKok. It's worthy of an investigation, because it serves some explanation and might even point to something clumsy in what is done in open_projectfile_data.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a relict of how read_projectfile parses the data: A list for each line in the file, split into elements for each comma in the line. However, in the "EXTRA FILES" text block, there is only one element per line, causing the unnecessary nested list. I'm not sure if I want complicate the logic in read_projectfile with edge cases, but I could concatenate the lists in open_projectfile_data as there all data processing is done.

Comment thread imod/msw/meteo_mapping.py Outdated
mete_grid_path = Path(mete_grid_path)

f = open(mete_grid_path, "r")
lines = f.readlines()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of readlines you can use readline to get the first line

Furthermore i wiould suggest using a context to make sure the file is closed when you're done reading it

with open(mete_grid_path, 'r') as file:
  first_line = file.readline()
  meteo_filepath = Path(first_line.split(",")[column_nr].replace('"', ""))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oomph good one! Missed the context when opening the file 😱

@HendrikKok HendrikKok left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work so far. I have one change request based on my recent experience.

In response to your question: I don't think we have to support relative paths in the mete_grid-file. Did not see this in practice.

assert_equal(results["svat"], np.array([1, 2, 3, 4, 5, 6]))
assert_equal(results["row"], np.array([1, 2, 1, 2, 2, 2]))
assert_equal(results["column"], np.array([2, 2, 2, 2, 2, 1]))
assert_equal(results["column"], np.array([2, 2, 2, 2, 2, 3]))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was there a mistake in the old test?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess because you moved the coordinates in the tests quite a bit it seems?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the original test fixtures had increasing y-coords, whereas most or all usecases are decreasing. This updates that.

Comment thread imod/msw/meteo_mapping.py Outdated
mete_grid_path = Path(mete_grid_path)

with open(mete_grid_path, "r") as f:
first_line = f.readline()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MetaSWAP also supports constants as input (I found out recently). According to the manual there should be at least one meteo-file present. I would suggest a open_first_present_meteo_grid method.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented!

@HendrikKok

Copy link
Copy Markdown
Contributor

Fixes #1292

Description

Context: MetaSWAP can map meteorological grids, which have a coarser grid but a very fine time resolution, to its svats. The mete_grid.inp file contains paths to meteorological information, stored in ASCII grids, and there are separate mapping files "svat2precgrid.inp" and "svat2etrefgrid.inp". Nota bene: MetaSWAP has hardcoded filenames for packages, hence why I can search for "mete_grid.inp".

Adds the following:

  • A MeteoGridCopy class to copy "mete_grid.inp" files, to avoid having to read and write the crazy amount of meteo files that exist in existing databases.
  • A MeteoMapping.from_imod5_data method which looks for the first ascii grid in the mete_grid.inp file, and reads that to derive mappings from meteorological cell to metaswap grid cell.
  • Add Imod5DataDict type alias, which can be implemented for other from_imod5_data methods in the future.

Only thing I doubt now is that I naively copy mete_grid.inp, which works when the paths to ASCII grids are absolute, but not when they are relative. I don't think iMOD5 does this as well, so present databases all contain absolute paths. @HendrikKok do you foresee a usecase to support relative paths in MeteoGridCopy?

Checklist

  • Links to correct issue
  • Update changelog, if changes affect users
  • PR title starts with Issue #nr, e.g. Issue #737
  • Unit tests were added
  • If feature added: Added/extended example

@HendrikKok HendrikKok closed this Nov 25, 2024
@HendrikKok HendrikKok reopened this Nov 25, 2024
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
32.0% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

@JoerivanEngelen JoerivanEngelen merged commit 72c0255 into issue_#1260_from_imod5_data_metaswap Nov 27, 2024
@JoerivanEngelen JoerivanEngelen deleted the issue_#1292_mete_grids_from_imod5 branch November 27, 2024 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add from_imod5_data for meteo grids

3 participants