Add lunar DEM file to remote server

[willschlitzer]

Per a discussion at the forum, I'm hoping to add a lunar topographic relief grid to the GMT remote server.

[PaulWessel]

Great. Just a heads up: I am actually trying to do some of the legwork for you and will do a PR soon to add the moon recipe. I thought I at least run in through a few resolutions before adding it.

[willschlitzer]

Thanks @PaulWessel! I'll hold off any work until your PR.

[PaulWessel]

PR safely updated and seems to work well.Unless you are on a mac with lots of RAM it would be good to see how well the process runs for you. Expect many hours.

[willschlitzer]

Currently running the script and it's converting the TIFF to a netCDF. You weren't joking about the RAM requirements; currently using 24 out of 32 GB. I'm on an Ubuntu 20.04 gaming desktop with an 8th gen i7, so I'm hoping it won't take too long. I did run into an issue the first time I ran the script; it stopped downloading after 4.6 GB and I received curl: (18) transfer closed with 3917136203 bytes remaining to read, so I'm not sure if that was a connection issue or something else. I deleted the downloaded portion and it seems to have completed the download the second time around!

[PaulWessel]

Interesting with curl doing that. Probably a glitch in the matrix somewhere as we say. If the downsampler step works well you will find the folder moon/moon_relief inside the staging folder and it will have all the grids at various resolutions. If you then run the tiler script then all the 5 min and higher resolutions will be split into tiles so there will then be directories with JP2 tiles inside that folder while the original global grids will be moved up to stain/tiled as safekeeping (we do not use those again or store them on the server).

[willschlitzer]

I just received this error and the process quit. Here is what I'm seeing on my terminal:

srv_downsampler_grid.sh: Must download original source https://planetarymaps.usgs.gov/mosaic/Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.tif
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8100M  100 8100M    0     0  28.7M      0  0:04:41  0:04:41 --:--:-- 28.1M
srv_downsampler_grid.sh: Must convert original tif source to Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc
scripts/srv_downsampler_grid.sh: line 117: [: NaN: integer expression expected
Convert Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_14s_p.grd=ns+s0.5+o0
Down-filter Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_15s_g.grd=ns+s0.5+o0
scripts/srv_downsampler_grid.sh: line 167: 21766 Killed                  gmt grdfilter -R-180/180/-90/0 ${SRC_FILE} -Fg${FILTER_WIDTH} -D${FMODE} -I${RES}${UNIT} -r${REG} -G${TMP}/s.grd --PROJ_ELLIPSOID=${DST_SPHERE}

It looks like the download went fine but it ran into an issue when trying to run grdfilter. It looks like my home environment still has GMT 6.0.0, so I'm trying it with my PyGMT development environment that has GMT 6.4.0.

[willschlitzer]

Same issue with GMT 6.4:

moon/moon_relief/moon_relief_14s_p.grd exists - skipping
Down-filter Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_15s_g.grd=ns+s0.5+o0
scripts/srv_downsampler_grid.sh: line 167: 22132 Killed                  gmt grdfilter -R-180/180/-90/0 ${SRC_FILE} -Fg${FILTER_WIDTH} -D${FMODE} -I${RES}${UNIT} -r${REG} -G${TMP}/s.grd --PROJ_ELLIPSOID=${DST_SPHERE}

Any idea what the culprit may be? Here is my environment info:

GMT library information:
  binary version: 6.4.0
  cores: 12
  grid layout: rows
  library path: /home/will/miniconda3/envs/pygmt-dev/lib/libgmt.so
  padding: 2
  plugin dir: /home/will/miniconda3/envs/pygmt-dev/lib/gmt/plugins
  share dir: /home/will/miniconda3/envs/pygmt-dev/share/gmt
  version: 6.4.0

[PaulWessel]

OK, I think you will actually need to run from GMT master - I do not know how to do that within conda since I am from the olden days. But I have made GMT source changes the last 24 hours in responses to issues with the moon processing. So master is a requirement for now.

[joa-quim]

The problem with big files is that it's not enough to have enough RAM. There needs to exist a contiguous chunk of memory to hold the entire array. Maybe if you reboot and start from a cleanest as possible state it may make a difference ... or not.

[PaulWessel]

True, @joa-quim but also a must to use master since I fixed some issues. They may be macOS-specific though - hard to tell. The crash is in the section where I had to run grdfilter separately on the N and S hemispheres otherwise I got strange horizontal lines that @Esteban82 reported. Doing it this way (until I can find some subtle trouble) works - at least for me - and is only done on the 30s or finer resolutions. This and other issues are most likely some subtle 64-bit counting issue but hard to debug.

[willschlitzer]

Installed gmt from master (showing 6.5.0_e43114b_2023.01.04) and rebooted; still got the same result. It also used all of my RAM (32 GB) for about a minute prior to crashing.

Down-filter Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_15s_g.grd=ns+s0.5+o0
scripts/srv_downsampler_grid.sh: line 167:  6018 Killed                  gmt grdfilter -R-180/180/-90/0 ${SRC_FILE} -Fg${FILTER_WIDTH} -D${FMODE} -I${RES}${UNIT} -r${REG} -G${TMP}/s.grd --PROJ_ELLIPSOID=${DST_SPHERE}

Unfortunately, I don't have any Apple hardware to try this out with. I'm not sure what troubleshooting to try next.

[PaulWessel]

Thanks for trying. What I guess you can try is to run the script with full verbose (bash -xv .....) so that you can see that command with all the actual arguments. Then copy paste and verify it still crashes (it should) but add -Vd so we might learn when it crashes. Then, just mess with -I and go from 15s to 30s to 01m and see if it stops crashing. I would think so as this seems to be related to trouble with 64-bit counting and not so much lack of memory since you have 32 Gb.