Accessing CMR Data in R

[smk0033]

Github ticket: https://github.com/NASA-IMPACT/active-maap-sprint/issues/1041

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[wildintellect]

should this print show some granule names, and example granule, or granule count?

Reply via ReviewNB

[wildintellect]

we should reference the page about DAAC AWS access, so people know how to find other DAAC credential urls

Reply via ReviewNB

[smk0033]

I may be incorrect or overlooking it, but I'm not sure we have a document that lists all other DAAC credential URLs or how to find them? The "Direct DAAC S3 Bucket Access (BETA)" notebook has S3 links to specific granules but not the general DAAC credential URL, though we do link to the cloud collections in Earthdata Search. "Accessing Data Providers by NASA's DAACs)" doesn't have S3

[wildintellect]

Openscapes has a list of these, but I think MAAP only supports 5 right now. If we don't have those listed in the docs we should.

[hrodmn]

Thanks for putting this together @HarshiniGirish and @smk0033! I left a few suggestions but I am left wondering if reading hdf files directly from S3 is possible from within R. Users really should not need to manually download files to read the bytes they want to use! If it is not possible it might be worth exploring a reticulate solution that can use a python library that can do that thing. I tried a few things like setting the AWS environment variables like this:

Sys.setenv(
  "AWS_ACCESS_KEY_ID" = credentials$accessKeyId,
  "AWS_SECRET_ACCESS_KEY" = credentials$secretAccessKey,
  "AWS_SESSION_TOKEN" = credentials$sessionToken,
  "AWS_REGION" = "us-west-2"
)

but I was not able to read the hdf file with /vsis3/{bucket}/{path}.hdf.

[wildintellect]

@hrodmn I think the whole using directly from S3 is unexplored territory in R. There are 2 issues, authenticating, and then the HDF reader library supporting network based reads. This works for Terra and SF because gdal/ogr handles it all. For some netcdf with Terra and Stars likely it would work too, not sure about this ncdf4 libraries abilities to do either of the required steps.

[hrodmn]

Thanks for putting this together, adding the collection search details in the "Searching CMR from R" would be great.

[smk0033]

Thanks Henry! Will do - I'll be sure to get a PR out for that this week