Skip to content

Some adjustments to simplify ingestion of data #158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mortenwh wants to merge 14 commits into
base: master
Choose a base branch
from
Open

Some adjustments to simplify ingestion of data #158

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
5dd4868
140: exclude datasets without geographic_location
mortenwh Dec 17, 2020
e3149f3
Merge branch 'master' of github.com:mortenwh/django-geo-spaas
mortenwh Mar 26, 2021
612aa94
exception on valueerror
mortenwh Mar 26, 2021
8c8ae05
allow a simpler way of providing the platfrom and instrument metadata
mortenwh Mar 27, 2021
d012a0f
corrected app name
Apr 22, 2021
edddb69
Merge branch 'master' of github.com:mortenwh/django-geo-spaas
mortenwh Apr 22, 2021
ca18a17
organise products in subfolders based on the dataset time_coverage_start
mortenwh Feb 2, 2023
fd4c33b
do not populate all vocabularies
mortenwh Jun 28, 2023
8ba94ec
change data_center to NO/MET instead of NERSC
mortenwh Jun 28, 2023
fcd0388
adjust to MET standard
mortenwh Aug 20, 2023
d25f47e
warnings to log at debug level
mortenwh Aug 20, 2023
eb5d5fb
close db connection
mortenwh Aug 20, 2023
f6be534
Hack to adapt to changes in gcmd platform list
mortenwh Oct 2, 2023
d17cdeb
handle FileExistsError
mortenwh Jun 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
organise products in subfolders based on the dataset time_coverage_start
  • Loading branch information
@mortenwh
mortenwh committed Feb 2, 2023
commit ca18a172c7b6a791d3ec91d18aa01d6050f9622f
12 changes: 9 additions & 3 deletions geospaas/utils/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,15 @@ def module_path(module, root):
return media_path


def path(module, filename, root):
def path(module, filename, root, date=None):
mp = module_path(module, root)

if date is not None:
for xx in [date.strftime('%Y'), date.strftime('%m'), date.strftime('%d')]:
mp = os.path.join(mp, xx)
if not os.path.exists(mp):
os.mkdir(mp)

# Get the path of media files created from <filename>
basename = os.path.split(filename)[-1].split('.')[0]
dataset_path = os.path.join(mp, basename)
Expand All @@ -39,8 +45,8 @@ def path(module, filename, root):
def media_path(module, filename):
return path(module, filename, settings.MEDIA_ROOT)

def product_path(module, filename):
return path(module, filename, settings.PRODUCTS_ROOT)
def product_path(module, filename, date=None):
return path(module, filename, settings.PRODUCTS_ROOT, date=date)

def validate_uri(uri):
""" Validation of URI and its existence
Expand Down