Develop

.env-template

@@ -0,0 +1,5 @@
+DBNAME=neotoma
+HOST=localhost
+PORT=5432
+USER=postgres
+PASSWORD=postgres

.githooks/pre-commit

@@ -0,0 +1,47 @@
+#!/bin/bash
+#
+# Pre-commit hook: refuse to commit secrets.
+#   1. Blocks staged .env-style files (allows .env-template / .env.example / .env.sample).
+#   2. Blocks staged ADDITIONS that look like a hardcoded credential.
+# This hook file is excluded from the scan so it can't flag its own patterns.
+# Override in a true emergency with:  git commit --no-verify
+#
+set -euo pipefail
+
+fail=0
+
+# 1) Block staged secret files.
+while IFS= read -r f; do
+    [ -z "$f" ] && continue
+    base=$(basename "$f")
+    case "$base" in
+        .env.template|.env.example|.env.sample) ;;      # explicitly allowed
+        .env|.env.*)
+            echo "BLOCKED: refusing to commit '$f' (looks like a secrets file)."
+            fail=1
+            ;;
+    esac
+done < <(git diff --cached --name-only --diff-filter=ACM)
+
+# 2) Scan staged ADDITIONS for hardcoded credentials. Ignore template files, this
+#    hook itself, and env-indirection (Sys.getenv / process.env / $VAR references).
+hits=$(git diff --cached -U0 --diff-filter=ACM -- . \
+        ':(exclude).env-template' ':(exclude).env.example' ':(exclude).env.sample' \
+        ':(exclude).githooks/pre-commit' \
+      | grep -E '^\+' \
+      | grep -Ei '(password|pgpassword|secret|token|api_?key)[[:space:]]*[:=]' \
+      | grep -viE 'getenv|environ|process\.env|\$|<|xxxxxx|placeholder|example|your_|changeme' \
+      | grep -Ei "[:=][[:space:]]*[\"']?[A-Za-z0-9]{6,}" || true)
+
+if [ -n "$hits" ]; then
+    echo "BLOCKED: staged changes look like a hardcoded credential:"
+    printf '  %s\n' "$hits"
+    echo "         Store secrets in .env (gitignored) and read them via environment variables."
+    fail=1
+fi
+
+if [ "$fail" -ne 0 ]; then
+    echo ""
+    echo "Commit aborted by pre-commit hook. To override (NOT recommended): git commit --no-verify"
+    exit 1
+fi

.gitignore

@@ -1,4 +1,10 @@
 connect_remote.json
 ceiwww/*
-.secrets.env
+.env
+.DS_Store
+*.Rproj
 .buildStats.sh
+.Rproj.user
+
+logs/
+*.md

README.md

@@ -13,6 +13,7 @@ An RMarkdown document used to derive intermediary statistics for the Neotoma dat
 This project is an open project, and contributions are welcome from any individual.  All contributors to this project are bound by a [code of conduct](CODE_OF_CONDUCT.md).  Please review and follow this code of conduct as part of your contribution.
 
 * [![orcid](https://img.shields.io/badge/orcid-0000--0002--2700--4605-brightgreen.svg)](https://orcid.org/0000-0002-2700-4605) [Simon Goring](http://goring.org)
+* [![ORCID](https://img.shields.io/badge/orcid-0000--0002--7926--4935-brightgreen.svg)](https://orcid.org/0000-0002-7926-4935) [Socorro Dominguez](https://ht-data.com/about)
 
 ### Tips for Contributing
 
@@ -39,16 +40,41 @@ rm -r ./dbout
 
 The above script creates a duplicate of the database locally and then cleans up the `sql` file extracted from the downloaded `tar` archive.
 
-The document expects environment variables (we use defaults here, assuming the Neotoma database has been restored locally):
+### Configuration
+
+The database connection variables are read from a `.env` file in the project root. Copy the provided template and fill in your values:
+
+```bash
+cp .env-template .env
+```
+
+The `.env` file defines the connection used by the build (defaults shown assume a locally restored Neotoma database):
 
 * DBNAME=neotoma
 * HOST=localhost
 * PORT=5432
 * USER=postgres
 * PASSWORD=postgres
 
-These variables are currently set by hand in the `buildStats.sh` bash script.
+### Scripts
+
+* **`run_cloudwatchquery.sh`** — pulls API and Tilia usage statistics from AWS CloudWatch Logs (the Neotoma/Tilia Elastic Beanstalk nginx access logs) and writes them to the `log_run*.json` files that the report reads for its usage charts. It runs several CloudWatch Logs Insights queries, each of which sleeps ~5 minutes while the query completes, so a full pull takes several minutes and requires working `aws` CLI credentials. It is invoked automatically by `buildStats.sh`.
+* **`buildStats.sh`** — the main build script. It first runs `run_cloudwatchquery.sh` to refresh the usage logs, then loads the database connection variables from `.env` and renders `StateoftheDB.Rmd` into HTML. It no longer hardcodes credentials; both the local and remote modes use whatever is in `.env`.
+
+### Running the build
+
+To execute and build the RMarkdown file, run:
 
-To execute and build the RMarkdown file, simply run `bash buildStats.sh` and a valid HTML document will be generated and output into the `outputs` folder.
+```bash
+bash buildStats.sh local
+```
+
+The `local` argument just renders the report; a valid HTML document is generated and output into the `outputs` folder.
+
+Running it with no argument renders the report **and then commits and pushes the rebuilt artifacts** (`git add --all && git commit && git push`), which is how the periodic builds are published:
+
+```bash
+bash buildStats.sh
+```
 
 ![The rendered Neotoma Stats document.](assets/docScreenshot.png)

StateoftheDB.Rmd

@@ -2,11 +2,11 @@
 title: "State of the Neotoma Paleoecology Database"
 shorttitle: "Neotoma Status"
 author:
+  - name: "Socorro Dominguez"
+    email: "dominguezvid@wisc.edu"
+    affiliation: "Department of Geography, University of Wisconsin--Madison"
   - name: "Simon Goring"
-    email: "goring@wisc.edu"
-  - affiliation:
-    - id: "1"
-      institution: "Department of Geography, University of Wisconsin--Madison"
+    affiliation: "Department of Geography, University of Wisconsin--Madison"
 date: "`r lubridate::today()`"
 description: |
   A summary of current Neotoma activity, potential data issues and other summary information.
@@ -28,7 +28,7 @@ output:
 knitr::opts_chunk$set(echo = FALSE, warning=FALSE, message=FALSE)
 
 library(pacman)
-p_load(lubridate, RPostgreSQL, DT, ggplot2, 
+p_load(lubridate, RPostgres, DT, ggplot2, 
        ggthemes, svglite,
        httr, jsonlite, dplyr, leaflet, geojsonsf, sf, purrr, forcats, scales,
        leaflet.providers, dotenv, snakecase)
@@ -37,12 +37,15 @@ load_dot_env()
 
 lastdate <- today() - years(1)
 
-con <- dbConnect(dbDriver("PostgreSQL"),
-                 dbname = Sys.getenv("DBNAME"), 
-                 host = Sys.getenv("HOST"), 
-                 port = Sys.getenv("PORT"), 
-                 user= Sys.getenv("USER"), 
-                 password= Sys.getenv("PASSWORD"))
+con <- dbConnect(
+  Postgres(),
+  host     = Sys.getenv("HOST"),
+  port     = Sys.getenv("PORT"),
+  dbname   = Sys.getenv("DBNAME"),
+  user     = Sys.getenv("USER"),
+  password = Sys.getenv("PASSWORD"),
+  sslmode  = "require"
+)
 
 p <- function(x) format(x, scientific=FALSE, big.mark=',')
 ```
@@ -68,7 +71,7 @@ Neotoma contains data from `r p(datasets)` datasets and `r p(sites)` unique site
 colnames(lastyear) = c('Datasets', 'Sites', 'Publications', 'Authors', 'Countries', 'Observations')
 outtable <- t(lastyear) %>% 
   as.data.frame %>% 
-  rename('Entries'='1') %>% 
+  rename('Entries'='V1') %>% 
   mutate('Entries' = p(`Entries`))
 DT::datatable(outtable,
   caption = htmltools::tags$caption(
@@ -231,15 +234,17 @@ Neotoma relies on significant the efforts of a volunteer group of data stewards
 
 ### API Calls
 
-```{r pullLogs, echo=FALSE, fig.caption="Volume of API calls to API and Tilia servers, aggregated by day over the last 6 months."}
-pullLog <- jsonlite::fromJSON('log_run.json')['results'][[1]] %>%
+```{r pullLogs, echo=FALSE, eval=FALSE, fig.caption="- Volume of API calls to API and Tilia servers, aggregated by day over the last 6 months. - Not updated info"}
+
+# Cannot use for now as logs were not being sent to AWS Cloudwatch Logs
+pullLog <- jsonlite::fromJSON('log_run.json')['results']$results %>%
   purrr::map(function(x) tidyr::pivot_wider(x,names_from=field)) %>%
   bind_rows() %>%
   mutate(date = lubridate::date(date),
     calls = as.integer(calls),
     volume = as.integer(volume))
 
-pullCalls <- jsonlite::fromJSON('log_run_calls.json')['results'][[1]] %>%
+pullCalls <- jsonlite::fromJSON('log_run_calls.json')['results']$results %>%
   purrr::map(function(x) tidyr::pivot_wider(x,names_from=field)) %>%
   bind_rows() %>%
   mutate(calls = as.integer(calls),
@@ -261,13 +266,27 @@ ggplot(pullLog) +
 
 ```
 
-Since the API has been implemented there have been a total of `r p(sum(pullLog$calls))` calls to the Neotoma API. These include calls to the core API ([`api.neotomadb.org`](https://api.neotomadb.org)), calls to support the Neotoma Landing Pages ([`data.neotomadb.org`](https://data.neotomadb.org)) and calls to support Neotoma Explorer ([`apps.neotomadb.org/explorer`](https://data.neotomadb.org)).
+```{r pullLogs2, echo=FALSE, fig.caption="- Volume of API calls to API and Tilia servers, aggregated by day over the last 8 months."}
 
-The main APIs delivered a total of `r p(floor(sum(pullLog$volume, na.rm=TRUE) /1000000000))` GB of data to users since `r min(pullLog$date)`.
+# Cannot use for now as logs were not being sent to AWS Cloudwatch Logs
+pullLog <- jsonlite::fromJSON('log_run_all.json')['results']$results %>%
+  purrr::map(function(x) tidyr::pivot_wider(x,names_from=field)) %>%
+  bind_rows() %>%
+  mutate(date = lubridate::date(date),
+    calls = as.integer(calls),
+    calls_2xx = as.integer(calls_2xx))
+```
 
-#### Specific API Calls
+In the last 8 months, there have been a total of `r p(sum(pullLog$calls_2xx))` calls to the Neotoma API. These include calls to the core API ([`api.neotomadb.org`](https://api.neotomadb.org)), calls to support the Neotoma Landing Pages ([`data.neotomadb.org`](https://data.neotomadb.org)) and calls to support Neotoma Explorer ([`apps.neotomadb.org/explorer`](https://data.neotomadb.org)).
 
-```{r callFrequency, echo=FALSE, warning=FALSE}
+
+
+
+
+```{r callFrequency, echo=FALSE, eval=FALSE, warning=FALSE}
+# The main APIs delivered a total of `r p(floor(sum(pullLog$volume, na.rm=TRUE) /1000000000))` GB of data to users since `r min(pullLog$date)`.
+
+#### Specific API Calls
 ggplot(pullCalls[1:20,]) +
   geom_bar(aes(x = 1:20, 
                y=fullcount/1000), stat='identity') +
@@ -276,9 +295,9 @@ ggplot(pullCalls[1:20,]) +
   theme(axis.text.x=element_blank())
 ```
 
-Several API calls are called thousands of times, but these are not necessarily the fastest, or slowest queries. There is no relationship between speed and the number of times an API endpoint is used. The most frequent API calls over the past week are:
+```{r freqCall, echo=FALSE, eval=FALSE}
 
-```{r freqCall, echo=FALSE}
+# Several API calls are called thousands of times, but these are not necessarily the fastest, or slowest queries. There is no relationship between speed and the number of times an API endpoint is used. The most frequent API calls over the past week are:
 pullCalls %>% 
   slice_max(fullcount, n = 20) %>% 
   select(path, fullcount) %>% 

buildStats.sh

@@ -1,25 +1,28 @@
 #!/bin/bash
 
-bash run_cloudwatchquery.sh
+# Need to run after I fix api_nodetest to send the logs to cloudwatch
+# bash run_cloudwatchquery.sh
+#bash run cloudwatchquery_partial.sh
+
+# Load the database connection variables (DBNAME, HOST, PORT, USER, PASSWORD)
+# from the .env file rather than hardcoding them here.
+if [ -f .env ]; then
+    set -a
+    source .env
+    set +a
+else
+    echo "No .env file found. Copy .env-template to .env and set your values." >&2
+    exit 1
+fi
 
 if [ "$1" == "local" ]; then
-    echo Running against the local DB.
-    export DBNAME=neotoma
-    export HOST=localhost
-    export PORT=5435
-    export USER=postgres
-    export PASSWORD=postgres
+    echo "Running against the local DB ($USER@$HOST:$PORT/$DBNAME)."
     Rscript -e "rmarkdown::render('StateoftheDB.Rmd')"
 else
-    echo Running against the remote.
-    export DBNAME=neotoma
-    export HOST=localhost
-    export PORT=5555
-    export USER=neotomaAdmin
-    export PASSWORD=REDACTED
+    echo "Running against the remote ($USER@$HOST:$PORT/$DBNAME)."
     Rscript -e "rmarkdown::render('StateoftheDB.Rmd')"
     git add --all
     git commit -m "Running the build"
     git push
 fi
-echo Done.
+echo Done.