Conversation
Modify README to make it easier to understand what needs to be done. Updated buildStats.sh to use a partial version of run_cloudwatchquery.sh as some logs had been lost from migration from elastic bean to App Runner
There was a problem hiding this comment.
Pull request overview
This PR updates the Neotoma Stats build/report workflow to improve credential handling and modernize the data pipeline, including moving DB credentials into a .env file, switching to RPostgres, and adding a secrets-blocking pre-commit hook plus an alternate CloudWatch metrics query script for API usage.
Changes:
- Load DB connection parameters from a gitignored
.envfile (with.env-template) and update build/docs accordingly. - Modernize R dependencies/DB connection (
RPostgres) and adjust report logic for updated API usage inputs. - Add a CloudWatch/AppRunner metrics-based query script and a custom
.githooks/pre-commitsecrets scanner.
Reviewed changes
Copilot reviewed 7 out of 12 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
StateoftheDB.Rmd |
Updates DB connection to RPostgres and revises API-calls reporting sections. |
run_cloudwatchquery_partial.sh |
New script to generate API usage stats from AWS/AppRunner CloudWatch metrics. |
requirements-src.R |
Switches dependency from RPostgreSQL to RPostgres. |
README.md |
Documents .env configuration and script usage. |
buildStats.sh |
Loads .env variables and clarifies local vs. remote build behavior. |
.gitignore |
Ignores .env and other local artifacts (and currently ignores *.md). |
.githooks/pre-commit |
Adds a pre-commit hook to block committing secret files/credentials. |
.env-template |
Provides a template for required DB environment variables. |
outputs/datasetsPerMonth.svg |
Updated generated report artifact output. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+8
to
+9
| start=`date -u -v-1y +"%Y-%m-%dT%H:%M:%SZ"` | ||
| end=`date -u +"%Y-%m-%dT%H:%M:%SZ"` |
Comment on lines
+34
to
+45
| printf '%s' "$points" | jq ' | ||
| group_by([.date[0:10], .service]) | ||
| | map({date: .[0].date[0:10], | ||
| service: .[0].service, | ||
| calls: (map(.calls // 0) | add | floor), | ||
| calls_2xx: (map(.calls_2xx // 0) | add | floor)}) | ||
| | sort_by([.date, .service]) | ||
| | {results: map([{field: "date", value: .date}, | ||
| {field: "calls", value: (.calls | tostring)}, | ||
| {field: "calls_2xx", value: (.calls_2xx | tostring)}, | ||
| {field: "service", value: .service}]), | ||
| status: "Complete"}' > log_run_all.json |
| [ -z "$f" ] && continue | ||
| base=$(basename "$f") | ||
| case "$base" in | ||
| .env.template|.env.example|.env.sample) ;; # explicitly allowed |
Comment on lines
+286
to
290
| ```{r callFrequency, echo=FALSE, eval=FALSE, warning=FALSE} | ||
| # The main APIs delivered a total of `r p(floor(sum(pullLog$volume, na.rm=TRUE) /1000000000))` GB of data to users since `r min(pullLog$date)`. | ||
|
|
||
| #### Specific API Calls | ||
| ggplot(pullCalls[1:20,]) + |
Comment on lines
+59
to
+62
| ### Scripts | ||
|
|
||
| * **`run_cloudwatchquery.sh`** — pulls API and Tilia usage statistics from AWS CloudWatch Logs (the Neotoma/Tilia Elastic Beanstalk nginx access logs) and writes them to the `log_run*.json` files that the report reads for its usage charts. It runs several CloudWatch Logs Insights queries, each of which sleeps ~5 minutes while the query completes, so a full pull takes several minutes and requires working `aws` CLI credentials. It is invoked automatically by `buildStats.sh`. | ||
| * **`buildStats.sh`** — the main build script. It first runs `run_cloudwatchquery.sh` to refresh the usage logs, then loads the database connection variables from `.env` and renders `StateoftheDB.Rmd` into HTML. It no longer hardcodes credentials; both the local and remote modes use whatever is in `.env`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements to the Neotoma Stats build process, focusing on better security, configuration management, and updated data handling. The key changes include switching from hardcoded to environment-based database credentials, updating R packages and connection code, and adding a pre-commit hook to prevent accidental commits of secrets. There are also documentation updates and code changes to support the new workflow and handle recent changes in API logging.
Security and Credential Management
.env-templatefile and updated documentation to guide users to store database credentials in a.envfile, which is loaded automatically bybuildStats.shinstead of hardcoding credentials. (.env-template,buildStats.sh,README.md) [1] [2] [3].githooks/pre-commitscript that blocks commits of secrets and scans staged changes for hardcoded credentials. (.githooks/pre-commit)Build and Data Pipeline Updates
buildStats.shto source credentials from.envand clarified script usage for local and remote builds. (buildStats.sh,README.md) [1] [2]run_cloudwatchquery_partial.shto fetch API usage stats from AWS App Runner metrics, reflecting infrastructure changes. (run_cloudwatchquery_partial.sh)R Code Modernization
RPostgreSQLtoRPostgresfor database connections and updated connection code inStateoftheDB.Rmdandrequirements-src.R. (StateoftheDB.Rmd,requirements-src.R) [1] [2] [3]StateoftheDB.Rmd) [1] [2] [3]Documentation and Attribution
README.md. (README.md,StateoftheDB.Rmd) [1] [2]These changes collectively improve the project's security posture, make the build process more robust and maintainable, and update the reporting to match current infrastructure.