Nosey Parker Parser by tpat13 · Pull Request #9067 · DefectDojo/django-DefectDojo

tpat13 · 2023-11-28T13:55:04Z

Hello!
This is a new parser for the secret scanning tool, Nosey Parker: https://github.com/praetorian-inc/noseyparker

Input into the parser is a JSON Lines file

Tests

Tests have been created for no vulnerabilities (empty jsonl), one vulnerability, many vulnerabilities, and malformed nosey output. NOTE: All secrets in the test jsonl files are fake

Documentation

Documentation created explaining the parser in noseyparker.md

Checklist

This checklist is for your information.

dryrunsecurity · 2023-11-28T13:55:31Z

Contextual Security Analysis

As DryRun Security performs checks, we’ll summarize them here. You can always dive into the detailed results in the section below for checks.

Status	DryRun Security Check
✅	Sensitive Functions Analyzer
✅	Configured Sensitive Files Analyzer
✅	Sensitive Files Analyzer

Chat with your AI-powered Security Buddy by typing @dryrunsecurity followed by your question into a comment.
Example: @dryrunsecurity What are common security issues with web application cookies?

Install and configure more repositories at DryRun Security

github-actions · 2023-12-04T05:13:56Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

* fixing up some links/etc * formatting * more formatting, links, etc * formatting table HTML * Fixing links * typo * formatting, links * typo; adding Aaron Weaver to hall of fame * reorganizing

Bumps [python-gitlab](https://github.com/python-gitlab/python-gitlab) from 3.15.0 to 4.2.0. - [Release notes](https://github.com/python-gitlab/python-gitlab/releases) - [Changelog](https://github.com/python-gitlab/python-gitlab/blob/main/CHANGELOG.md) - [Commits](python-gitlab/python-gitlab@v3.15.0...v4.2.0) --- updated-dependencies: - dependency-name: python-gitlab dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [fontawesomefree](https://github.com/FortAwesome/Font-Awesome) from 6.4.2 to 6.5.0. - [Release notes](https://github.com/FortAwesome/Font-Awesome/releases) - [Changelog](https://github.com/FortAwesome/Font-Awesome/blob/6.x/CHANGELOG.md) - [Commits](FortAwesome/Font-Awesome@6.4.2...6.5.0) --- updated-dependencies: - dependency-name: fontawesomefree dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* 🎉 added humble * fixed humble * added endpoints * fix according to comment * fix according to review * update * added deduplication setting * fix

Bumps [social-auth-core](https://github.com/python-social-auth/social-core) from 4.5.0 to 4.5.1. - [Release notes](https://github.com/python-social-auth/social-core/releases) - [Changelog](https://github.com/python-social-auth/social-core/blob/master/CHANGELOG.md) - [Commits](python-social-auth/social-core@4.5.0...4.5.1) --- updated-dependencies: - dependency-name: social-auth-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…l) (DefectDojo#9075) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…ocker-compose.yml) (DefectDojo#9082) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…ocker-compose.yml) (DefectDojo#9083) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Bumps [boto3](https://github.com/boto/boto3) from 1.29.7 to 1.33.5. - [Release notes](https://github.com/boto/boto3/releases) - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst) - [Commits](boto/boto3@1.29.7...1.33.5) --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [fontawesomefree](https://github.com/FortAwesome/Font-Awesome) from 6.5.0 to 6.5.1. - [Release notes](https://github.com/FortAwesome/Font-Awesome/releases) - [Changelog](https://github.com/FortAwesome/Font-Awesome/blob/6.x/CHANGELOG.md) - [Commits](FortAwesome/Font-Awesome@6.5.0...6.5.1) --- updated-dependencies: - dependency-name: fontawesomefree dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ectDojo#9087)

* new parser Jfrog Xray on Demand Binary Scan * new parser Jfrog Xray on Demand Binary Scan * delete blank line at end of file * rename function * More sample reports * Update docs/content/en/integrations/parsers/file/jfrog_xray_on_demand_binary_scan.md Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update docs/content/en/integrations/parsers/file/jfrog_xray_on_demand_binary_scan.md Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update docs/content/en/integrations/parsers/file/jfrog_xray_on_demand_binary_scan.md Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update dojo/settings/settings.dist.py Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update dojo/settings/settings.dist.py Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: kiblik <kiblik@gjh.sk> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: kiblik <kiblik@gjh.sk> * First round of Improvements * Drop duplicates in component_id and full_path * Process per component * Visual improvements * Use+clean summary in Title, fix dedup, parse version, drop useless functions * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: kiblik <kiblik@gjh.sk> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: kiblik <kiblik@gjh.sk> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: kiblik <kiblik@gjh.sk> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: kiblik <kiblik@gjh.sk> * Update dojo/tools/jfrog_xray_on_demand_binary_scan/parser.py Co-authored-by: kiblik <kiblik@gjh.sk> * fix test rename class * Last Improvements and tests * capitalization skills --------- Co-authored-by: Tomas Kubla <tomas@kubla.sk> Co-authored-by: Charles Neill <1749665+cneill@users.noreply.github.com> Co-authored-by: kiblik <kiblik@gjh.sk>

…ocker-compose.yml) (DefectDojo#9089) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

mtesauro · 2024-02-07T15:52:11Z

@tpat13 Thanks for returning to this PR. Waiting for the tests to finish and then I'll review.

tpat13 · 2024-02-12T17:12:36Z

Hi @mtesauro, I see some failing checks but not sure those are related to my changes.

tpat13 · 2024-02-13T18:47:50Z

Hi! @cneill @Maffooch would you be able to review when you get a chance? Thanks!

github-actions · 2024-02-16T02:48:41Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

# Conflicts: # dojo/settings/settings.dist.py

github-actions · 2024-02-20T16:19:52Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

cneill

Just a few small things, otherwise looks good!

cneill

Just a quick fix for the failing docs unit test

tpat13 · 2024-02-23T20:26:18Z

Thank you @cneill!!

manuel-sommer · 2024-02-26T00:29:39Z

+
+The following is an example of an acceptable JSON lines file:
+~~~
+{"type": "finding", "rule_name": "Generic Password (double quoted)", "match_content": "32ui1ffdasfhu239b4df2ac6609a9919", "num_matches": 2, "status": null, "comment": null, "matches": [ { "provenance": [ { "kind": "file", "path": "app/schema/config.py" }, { "kind": "git_repo", "repo_path": "./.git", "commit_provenance": { "commit_kind": "first_seen", "commit_metadata": { "commit_id": "0ef84b84c29924b210e3576f69d1e8632948bedc", "committer_name": "Princess Leia", "committer_email": "leia@test.com", "committer_timestamp": "1685495256 +0000", "author_name": "Princess Leia", "author_email": "leia@test.com", "author_timestamp": "1685495256 +0000", "message": "first commit\n" }, "blob_path": "app/schema/config.py" } } ], "blob_metadata": { "id": "0ee84b84c29924b210e3576fe9d1e8632948bedc", "num_bytes": 664, "mime_essence": "text/plain", "charset": null }, "blob_id": "0ee84b84c29924b210e3576fe9d1e8632948bedc", "location": { "offset_span": { "start": 617, "end": 660 }, "source_span": { "start": { "line": 16, "column": 17 }, "end": { "line": 16, "column": 59 } } }, "capture_group_index": 1, "match_content": "32ui1ffdasfhu239b4df2ac6609a9919", "snippet": { "before": "E = \"https://testwebsite.com\"\n ", "matching": "API_KEY = \"32ui1ffdasfhu239b4df2ac6609a9919", "after": "\"\n\n\n" }, "rule_name": "Generic API Key" } ] }{"type":"finding","rule_name":"Generic Username and Password (unquoted)","match_content":"secret","num_matches":1,"matches":[{"provenance":[{"kind":"file","path":"./app/schema/config.py"},{"kind":"git_repo","repo_path":"./.git","commit_provenance":{"commit_kind":"first_seen","commit_metadata":{"commit_id":"0ee84b84c29924b210e3576fe9d1e8632948bedc","committer_name":"Princess Leia","committer_email":"leia@test.com","committer_timestamp":"1685495256 +0000","author_name":"Princess Leia","author_email":"leia@test.com","author_timestamp":"1685495256 +0000","message":"framework\n"},"blob_path":"app/schema/config.py"}}],"blob_metadata":{"id":"0ee84b84c29924b210e3576fe9d1e8632948bedc","num_bytes":664,"mime_essence":"text/plain","charset":null},"blob_id":"0ee84b84c29924b210e3576fe9d1e8632948bedc","location":{"offset_span":{"start":617,"end":660},"source_span":{"start":{"line":16,"column":17},"end":{"line":16,"column":59}}},"capture_group_index":1,"match_content":"secret","snippet":{"before":"E = \"https://testwebsite.com\"\n ","matching":"secret","after":"testing\"\n\n\n"},"rule_name":"Generic Username and Password (unquoted)"}]}


The question is if you really need a plaintext JSON file example if you have a whole link in

Sample Scan Data

I guess you can remove this in order to make the md slimmer and only share relevant information.

Fixed @manuel-sommer, thanks for the suggestion!

tpat13 · 2024-02-27T23:07:21Z

Thanks for the approval @manuel-sommer! It seems like merging is still blocked because I need "4 reviewers with write access." Do you know if this is a permissions issue or do I just need 1 more admin reviewer?

manuel-sommer · 2024-02-27T23:16:15Z

You need anorher maintainer approval

tpat13 · 2024-03-08T21:03:29Z

+                                  f"Commit ID: {json_path['commit_provenance']['commit_metadata']['commit_id']}  \n" \
+                                  f"Location: {filepath} line #{line_num} \n " \
+                                  f"Line #{line_num} \n " \
+                                  f"Code Snippet Containing Secret: {match['snippet']['before']}***SECRET***{match['snippet']['after']} \n"


Missing new line

tpat13 added 2 commits November 28, 2023 08:30

Created _init_.py

a2d389d

Created parser.py

6052a25

github-actions Bot added docs parser labels Nov 28, 2023

Merge branch 'DefectDojo:master' into nosey-parker

94f0125

github-actions Bot added the conflicts-detected label Dec 4, 2023

devGregA and others added 18 commits December 4, 2023 00:14

Update README.md (DefectDojo#9048)

be31c39

Fixing README links and formatting (DefectDojo#9022)

0aafba0

* fixing up some links/etc * formatting * more formatting, links, etc * formatting table HTML * Fixing links * typo * formatting, links * typo; adding Aaron Weaver to hall of fame * reorganizing

🎉 added humble DefectDojo#8988 (DefectDojo#8989)

f8426ef

* 🎉 added humble * fixed humble * added endpoints * fix according to comment * fix according to review * update * added deduplication setting * fix

Adding subcomponent labels for celery beat and worker (DefectDojo#9078)

12c20c4

Update rabbitmq Docker tag from 3.12.9 to v3.12.10 (docker-compose.ym…

e2885d5

…l) (DefectDojo#9075) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update postgres:16.1-alpine Docker digest from 16.1 to 16.1-alpine (d…

55f5573

…ocker-compose.yml) (DefectDojo#9082) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update redis:7.2.3-alpine Docker digest from 7.2.3 to 7.2.3-alpine (d…

4b80f78

…ocker-compose.yml) (DefectDojo#9083) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Add logging statement for failed password reset validation logic (Def…

cd83b00

…ectDojo#9087)

Finding Template: Correct save ordering (DefectDojo#9088)

9ee0605

Update postgres:16.1-alpine Docker digest from 16.1 to 16.1-alpine (d…

1f36b42

…ocker-compose.yml) (DefectDojo#9089) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Nosey Parker Test Cases

526319f

Updated Parser

0ba7a9d

github-actions Bot added docker settings_changes Needs changes to settings.py based on changes in settings.dist.py included in this PR unittests helm labels Dec 4, 2023

github-actions Bot added the conflicts-detected label Feb 16, 2024

kiblik mentioned this pull request Feb 16, 2024

Labeler: Add sync-labels #9565

Merged

Merge branch 'dev' into nosey-parker

a8c7358

# Conflicts: # dojo/settings/settings.dist.py

github-actions Bot removed the conflicts-detected label Feb 20, 2024

cneill reviewed Feb 21, 2024

View reviewed changes

Comment thread docs/content/en/integrations/parsers/file/noseyparker.md Outdated

Comment thread dojo/tools/noseyparker/parser.py Outdated

cneill added 2 commits February 21, 2024 12:39

Update docs/content/en/integrations/parsers/file/noseyparker.md

8d16815

Update dojo/tools/noseyparker/parser.py

02f017a

github-actions Bot removed docker helm labels Feb 21, 2024

cneill reviewed Feb 21, 2024

View reviewed changes

Comment thread docs/content/en/integrations/parsers/file/noseyparker.md Outdated

Update docs/content/en/integrations/parsers/file/noseyparker.md

068106a

cneill approved these changes Feb 21, 2024

View reviewed changes

manuel-sommer reviewed Feb 26, 2024

View reviewed changes

tpat13 added 4 commits February 26, 2024 17:29

Merge branch 'dev' into nosey-parker

9219e44

Removed example JSONL file

8cb6795

Add link to 0.16.0 Release

4825a14

Spacing

3206c2c

manuel-sommer approved these changes Feb 27, 2024

View reviewed changes

Merge branch 'dev' into nosey-parker

b387b19

blakeaowens approved these changes Feb 28, 2024

View reviewed changes

blakeaowens merged commit 18ff583 into DefectDojo:dev Feb 28, 2024

tpat13 commented Mar 8, 2024

View reviewed changes

Conversation

tpat13 commented Nov 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dryrunsecurity Bot commented Nov 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contextual Security Analysis

Uh oh!

github-actions Bot commented Dec 4, 2023

Uh oh!

mtesauro commented Feb 7, 2024

Uh oh!

tpat13 commented Feb 12, 2024

Uh oh!

tpat13 commented Feb 13, 2024

Uh oh!

github-actions Bot commented Feb 16, 2024

Uh oh!

github-actions Bot commented Feb 20, 2024

Uh oh!

cneill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cneill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tpat13 commented Feb 23, 2024

Uh oh!

manuel-sommer Feb 26, 2024

Choose a reason for hiding this comment

Sample Scan Data

Uh oh!

tpat13 Feb 26, 2024

Choose a reason for hiding this comment

Uh oh!

tpat13 commented Feb 27, 2024

Uh oh!

manuel-sommer commented Feb 27, 2024

Uh oh!

tpat13 Mar 8, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

tpat13 commented Nov 28, 2023 •

edited

Loading

dryrunsecurity Bot commented Nov 28, 2023 •

edited

Loading