Prowler v3 Importer

[finventario]

Scanner Name
In prowler v3, the fields have changed.

See https://github.com/prowler-cloud/prowler/releases/tag/3.0.0 and prowler-cloud/prowler@7b9fae5

Sample File
csv:

ASSESSMENT_START_TIME;FINDING_UNIQUE_ID;PROVIDER;CHECK_ID;CHECK_TITLE;CHECK_TYPE;STATUS;STATUS_EXTENDED;SERVICE_NAME;SUBSERVICE_NAME;SEVERITY;RESOURCE_TYPE;RESOURCE_DETAILS;RESOURCE_TAGS;DESCRIPTION;RISK;RELATED_URL;REMEDIATION_RECOMMENDATION_TEXT;REMEDIATION_RECOMMENDATION_URL;REMEDIATION_RECOMMENDATION_CODE_NATIVEIAC;REMEDIATION_RECOMMENDATION_CODE_TERRAFORM;REMEDIATION_RECOMMENDATION_CODE_CLI;REMEDIATION_RECOMMENDATION_CODE_OTHER;CATEGORIES;DEPENDS_ON;RELATED_TO;NOTES;PROFILE;ACCOUNT_ID;ACCOUNT_NAME;ACCOUNT_EMAIL;ACCOUNT_ARN;ACCOUNT_ORG;ACCOUNT_TAGS;REGION;RESOURCE_ID;RESOURCE_ARN
2023-01-02T17:43:47.486212;prowler-aws-accessanalyzer_enabled_without_findings-xxxxx;aws;accessanalyzer_enabled_without_findings;Check if IAM Access Analyzer is enabled without findings;IAM;FAIL;IAM Access Analyzer is not enabled;accessanalyzer;;low;Other;;[];Check if IAM Access Analyzer is enabled without findings;AWS IAM Access Analyzer helps you identify the resources in your organization and accounts, such as Amazon S3 buckets or IAM roles, that are shared with an external entity. This lets you identify unintended access to your resources and data, which is a security risk. IAM Access Analyzer uses a form of mathematical analysis called automated reasoning, which applies logic and mathematical inference to determine all possible access paths allowed by a resource policy.;https://docs.aws.amazon.com/IAM/latest/UserGuide/what-is-access-analyzer.html;Enable IAM Access Analyzer for all accounts, create analyzer and take action over it is recommendations (IAM Access Analyzer is available at no additional cost).;https://docs.aws.amazon.com/IAM/latest/UserGuide/what-is-access-analyzer.html;;;aws accessanalyzer create-analyzer --analyzer-name <NAME> --type <ACCOUNT|ORGANIZATION>;;;;;;ENV;xxxxxx;;;;;;xxregio;xxxxx;

json:

[{
    "AssessmentStartTime": "2023-01-02T17:22:49.730532",
    "FindingUniqueId": "prowler-aws-accessanalyzer_enabled_without_findings-xxxxx",
    "Provider": "aws",
    "CheckID": "accessanalyzer_enabled_without_findings",
    "CheckTitle": "Check if IAM Access Analyzer is enabled without findings",
    "CheckType": [
        "IAM"
    ],
    "ServiceName": "accessanalyzer",
    "SubServiceName": "",
    "Status": "FAIL",
    "StatusExtended": "IAM Access Analyzer is not enabled",
    "Severity": "low",
    "ResourceType": "Other",
    "ResourceDetails": "",
    "Tags": {
        "Tag1Key": "value",
        "Tag2Key": "value"
    },
    "Description": "Check if IAM Access Analyzer is enabled without findings",
    "Risk": "AWS IAM Access Analyzer helps you identify the resources in your organization and accounts, such as Amazon S3 buckets or IAM roles, that are shared with an external entity. This lets you identify unintended access to your resources and data, which is a security risk. IAM Access Analyzer uses a form of mathematical analysis called automated reasoning, which applies logic and mathematical inference to determine all possible access paths allowed by a resource policy.",
    "RelatedUrl": "https://docs.aws.amazon.com/IAM/latest/UserGuide/what-is-access-analyzer.html",
    "Remediation": {
        "Code": {
            "NativeIaC": "",
            "Terraform": "",
            "CLI": "aws accessanalyzer create-analyzer --analyzer-name <NAME> --type <ACCOUNT|ORGANIZATION>",
            "Other": ""
        },
        "Recommendation": {
            "Text": "Enable IAM Access Analyzer for all accounts, create analyzer and take action over it is recommendations (IAM Access Analyzer is available at no additional cost).",
            "Url": "https://docs.aws.amazon.com/IAM/latest/UserGuide/what-is-access-analyzer.html"
        }
    },
    "Categories": [],
    "DependsOn": [],
    "RelatedTo": [],
    "Notes": "",
    "Profile": "ENV",
    "AccountId": "xxxxx",
    "OrganizationsInfo": null,
    "Region": "xxxxx-regio",
    "ResourceId": "xxxxx",
    "ResourceArn": ""
}]

[security101]

Using toniblyx/prowler v3.2.4 and DefectDojo v. 2.18.3 I get the following error messages when I import a Prowler result file. The error messages provided by DD vary depending on the filetype:
CSV result file: An exception error occurred during the report import:expected string or bytes-like object, got 'NoneType'
JSON: An exception error occurred during the report import:Expecting property name enclosed in double quotes: line 2 column 1 (char 4)

Any advice?

[damiencarol]

@security101 we need a stack track to understand what's going on.

[damiencarol]

Made some tests and the format of Prowler changed, this need maintenance work.

[security101]

The prowler team responded to prowler-cloud/prowler#2076

The new format specification is detailed here:

CSV https://docs.prowler.cloud/en/latest/tutorials/reporting/#csv
JSON https://docs.prowler.cloud/en/latest/tutorials/reporting/#json
JSON-ASFF https://docs.prowler.cloud/en/latest/tutorials/reporting/#json-asff

An idea:

• Prowler supports as report format json-asff
• DD supports json-asff as import (see AWS Security Hub https://defectdojo.github.io/django-DefectDojo/integrations/parsers/file/awssecurityhub/)
• May be using the AWS Security Import could solve the issue and might we a recommendation for Prowler v3 reports?

Update: Idea did not work. Got the error message: An exception error occurred during the report import:'list' object has no attribute 'get'

[damiencarol]

@security101 I think I should fix the JSON-ASFF parser and extend the Prowler to support v3. Could you push me an ASFF reportin DM/email/Slack?

[damiencarol]

I took a look at the different file format and parsers and the best way to fix your issue is to add a parser for ASFF format.
This way, we stay generic and ASFF format is more stable.

@security101 will it be ok for you to use this new one?

[security101]

Dear @damiencarol, yes I think this a good choice and more stable in the future, too.

[security101]

@security101 I think I should fix the JSON-ASFF parser and extend the Prowler to support v3. Could you push me an ASFF reportin DM/email/Slack?

I will provide a report via DM. Upfront the associated stack trace:

django-defectdojo-218-nginx-1         | 2023/03/20 13:50:14 [warn] 7#7: *12 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000010, client: 172.20.0.1, server: , request: "POST /engagement/148/import_scan_results HTTP/1.1", host: "<my_domain>", referrer: "https://<my_domain>/engagement/148/import_scan_results"
django-defectdojo-218-uwsgi-1         | /usr/local/lib/python3.11/site-packages/django/db/models/fields/__init__.py:1358: RuntimeWarning: DateTimeField Test.target_start received a naive datetime (2023-03-20 00:00:00) while time zone support is active.
django-defectdojo-218-uwsgi-1         |   warnings.warn("DateTimeField %s.%s received a naive datetime "
django-defectdojo-218-uwsgi-1         | /usr/local/lib/python3.11/site-packages/django/db/models/fields/__init__.py:1358: RuntimeWarning: DateTimeField Test.target_end received a naive datetime (2023-03-20 00:00:00) while time zone support is active.
django-defectdojo-218-uwsgi-1         |   warnings.warn("DateTimeField %s.%s received a naive datetime "
django-defectdojo-218-uwsgi-1         | [20/Mar/2023 13:50:14] ERROR [dojo.engagement.views:698] Expecting property name enclosed in double quotes: line 2 column 1 (char 4)
django-defectdojo-218-uwsgi-1         | Traceback (most recent call last):
django-defectdojo-218-uwsgi-1         |   File "/app/dojo/engagement/views.py", line 682, in import_scan_results
django-defectdojo-218-uwsgi-1         |     test, finding_count, closed_finding_count, _ = importer.import_scan(scan, scan_type, engagement, user, environment, active=active, verified=verified, tags=tags,
django-defectdojo-218-uwsgi-1         |                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/app/dojo/importers/importer/importer.py", line 305, in import_scan
django-defectdojo-218-uwsgi-1         |     parsed_findings = parser.get_findings(scan, test)
django-defectdojo-218-uwsgi-1         |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/app/dojo/tools/aws_prowler/parser.py", line 29, in get_findings
django-defectdojo-218-uwsgi-1         |     return self.process_json(file, test)
django-defectdojo-218-uwsgi-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/app/dojo/tools/aws_prowler/parser.py", line 123, in process_json
django-defectdojo-218-uwsgi-1         |     deserialized = json.loads(issue)
django-defectdojo-218-uwsgi-1         |                    ^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
django-defectdojo-218-uwsgi-1         |     return _default_decoder.decode(s)
django-defectdojo-218-uwsgi-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
django-defectdojo-218-uwsgi-1         |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
django-defectdojo-218-uwsgi-1         |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         |   File "/usr/local/lib/python3.11/json/decoder.py", line 353, in raw_decode
django-defectdojo-218-uwsgi-1         |     obj, end = self.scan_once(s, idx)
django-defectdojo-218-uwsgi-1         |                ^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-218-uwsgi-1         | json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 4)
django-defectdojo-218-uwsgi-1         | [pid: 41|app: -|req: -/-] 172.20.0.1 (admin) {74 vars in 1738 bytes} [Mon Mar 20 13:50:14 2023] POST /engagement/148/import_scan_results => generated 145334 bytes in 770 msecs (HTTP/1.1 200) 7 headers in 361 bytes (1 switches on core 0)

[awakenine]

@damiencarol, thank you for the parser. I have tested it a bit for prowler, and assume it's better to separate parser for prowlerV3, because some fields are used in a different way in prowler, than in GuardDuty.

Parsing Regular JSON or CSV output is preferable, because ASFF JSON output format miss some important fields (like Risk).

Currently if we use general ASFF parser for Prowler V3 report it creates a lot of PASSED active findings in Dojo that look the same as discovered vulnerabilities in Dojo and only can find Title, Description and Severity missing a lot of useful details.
Below you can see my notes about mapping I think would make sense:

• Mapping of "Remediation": { "Recommendation" to Mitigation in Dojo may be also useful as it provides info about the fix.
• Resources field can be appended to Description in Dojo just as array.
• Description is better to map to both finding title and description in Dojo, because title from PrawlerV3 ASFF is just a check name, which looks confusing.

Here is the sample of prowlerV3 finding in ASFF format

{
    "SchemaVersion": "2018-10-08",
    "Id": "prowler-acm_certificates_expiration_check-HIDDEN",
    "ProductArn": "arn:aws:securityhub:eu-west-1::product/prowler/prowler",
    "RecordState": "ACTIVE",
    "ProductFields": {
      "ProviderName": "Prowler",
      "ProviderVersion": "3.3.2",
      "ProwlerResourceName": "HIDDEN"
    },
    "GeneratorId": "prowler-acm_certificates_expiration_check",
    "AwsAccountId": "HIDDEN",
    "Types": [
      "Data Protection"
    ],
    "FirstObservedAt": "2023-04-01T13:40:45Z",
    "UpdatedAt": "2023-04-01T13:40:45Z",
    "CreatedAt": "2023-04-01T13:40:45Z",
    "Severity": {
      "Label": "HIGH"
    },
    "Title": "Check if ACM Certificates are about to expire in specific days or less",
    "Description": "ACM Certificate for example.com is about to expire in 7 days.",
    "Resources": [
      {
        "Type": "AwsCertificateManagerCertificate",
        "Id": "HIDDEN",
        "Partition": "aws",
        "Region": "eu-west-1"
      }
    ],
    "Compliance": {
      "Status": "FAILED",
      "RelatedRequirements": [
        "CISA your-data-2",
        "SOC2 cc_6_7",
        "GDPR article_32",
        "HIPAA 164_308_a_4_ii_a 164_312_e_1",
        "NIST-800-171-Revision-2 3_13_1 3_13_2 3_13_8 3_13_11",
        "NIST-800-53-Revision-4 ac_4 ac_17_2 sc_12",
        "NIST-800-53-Revision-5 sc_7_12 sc_7_16",
        "NIST-CSF-1.1 ac_5 ds_2",
        "RBI-Cyber-Security-Framework annex_i_1_3",
        "FFIEC d3-pc-im-b-1",
        "FedRamp-Moderate-Revision-4 ac-4 ac-17-2 sc-12",
        "FedRAMP-Low-Revision-4 ac-17 sc-12"
      ],
      "AssociatedStandards": [
        {
          "StandardsId": "CISA"
        },
        {
          "StandardsId": "SOC2"
        },
        {
          "StandardsId": "GDPR"
        },
        {
          "StandardsId": "HIPAA"
        },
        {
          "StandardsId": "NIST-800-171-Revision-2"
        },
        {
          "StandardsId": "NIST-800-53-Revision-4"
        },
        {
          "StandardsId": "NIST-800-53-Revision-5"
        },
        {
          "StandardsId": "NIST-CSF-1.1"
        },
        {
          "StandardsId": "RBI-Cyber-Security-Framework"
        },
        {
          "StandardsId": "FFIEC"
        },
        {
          "StandardsId": "FedRamp-Moderate-Revision-4"
        },
        {
          "StandardsId": "FedRAMP-Low-Revision-4"
        }
      ]
    },
    "Remediation": {
      "Recommendation": {
        "Text": "Monitor certificate expiration and take automated action to renew; replace or remove. Having shorter TTL for any security artifact is a general recommendation; but requires additional automation in place. If not longer required delete certificate. Use AWS config using the managed rule: acm-certificate-expiration-check.",
        "Url": "https://docs.aws.amazon.com/config/latest/developerguide/acm-certificate-expiration-check.html"
      }
    }
  },