Skip to content

[Darktrace] Initial Release for the Darktrace#4001

Merged
P1llus merged 5 commits into
elastic:mainfrom
vinit-chauhan:package_darktrace
Oct 4, 2022
Merged

[Darktrace] Initial Release for the Darktrace#4001
P1llus merged 5 commits into
elastic:mainfrom
vinit-chauhan:package_darktrace

Conversation

@vinit-chauhan

@vinit-chauhan vinit-chauhan commented Aug 16, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

  • Generated the skeleton of the Darktrace integration package.
  • Added data streams.
  • Added data collection logic for all the data streams.
  • Added the ingest pipeline for all the data streams.
  • Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
  • Added dashboards and visualizations.
  • Added test for pipeline for all the data streams.
  • Added system test cases for all the data streams.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

  • Change follows the contributing guidelines
  • Supported versions of the monitoring target are documented
  • Supported operating systems are documented (if applicable)
  • Integration or System tests exist
  • Documentation exists
  • Fields follow ECS and naming conventions
  • At least a manual test with ES / Kibana / Agent has been performed.
  • Required Kibana version set to: ^8.2.1

New Package

  • Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

  • Dashboards exists
  • Screenshots added or updated
  • Datastream filters added to visualizations

Log dataset changes

  • Pipeline tests exist (if applicable)
  • Generated output for at least 1 log file exists
  • Sample event (sample_event.json) exists

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/darktrace directory.
  • Run the following command to run tests.

elastic-package test

Related issues

Screenshots

image
image
image
image
image

@vinit-chauhan vinit-chauhan added enhancement New feature or request Team:Security-External Integrations New Integration Issue or pull request for creating a new integration package. labels Aug 16, 2022
@elasticmachine

elasticmachine commented Aug 16, 2022

Copy link
Copy Markdown

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@elasticmachine

elasticmachine commented Aug 16, 2022 •
edited
Loading

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-10-03T12:24:09.310+0000

  • Duration: 25 min 21 sec

Test stats 🧪

Test Results
Failed 0
Passed 29
Skipped 0
Total 29

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine

elasticmachine commented Aug 16, 2022 •
edited
Loading

Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (3/3) 💚
Files 100.0% (3/3) 💚 2.564
Classes 100.0% (3/3) 💚 2.564
Methods 100.0% (46/46) 💚 9.946
Lines 95.645% (2350/2457) 👍 4.116
Conditionals 100.0% (0/0) 💚

efd6
efd6 approved these changes Aug 17, 2022
View reviewed changes

@efd6 efd6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Query: There are a lot of object arrays in the documents produced here under the darktrace key path. Is this a concern?

@vinit-chauhan

vinit-chauhan commented Aug 24, 2022

Copy link
Copy Markdown
Contributor Author

Hey @efd6, Yes the Darktrace API sends the data in such a way. And We can't flatten in as it would result in conflicts in index mappings. So that doesn't seem to be a good idea to flatten it.

@efd6

efd6 commented Aug 24, 2022

Copy link
Copy Markdown
Contributor

Thanks

@jamiehynds

jamiehynds commented Sep 13, 2022

Copy link
Copy Markdown

@andrewkroh @efd6 could we prioritise review on this integration next please? Thanks!

P1llus
P1llus approved these changes Sep 28, 2022
View reviewed changes

@P1llus P1llus left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit concerned about the amount of arrays in the resulting data, but I don't feel there is any good way around it, since creating single documents for each would simply be too many.

I presume there has not been any big challenges in still using them for visualizations?

I am also a bit unsure how this impacts possibility to create SIEM rules for certain content, seeing as they are all stored in arrays.

If this is something that has already been discussed/thought about, feel free to ignore that.

Outside of that I added a few small comments, the rest seems good to go, great work! :)

Comment thread packages/darktrace/_dev/deploy/docker/files/config.yml Outdated
@@ -0,0 +1,13 @@
rules:
- path: /modelbreaches
methods: ["GET"]

@P1llus P1llus Sep 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this API use any sort of authentication? Any way we could add that in as well?

Comment thread packages/darktrace/data_stream/ai_analyst_alert/agent/stream/httpjson.yml.hbs Outdated
@@ -0,0 +1,48 @@
config_version: 2
interval: {{interval}}
request.timeout: 5m

@P1llus P1llus Sep 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be a configurable advanced option which defaults to 5min? We usually allow users to override this in some niche usecases.

Comment thread packages/darktrace/data_stream/model_breach_alert/agent/stream/httpjson.yml.hbs Outdated
@@ -0,0 +1,51 @@
config_version: 2
interval: {{interval}}
request.timeout: 5m

@P1llus P1llus Sep 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, please make this configurable

@P1llus

P1llus commented Sep 28, 2022

Copy link
Copy Markdown
Member

I think maybe we have to discuss the duplicate custom fields, as has been brought up in some other PR comments as well

@vinit-chauhan

vinit-chauhan commented Sep 30, 2022

Copy link
Copy Markdown
Contributor Author

/test

@elasticmachine

elasticmachine commented Oct 3, 2022

Copy link
Copy Markdown

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@P1llus P1llus merged commit b5e9f7e into elastic:main Oct 4, 2022
@andrewkroh andrewkroh mentioned this pull request Oct 4, 2022
Merged
@vinit-chauhan vinit-chauhan deleted the package_darktrace branch April 1, 2026 23:57
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
@vinit-chauhan
[Darktrace] Initial Release for the Darktrace (elastic#4001)
bfbc83f 
* Initial Release for the Darktrace

* Update changelog file

* updated the file names for the pipeline test

* Update request.timeout parameter, tag & user guide

* resolved build failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@P1llus P1llus P1llus approved these changes
@efd6 efd6 efd6 approved these changes

Assignees

@P1llus P1llus

Labels

enhancement New feature or request New Integration Issue or pull request for creating a new integration package.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Darktrace

5 participants

@vinit-chauhan @elasticmachine @efd6 @jamiehynds @P1llus