[Beaconing] Packaging Network Beaconing Detection by sodhikirti07 · Pull Request #7418 · elastic/integrations

sodhikirti07 · 2023-08-16T15:02:24Z

What does this PR do?

Package the beaconing detection model previously installed via dreml repo.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Points to review:

The README covers the description and steps to configure this package
Check for transform.yml and dashboard objects
Manifest has the right license version and description. Changelog points to correct PR.

How to test this PR locally

Use elastic-package build to build the package and stand up the stack locally using the following command:

                                    elastic-package stack up -d -v --version 8.8.0

Make sure the transform is installed and started before checking the dashboards.

Related issues

Screenshots

Screenshots are tracked in the comments below.

elasticmachine · 2023-08-16T15:08:08Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-04T16:21:20.744+0000
Duration: 17 min 52 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4
Skipped	0
Total	4

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

sodhikirti07 · 2023-08-16T15:09:47Z

Screenshots:

Package's README

Transform installed and collecting data successfully

Destination index of transform collecting data

Dashboards installed

Network Beaconing

Beaconing Drilldown

Hosts Affected Over Time By Process Name

susan-shu-c · 2023-08-16T15:54:04Z

+    - Name: `ml_beaconing_default`
+    - Custom data view ID: `ml_beaconing_default`
+## Licensing
+Usage in production requires that you have a license key that permits use of machine learning features.


I think we didn't use to do it, but it might be good to start linking back to this page (or equivalent)
https://www.elastic.co/subscriptions

Re-thinking: Guess we can remove this line as this package isn't using any ML features.

susan-shu-c · 2023-08-16T15:56:45Z

+                process.name: ["lsass.exe", "Microsoft Outlook", "omadmclient.exe", "OUTLOOK.EXE", "SettingSyncHost.exe", "SIHClient.exe", "POWERPNT.EXE", "svchost.exe", "uhssvc.exe"]
+            must_not:
+              terms:
+                destination.ip: ["10.0.0.0/8", "169.254.0.0/16", "172.16.0.0/12", "127.0.0.0/8", "192.0.0.0/24", "192.0.0.0/29", "192.0.0.8/32", "192.0.0.9/32", "192.0.0.10/32", "192.0.0.170/32", "192.0.0.171/32", "192.0.2.0/24", "192.31.196.0/24", "192.52.193.0/24", "192.168.0.0/16", "192.88.99.0/24", "224.0.0.0/4", "100.64.0.0/10", "192.175.48.0/24", "198.18.0.0/15", "198.51.100.0/24", "203.0.113.0/24", "240.0.0.0/4", "::1", "FE80::/10", "FF00::/8", "13.64.0.0/11", "13.104.0.0/14", "13.96.0.0/13", "18.209.113.128/26", "20.33.0.0/16", "20.34.0.0/15", "20.36.0.0/14", "20.40.0.0/13", "20.48.0.0/12", "20.64.0.0/10", "20.128.0.0/16", "20.36.0.0/14", "20.34.0.0/15", "20.40.0.0/13", "20.128.0.0/16", "20.48.0.0/12", "20.33.0.0/16", "20.180.0.0/14", "20.184.0.0/13", "23.64.0.0/14", "23.32.0.0/11", "40.74.0.0/15", "40.76.0.0/14", "40.80.0.0/12", "40.96.0.0/12", "40.112.0.0/13", "40.120.0.0/14", "40.124.0.0/16", "40.126.0.0/18", "40.125.0.0/17", "52.132.0.0/14", "52.136.0.0/13", "52.148.0.0/14", "52.145.0.0/16", "52.146.0.0/15", "52.160.0.0/11", "52.152.0.0/13", "52.224.0.0/11"]


Quick spot check: did we run linting on this package?

Yes, I did!

Nice! I assumed you did, but hard to tell how the packages gets formatted (in terms of line breaks, lists etc.) as we've seen in the past xD

Co-authored-by: Susan <23287722+susan-shu-c@users.noreply.github.com>

ajosh0504

Awesome work, @sodhikirti07! Thanks for doing this! 🚀

ajosh0504 · 2023-08-22T14:16:52Z

+1. Ensure the pivot transform is installed and running.
+2. Go to **Management > Stack Management > Kibana > Data Views**. Click on **Create data view** button and enable **Allow hidden and system indices** under the **Show Advanced settings**.
+3. Create a data view with the following settings:
+    - Index pattern : `ml_beaconing_default`


Same comment as above re: including space in the index name.

ajosh0504 · 2023-08-22T14:23:52Z

Wonder if we should also include a link to the Beaconing blog if users are curious about the inner workings of the transform itself, and the signals it produces?

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

…ee-b990-0d17f5be0d77.json Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

qn895 · 2023-08-22T23:12:25Z

Transform yml formats LGTM 🎉 Please note that with the use of the ingest pipeline in the transform, every package version update will also require a change in the transform version. This is because the ingest pipeline is named automatically by default by Fleet (e.g. in package version 2.0.0, the pipeline will be named 2.0.0-ml_beaconing_ingest_pipeline). So, even if the content of the ingest pipeline and transform rename the same in future versions, the names in the yml file will still have to be updated.

sodhikirti07 · 2023-08-23T15:21:48Z

Please note that with the use of the ingest pipeline in the transform, every package version update will also require a change in the transform version.

@qn895 I understand that upgrading the package will install a new ingest pipeline and transform named 2.0.0-ml_beaconing_ingest_pipeline and logs-beaconing.pivot_transform-default-2.0.0 respectively, but what would happen to the data collected by the previous transform given that the destination index remains the same?

susan-shu-c · 2023-08-23T15:35:37Z

Curious about this too

what would happen to the data collected by the previous transform given that the destination index remains the same?

If the destination index is set to index-name.all, then would the data just be put into the same index as before? Based on my understanding of the best practices document

sodhikirti07 · 2023-08-24T13:44:36Z

@qn895 Could you take a final look at the transform.yml? I tested the aliases by upgrading and downgrading the package and everything is working as expected.

qn895 · 2023-08-24T14:48:57Z

Latest changes with the aliases and format version LGTM 🎉 To clarify regarding:

If the destination index is set to index-name.all, then would the data just be put into the same index as before? Based on my understanding of the best practices document

The .all postfix is for index aliases. So for the following settings:

  aliases:
    - alias: ml_beaconing.latest
      move_on_creation: true
    - alias: ml_beaconing.all
      move_on_creation: false

In the next version upgrade (let's say for package v2.0.0 and transform version v2.0.0), ml_beaconing.latest alias will be updated to refer to ml_beaconing-2.0.0 index and ml_beaconing.all will refer to both ml_beaconing-1.0.0 and ml_beaconing-2.0.0 indices.

sodhikirti07 · 2023-09-05T20:19:35Z

There's a bug in the latest version release of elastic-package where aliases for the transforms aren't working as intended. Discussed it with @qn895 and she has opened a PR to fix the bug from fleet's end. She'll also spend some time troubleshooting it with the engineer responsible for elasticsearch/backend side of things.

sodhikirti07 · 2023-10-04T15:47:52Z

Tested and installed the package in a serverless environment. The following screenshots confirm that all assets have been successfully installed without encountering any errors.

ajosh0504

LGTM! 🚀

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

elasticmachine · 2023-10-04T20:11:30Z

Package beaconing - 1.0.0 containing this change is available at https://epr.elastic.co/search?package=beaconing

* Packaging network beaconing (with serverless support)

sodhikirti07 added 5 commits July 28, 2023 17:02

Packaging network beaconing

b499e0c

README for beaconing

b573178

Added dashboards for beaconing

65ea1f7

Added search object for beaconing drilldown

625b309

Transform freq change

6370376

sodhikirti07 requested a review from a team August 16, 2023 15:04

sodhikirti07 added 2 commits August 16, 2023 11:12

Updated CODEOWNERS

28ff3a4

Updated changelog

4e8355f

sodhikirti07 requested a review from a team August 16, 2023 15:14

susan-shu-c reviewed Aug 16, 2023

View reviewed changes

sodhikirti07 and others added 3 commits August 16, 2023 13:08

Update packages/beaconing/docs/README.md

f5922dd

Co-authored-by: Susan <23287722+susan-shu-c@users.noreply.github.com>

Update packages/beaconing/docs/README.md

e1b6564

Co-authored-by: Susan <23287722+susan-shu-c@users.noreply.github.com>

Removed ML licensing requirements

e553979

ajosh0504 approved these changes Aug 22, 2023

View reviewed changes

sodhikirti07 and others added 9 commits August 22, 2023 10:26

Update packages/beaconing/docs/README.md

a48f7db

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

Update packages/beaconing/docs/README.md

5315778

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

Update packages/beaconing/docs/README.md

1c963cc

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

Update packages/beaconing/docs/README.md

023da85

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

Update packages/beaconing/kibana/dashboard/beaconing-7a2c6260-2d65-11…

cce5734

…ee-b990-0d17f5be0d77.json Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

Update README.md

0e649ac

Changed destination index name in transform.yml

48ffd72

Update package name in manifest.yml

74d0260

Changed index name in dashboards and minor correction to README

fe43bc1

qn895 self-requested a review August 22, 2023 15:27

sodhikirti07 added 3 commits August 23, 2023 14:10

Added version and aliases for dest index

867d9f3

Added alias info in README

ad2c3a4

Updated format version

350c18e

sodhikirti07 requested a review from susan-shu-c August 24, 2023 13:44

qn895 approved these changes Aug 24, 2023

View reviewed changes

sodhikirti07 requested a review from SourinPaul August 30, 2023 20:29

sodhikirti07 added 2 commits October 3, 2023 09:23

Updated kibana version in manifest.yml

d82a811

Update manifest to work with serverless

73c7722

sodhikirti07 requested a review from ajosh0504 October 4, 2023 13:31

Changed format version and added filter to avoid validation errors

8481c33

ajosh0504 reviewed Oct 4, 2023

View reviewed changes

Comment thread packages/beaconing/manifest.yml

Comment thread packages/beaconing/docs/README.md

Comment thread packages/beaconing/docs/README.md

Comment thread packages/beaconing/changelog.yml Outdated

sodhikirti07 and others added 3 commits October 4, 2023 12:19

Update packages/beaconing/manifest.yml

6b4343d

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

Update packages/beaconing/docs/README.md

20441ad

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

Update packages/beaconing/changelog.yml

5ce96cd

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

sodhikirti07 merged commit e199608 into main Oct 4, 2023

andrewkroh added the Integration:beaconing Network Beaconing Identification label Jul 22, 2024

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024

orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026

[Beaconing] Packaging Network Beaconing Detection (elastic#7418)

c9f4daf

* Packaging network beaconing (with serverless support)

Uh oh!

Conversation

sodhikirti07 commented Aug 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Aug 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

sodhikirti07 commented Aug 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

susan-shu-c Aug 16, 2023

Choose a reason for hiding this comment

Uh oh!

sodhikirti07 Aug 16, 2023

Choose a reason for hiding this comment

Uh oh!

susan-shu-c Aug 16, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

susan-shu-c Aug 16, 2023

Choose a reason for hiding this comment

Uh oh!

sodhikirti07 Aug 16, 2023

Choose a reason for hiding this comment

Uh oh!

susan-shu-c Aug 16, 2023

Choose a reason for hiding this comment

Uh oh!

ajosh0504 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajosh0504 Aug 22, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ajosh0504 commented Aug 22, 2023

Uh oh!

qn895 commented Aug 22, 2023

Uh oh!

sodhikirti07 commented Aug 23, 2023

Uh oh!

susan-shu-c commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sodhikirti07 commented Aug 24, 2023

Uh oh!

qn895 commented Aug 24, 2023

Uh oh!

sodhikirti07 commented Sep 5, 2023

Uh oh!

sodhikirti07 commented Oct 4, 2023

Uh oh!

ajosh0504 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Oct 4, 2023

Uh oh!

sodhikirti07 commented Aug 16, 2023 •

edited

Loading

elasticmachine commented Aug 16, 2023 •

edited

Loading

sodhikirti07 commented Aug 16, 2023 •

edited

Loading

susan-shu-c commented Aug 23, 2023 •

edited

Loading