Skip to content

[Logs] Remove logpull refs in logpush + Add Datadog/Splunk to ToC #803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
crwaters16 merged 6 commits into from
Feb 23, 2021
Merged

[Logs] Remove logpull refs in logpush + Add Datadog/Splunk to ToC #803

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c590858
Removed logpull references, added datadog and splunk references to va…
crwaters16 Feb 18, 2021
964451d
Fixing small formatting error
crwaters16 Feb 18, 2021
5225028
Small content edits
crwaters16 Feb 18, 2021
8148936
Implementing edits and merging in recently updated files
crwaters16 Feb 23, 2021
28becb7
Small typo fix
crwaters16 Feb 23, 2021
fe3b673
small formatting errors
crwaters16 Feb 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions products/logs/src/content/logpush/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,21 @@ order: 40

# Logpush Service

Have logs of your HTTP requests, Spectrum events, or Firewall events pushed to your cloud service in batches as soon as possible.
Push logs of your HTTP requests, Spectrum events, or Firewall events to your cloud service in batches as soon as possible.

Logpush is available to customers on Cloudflare's Enterprise plan.

Logpush can be configured using the [Cloudflare UI (Dashboard)](/logpush/logpush-dashboard/) or [API](/logpush/logpush-configuration-api/).
Configure Logpush using the [Cloudflare UI (Dashboard)](/logpush/logpush-dashboard/) or [API](/logpush/logpush-configuration-api/).

If enabling via the API, you can push to the following storage services:
- [AWS S3](/logpush/aws-s3/)
- [S3-compatible endpoints](/logpush/s3-compatible-endpoints)
- [Google Cloud Storage](/logpush/google-cloud-storage/)
- [Microsoft Azure Blobg Storage](/logpush/azure/)
- [Microsoft Azure Blob Storage](/logpush/azure/)

You can also push to the following SIEMs and log management providers:
You also can push to the following SIEMs and log management providers:
- [Datadog](/logpush/datadog)
- [Splunk](/logpush/splunk)
- [Sumo Logic](/logpush/sumo-logic/)

Cloudflare aims to support additional plans and services in the future. Interested in a particular service? Please take this [survey](https://goo.gl/forms/0KpMfae63WMPjBmD2).
Cloudflare aims to support additional plans and services in the future. Interested in a particular service? Take this [survey](https://goo.gl/forms/0KpMfae63WMPjBmD2).
2 changes: 2 additions & 0 deletions products/logs/src/content/logpush/logpush-configuration-api/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ To get started:
1. Set up a storage provider and grant Cloudflare access. For specific instructions:

- [Enable Amazon S3](/logpush/aws-s3/)
- [Enable Datadog](/logpush/datadog/)
- [Enable Google Cloud Storage](/logpush/google-cloud-storage/)
- [Enable Microsoft Azure](/logpush/azure/)
- [Enable Splunk](/logpush/splunk/)
- [Enable Sumo Logic](/logpush/sumo-logic/)
- [Enable S3-compatible endpoints](/logpush/s3-compatible-endpoints/)

Expand Down
14 changes: 8 additions & 6 deletions ...logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,11 @@ When using Sumo Logic, you may find it helpful to have [Live Tail](https://help.
You can specify your cloud service provider destination via the required `destination_conf` parameter.

* **AWS S3**: bucket + optional directory + region + optional encryption parameter (if required by your policy); for example: `s3://bucket/[dir]?region=<region>[&sse=AES256]`
* **Datadog**: Datadog endpoint URL + Datadog API key + optional parameters; for example: `datadog://<DATADOG-ENDPOINT-URL>?header_DD-API_KEY=<DATADOG-API-KEY>&service=<SERVICE>&host=<HOST>&ddsource=<SOURCE>`

@crwaters16 crwaters16 Feb 18, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@soheiokamoto , I added the examples from the Enable pages we just created for Splunk and Datadog, but they're formatted differently than the other examples. Can I change the <> to [] to follow the other examples, or what do you recommend here?

@soheiokamoto soheiokamoto Feb 19, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know <> indicates parameter value, and [] indicates optional value, so I don't think you should change those for these.

@soheiokamoto soheiokamoto Feb 19, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand, it seems like not all of them are following the convention properly, and use [] for required part, so may be we should fix those.
In any case, I don't have strong opinion, as long as they are consistent throughout this docs (at least within Logs).

* **Google Cloud Storage**: bucket + optional directory; for example: `gs://bucket/[dir]`
* **Microsoft Azure**: service-level SAS URL with `https` replaced by `azure` + optional directory added before query string; for example: `azure://[BlobContainerPath]/[dir]?[QueryString]`
* **Sumo Logic**: HTTP source address URL with `https` replaced by `sumo`; for example: `sumo://[SumoEndpoint]/receiver/v1/http/[UniqueHTTPCollectorCode]`
* **Microsoft Azure**: service-level SAS URL with `https` replaced by `azure` + optional directory added before query string; for example: `azure://<BlobContainerPath>/[dir]?<QueryString>`
* **Splunk**: Splunk endpoint URL + Splunk channel ID + insecure-skip-verify flag + Splunk sourcetype + Splunk authorization token; for example: `splunk://<SPLUNK-ENDPOINT-URL>?channel=<SPLUNK-CHANNEL-ID>&insecure-skip-verify=<INSECURE-SKIP-VERIFY>&sourcetype=<SOURCE-TYPE>&header_Authorization=<SPLUNK-AUTH-TOKEN>`
* **Sumo Logic**: HTTP source address URL with `https` replaced by `sumo`; for example: `sumo://<SumoEndpoint>/receiver/v1/http/<UniqueHTTPCollectorCode>`

For S3, Google Cloud Storage, and Azure, logs can be separated into daily subdirectories by using the special string `{DATE}` in the URL path; for example: `s3://mybucket/logs/{DATE}?region=us-east-1&sse=AES256` or `azure://myblobcontainer/logs/{DATE}?[QueryString]`. It will be substituted with the date in `YYYYMMDD` format, like `20180523`.

Expand Down Expand Up @@ -124,7 +126,7 @@ See a detailed description of the [Logpush object JSON schema](/logpush/logpush-

Logpush repeatedly pulls logs on your behalf and uploads them to your destination.

Log options, such fields or sampling rate, are configured in the `logpull_options` job parameter (*see [Logpush job object schema](/logpush/logpush-configuration-api/job-json-schema/)*). If you're migrating from the Logpull API, `logpull_options` is simply the query string for the API call. For example, the following query gets data from the Logpull API:
Log options, such as fields or sampling rate, are configured in the `logpull_options` job parameter (*see [Logpush job object schema](/logpush/logpush-configuration-api/job-json-schema/)*). For example, the following query gets data from the Logpull API:

```bash
curl -sv \
Expand All @@ -139,9 +141,9 @@ If you don't change any options, you will receive logs with default fields that

The three options that you can customize are:

1. Fields; *see [Log fields](/log-fields/)* for the currently available fields. The list of fields is also accessible directly from the API: `https://api.cloudflare.com/client/v4/zones/<zone_id>/logpush/datasets/<dataset>/fields`. Default fields: `https://api.cloudflare.com/client/v4/zones/<zone_id>/logpush/datasets/<dataset>/fields/default`.
2. Sampling rate; value can range from 0.001 to 1.0 (inclusive). `sample=0.1` means return 10% (1 in 10) of all records.
3. Timestamp format; the format in which timestamp fields will be returned. Value options: unixnano (default), unix, rfc3339.
1. Fields: See *[Log fields](/log-fields/)* for the currently available fields. The list of fields is also accessible directly from the API: `https://api.cloudflare.com/client/v4/zones/<zone_id>/logpush/datasets/<dataset>/fields`. Default fields: `https://api.cloudflare.com/client/v4/zones/<zone_id>/logpush/datasets/<dataset>/fields/default`.
2. Sampling rate: Value can range from 0.001 to 1.0 (inclusive). `sample=0.1` means return 10% (1 in 10) of all records.
3. Timestamp format: The format in which timestamp fields will be returned. Value options: unixnano (default), unix, rfc3339.

To check if `logpull_options` is valid:

Expand Down
53 changes: 25 additions & 28 deletions products/logs/src/content/logpush/logpush-dashboard/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,38 +32,35 @@ To enable the Clouflare Logpush service:

6. Under **Select service**, pick your cloud service provider and click **Next**

<Aside type="note" header="Note">

Complete either *step 7*, *8*, *9*, or *10* below based on you provider, then continue to *step 11*
</Aside>

7. For **Amazon S3**, enter or select the following:
* **Bucket path**
* **Daily subfolders**
* **Bucket region**
**Encryption constraint in bucket policy**
* For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a policy (if you did not add it already):
7. Complete the steps below based on your provider, then continue to *step 8*

* For **Amazon S3**, enter or select the following:
* **Bucket path**
* **Daily subfolders**
* **Bucket region**
* **Encryption constraint in bucket policy**
* For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a policy (if you did not add it already):
* Copy the JSON policy, then go to your bucket in the Amazon S3 console and paste the policy in **Permissions** > **Bucket Policy** and click **Save**

8. For **Google Cloud Storage**, enter or select the following:
* **Bucket path**
* **Daily subfolders**
* For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a Cloudflare IAM user (if you did not add it already):
* See *step 2* in [Enable Google Cloud Storage](/logpush/google-cloud-storage/)
* For **Google Cloud Storage**, enter or select the following:
* **Bucket path**
* **Daily subfolders**
* For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a Cloudflare IAM user (if you did not add it already):
* See *step 2* in [Enable Google Cloud Storage](/logpush/google-cloud-storage/)

9. For **Microsoft Azure**, enter or select the following:
* **SAS URL**
* See [Enable Microsoft Azure](/logpush/azure/) for instructions on creating this URL
* **Blob container subpath (optional)**
* **Daily subfolders**
* For **Microsoft Azure**, enter or select the following:
* **SAS URL**
* See [Enable Microsoft Azure](/logpush/azure/) for instructions on creating this URL
* **Blob container subpath (optional)**
* **Daily subfolders**

10. For **Sumo Logic**, enter or select the following:
* **HTTP Source Address**
* See [Enable Sumo Logic](/logpush/sumo-logic/) for instructions on getting this address
* For **Sumo Logic**, enter or select the following:
* **HTTP Source Address**
* See [Enable Sumo Logic](/logpush/sumo-logic/) for instructions on getting this address

11. In the Cloudflare dashboard, click **Validate access**
8. In the Cloudflare dashboard, click **Validate access**

12. Follow the on-screen instructions to enter the **Ownership token** (included in a file or log Cloudflare sends to your provider), then click **Prove ownership**
9. Follow the on-screen instructions to enter the **Ownership token** (included in a file or log Cloudflare sends to your provider), then click **Prove ownership**

<Aside type="note" header="Note">

Expand All @@ -75,8 +72,8 @@ Tips for seeing the token:

</Aside>

13. Next in the **Customize log** screen, select the data set in which you're interested (currently `HTTP requests` or `Spectrum events`). You can keep the default fields to include in your log or make changes. You can add or remove fields at a later time by modifying your settings in **Logs** > **Logpush** (select the wrench icon).
10. Next in the **Customize log** screen, select the data set in which you're interested (currently `HTTP requests` or `Spectrum events`). You can keep the default fields to include in your log or make changes. You can add or remove fields at a later time by modifying your settings in **Logs** > **Logpush** (select the wrench icon).

14. To finish enabling Logpush, click **Save and Start Pushing**
11. To finish enabling Logpush, click **Save and Start Pushing**

Once connected, Cloudflare lists the provider you just configured under **Logs** > **Logpush**. This is where you can make changes or remove the provider.