From c590858228b406fd7ba176c289c65e6cfb6e156e Mon Sep 17 00:00:00 2001 From: Claire Waters Date: Thu, 18 Feb 2021 10:31:59 -0600 Subject: [PATCH 1/5] Removed logpull references, added datadog and splunk references to various pages --- products/logs/src/content/logpush/index.md | 13 ++++++------ .../logpush-configuration-api/index.md | 2 ++ .../understanding-logpush-api.md | 10 +++++---- .../logpush/logpush-dashboard/index.md | 21 ++++++++----------- 4 files changed, 24 insertions(+), 22 deletions(-) diff --git a/products/logs/src/content/logpush/index.md b/products/logs/src/content/logpush/index.md index 11f9d7d748e..c254bfa7077 100644 --- a/products/logs/src/content/logpush/index.md +++ b/products/logs/src/content/logpush/index.md @@ -5,11 +5,9 @@ order: 40 # Logpush Service -Have logs of your HTTP requests, Spectrum events, or Firewall events pushed to your cloud service in batches as soon as possible. +Push logs of your HTTP requests, Spectrum events, or Firewall events to your cloud service in batches as soon as possible. -If you're already using the Logpull API to download HTTP request logs, you can get the same functionality—including selecting fields and sampling—in Logpush. - -Logpush can be configured using one of the following methods: +Configure Logpush using one of the following methods: - [Cloudflare UI (Dashboard)](/logpush/logpush-dashboard/) - [Cloudflare Logpush API](/logpush/logpush-configuration-api/) @@ -23,10 +21,13 @@ If following the UI method, you'll be prompted to allow Cloudflare to access you Currently, Logpush is available to customers on the Cloudflare Enterprise plan and for the following services: - Amazon S3 +- Datadog - Google Cloud Storage (GCS) - Microsoft Azure +- Splunk - Sumo Logic -Customers using IBM Cloud Internet Services (CIS) on Enterprise accounts, powered by Cloudflare, can push their logs to [IBM's Cloud Object Storage (COS)](https://cloud.ibm.com/docs/cis?topic=cis-logpush) + +Customers using IBM Cloud Internet Services (CIS) on Enterprise accounts powered by Cloudflare can push their logs to [IBM's Cloud Object Storage (COS)](https://cloud.ibm.com/docs/cis?topic=cis-logpush) Cloudflare Logpush also supports S3-compatible endpoints in an API-only beta, which includes: - [Digital Ocean Spaces](https://www.digitalocean.com/docs/spaces/) @@ -39,4 +40,4 @@ Cloudflare Logpush also supports S3-compatible endpoints in an API-only beta, wh See instructions to [Enable S3-compatible endpoints](/logpush/s3-compatible-endpoints). -Cloudflare aims to support additional plans and services in the future. Interested in a particular service? Please take this [survey](https://goo.gl/forms/0KpMfae63WMPjBmD2). +Cloudflare aims to support additional plans and services in the future. Interested in a particular service? Take this [survey](https://goo.gl/forms/0KpMfae63WMPjBmD2). diff --git a/products/logs/src/content/logpush/logpush-configuration-api/index.md b/products/logs/src/content/logpush/logpush-configuration-api/index.md index 3ea524bcb67..28a566ffc5e 100644 --- a/products/logs/src/content/logpush/logpush-configuration-api/index.md +++ b/products/logs/src/content/logpush/logpush-configuration-api/index.md @@ -14,8 +14,10 @@ To get started: 1. Set up a storage provider and grant Cloudflare access. For specific instructions: - [Enable Amazon S3](/logpush/aws-s3/) + - [Enable Datadog](/logpush/datadog/) - [Enable Google Cloud Storage](/logpush/google-cloud-storage/) - [Enable Microsoft Azure](/logpush/azure/) + - [Enable Splunk](/logpush/splunk/) - [Enable Sumo Logic](/logpush/sumo-logic/) - [Enable S3-compatible endpoints](/logpush/s3-compatible-endpoints/) diff --git a/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md b/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md index 5d93c5e8987..68cf60fd077 100644 --- a/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md +++ b/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md @@ -79,8 +79,10 @@ When using Sumo Logic, you may find it helpful to have [Live Tail](https://help. You can specify your cloud service provider destination via the required `destination_conf` parameter. * **AWS S3**: bucket + optional directory + region + optional encryption parameter (if required by your policy); for example: `s3://bucket/[dir]?region=[&sse=AES256]` +* **Datadog**: Datadog endpoint URL + Datadog API key + optional parameters; for example: `datadog://?header_DD-API_KEY=&service=&host=&ddsource=` * **Google Cloud Storage**: bucket + optional directory; for example: `gs://bucket/[dir]` * **Microsoft Azure**: service-level SAS URL with `https` replaced by `azure` + optional directory added before query string; for example: `azure://[BlobContainerPath]/[dir]?[QueryString]` +* **Splunk**: Splunk endpoint URL + Splunk channel ID + insecure-skip-verify flag + Splunk sourcetype + Splunk authorization token; for example: `splunk://?channel=&insecure-skip-verify=&sourcetype=&header_Authorization=` * **Sumo Logic**: HTTP source address URL with `https` replaced by `sumo`; for example: `sumo://[SumoEndpoint]/receiver/v1/http/[UniqueHTTPCollectorCode]` For S3, Google Cloud Storage, and Azure, logs can be separated into daily subdirectories by using the special string `{DATE}` in the URL path; for example: `s3://mybucket/logs/{DATE}?region=us-east-1&sse=AES256` or `azure://myblobcontainer/logs/{DATE}?[QueryString]`. It will be substituted with the date in `YYYYMMDD` format, like `20180523`. @@ -124,7 +126,7 @@ See a detailed description of the [Logpush object JSON schema](/logpush/logpush- Logpush repeatedly pulls logs on your behalf and uploads them to your destination. -Log options, such fields or sampling rate, are configured in the `logpull_options` job parameter (*see [Logpush job object schema](/logpush/logpush-configuration-api/job-json-schema/)*). If you're migrating from the Logpull API, `logpull_options` is simply the query string for the API call. For example, the following query gets data from the Logpull API: +Log options, such fields or sampling rate, are configured in the `logpull_options` job parameter (*see [Logpush job object schema](/logpush/logpush-configuration-api/job-json-schema/)*). For example, the following query gets data from the Logpull API: ```bash curl -sv \ @@ -139,9 +141,9 @@ If you don't change any options, you will receive logs with default fields that The three options that you can customize are: -1. Fields; *see [Log fields](/log-fields/)* for the currently available fields. The list of fields is also accessible directly from the API: `https://api.cloudflare.com/client/v4/zones//logpush/datasets//fields`. Default fields: `https://api.cloudflare.com/client/v4/zones//logpush/datasets//fields/default`. -2. Sampling rate; value can range from 0.001 to 1.0 (inclusive). `sample=0.1` means return 10% (1 in 10) of all records. -3. Timestamp format; the format in which timestamp fields will be returned. Value options: unixnano (default), unix, rfc3339. +1. Fields: See *[Log fields](/log-fields/)* for the currently available fields. The list of fields is also accessible directly from the API: `https://api.cloudflare.com/client/v4/zones//logpush/datasets//fields`. Default fields: `https://api.cloudflare.com/client/v4/zones//logpush/datasets//fields/default`. +2. Sampling rate: Value can range from 0.001 to 1.0 (inclusive). `sample=0.1` means return 10% (1 in 10) of all records. +3. Timestamp format: The format in which timestamp fields will be returned. Value options: unixnano (default), unix, rfc3339. To check if `logpull_options` is valid: diff --git a/products/logs/src/content/logpush/logpush-dashboard/index.md b/products/logs/src/content/logpush/logpush-dashboard/index.md index 485b096487d..c245dc237de 100644 --- a/products/logs/src/content/logpush/logpush-dashboard/index.md +++ b/products/logs/src/content/logpush/logpush-dashboard/index.md @@ -32,12 +32,9 @@ To enable the Clouflare Logpush service: 6. Under **Select service**, pick your cloud service provider and click **Next** - +7. Complete the steps below based on your provider, then continue to *step 8* -7. For **Amazon S3**, enter or select the following: + * For **Amazon S3**, enter or select the following: * **Bucket path** * **Daily subfolders** * **Bucket region** @@ -45,25 +42,25 @@ Complete either *step 7*, *8*, *9*, or *10* below based on you provider, then co * For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a policy (if you did not add it already): * Copy the JSON policy, then go to your bucket in the Amazon S3 console and paste the policy in **Permissions** > **Bucket Policy** and click **Save** -8. For **Google Cloud Storage**, enter or select the following: + * For **Google Cloud Storage**, enter or select the following: * **Bucket path** * **Daily subfolders** * For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a Cloudflare IAM user (if you did not add it already): * See *step 2* in [Enable Google Cloud Storage](/logpush/google-cloud-storage/) -9. For **Microsoft Azure**, enter or select the following: + * For **Microsoft Azure**, enter or select the following: * **SAS URL** * See [Enable Microsoft Azure](/logpush/azure/) for instructions on creating this URL * **Blob container subpath (optional)** * **Daily subfolders** -10. For **Sumo Logic**, enter or select the following: + * For **Sumo Logic**, enter or select the following: * **HTTP Source Address** * See [Enable Sumo Logic](/logpush/sumo-logic/) for instructions on getting this address -11. In the Cloudflare dashboard, click **Validate access** +8. In the Cloudflare dashboard, click **Validate access** -12. Follow the on-screen instructions to enter the **Ownership token** (included in a file or log Cloudflare sends to your provider), then click **Prove ownership** +9. Follow the on-screen instructions to enter the **Ownership token** (included in a file or log Cloudflare sends to your provider), then click **Prove ownership** -13. Next in the **Customize log** screen, select the data set in which you're interested (currently `HTTP requests` or `Spectrum events`). You can keep the default fields to include in your log or make changes. You can add or remove fields at a later time by modifying your settings in **Logs** > **Logpush** (select the wrench icon). +10. Next in the **Customize log** screen, select the data set in which you're interested (currently `HTTP requests` or `Spectrum events`). You can keep the default fields to include in your log or make changes. You can add or remove fields at a later time by modifying your settings in **Logs** > **Logpush** (select the wrench icon). -14. To finish enabling Logpush, click **Save and Start Pushing** +11. To finish enabling Logpush, click **Save and Start Pushing**w Once connected, Cloudflare lists the provider you just configured under **Logs** > **Logpush**. This is where you can make changes or remove the provider. \ No newline at end of file From 964451ddd1402c2c2494aecbaae83097bb954dd9 Mon Sep 17 00:00:00 2001 From: Claire Waters Date: Thu, 18 Feb 2021 10:44:15 -0600 Subject: [PATCH 2/5] Fixing small formatting error --- .../logpush/logpush-dashboard/index.md | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/products/logs/src/content/logpush/logpush-dashboard/index.md b/products/logs/src/content/logpush/logpush-dashboard/index.md index c245dc237de..9a147307e8c 100644 --- a/products/logs/src/content/logpush/logpush-dashboard/index.md +++ b/products/logs/src/content/logpush/logpush-dashboard/index.md @@ -35,28 +35,28 @@ To enable the Clouflare Logpush service: 7. Complete the steps below based on your provider, then continue to *step 8* * For **Amazon S3**, enter or select the following: - * **Bucket path** - * **Daily subfolders** - * **Bucket region** - **Encryption constraint in bucket policy** + * **Bucket path** + * * **Daily subfolders** + * * **Bucket region** + * **Encryption constraint in bucket policy** * For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a policy (if you did not add it already): - * Copy the JSON policy, then go to your bucket in the Amazon S3 console and paste the policy in **Permissions** > **Bucket Policy** and click **Save** + * Copy the JSON policy, then go to your bucket in the Amazon S3 console and paste the policy in **Permissions** > **Bucket Policy** and click **Save** * For **Google Cloud Storage**, enter or select the following: - * **Bucket path** - * **Daily subfolders** - * For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a Cloudflare IAM user (if you did not add it already): - * See *step 2* in [Enable Google Cloud Storage](/logpush/google-cloud-storage/) + * **Bucket path** + * **Daily subfolders** + * For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a Cloudflare IAM user (if you did not add it already): + * See *step 2* in [Enable Google Cloud Storage](/logpush/google-cloud-storage/) * For **Microsoft Azure**, enter or select the following: - * **SAS URL** - * See [Enable Microsoft Azure](/logpush/azure/) for instructions on creating this URL - * **Blob container subpath (optional)** - * **Daily subfolders** + * **SAS URL** + * See [Enable Microsoft Azure](/logpush/azure/) for instructions on creating this URL + * **Blob container subpath (optional)** + * **Daily subfolders** * For **Sumo Logic**, enter or select the following: - * **HTTP Source Address** - * See [Enable Sumo Logic](/logpush/sumo-logic/) for instructions on getting this address + * **HTTP Source Address** + * See [Enable Sumo Logic](/logpush/sumo-logic/) for instructions on getting this address 8. In the Cloudflare dashboard, click **Validate access** From 5225028a235472f7e1ef851d9fa9f7a8a1168138 Mon Sep 17 00:00:00 2001 From: Claire Waters Date: Thu, 18 Feb 2021 10:57:19 -0600 Subject: [PATCH 3/5] Small content edits --- .../logs/src/content/logpush/logpush-dashboard/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/products/logs/src/content/logpush/logpush-dashboard/index.md b/products/logs/src/content/logpush/logpush-dashboard/index.md index 9a147307e8c..391797b3475 100644 --- a/products/logs/src/content/logpush/logpush-dashboard/index.md +++ b/products/logs/src/content/logpush/logpush-dashboard/index.md @@ -36,8 +36,8 @@ To enable the Clouflare Logpush service: * For **Amazon S3**, enter or select the following: * **Bucket path** - * * **Daily subfolders** - * * **Bucket region** + * **Daily subfolders** + * **Bucket region** * **Encryption constraint in bucket policy** * For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a policy (if you did not add it already): * Copy the JSON policy, then go to your bucket in the Amazon S3 console and paste the policy in **Permissions** > **Bucket Policy** and click **Save** @@ -74,6 +74,6 @@ Tips for seeing the token: 10. Next in the **Customize log** screen, select the data set in which you're interested (currently `HTTP requests` or `Spectrum events`). You can keep the default fields to include in your log or make changes. You can add or remove fields at a later time by modifying your settings in **Logs** > **Logpush** (select the wrench icon). -11. To finish enabling Logpush, click **Save and Start Pushing**w +11. To finish enabling Logpush, click **Save and Start Pushing**s Once connected, Cloudflare lists the provider you just configured under **Logs** > **Logpush**. This is where you can make changes or remove the provider. \ No newline at end of file From 28becb73b56a607e87b39cb9cd947ecdf2e2f09e Mon Sep 17 00:00:00 2001 From: Claire Waters Date: Tue, 23 Feb 2021 09:00:38 -0600 Subject: [PATCH 4/5] Small typo fix --- .../logpush-configuration-api/understanding-logpush-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md b/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md index 320fe52c478..f4b6664d27c 100644 --- a/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md +++ b/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md @@ -126,7 +126,7 @@ See a detailed description of the [Logpush object JSON schema](/logpush/logpush- Logpush repeatedly pulls logs on your behalf and uploads them to your destination. -Log options, such fields or sampling rate, are configured in the `logpull_options` job parameter (*see [Logpush job object schema](/logpush/logpush-configuration-api/job-json-schema/)*). For example, the following query gets data from the Logpull API: +Log options, such as fields or sampling rate, are configured in the `logpull_options` job parameter (*see [Logpush job object schema](/logpush/logpush-configuration-api/job-json-schema/)*). For example, the following query gets data from the Logpull API: ```bash curl -sv \ From fe3b67313b295e8710f0cee115e298ea467da292 Mon Sep 17 00:00:00 2001 From: Claire Waters Date: Tue, 23 Feb 2021 09:10:49 -0600 Subject: [PATCH 5/5] small formatting errors --- .../logpush-configuration-api/understanding-logpush-api.md | 2 +- products/logs/src/content/logpush/logpush-dashboard/index.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md b/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md index f4b6664d27c..e096233bee1 100644 --- a/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md +++ b/products/logs/src/content/logpush/logpush-configuration-api/understanding-logpush-api.md @@ -81,7 +81,7 @@ You can specify your cloud service provider destination via the required `destin * **AWS S3**: bucket + optional directory + region + optional encryption parameter (if required by your policy); for example: `s3://bucket/[dir]?region=[&sse=AES256]` * **Datadog**: Datadog endpoint URL + Datadog API key + optional parameters; for example: `datadog://?header_DD-API_KEY=&service=&host=&ddsource=` * **Google Cloud Storage**: bucket + optional directory; for example: `gs://bucket/[dir]` -* **Microsoft Azure**: service-level SAS URL with `https` replaced by `azure` + optional directory added before query string; for example: `azure:///[dir]?` +* **Microsoft Azure**: service-level SAS URL with `https` replaced by `azure` + optional directory added before query string; for example: `azure:///[dir]?` * **Splunk**: Splunk endpoint URL + Splunk channel ID + insecure-skip-verify flag + Splunk sourcetype + Splunk authorization token; for example: `splunk://?channel=&insecure-skip-verify=&sourcetype=&header_Authorization=` * **Sumo Logic**: HTTP source address URL with `https` replaced by `sumo`; for example: `sumo:///receiver/v1/http/` diff --git a/products/logs/src/content/logpush/logpush-dashboard/index.md b/products/logs/src/content/logpush/logpush-dashboard/index.md index 9ab0b675d81..8d25206a8c0 100644 --- a/products/logs/src/content/logpush/logpush-dashboard/index.md +++ b/products/logs/src/content/logpush/logpush-dashboard/index.md @@ -39,8 +39,8 @@ To enable the Clouflare Logpush service: * **Daily subfolders** * **Bucket region** * **Encryption constraint in bucket policy** - * For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a policy (if you did not add it already): - * Copy the JSON policy, then go to your bucket in the Amazon S3 console and paste the policy in **Permissions** > **Bucket Policy** and click **Save** + * For **Grant Cloudflare access to upload files to your bucket**, make sure your bucket has a policy (if you did not add it already): + * Copy the JSON policy, then go to your bucket in the Amazon S3 console and paste the policy in **Permissions** > **Bucket Policy** and click **Save** * For **Google Cloud Storage**, enter or select the following: * **Bucket path**