From 7babf6f14c160936bf50bed98a7ffbddf17cc650 Mon Sep 17 00:00:00 2001 From: Jill Osborne Date: Fri, 26 Aug 2022 15:16:51 +0100 Subject: [PATCH 1/6] Firehose migration doc --- .../migrate-from-firehose-ingestion.md | 196 ++++++++++++++++++ docs/ingestion/native-batch-firehose.md | 11 +- website/sidebars.json | 1 + 3 files changed, 202 insertions(+), 6 deletions(-) create mode 100644 docs/ingestion/migrate-from-firehose-ingestion.md diff --git a/docs/ingestion/migrate-from-firehose-ingestion.md b/docs/ingestion/migrate-from-firehose-ingestion.md new file mode 100644 index 000000000000..035d63cc75e2 --- /dev/null +++ b/docs/ingestion/migrate-from-firehose-ingestion.md @@ -0,0 +1,196 @@ +--- +id: migrate-from-firehose +title: "Migrate from firehose to input source ingestion" +sidebar_label: "Migrate from firehose" +--- + + + +Apache deprecated support for Druid firehoses in version 0.17. Support for firehose ingestion was removed in LTS version 0.24. + +If you're using a firehose for batch ingestion, we strongly recommend that you follow the instructions on this page to transition to using native batch ingestion input sources as soon as possible. + +Firehose ingestion doesn't work with newer Druid versions, so you must be using an ingestion spec with a defined input source before you upgrade. + +## Migrate from firehose ingestion to an input source + +To migrate from firehose ingestion, copy your existing ingestion spec into a new file. Refer to [Native batch ingestion with firehose (Deprecated)](./native-batch-firehose.md) for a description of firehose properties. + +Edit the new file as follows: + +1. In the `ioConfig` component, replace the `firehose` definition with an `inputSource` definition for your chosen input source. See [Native batch input sources](./native-batch-input-sources) for details. +2. Move the `timeStampSpec` definition from `parser.parseSpec` to the `dataSchema` component. +3. Move the `dimensionsSpec` definition from `parser.parseSpec` to the `dataSchema` component. +4. Move the `format` definition from `parser.parseSpec` to an `inputFormat` definition in `ioConfig`. +5. Delete the `parser` definition. +6. Save the file. +
You can check the format of your new ingestion file against the [migrated example](#example-ingestion-spec-after-migration) below. +7. Test the new ingestion spec with a temporary data source. +8. Once you've successfully ingested sample data with the new spec, stop firehose ingestion and switch to the new spec. + +When the transition is complete, you can upgrade Druid to the latest version. See the [Druid release notes](https://druid.apache.org/downloads.html) for upgrade instructions. + +### Example firehose ingestion spec + +An example firehose ingestion spec is as follows: + +```json +{ + "type" : "index", + "spec" : { + "dataSchema" : { + "dataSource" : "wikipedia", + "metricsSpec" : [ + { + "type" : "count", + "name" : "count" + }, + { + "type" : "doubleSum", + "name" : "added", + "fieldName" : "added" + }, + { + "type" : "doubleSum", + "name" : "deleted", + "fieldName" : "deleted" + }, + { + "type" : "doubleSum", + "name" : "delta", + "fieldName" : "delta" + } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "DAY", + "queryGranularity" : "NONE", + "intervals" : [ "2013-08-31/2013-09-01" ] + }, + "parser": { + "parseSpec": { + "format": "json", + "timestampSpec" : { + "column" : "timestamp", + "format" : "auto" + }, + "dimensionsSpec" : { + "dimensions": ["country", "page","language","user","unpatrolled","newPage","robot","anonymous","namespace","continent","region","city"], + "dimensionExclusions" : [] + } + } + } + }, + "ioConfig" : { + "type" : "index", + "firehose" : { + "type" : "local", + "baseDir" : "examples/indexing/", + "filter" : "wikipedia_data.json" + } + }, + "tuningConfig" : { + "type" : "index", + "partitionsSpec": { + "type": "single_dim", + "partitionDimension": "country", + "targetRowsPerSegment": 5000000 + } + } + } +} +``` + +### Example ingestion spec after migration + +The following example illustrates the result of migrating the [example firehose ingestion spec](#example-firehose-ingestion-spec) to a spec with an input source: + +```json +{ + "type" : "index", + "spec" : { + "dataSchema" : { + "dataSource" : "wikipedia", + "timestampSpec" : { + "column" : "timestamp", + "format" : "auto" + }, + "dimensionsSpec" : { + "dimensions": ["country", "page","language","user","unpatrolled","newPage","robot","anonymous","namespace","continent","region","city"], + "dimensionExclusions" : [] + }, + "metricsSpec" : [ + { + "type" : "count", + "name" : "count" + }, + { + "type" : "doubleSum", + "name" : "added", + "fieldName" : "added" + }, + { + "type" : "doubleSum", + "name" : "deleted", + "fieldName" : "deleted" + }, + { + "type" : "doubleSum", + "name" : "delta", + "fieldName" : "delta" + } + ], + "granularitySpec" : { + "type" : "uniform", + "segmentGranularity" : "DAY", + "queryGranularity" : "NONE", + "intervals" : [ "2013-08-31/2013-09-01" ] + } + }, + "ioConfig" : { + "type" : "index", + "inputSource" : { + "type" : "local", + "baseDir" : "examples/indexing/", + "filter" : "wikipedia_data.json" + }, + "inputFormat": { + "type": "json" + } + }, + "tuningConfig" : { + "type" : "index", + "partitionsSpec": { + "type": "single_dim", + "partitionDimension": "country", + "targetRowsPerSegment": 5000000 + } + } + } +} +``` + +## Learn more + +For more information, see the following pages: + +- [Ingestion](./index.md): Overview of the Druid ingestion process. +- [Native batch ingestion](./native-batch.md): Description of the supported native batch indexing tasks. +- [Ingestion spec reference](./ingestion-spec.md): Description of the components and properties in the ingestion spec. diff --git a/docs/ingestion/native-batch-firehose.md b/docs/ingestion/native-batch-firehose.md index a6c76b53ccc8..63970a6bb3bc 100644 --- a/docs/ingestion/native-batch-firehose.md +++ b/docs/ingestion/native-batch-firehose.md @@ -1,6 +1,6 @@ --- id: native-batch-firehose -title: "Native batch ingestion with firehose" +title: "Native batch ingestion with firehose (Deprecated)" sidebar_label: "Firehose" --- @@ -23,14 +23,13 @@ sidebar_label: "Firehose" ~ under the License. --> - -Firehoses are deprecated in 0.17.0. It's highly recommended to use the [Native batch ingestion input sources](./native-batch-input-source.md) instead. +> Firehose ingestion is deprecated. See [Migrate from firehose to input source ingestion](./migrate-from-firehose-ingestion.md) for instructions on migrating from firehose ingestion to using native batch ingestion input sources. There are several firehoses readily available in Druid, some are meant for examples, others can be used directly in a production environment. ## StaticS3Firehose -> You need to include the [`druid-s3-extensions`](../development/extensions-core/s3.md) as an extension to use the StaticS3Firehose. +You need to include the [`druid-s3-extensions`](../development/extensions-core/s3.md) as an extension to use the StaticS3Firehose. This firehose ingests events from a predefined list of S3 objects. This firehose is _splittable_ and can be used by the [Parallel task](./native-batch.md). @@ -62,7 +61,7 @@ Note that prefetching or caching isn't that useful in the Parallel task. ## StaticGoogleBlobStoreFirehose -> You need to include the [`druid-google-extensions`](../development/extensions-core/google.md) as an extension to use the StaticGoogleBlobStoreFirehose. +You need to include the [`druid-google-extensions`](../development/extensions-core/google.md) as an extension to use the StaticGoogleBlobStoreFirehose. This firehose ingests events, similar to the StaticS3Firehose, but from an Google Cloud Store. @@ -112,7 +111,7 @@ Google Blobs: ## HDFSFirehose -> You need to include the [`druid-hdfs-storage`](../development/extensions-core/hdfs.md) as an extension to use the HDFSFirehose. +You need to include the [`druid-hdfs-storage`](../development/extensions-core/hdfs.md) as an extension to use the HDFSFirehose. This firehose ingests events from a predefined list of files from the HDFS storage. This firehose is _splittable_ and can be used by the [Parallel task](./native-batch.md). diff --git a/website/sidebars.json b/website/sidebars.json index c414181d6180..f6d2e167d17c 100644 --- a/website/sidebars.json +++ b/website/sidebars.json @@ -59,6 +59,7 @@ "ingestion/native-batch", "ingestion/native-batch-simple-task", "ingestion/native-batch-input-sources", + "ingestion/migrate-from-firehose", "ingestion/native-batch-firehose", "ingestion/hadoop" ] From eb4ade7ff49cbf871b6c563f2c951a037cf412d8 Mon Sep 17 00:00:00 2001 From: Jill Osborne Date: Mon, 26 Sep 2022 13:44:04 +0100 Subject: [PATCH 2/6] Update migrate-from-firehose-ingestion.md --- docs/ingestion/migrate-from-firehose-ingestion.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/ingestion/migrate-from-firehose-ingestion.md b/docs/ingestion/migrate-from-firehose-ingestion.md index 035d63cc75e2..ca7a24fa1568 100644 --- a/docs/ingestion/migrate-from-firehose-ingestion.md +++ b/docs/ingestion/migrate-from-firehose-ingestion.md @@ -31,7 +31,15 @@ Firehose ingestion doesn't work with newer Druid versions, so you must be using ## Migrate from firehose ingestion to an input source -To migrate from firehose ingestion, copy your existing ingestion spec into a new file. Refer to [Native batch ingestion with firehose (Deprecated)](./native-batch-firehose.md) for a description of firehose properties. +To migrate from firehose ingestion, you can use the Druid console to update your ingestion spec, or you can update it manally. + +### Use the Druid console + + + +### Update your ingestion spec manually + +To update your ingestion spec manually, copy your existing spec into a new file. Refer to [Native batch ingestion with firehose (Deprecated)](./native-batch-firehose.md) for a description of firehose properties. Edit the new file as follows: From c5fa67b9aaffb84af96346be500890beb98c05a7 Mon Sep 17 00:00:00 2001 From: Jill Osborne Date: Mon, 26 Sep 2022 13:53:47 +0100 Subject: [PATCH 3/6] Updated with review comments and suggestions --- docs/ingestion/migrate-from-firehose-ingestion.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/ingestion/migrate-from-firehose-ingestion.md b/docs/ingestion/migrate-from-firehose-ingestion.md index ca7a24fa1568..b4b7419ef05a 100644 --- a/docs/ingestion/migrate-from-firehose-ingestion.md +++ b/docs/ingestion/migrate-from-firehose-ingestion.md @@ -35,7 +35,11 @@ To migrate from firehose ingestion, you can use the Druid console to update your ### Use the Druid console +To update your ingestion spec using the Druid console, open the console and copy your spec into the **Edit spec** stage of the data loader. +Druid converts the spec into one with a defined input source. For example, it converts the [example firehose ingestion spec](#example-firehose-ingestion-spec) below into the [example ingestion spec after migration](#example-ingestion-spec-after-migration). + +If you're unable to use the console or you have problems with the console method, the alternative is to update your ingestion spec manually. ### Update your ingestion spec manually @@ -93,6 +97,7 @@ An example firehose ingestion spec is as follows: "intervals" : [ "2013-08-31/2013-09-01" ] }, "parser": { + "type": "string", "parseSpec": { "format": "json", "timestampSpec" : { From cd22ac80fac88e46474a477cb9a90be7c6eecefb Mon Sep 17 00:00:00 2001 From: Jill Osborne Date: Thu, 10 Nov 2022 11:37:31 +0000 Subject: [PATCH 4/6] Update migrate-from-firehose-ingestion.md --- docs/ingestion/migrate-from-firehose-ingestion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ingestion/migrate-from-firehose-ingestion.md b/docs/ingestion/migrate-from-firehose-ingestion.md index b4b7419ef05a..256e7ffa4215 100644 --- a/docs/ingestion/migrate-from-firehose-ingestion.md +++ b/docs/ingestion/migrate-from-firehose-ingestion.md @@ -47,7 +47,7 @@ To update your ingestion spec manually, copy your existing spec into a new file. Edit the new file as follows: -1. In the `ioConfig` component, replace the `firehose` definition with an `inputSource` definition for your chosen input source. See [Native batch input sources](./native-batch-input-sources) for details. +1. In the `ioConfig` component, replace the `firehose` definition with an `inputSource` definition for your chosen input source. See [Native batch input sources](./native-batch-input-sources.md) for details. 2. Move the `timeStampSpec` definition from `parser.parseSpec` to the `dataSchema` component. 3. Move the `dimensionsSpec` definition from `parser.parseSpec` to the `dataSchema` component. 4. Move the `format` definition from `parser.parseSpec` to an `inputFormat` definition in `ioConfig`. From e56819490a6de51915b9f48c3a73b76eb4a28f6d Mon Sep 17 00:00:00 2001 From: Jill Osborne Date: Thu, 10 Nov 2022 13:55:49 +0000 Subject: [PATCH 5/6] Update migrate-from-firehose-ingestion.md --- docs/ingestion/migrate-from-firehose-ingestion.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ingestion/migrate-from-firehose-ingestion.md b/docs/ingestion/migrate-from-firehose-ingestion.md index 256e7ffa4215..60463b51ae45 100644 --- a/docs/ingestion/migrate-from-firehose-ingestion.md +++ b/docs/ingestion/migrate-from-firehose-ingestion.md @@ -23,7 +23,7 @@ sidebar_label: "Migrate from firehose" ~ under the License. --> -Apache deprecated support for Druid firehoses in version 0.17. Support for firehose ingestion was removed in LTS version 0.24. +Apache deprecated support for Druid firehoses in version 0.17. Support for firehose ingestion was removed in version 24.0. If you're using a firehose for batch ingestion, we strongly recommend that you follow the instructions on this page to transition to using native batch ingestion input sources as soon as possible. @@ -31,7 +31,7 @@ Firehose ingestion doesn't work with newer Druid versions, so you must be using ## Migrate from firehose ingestion to an input source -To migrate from firehose ingestion, you can use the Druid console to update your ingestion spec, or you can update it manally. +To migrate from firehose ingestion, you can use the Druid console to update your ingestion spec, or you can update it manually. ### Use the Druid console From 8770ebb9a88af3b3043232c0d49986adc34af75d Mon Sep 17 00:00:00 2001 From: Jill Osborne Date: Thu, 10 Nov 2022 15:48:34 +0000 Subject: [PATCH 6/6] Update migrate-from-firehose-ingestion.md --- docs/ingestion/migrate-from-firehose-ingestion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ingestion/migrate-from-firehose-ingestion.md b/docs/ingestion/migrate-from-firehose-ingestion.md index 60463b51ae45..2aa77772652c 100644 --- a/docs/ingestion/migrate-from-firehose-ingestion.md +++ b/docs/ingestion/migrate-from-firehose-ingestion.md @@ -47,7 +47,7 @@ To update your ingestion spec manually, copy your existing spec into a new file. Edit the new file as follows: -1. In the `ioConfig` component, replace the `firehose` definition with an `inputSource` definition for your chosen input source. See [Native batch input sources](./native-batch-input-sources.md) for details. +1. In the `ioConfig` component, replace the `firehose` definition with an `inputSource` definition for your chosen input source. See [Native batch input sources](./native-batch-input-source.md) for details. 2. Move the `timeStampSpec` definition from `parser.parseSpec` to the `dataSchema` component. 3. Move the `dimensionsSpec` definition from `parser.parseSpec` to the `dataSchema` component. 4. Move the `format` definition from `parser.parseSpec` to an `inputFormat` definition in `ioConfig`.