Lookup module not registered with overlord

[2PacIsAlive]

Currently, the lookup module is only registered with peons, historicals, and brokers.

This makes it impossible to use lookups in a reindexing task filter.

Here is an example reindexing task spec that showcases the failure:

{
  "type": "index_hadoop",
  "spec": {
    "dataSchema": {
      "dataSource": "REDACTED",
      "ignoreWhenNoSegments": true,
      "parser": {
        "type": "string",
        "parseSpec": {
          "format": "json",
          "timestampSpec": {
            "column": "timestamp",
            "format": "auto"
          },
          "dimensionsSpec": {
            "dimensions": [
              "example",
              "timestamp"
            ],
            "dimensionExclusions": [],
            "spatialDimensions": []
          }
        }
      },
      "metricsSpec": [
        {
          "type": "count",
          "name": "count"
        }
      ],
      "granularitySpec": {
        "segmentGranularity": "DAY",
        "queryGranularity": "NONE",
        "intervals": [
          "2017-04-22T16:52:31.373Z/2018-04-23T14:52:31.373Z"
        ]
      }
    },
    "ioConfig": {
      "type": "hadoop",
      "inputSpec": {
        "type": "dataSource",
        "ingestionSpec": {
          "dataSource": "REDACTED",
          "intervals": [
            "2017-04-22T16:52:31.373Z/2018-04-23T14:52:31.373Z"
          ],
          "filter": {
            "type": "or",
            "fields": [
              {
                "type": "selector",
                "dimension": "example",
                "value": "example_lookup_value",
                "extractionFn": {
                  "type": "registeredLookup",
                  "lookup": "example_lookup",
                  "retainMissingValue": true
                }
              },
              {
                "type": "not",
                "field": {
                  "type": "or",
                  "fields": [
                    {
                      "type": "selector",
                      "dimension": "example",
                      "value": "123"
                    },
                    {
                      "type": "selector",
                      "dimension": "example",
                      "value": "456"
                    }
                  ]
                }
              }
            ]
          }
        },
        "dataSource": "REDACTED"
      }
    },
    "tuningConfig": {
      "type": "hadoop",
      "jobProperties": {
        "fs.s3.awsAccessKeyId": "REDACTED",
        "fs.s3.awsSecretAccessKey": "REDACTED",
        "fs.s3n.awsAccessKeyId": "REDACTED",
        "fs.s3n.awsSecretAccessKey": "REDACTED",
        "fs.s3.impl": "org.apache.hadoop.fs.s3native.NativeS3FileSystem",
        "fs.s3n.impl": "org.apache.hadoop.fs.s3native.NativeS3FileSystem",
        "mapreduce.job.classloader": "true",
        "mapreduce.job.classloader.system.classes": "-javax.validation.,java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop."
      },
      "forceExtendableShardSpecs": true
    }
  },
  "hadoopDependencyCoordinates": [
    "org.apache.hadoop:hadoop-client:2.7.1"
  ]
}

This task will produce the following error:

2018-04-23T21:29:51,442 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_REDACTED_2018-04-23T21:29:43.610Z, type=index_hadoop, dataSource=REDACTED}]
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:211) ~[druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:176) ~[druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_162]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:208) ~[druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	... 7 more
Caused by: java.lang.IllegalArgumentException: Guice configuration errors:

1) Could not find a suitable constructor in io.druid.query.lookup.LookupConfig. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at io.druid.query.lookup.LookupConfig.class(LookupConfig.java:39)
  while locating io.druid.query.lookup.LookupConfig
    for the 1st parameter of io.druid.query.lookup.LookupReferencesManager.<init>(LookupReferencesManager.java:69)
  while locating io.druid.query.lookup.LookupReferencesManager

1 error (through reference chain: java.util.ArrayList[0])
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6]
	at io.druid.indexer.HadoopDruidIndexerConfig.<init>(HadoopDruidIndexerConfig.java:228) ~[druid-indexing-hadoop-0.10.0-iap5.jar:0.10.0-iap5]
	at io.druid.indexer.HadoopDruidIndexerConfig.fromSpec(HadoopDruidIndexerConfig.java:130) ~[druid-indexing-hadoop-0.10.0-iap5.jar:0.10.0-iap5]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:297) ~[druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:208) ~[druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	... 7 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Guice configuration errors:

1) Could not find a suitable constructor in io.druid.query.lookup.LookupConfig. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at io.druid.query.lookup.LookupConfig.class(LookupConfig.java:39)
  while locating io.druid.query.lookup.LookupConfig
    for the 1st parameter of io.druid.query.lookup.LookupReferencesManager.<init>(LookupReferencesManager.java:69)
  while locating io.druid.query.lookup.LookupReferencesManager

1 error (through reference chain: java.util.ArrayList[0])
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:189) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:206) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6]
	at io.druid.indexer.HadoopDruidIndexerConfig.<init>(HadoopDruidIndexerConfig.java:228) ~[druid-indexing-hadoop-0.10.0-iap5.jar:0.10.0-iap5]
	at io.druid.indexer.HadoopDruidIndexerConfig.fromSpec(HadoopDruidIndexerConfig.java:130) ~[druid-indexing-hadoop-0.10.0-iap5.jar:0.10.0-iap5]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:297) ~[druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:208) ~[druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	... 7 more
Caused by: com.google.inject.ConfigurationException: Guice configuration errors:

1) Could not find a suitable constructor in io.druid.query.lookup.LookupConfig. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at io.druid.query.lookup.LookupConfig.class(LookupConfig.java:39)
  while locating io.druid.query.lookup.LookupConfig
    for the 1st parameter of io.druid.query.lookup.LookupReferencesManager.<init>(LookupReferencesManager.java:69)
  while locating io.druid.query.lookup.LookupReferencesManager

1 error
	at com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1045) ~[guice-4.1.0.jar:?]
	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1050) ~[guice-4.1.0.jar:?]
	at io.druid.guice.GuiceInjectableValues.findInjectableValue(GuiceInjectableValues.java:48) ~[druid-common-0.10.0-iap5.jar:0.10.0-iap5]
	at com.fasterxml.jackson.databind.DeserializationContext.findInjectableValue(DeserializationContext.java:293) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.PropertyValueBuffer.inject(PropertyValueBuffer.java:66) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.startBuilding(PropertyBasedCreator.java:160) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:331) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:234) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:206) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6]
	at io.druid.indexer.HadoopDruidIndexerConfig.<init>(HadoopDruidIndexerConfig.java:228) ~[druid-indexing-hadoop-0.10.0-iap5.jar:0.10.0-iap5]
	at io.druid.indexer.HadoopDruidIndexerConfig.fromSpec(HadoopDruidIndexerConfig.java:130) ~[druid-indexing-hadoop-0.10.0-iap5.jar:0.10.0-iap5]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:297) ~[druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:208) ~[druid-indexing-service-0.10.0-iap5.jar:0.10.0-iap5]
	... 7 more
2018-04-23T21:29:51,450 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_REDACTED_2018-04-23T21:29:43.610Z] status changed to [FAILED].
2018-04-23T21:29:51,453 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_hadoop_REDACTED_2018-04-23T21:29:43.610Z",
  "status" : "FAILED",
  "duration" : 2838
}

It might be sufficient to add a line like this to the overlord startup file, but I have not tested this

[drcrallen]

I think you are correct, would you be able to do a PR?

[2PacIsAlive]

👍Sure thing I will do that this weekend

[jihoonson]

I think this isn't a bug but a missing feature. I'll change the label.

[scrawfor]

I'm taking a stab at this. In my first attempt I added the new LookupModule() to the CliOverlord.java file as @2PacIsAlive suggested. This got me further, but the middle manager wouldn't run any tasks and threw this error.

com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class org.apache.druid.indexing.kafka.KafkaIndexTask] value failed: function 'lookup' is not defined.

So I added new LookupModule() to CliMiddleManager.java as well. This worked as the tasks get created and executed, and lookups are mapping data.

However, it also attempts to load lookup values on the middle manager, which throws out of memory errors

2019-02-13T13:24:56,227 ERROR [NamespaceExtractionCacheManager-1] org.apache.druid.server.lookup.namespace.cache.CacheScheduler - Failed to update namespace [JdbcExtractionNamespace{connectorConfig=DbConnectorConfig{cr
eateTables=true, connectURI='Redacted', user='Redacted', passwordProvider=org.apache.druid.metadata.DefaultPasswordProvider}, table='Redacted', keyC
olumn='Redacted', valueColumn='Redacted', tsColumn='null', filter='null', pollPeriod=P1D}] : org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl@6fc24ef7
org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: org.postgresql.util.PSQLException: Ran out of memory retrieving query results. [statement:"Redacted", located:"Redacted", rewritten:"Redacted", arguments:{ positional:{}, named:{}, finder:[]}]
        at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator.generateCache(JdbcCacheGenerator.java:77) ~[druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
        at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator.generateCache(JdbcCacheGenerator.java:49) ~[druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
        at org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl.tryUpdateCache(CacheScheduler.java:228) [druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
        at org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl.updateCache(CacheScheduler.java:207) [druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
        at org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl.access$600(CacheScheduler.java:143) [druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
        at org.apache.druid.server.lookup.namespace.cache.CacheScheduler$EntryImpl$2.run(CacheScheduler.java:189) [druid-lookups-cached-global-0.13.0-incubating.jar:0.13.0-incubating]
Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: org.postgresql.util.PSQLException: Ran out of memory retrieving query results. [statement:"Redacted", located:"Redacted", rewritten:"Redacted", arguments:{ posi
tional:{}, named:{}, finder:[]}]
        at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator$1.withHandle(JdbcCacheGenerator.java:99) ~[?:?]
        at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator$1.withHandle(JdbcCacheGenerator.java:79) ~[?:?]
        at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator$1.withHandle(JdbcCacheGenerator.java:99) ~[?:?]
        at org.apache.druid.server.lookup.namespace.JdbcCacheGenerator$1.withHandle(JdbcCacheGenerator.java:79) ~[?:?]

Does anyone with a better understanding of the architecture have any ideas on how we could get the MiddleManager to recognize the lookup syntax without attempting to load them? We can't set druid.lookup.enableLookupSyncOnStartup=false because the middle manager will pass that property to the peon right?

For reference, my Indexing Task Spec:

{
  "type": "kafka",
   "dataSchema": {
     "dataSource": "REDACTED",
     "parser": {
       "type": "string",
       "parseSpec": {
         "format": "json",
         "timestampSpec": {
           "column": "timestamp",
           "format": "iso"
         },    
         "dimensionsSpec": {
           "dimensions": [
             "user_type",
             "client_id",
           ]
         }
       }
     },
     "metricsSpec": [
       {
         "type": "hyperUnique",
         "name": "unique_views",
         "fieldName": "view_id"
       },
     ],
     "transformSpec": {
       "transforms": [
         {
           "type": "expression",
           "name": "user_type",
           "expression": "lookup(client_id, 'user_type_lookup')"
         },
       ],
       "filter": {
         "type": "and",
         "fields": [
           {
             "type": "not",
             "field": {
               "type": "in",
               "dimension": "profile",
               "values": ["REDACTED"]
             }
           }          
         ]
       }
     },
     "granularitySpec": {
       "type": "uniform",
       "segmentGranularity": "DAY",
       "queryGranularity": "NONE",
       "rollup": true
     }
   },
  "tuningConfig": {
    "type": "kafka",
    "maxRowsInMemory": 5000,
    "maxRowsPerSegment": 5000000,
    "intermediatePersistPeriod": "PT2M",
    "resetOffsetAutomatically": true
  },
  "ioConfig": {
    "topic": "REDACTED",
    "consumerProperties": {
      "bootstrap.servers": "REDACTED"
    },
    "useEarliestOffset": false,
    "skipOffsetGaps": true
  }
}

[jihoonson]

How about setting druid.lookup.enableLookupSyncOnStartup = false for MMs but adding -Ddruid.lookup.enableLookupSyncOnStartup=true to druid.indexer.runner.javaOpts for tasks? Does it work?

[scrawfor]

Unfortunately it doesn't. It prevents lookups from loading on both the middle manager and the peons. The option is passed, but it's passed first, and then overridden by the configuration.

Job Log:

[LookupConfig{snapshotWorkingDir='', enableLookupSyncOnStartup=false, numLookupLoadingThreads=4, coordinatorFetchRetries=3, lookupStartRetries=3, coordinatorRetryDelay=60000}]
...
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Ddruid.lookup.enableLookupSyncOnStartup=true
...
2019-02-14T14:14:37,380 INFO [main] org.apache.druid.cli.CliPeon - * druid.lookup.enableLookupSyncOnStartup: false  
...

[scrawfor]

I found the magic settings to prevent the lookups from loading on the middle manager.

# Disable lookups on middleManager, but enable on peon.
druid.lookup.enableLookupSyncOnStartup=false
druid.indexer.fork.property.druid.lookup.enableLookupSyncOnStartup=true

Created #7082