Logger does not find levels specified in configuration file

[deniederhut]

In distributed.yaml, the logging configurations are listed under the 'distributed' key:

distributed:
  version: 2
  # logging:
  #   distributed: info
  #   distributed.client: warning
  #   bokeh: critical
  #   # http://stackoverflow.com/questions/21234772/python-tornado-disable-logging-to-stderr
  #   tornado: critical
  #   tornado.application: error

but the logger looks for a key called 'logging' at the top level of dask.config.config:

    loggers = {  # default values
        "distributed": "info",
        "distributed.client": "warning",
        "bokeh": "critical",
        "tornado": "critical",
        "tornado.application": "error",
    }
    loggers.update(config.get("logging", {}))

this causes the logger to ignore changes to the logging levels in the configuration file, unless they are moved to the top level, like this:

logging:
  distributed: debug
  distributed.client: debug
  distributed.worker: debug
  distributed.scheduler: debug
distributed:
  version: 2

It's not clear to me if the intention was for dask.config to update its configuration dictionary with the mapping underneath the 'distributed' key; or, for the logger to look under the 'distributed' key for library-specific logging; or, if the 'distributed' key was not meant to be used for user configs placed in /etc/dask or specified via the DASK_CONFIG environment variable.

[TomAugspurger]

I think you're correct that the logging key needs to be at the top level to have them take effect. Does that sound right @mrocklin, or was the intention for distributed.config.initialize_logging to look under the distributed key?

[mrocklin]

Honestly I don't recall. There is also some history here where the logging config predates our current nesting, so there might still be support for both in some odd way.

…

On Wed, Aug 7, 2019 at 12:23 PM Tom Augspurger ***@***.***> wrote: I think you're correct that the logging key needs to be at the top level to have them take effect. Does that sound right @mrocklin <https://github.com/mrocklin>, or was the intention for distributed.config.initialize_logging to look under the distributed key? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2937?email_source=notifications&email_token=AACKZTGNQQI2W4OTYXRRRKDQDMOKDA5CNFSM4IKCNP32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3ZON6A#issuecomment-519235320>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACKZTHWPJOYVUEV3JDUUB3QDMOKDANCNFSM4IKCNP3Q> .

[deniederhut]

Hm. So the preferred solution would be to have distributed.config first look under the distributed key, then fall back to a top-level logging config? If you like, I can take a stab at implementing this.

[TomAugspurger]

That sounds right to me, but I'm a little perplexed by the current state of things.

[deniederhut]

Hm, okay. Well here is a PR that implements what I believe is the wanted behavior.