Configuration

Taking full advantage of Dask sometimes requires user configuration. This might be to control logging verbosity, specify cluster configuration, provide credentials for security, or any of several other options that arise in production.

Configuration is specified in one of the following ways:

  1. YAML files in ~/.config/dask/ or /etc/dask/
  2. Environment variables like DASK_DISTRIBUTED__SCHEDULER__WORK_STEALING=True
  3. Default settings within sub-libraries

This combination makes it easy to specify configuration in a variety of settings ranging from personal workstations, to IT-mandated configuration, to docker images.

Access Configuration

dask.config.get(key[, default, config]) Get elements from global config

Configuration is usually read by using the dask.config module, either with the config dictionary or the get function:

>>> import dask
>>> import dask.distributed  # populate config with distributed defaults
>>> dask.config.config
{
  'logging': {
    'distributed': 'info',
    'bokeh': 'critical',
    'tornado': 'critical',
  }
  'admin': {
    'log-format': '%(name)s - %(levelname)s - %(message)s'
  }
}

>>> dask.config.get('logging')
{'distributed': 'info',
 'bokeh': 'critical',
 'tornado': 'critical'}

>>> dask.config.get('logging.bokeh')  # use `.` for nested access
'critical'

You may wish to inspect the dask.config.config dictionary to get a sense for what configuration is being used by your current system.

Specify Configuration

YAML files

You can specify configuration values in YAML files like the following:

logging:
  distributed: info
  bokeh: critical
  tornado: critical

scheduler:
  work-stealing: True
  allowed-failures: 5

 admin:
   log-format: '%(name)s - %(levelname)s - %(message)s'

These files can live in any of the following locations:

  1. The ~/.config/dask directory in the user’s home directory
  2. The {sys.prefix}/etc/dask directory local to Python
  3. The root /etc/dask/ directory

Dask searches for all YAML files within each of these directories and merges them together, preferring configuration files closer to the user over system configuration files (preference follows the order in the list above). Additionally users can specify a path with the DASK_CONFIG environment variable, that takes precedence at the top of the list above.

The contents of these YAML files are merged together, allowing different dask subprojects like dask-kubernetes or dask-ml to manage configuration files separately, but have them merge into the same global configuration.

Note: for historical reasons we also look in the ``~/.dask`` directory for config files. This is deprecated and will soon be removed.

Environment Variables

You can also specify configuration values with environment variables like the following:

export DASK_DISTRIBUTED__SCHEDULER__WORK_STEALING=True
export DASK_DISTRIBUTED__SCHEDULER__ALLOWED_FAILURES=5

resulting in configuration values like the following:

{'distributed':
  {'scheduler':
    {'work-stealing': True,
     'allowed-failures': 5}
  }
}

Dask searches for all environment variables that start with DASK_, then transforms keys by converting to lower case, changing double-underscores to nested structures, and changing single underscores to hyphens.

Dask tries to parse all values with ast.literal_eval, letting users pass numeric and boolean values (such as True in the example above) as well as lists, dictionaries, and so on with normal Python syntax.

Environment variables take precedence over configuration values found in YAML files.

Defaults

Additionally, individual subprojects may add their own default values when they are imported. These are always added with lower priority than the YAML files or environment variables mentioned above

>>> import dask.config
>>> dask.config.config  # no configuration by default
{}

>>> import dask.distributed
>>> dask.config.config  # New values have been added
{'scheduler': ...,
 'worker': ...,
 'tls': ...}

Directly within Python

dask.config.set([arg, config, lock]) Temporarily set configuration values within a context manager

Configuration is stored within a normal Python dictionary in dask.config.config and can be modified using normal Python operations.

Additionally, you can temporarily set a configuration value using the dask.config.set function. This function accepts a dictionary as an input and interprets "." as nested access

>>> dask.config.set({'scheduler.work-stealing': True})

This function can also be used as a context manager for consistent cleanup.

with dask.config.set({'scheduler.work-stealing': True}):
    ...

Updating Configuration

Manipulating configuration dictionaries

dask.config.merge(*dicts) Update a sequence of nested dictionaries
dask.config.update(old, new[, priority]) Update a nested dictionary with values from another

As described above, configuration can come from many places, including several YAML files, environment variables, and project defaults. Each of these provides a configuration that is possibly nested like the following:

x = {'a': 0, 'c': {'d': 4}}
y = {'a': 1, 'b': 2, 'c': {'e': 5}}

Dask will merge these configurations respecting nested data structures, and respecting order.

>>> dask.config.merge(x, y)
{'a': 1, 'b': 2, 'c': {'d': 4, 'e': 5}}

You can also use the update function to update the existing configuration in place with a new configuration. This can be done with priority being given to either config. This is often used to update the global configuration in dask.config.config

dask.config.update(dask.config, new, priority='new')  # Give priority to new values
dask.config.update(dask.config, new, priority='old')  # Give priority to old values

Refreshing Configuration

dask.config.collect([paths, env]) Collect configuration from paths and environment variables
dask.config.refresh([config, defaults]) Update configuration by re-reading yaml files and env variables

If you change your environment variables or YAML files Dask will not immediately see the changes. Instead, you can call refresh to go through the configuration collection process and update the default configuration.

>>> dask.config.config
{}

>>> # make some changes to yaml files

>>> dask.config.refresh()
>>> dask.config.config
{...}

This function uses dask.config.collect, which returns the configuration without modifying the global configuration. You might use this to determine the configuration of particular paths not yet on the config path.

>>> dask.config.collect(paths=[...])
{...}

Downstream Libraries

dask.config.ensure_file(source[, …]) Copy file to default location if it does not already exist
dask.config.update(old, new[, priority]) Update a nested dictionary with values from another
dask.config.update_defaults(new[, config, …]) Add a new set of defaults to the configuration

Downstream Dask libraries often follow a standard convention to use the central Dask configuration. This section provides recommendations for integration, using a fictional project, dask-foo, as an example.

Downstream projects typically follow the following convention:

  1. Maintain default configuration in a YAML file within their source directory:

    setup.py
    dask_foo/__init__.py
    dask_foo/config.py
    dask_foo/core.py
    dask_foo/foo.yaml  # <---
    
  2. Place configuration in that file within a namespace for the project

    # dask_foo/foo.yaml
    
    foo:
      color: red
      admin:
        a: 1
        b: 2
    
  3. Within a config.py file (or anywhere) load that default config file and update it into the global configuration

    # dask_foo/config.py
    import os
    import yaml
    
    import dask.config
    
    fn = os.path.join(os.path.dirname(__file__), 'foo.yaml')
    
    with open(fn) as f:
        defaults = yaml.load(f)
    
    dask.config.update_defaults(defaults)
    
  4. Within that same config.py file, copy the 'foo.yaml' file to the user’s configuration directory if it doesn’t already exist.

    We also comment the file to make it easier for us to change defaults in the future.

    # ... continued from above
    
    dask.config.ensure_file(source=fn, comment=True)
    

    The user can investigate ~/.config/dask/*.yaml to see all of the commented out configuration files to which they have access.

  5. Ensure that this file is run on import by including it in __init__.py

    # dask_foo/__init__.py
    
    from . import config
    
  6. Within dask_foo code, use the dask.config.get function to access configuration values

    # dask_foo/core.py
    
    def process(fn, color=dask.config.get('foo.color')):
        ...
    
  7. You may also want to ensure that your yaml configuration files are included in your package. This can be accomplished by including the following line in your MANIFEST.in:

    recursive-include <PACKAGE_NAME> *.yaml
    

    and the following in your setup.py setup call

    from setuptools import setup
    
    setup(...,
          include_package_data=True,
          ...)
    

This process keeps configuration in a central place, but also keeps it safe within namespaces. It places config files in an easy to access location ,``~/.config/dask/*.yaml`` by default so that users can easily discover what they can change, but maintains the actual defaults within the source code, so that they more closely track changes in the library.

However, downstream libraries may choose alternative solutions, such as isolating their configuration within their library, rather than using the global dask.config system. All functions in the dask.config module also work with parameters, and do not need to mutate global state.

API

dask.config.get(key, default='__no_default__', config={'array': {'rechunk-threshold': 4, 'chunk-size': '128MiB'}})

Get elements from global config

Use ‘.’ for nested access

See also

dask.config.set

Examples

>>> from dask import config
>>> config.get('foo')  
{'x': 1, 'y': 2}
>>> config.get('foo.x')  
1
>>> config.get('foo.x.y', default=123)  
123
dask.config.set(arg=None, config={'array': {'rechunk-threshold': 4, 'chunk-size': '128MiB'}}, lock=<unlocked _thread.lock object>, **kwargs)

Temporarily set configuration values within a context manager

See also

dask.config.get

Examples

>>> import dask
>>> with dask.config.set({'foo': 123}):
...     pass
dask.config.merge(*dicts)

Update a sequence of nested dictionaries

This prefers the values in the latter dictionaries to those in the former

Examples

>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'y': {'b': 3}}
>>> merge(a, b)  
{'x': 1, 'y': {'a': 2, 'b': 3}}
dask.config.update(old, new, priority='new')

Update a nested dictionary with values from another

This is like dict.update except that it smoothly merges nested values

This operates in-place and modifies old

Parameters:
priority: string {‘old’, ‘new’}

If new (default) then the new dictionary has preference. Otherwise the old dictionary does.

Examples

>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'x': 2, 'y': {'b': 3}}
>>> update(a, b)  
{'x': 2, 'y': {'a': 2, 'b': 3}}
>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'x': 2, 'y': {'b': 3}}
>>> update(a, b, priority='old')  
{'x': 1, 'y': {'a': 2, 'b': 3}}
dask.config.collect(paths=['/etc/dask', '/home/docs/checkouts/readthedocs.org/user_builds/dask/envs/latest/etc/dask', '/home/docs/.config/dask', '/home/docs/.dask'], env=None)

Collect configuration from paths and environment variables

Parameters:
paths : List[str]

A list of paths to search for yaml config files

env : dict

The system environment variables

Returns:
config: dict

See also

dask.config.refresh
collect configuration and update into primary config
dask.config.refresh(config={'array': {'rechunk-threshold': 4, 'chunk-size': '128MiB'}}, defaults=[{'array': {'rechunk-threshold': 4, 'chunk-size': '128MiB'}}], **kwargs)

Update configuration by re-reading yaml files and env variables

This mutates the global dask.config.config, or the config parameter if passed in.

This goes through the following stages:

  1. Clearing out all old configuration
  2. Updating from the stored defaults from downstream libraries (see update_defaults)
  3. Updating from yaml files and environment variables

Note that some functionality only checks configuration once at startup and may not change behavior, even if configuration changes. It is recommended to restart your python process if convenient to ensure that new configuration changes take place.

See also

dask.config.collect
for parameters

dask.config.update_defaults

dask.config.ensure_file(source, destination=None, comment=True)

Copy file to default location if it does not already exist

This tries to move a default configuration file to a default location if if does not already exist. It also comments out that file by default.

This is to be used by downstream modules (like dask.distributed) that may have default configuration files that they wish to include in the default configuration path.

Parameters:
source : string, filename

Source configuration file, typically within a source directory.

destination : string, directory

Destination directory. Configurable by DASK_CONFIG environment variable, falling back to ~/.config/dask.

comment : bool, True by default

Whether or not to comment out the config file when copying.