DagsterDocs
GitHub iconStack Overflow iconSlack icon

Config

Config Types

The following types are used to describe the schema of configuration data via config. They are used in conjunction with the builtin types above.

class dagster.ConfigSchema[source]

This is a placeholder type. Any time that it appears in documentation, it means that any of the following types are acceptable:

  1. A Python scalar type that resolves to a Dagster config type (int, float, bool, or str). For example:

    • @solid(config_schema=int)

    • @solid(config_schema=str)

  2. A built-in python collection (list, or dict). list is exactly equivalent to Array [ Any ] and dict is equivalent to Permissive. For example:

    • @solid(config_schema=list)

    • @solid(config_schema=dict)

  3. A Dagster config type:

  4. A bare python dictionary, which will be automatically wrapped in Shape. Values of the dictionary are resolved recursively according to the same rules. For example:

    • {'some_config': str} is equivalent to Shape({'some_config: str}).

    • {'some_config1': {'some_config2': str}} is equivalent to

      Shape({'some_config1: Shape({'some_config2: str})}).

  5. A bare python list of length one, whose single element will be wrapped in a Array is resolved recursively according to the same rules. For example:

    • [str] is equivalent to Array[str].

    • [[str]] is equivalent to Array[Array[str]].

    • [{'some_config': str}] is equivalent to Array(Shape({'some_config: str})).

  6. An instance of Field.

class dagster.Field(config, default_value=<class 'dagster.config.field_utils.__FieldValueSentinel'>, is_required=None, description=None)[source]

Defines the schema for a configuration field.

Fields are used in config schema instead of bare types when one wants to add a description, a default value, or to mark it as not required.

Config fields are parsed according to their schemas in order to yield values available at pipeline execution time through the config system. Config fields can be set on solids, on loaders and materializers for custom, and on other pluggable components of the system, such as resources, loggers, and executors.

Parameters

  • config (Any) –

    The schema for the config. This value can be any of:

    1. A Python primitive type that resolves to a Dagster config type (int, float, bool, str, or list).

    2. A Dagster config type:

    3. A bare python dictionary, which will be automatically wrapped in Shape. Values of the dictionary are resolved recursively according to the same rules.

    4. A bare python list of length one which itself is config type. Becomes Array with list element as an argument.

  • default_value (Any) –

    A default value for this field, conformant to the schema set by the dagster_type argument. If a default value is provided, is_required should be False.

    Note: for config types that do post processing such as Enum, this value must be the pre processed version, ie use ExampleEnum.VALUE.name instead of ExampleEnum.VALUE

  • is_required (bool) – Whether the presence of this field is required. Defaults to true. If is_required is True, no default value should be provided.

  • description (str) – A human-readable description of this config field.

Examples:

@solid(
    config_schema={
        'word': Field(str, description='I am a word.'),
        'repeats': Field(Int, default_value=1, is_required=False),
    }
)
def repeat_word(context):
    return context.solid_config['word'] * context.solid_config['repeats']
class dagster.Selector(fields, description=None)[source]

Define a config field requiring the user to select one option.

Selectors are used when you want to be able to present several different options in config but allow only one to be selected. For example, a single input might be read in from either a csv file or a parquet file, but not both at once.

Note that in some other type systems this might be called an ‘input union’.

Functionally, a selector is like a Dict, except that only one key from the dict can be specified in valid config.

Parameters

fields (Dict[str, Field]) – The fields from which the user must select.

Examples:

@solid(
    config_schema=Field(
        Selector(
            {
                'haw': {'whom': Field(String, default_value='honua', is_required=False)},
                'cn': {'whom': Field(String, default_value='世界', is_required=False)},
                'en': {'whom': Field(String, default_value='world', is_required=False)},
            }
        ),
        is_required=False,
        default_value={'en': {'whom': 'world'}},
    )
)
def hello_world_with_default(context):
    if 'haw' in context.solid_config:
        return 'Aloha {whom}!'.format(whom=context.solid_config['haw']['whom'])
    if 'cn' in context.solid_config:
        return '你好，{whom}!'.format(whom=context.solid_config['cn']['whom'])
    if 'en' in context.solid_config:
        return 'Hello, {whom}!'.format(whom=context.solid_config['en']['whom'])
class dagster.Permissive(fields=None, description=None)[source]

Defines a config dict with a partially specified schema.

A permissive dict allows partial specification of the config schema. Any fields with a specified schema will be type checked. Other fields will be allowed, but will be ignored by the type checker.

Parameters

fields (Dict[str, Field]) – The partial specification of the config dict.

Examples:

@solid(config_schema=Field(Permissive({'required': Field(String)})))
def partially_specified_config(context) -> List:
    return sorted(list(context.solid_config.items()))
class dagster.Shape(fields, description=None)[source]

Schema for configuration data with string keys and typed values via Field.

Unlike Permissive, unspecified fields are not allowed and will throw a DagsterInvalidConfigError.

Parameters

fields (Dict[str, Field]) – The specification of the config dict.

class dagster.Array(inner_type)[source]

Defines an array (list) configuration type that contains values of type inner_type.

Parameters

inner_type (type) – The type of the values that this configuration type can contain.

class dagster.Noneable(inner_type)[source]

Defines a configuration type that is the union of NoneType and the type inner_type.

Parameters

inner_type (type) – The type of the values that this configuration type can contain.

Examples:

config_schema={"name": Noneable(str)}

config={"name": "Hello"}  # Ok
config={"name": None}     # Ok
config={}                 # Error
class dagster.Enum(name, enum_values)[source]

Defines a enum configuration type that allows one of a defined set of possible values.

Parameters

  • name (str) – The name of the enum configuration type.

  • enum_values (List[EnumValue]) – The set of possible values for the enum configuration type.

Examples:

@solid(
    config_schema=Field(
        Enum(
            'CowboyType',
            [
                EnumValue('good'),
                EnumValue('bad'),
                EnumValue('ugly'),
            ]
        )
    )
)
def resolve_standoff(context):
    # ...
classmethod from_python_enum(enum, name=None)[source]

Create a Dagster enum corresponding to an existing Python enum.

Parameters

  • enum (enum.EnumMeta) – The class representing the enum.

  • name (Optional[str]) – The name for the enum. If not present, enum.__name__ will be used.

Example:

class Color(enum.Enum):
    RED = enum.auto()
    GREEN = enum.auto()
    BLUE = enum.auto()

@solid(
    config_schema={"color": Field(Enum.from_python_enum(Color))}
)
def select_color(context):
    # ...
class dagster.EnumValue(config_value, python_value=None, description=None)[source]

Define an entry in a Enum.

Parameters

  • config_value (str) – The string representation of the config to accept when passed.

  • python_value (Optional[Any]) – The python value to convert the enum entry in to. Defaults to the config_value.

  • description (Optional[str]) – A human-readable description of the enum entry.

class dagster.ScalarUnion(scalar_type, non_scalar_schema, _key=None)[source]

Defines a configuration type that accepts a scalar value OR a non-scalar value like a List, Dict, or Selector.

This allows runtime scalars to be configured without a dictionary with the key value and instead just use the scalar value directly. However this still leaves the option to load scalars from a json or pickle file.

Parameters

  • scalar_type (type) – The scalar type of values that this configuration type can hold. For example, int, float, bool, or str.

  • non_scalar_schema (ConfigSchema) – The schema of a non-scalar Dagster configuration type. For example, List, Dict, or Selector.

  • key (Optional[str]) – The configuation type’s unique key. If not set, then the key will be set to ScalarUnion.{scalar_type}-{non_scalar_schema}.

Examples:

solids:
  transform_word:
    inputs:
      word:
        value: foobar

becomes, optionally,

solids:
  transform_word:
    inputs:
      word: foobar
dagster.StringSource

Use this type when you want to read a string config value from an environment variable. The value passed to a config field of this type may either be a string literal, or a selector describing how to look up the value from the executing process’s environment variables.

Examples:

@solid(config_schema=StringSource)
def secret_solid(context) -> str:
    return context.solid_config

execute_solid(
    secret_solid,
    run_config={
        'solids': {'secret_solid': {'config': 'test_value'}}
    }
)

execute_solid(
    secret_solid,
    run_config={
        'solids': {'secret_solid': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE'}}}
    }
)
dagster.IntSource

Use this type when you want to read an integer config value from an environment variable. The value passed to a config field of this type may either be a integer literal, or a selector describing how to look up the value from the executing process’s environment variables.

Examples:

@solid(config_schema=IntSource)
def secret_int_solid(context) -> str:
    return context.solid_config

execute_solid(
    secret_int_solid,
    run_config={
        'solids': {'secret_int_solid': {'config': 3}}
    }
)

execute_solid(
    secret_int_solid,
    run_config={
        'solids': {'secret_int_solid': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE_INT'}}}
    }
)
dagster.BoolSource

Use this type when you want to read an boolean config value from an environment variable. The value passed to a config field of this type may either be a boolean literal, or a selector describing how to look up the value from the executing process’s environment variables. Set the value of the corresponding environment variable to "" to indicate False.

Examples:

@solid(config_schema=BoolSource)
def secret_bool_solid(context) -> str:
    return context.solid_config

execute_solid(
    secret_bool_solid,
    run_config={
        'solids': {'secret_bool_solid': {'config': True}}
    }
)

execute_solid(
    secret_bool_solid,
    run_config={
        'solids': {'secret_bool_solid': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE_BOOL'}}}
    }
)

Config Utilities

@dagster.configured(configurable, config_schema=None, **kwargs)[source]

A decorator that makes it easy to create a function-configured version of an object. The following definition types can be configured using this function:

If the config that will be supplied to the object is constant, you may alternatively invoke this and call the result with a dict of config values to be curried. Examples of both strategies below.

Parameters

  • configurable (ConfigurableDefinition) – An object that can be configured.

  • config_schema (ConfigSchema) – The config schema that the inputs to the decorated function must satisfy.

  • **kwargs – Arbitrary keyword arguments that will be passed to the initializer of the returned object.

Returns

(Callable[[Union[Any, Callable[[Any], Any]]], ConfigurableDefinition])

Examples:

dev_s3 = configured(s3_resource, name="dev_s3")({'bucket': 'dev'})

@configured(s3_resource):
def dev_s3(_):
    return {'bucket': 'dev'}

@configured(s3_resource, {'bucket_prefix', str}):
def dev_s3(config):
    return {'bucket': config['bucket_prefix'] + 'dev'}