Data Types

Data types supported by DataChain must be of type DataType. DataType includes most Python types supported in Pydantic fields, as well as any class that inherits from Pydantic BaseModel.

Pydantic models can be used to group and nest multiple fields together into a single type object. Any Pydantic model must be registered so that the chain knows the expected schema of the model. Alternatively, models may inherit from DataModel, which is a lightweight wrapper around Pydantic's BaseModel that automatically handles registering the model.

DataModel

Bases: BaseModel

Pydantic model wrapper that registers model with DataChain.

__pydantic_init_subclass__ classmethod

__pydantic_init_subclass__()

It automatically registers every declared DataModel child class.

Source code in datachain/lib/data_model.py

@classmethod
def __pydantic_init_subclass__(cls):
    """It automatically registers every declared DataModel child class."""
    promote_default_none(cls)
    ModelStore.register(cls)

hidden_fields classmethod

hidden_fields() -> list[str]

Returns a list of fields that should be hidden from the user.

Source code in datachain/lib/data_model.py

@classmethod
def hidden_fields(cls) -> list[str]:
    """Returns a list of fields that should be hidden from the user."""
    return cls._hidden_fields

register staticmethod

register(models: DataType | Sequence[DataType])

For registering classes manually. It accepts a single class or a sequence of classes.

Source code in datachain/lib/data_model.py

@staticmethod
def register(models: DataType | Sequence[DataType]):
    """For registering classes manually. It accepts a single class or a sequence of
    classes."""
    if not isinstance(models, Sequence):
        models = [models]
    for val in models:
        ModelStore.register(val)

DataType module-attribute

DataType = type[BaseModel] | StandardType

is_chain_type

is_chain_type(t: type) -> bool

Return true if type is supported by DataChain.

Source code in datachain/lib/data_model.py

def is_chain_type(t: type) -> bool:
    """Return true if type is supported by `DataChain`."""
    if ModelStore.is_pydantic(t):
        return True
    if any(t is ft or t is get_args(ft)[0] for ft in get_args(StandardType)):
        return True

    inner, is_optional = unwrap_optional(t)
    if is_optional:
        return is_chain_type(inner)

    # Deliberately not using `annotation_parts` here. This is validation, not
    # traversal: it must see the raw args, since normalising away `Ellipsis` would
    # let `list[int, ...]` through to be serialized as `list[int]`. Only `list` and
    # `dict` at the exact arity `type_to_str` can write back out are accepted --
    # abstract origins serialize as a bare "Sequence"/"Mapping", and `python_to_sql`
    # mis-types tuples. Matching on the origin identity also avoids `issubclass`
    # against generics that reject it (TypedDicts, some protocols).
    orig = get_origin(t)
    args = get_args(t)
    if orig is list:
        return len(args) == 1 and is_chain_type(args[0])
    if orig is dict:
        return len(args) == 2 and all(is_chain_type(arg) for arg in args)

    return False