[Bug] ModelTrainer with no input channels emits InputDataConfig: [], which CreatePipeline rejects (min=1) — v2 omitted the key

[thejesterscap]

PySDK Version

• PySDK V2 (2.x)
• PySDK V3 (3.x)

Describe the bug
A TrainingStep built from a ModelTrainer that has no input channels serializes "InputDataConfig": [] into the pipeline definition. CreatePipeline rejects the whole definition:

botocore.exceptions.ClientError: An error occurred (ValidationException) when
calling the CreatePipeline operation: Unable to parse pipeline definition.
Model Validation failed: Length of container InputDataConfig=0 cannot be less
than min=1.

The SageMaker API accepts an absent InputDataConfig — it is not among CreateTrainingJob's required members — but rejects an empty one, because botocore's own service model gives that member min=1. It is the only min=1 list in the CreateTrainingJob shape.

Under the v2 SDK, TrainingStep(name=..., estimator=...) with no inputs= omitted the key entirely and the same pipeline created successfully. So this is a v2 → v3 behaviour regression for any training job whose data does not arrive over an S3 channel — in our case a fine-tuning job that pulls its dataset, base model and resume checkpoint from the HuggingFace Hub inside the container.

Cause — sagemaker-train, src/sagemaker/train/model_trainer.py:

# :583
final_input_data_config = self.input_data_config.copy() if self.input_data_config else []
...
# :739
"input_data_config": final_input_data_config,

The else [] makes "no channels" indistinguishable from "an empty list of channels" downstream, and the empty list is serialized rather than dropped. input_data_config=None — the default, and what the reproduction passes — takes that branch.

Suggested fix: omit the key when there are no channels, e.g. build the request without input_data_config when final_input_data_config is falsy. None would also work if the serializer drops None members.

To reproduce
Standalone, no AWS calls, no credentials — the two mock.patch calls only stop the SDK reaching IAM/S3 during construction:

import json, os
os.environ.setdefault("AWS_DEFAULT_REGION", "us-west-2")
from unittest import mock
import sagemaker.train.defaults as td
from sagemaker.core.workflow.pipeline_context import PipelineSession
from sagemaker.mlops.workflow.pipeline import Pipeline
from sagemaker.mlops.workflow.steps import TrainingStep
from sagemaker.train import ModelTrainer
from sagemaker.train.configs import Compute

ROLE = "arn:aws:iam::000000000000:role/example"
with mock.patch.object(td, "resolve_and_validate_role",
                       lambda provided_role=None, **kw: provided_role or ROLE), \
     mock.patch.object(PipelineSession, "default_bucket", lambda self: "example-bucket"):
    session = PipelineSession()
    trainer = ModelTrainer(                      # no input_data_config: none needed
        sagemaker_session=session, role=ROLE, base_job_name="repro",
        training_image="000000000000.dkr.ecr.us-west-2.amazonaws.com/example:latest",
        compute=Compute(instance_type="ml.m5.large", instance_count=1),
    )
    step = TrainingStep(name="NoChannels", step_args=trainer.train(wait=False))
    definition = json.loads(
        Pipeline(name="repro", steps=[step], sagemaker_session=session).definition())

args = definition["Steps"][0]["Arguments"]
print("InputDataConfig present:", "InputDataConfig" in args)
print("InputDataConfig value  :", json.dumps(args.get("InputDataConfig")))

Output:

InputDataConfig present: True
InputDataConfig value  : []

Calling pipeline.upsert(role_arn=...) on that definition raises the ValidationException quoted above.

Expected behavior
With no input channels, InputDataConfig is omitted from the serialized definition, matching v2 and matching what the API accepts.

Screenshots or logs
See the ValidationException and reproduction output above.

System information

• SageMaker Python SDK version: sagemaker 3.18.0 (PyPI latest at time of writing); sagemaker-core 2.18.0; sagemaker-train 1.18.0; sagemaker-mlops 1.18.0; sagemaker-serve 1.18.0; boto3/botocore 1.43.53
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): custom training image (data pulled from HuggingFace Hub inside the container)
• Framework version: N/A
• Python version: 3.12
• CPU or GPU: N/A — bug is SDK-side serialization, no job runs
• Custom Docker image (Y/N): Y

Additional context
master carries the byte-identical else [], so this is not fixed in an unreleased commit. Pinning back to the 3.11.0 family avoids it, but that reverts a deliberate change and alters other parts of the definition.

Note for anyone reproducing: sagemaker.__version__ no longer exists on v3 (the sagemaker 3.x wheel is a namespace shim), so version-report snippets that read it raise AttributeError.

Workaround we are using — subclass TrainingStep and drop the empty container at the point the request first exists as a plain dict (arguments is an abstract member of the SDK's own Step ABC, so it is the documented seam):

from sagemaker.mlops.workflow.steps import TrainingStep as _SdkTrainingStep

class TrainingStep(_SdkTrainingStep):
    @property
    def arguments(self) -> dict:
        request = super().arguments
        if not request.get("InputDataConfig"):
            request.pop("InputDataConfig", None)
        return request

Guarding on emptiness rather than popping unconditionally means a step that does take a channel is unaffected.

[thejesterscap]

A second defect of the same family, at the same seam

Fixing the empty InputDataConfig let our dispatch get one step further, into a second rejection from the same CreatePipeline call:

ValidationException calling CreatePipeline:
Unable to parse pipeline definition. Unknown Argument member 'key'.

Cause. ModelTrainer validates tags as List[shapes.Tag] (pydantic) rather than the Key/Value dicts the boto path takes. The definition serializer then emits those objects under their Python field names:

"Tags": [{"key": "project", "value": "lab-1-mlops-pilot"}]

while the API's Tag shape declares exactly Key and Value.

There is no correct path through the SDK. shapes.Tag declares neither alias nor serialization_alias, so model_dump(by_alias=True) returns the same lowercase keys — verified directly against the installed wheel. Dropping the tags is not an option for anyone doing cost allocation.

Why this belongs on this issue rather than its own. Same class — a shapes.* object that cannot be serialized into a valid request — reached through the same TrainingStep seam and fixed by the same override. Concretely: a fix for InputDataConfig alone would let people delete a workaround that is still required for Tags, and the two failures are indistinguishable from outside (both are CreatePipeline rejecting a definition the SDK built happily).

Workaround, extending the one already in this issue:

class TrainingStep(_SdkTrainingStep):
    @property
    def arguments(self) -> dict:
        request = super().arguments
        if not request.get("InputDataConfig"):
            request.pop("InputDataConfig", None)
        tags = request.get("Tags")
        if tags:
            request["Tags"] = [
                {"Key": t["key"], "Value": t["value"]}
                if isinstance(t, dict) and "key" in t else t
                for t in tags
            ]
        return request

A note on detection that may be useful to maintainers. A test asserting the value constraint that caught InputDataConfig (min-length) cannot catch this, because this is a name problem. Validating the whole serialized definition against botocore's own service model — failing on any member the shape does not declare — catches both, and is what established that Tags was the last offender rather than the next in a queue. Pipeline expressions (Get, Std:Join, Std:JsonGet) have to be skipped; they are substituted at execution time and are not API payloads.

[danb27]

Hi, I actually submitted a PR for the original issue yesterday: #6154

Awaiting review.