Feat/ecs agentcore container builds#9012
Conversation
fix: Pin cython to a version that is less than 3.0.0
| if not container_defs: | ||
| return | ||
|
|
||
| property_value = container_defs[0].get("Image") |
There was a problem hiding this comment.
[BUG] ECSTaskDefinitionImageResource.export hardcodes container_defs[0] for both reading the image and writing back the uploaded URL, ignoring the ContainerName metadata. This is inconsistent with the build path: ApplicationBuilder._update_built_resource in app_builder.py (lines 423–430) reads metadata.get("ContainerName") and updates the matching container by name, falling back to index 0 only when no name is provided.
As a result, for a multi-container TaskDefinition where the user sets Metadata.ContainerName to target a non-first container, sam package (which calls this export) will overwrite the wrong container's Image. The same bug exists in delete at line 726. The PR description explicitly advertises ContainerName as the way to target a specific container, so the package path needs to honor it too.
target_name = (resource_dict.get("Metadata") or {}).get("ContainerName") or \
(resource_dict.get("Properties", {}).get("Metadata") or {}).get("ContainerName")
target_index = 0
if target_name:
for i, cd in enumerate(container_defs):
if cd.get("Name") == target_name:
target_index = i
break
property_value = container_defs[target_index].get("Image")
...
container_defs[target_index]["Image"] = uploaded_url(Note: Metadata is sibling to Properties in the resource dict, so check the right level.)
| try: | ||
| svc_response = ecs_client.describe_services(cluster=cluster_arn, services=[service_arn]) | ||
| for svc in svc_response.get("services", []): | ||
| if family in svc.get("taskDefinition", ""): |
There was a problem hiding this comment.
[BUG] if family in svc.get("taskDefinition", "") uses a substring match against the task-definition ARN (which looks like arn:aws:ecs:...:task-definition/<family>:<rev>). If two families share a common prefix or one is a substring of another (e.g. web and webapp, or api and api-v2), syncing one will incorrectly force a new deployment of services running the other family.
Parse the family out of the ARN and compare for equality:
deployed_family = svc.get("taskDefinition", "").rsplit("/", 1)[-1].split(":", 1)[0]
if deployed_family == family:
ecs_client.update_service(...)|
|
||
| if family: | ||
| # Find services using this family | ||
| clusters_response = ecs_client.list_clusters() |
There was a problem hiding this comment.
[BUG] ecs_client.list_clusters() and ecs_client.list_services(cluster=...) on lines 180 and 182 are not paginated. ECS returns at most 100 results per page and uses nextToken for the rest. In any account that exceeds 100 clusters or 100 services in a cluster, services on later pages will be silently missed and never get a forced deployment — making sync unreliable as the account grows.
Use ecs_client.get_paginator("list_clusters") and get_paginator("list_services"), or loop on nextToken, to enumerate every page.
| try: | ||
| container_provider = SamContainerServiceProvider(self.stacks) | ||
| container_services = list(container_provider.get_all()) | ||
| except (TypeError, AttributeError): |
There was a problem hiding this comment.
[ERROR_HANDLING] BuildContext._build_container_images wraps the SamContainerServiceProvider construction in try / except (TypeError, AttributeError) and silently returns {}. These exception types are not narrow indicators of a missing container — they indicate programming errors in the provider or an unexpected stack shape. Swallowing them here means a bug that prevents container builds from being discovered will surface as a successful build with no container artifacts and no diagnostic, leaving the user wondering why their sam deploy still references the placeholder image.
If the intent is to handle the case where there are no container resources, that is already covered by the explicit if not container_services: return {} check immediately below — the broad except is unnecessary. Either remove it entirely or narrow it and at minimum log the exception with LOG.debug(..., exc_info=True) so the failure is observable.
6c74d2a to
a771de9
Compare
| return | ||
|
|
||
| try: | ||
| uploaded_url = upload_local_image_artifacts( |
There was a problem hiding this comment.
[BUG] ECSTaskDefinitionImageResource.export calls upload_local_image_artifacts(resource_id, resource_dict, self.PROPERTY_NAME, parent_dir, self.uploader) with PROPERTY_NAME = "ContainerDefinitions.Image". Inside upload_local_image_artifacts this becomes jmespath.search("ContainerDefinitions.Image", resource_dict), but ContainerDefinitions is a list — JMESPath's .Image step on a list evaluates to null, so image_path is None and the function raises ImageNotFoundError (wrapped as ExportFailedError). The image at target_idx that the surrounding code carefully selected is never actually uploaded.
You either need to upload the image directly (you already have property_value in hand), or use a JMESPath expression like ContainerDefinitions[{target_idx}].Image and pass it as the property path. The simplest fix is something like:
image_path = property_value
if not is_ecr_url(image_path):
abs_path = make_abs_path(parent_dir, image_path)
if is_local_file(abs_path):
image_path = abs_path
uploaded_url = self.uploader.upload(image_path, resource_id)
container_defs[target_idx]["Image"] = uploaded_urlThere is no unit test exercising this path, so the bug isn't caught by the test suite — please add a test that runs export against a TaskDefinition with a local Dockerfile path and asserts that the resulting Image is the ECR URL returned by uploader.upload.
| target_idx = i | ||
| break | ||
|
|
||
| property_value = container_defs[target_idx].get("Image") |
There was a problem hiding this comment.
[BUG] ECSTaskDefinitionImageResource.export ignores the ContainerName metadata when selecting which container to update — it picks "the first container whose Image is not already an ECR URL", with index 0 as the fallback. The build path in ApplicationBuilder._update_built_resource (samcli/lib/build/app_builder.py lines 422–430) honors metadata.get("ContainerName") and updates the matching container by name. The two paths can disagree.
Concrete failing case: a TaskDefinition with two locally-pointing containers, e.g.
Metadata:
Dockerfile: Dockerfile
DockerContext: ./web
ContainerName: web
Properties:
ContainerDefinitions:
- Name: api # references a separate, locally-tagged sidecar image
Image: ./api-sidecar
- Name: web
Image: ./webBuild writes the built tag into web (correct). At package time, export iterates and picks the first non-ECR image — api at index 0 — and overwrites api's Image with the URL of the web container's pushed image. The deployed task ends up with web's image running under the api container name and web still pointing at a local-only string.
The export logic must read ContainerName from the resource's Metadata (passed in via the resource dict — see how _update_built_resource does it) and target that container, falling back to index 0 only when no name is provided.
| if not container_defs: | ||
| return | ||
| # Delete any ECR images we manage | ||
| for cd in container_defs: |
There was a problem hiding this comment.
[BUG] -739
ECSTaskDefinitionImageResource.delete iterates all containers and calls self.uploader.delete_artifact(image_uri=remote_path, ...) for any whose Image is an ECR URL. SAM only manages the single container identified by ContainerName (or index 0); the other containers' images may be unrelated ECR repositories the user owns, third-party images (e.g. a sidecar like 123.dkr.ecr.us-east-1.amazonaws.com/aws-otel-collector:latest), or images managed by another stack. sam delete will attempt to delete those too.
Restrict the deletion to the single container SAM is responsible for — the same one selected by export (after the fix in comment #2, the container matching Metadata.ContainerName, falling back to index 0).
a771de9 to
699c10a
Compare
| if not service_arns: | ||
| continue | ||
| try: | ||
| svc_response = ecs_client.describe_services(cluster=cluster_arn, services=service_arns) |
There was a problem hiding this comment.
[BUG] describe_services(cluster=cluster_arn, services=service_arns) is called with all service ARNs returned by a single list_services page, but the ECS DescribeServices API rejects requests with more than 10 service identifiers (InvalidParameterException: Services cannot have more than 10 items). list_services can return up to 100 ARNs per page, so any cluster with more than 10 services will cause the call to fail. The exception is then caught by the broad except Exception on line 200 and logged at debug level, so syncs in those clusters silently never trigger a forced deployment.
Batch service_arns into chunks of at most 10 before calling describe_services, e.g.:
for i in range(0, len(service_arns), 10):
batch = service_arns[i : i + 10]
svc_response = ecs_client.describe_services(cluster=cluster_arn, services=batch)
for svc in svc_response.get("services", []):
...| if not container_defs: | ||
| return | ||
|
|
||
| # Use ContainerName from Metadata to find the target container, fall back to index 0 |
There was a problem hiding this comment.
[BUG] ECSTaskDefinitionImageResource.export picks the target container by "first ContainerDefinitions entry whose Image is not already an ECR URL" and ignores the ContainerName metadata. The build path in samcli/lib/build/app_builder.py:423-430 honors metadata.get("ContainerName") and writes the built image to the matching container by name (falling back to index 0 only when no name is set). For any multi-container TaskDefinition where the SAM-managed container is not the first non-ECR entry — for example a sidecar at index 0 with a public Docker Hub image and the SAM-managed container at index 1 — the build will set the right container's Image, but export will rewrite the sidecar's Image with the SAM-built ECR URL. The deployed task then runs the SAM image in the sidecar slot and the wrong image (the unmodified built tag) in the SAM-managed slot.
The author's comment says "Metadata is not available here at package time", but ResourceTypeBaseExporter already exposes mutable per-instance context (e.g. parent_parameter_values set by the caller in artifact_exporter.py just before export). The same pattern can be used to pass resource.get("Metadata") to the exporter, and then select the container by ContainerName using identical logic to _update_built_resource. Fixing this also fixes the analogous problem in delete (next comment).
| # Only delete the container SAM manages (same logic as export: first non-ECR, i.e. index 0 after deploy) | ||
| # After a successful deploy, the managed container will have an ECR URL that SAM pushed. | ||
| # We only delete index 0 by default since we can't access Metadata here. | ||
| remote_path = container_defs[0].get("Image") |
There was a problem hiding this comment.
[BUG] ECSTaskDefinitionImageResource.delete hardcodes container_defs[0] when choosing which image to delete, which is inconsistent with both the build path (selects by ContainerName) and the export path in the same class (selects by "first non-ECR"). After a successful deploy, every SAM-managed container has an ECR URL — but so do any user-owned ECR images for sidecars, and so do third-party images pulled through pull-through cache repos. If the SAM-managed container is at index 1 and an unrelated ECR image (e.g. a customer-owned aws-otel-collector repo, or a different team's repo) is at index 0, sam delete will call uploader.delete_artifact on an image SAM never pushed and does not own.
Use the same ContainerName-aware selection as the build path; only delete the image in the container that SAM manages.
| self.log_prefix, | ||
| svc["serviceName"], | ||
| ) | ||
| except Exception: |
There was a problem hiding this comment.
[ERROR_HANDLING] -203
The two broad except Exception blocks in _force_ecs_deployment swallow every failure path (the 10-item limit above, IAM denials on ecs:ListClusters/ecs:ListServices/ecs:UpdateService, throttling, network errors, etc.) and only log at debug level. The ECS service redeploy is the step that actually makes the new image take effect — if it fails, the user sees a successful image push and assumes their container has been updated, when in reality the running task is still on the old image. The mismatch is invisible without --debug.
At minimum, log at WARNING when these blocks fire so the user can see that the redeploy did not happen, and consider catching botocore.exceptions.ClientError specifically instead of bare Exception so that programming errors (AttributeError, KeyError) still surface.
699c10a to
890ee38
Compare
| # (ECR URLs, public images, etc.). Therefore, selecting the first container with a | ||
| # non-ECR Image value correctly identifies the SAM-managed container in all cases | ||
| # where 'sam build' has run before 'sam package/deploy'. | ||
| target_idx = 0 |
There was a problem hiding this comment.
[BUG] -720
ECSTaskDefinitionImageResource.export selects the target container as "the first ContainerDefinitions entry whose Image is not an ECR URL," but is_ecr_url (in samcli/lib/package/ecr_utils.py) only matches account-private ECR hostnames (*.dkr.ecr.<region>.amazonaws.com); it does not match public.ecr.aws/..., Docker Hub images (nginx:alpine), or any other non-private-ECR URL. After sam build, sidecars retain their original (often public) image references, so the heuristic can pick a sidecar instead of the SAM-managed container.
This is reproducible with the PR's own integration test template at tests/integration/testdata/buildcmd/container_image/template.yaml:
ContainerDefinitions:
- Name: sidecar
Image: public.ecr.aws/envoy:latest # index 0 — not a private ECR URL
- Name: web
Image: placeholder # index 1 — SAM-managed (ContainerName: web)After build, container 0 still has public.ecr.aws/envoy:latest. is_ecr_url("public.ecr.aws/envoy:latest") returns False, the loop picks target_idx = 0, and SAM will attempt to upload the public envoy image to the user's ECR repository instead of the actually built web image (which then never gets pushed/wired up).
The build path already honors metadata["ContainerName"] to write the built tag to the correct index (app_builder.py:423–430); the export path needs the same information. Plumb Metadata down to export/do_export for this resource (or look it up via the parent resource dict) and select the target container by ContainerName, falling back to index 0 only when no ContainerName is set — matching the build behavior.
This was raised in earlier review rounds and not addressed; the "first non-ECR" comment in the code defends an approach that does not work for non-private-ECR sidecars.
| # managed container's Image is the ECR URL SAM pushed, we delete only index 0. | ||
| # This is safe because sam build always places the built image at the correct index | ||
| # via _update_built_resource which honors ContainerName. | ||
| remote_path = container_defs[0].get("Image") |
There was a problem hiding this comment.
[BUG] ECSTaskDefinitionImageResource.delete hardcodes container_defs[0] when choosing which image to clean up. The accompanying comment claims "sam build always places the built image at the correct index via _update_built_resource which honors ContainerName" — but _update_built_resource in samcli/lib/build/app_builder.py:423–430 does the opposite: it writes the built image to the index of the container whose Name matches metadata["ContainerName"], which can be any index, not necessarily 0.
Concrete failure case using the same integration test template:
ContainerDefinitions:
- Name: sidecar # index 0
Image: <unrelated image>
- Name: web # index 1 — Metadata.ContainerName: web
Image: <SAM-pushed ECR URL after deploy>On sam delete:
- If the sidecar's image is a private ECR URL (e.g. an unrelated repo or a third-party pull-through-cache image like
123.dkr.ecr.us-east-1.amazonaws.com/aws-otel-collector),deletewill erroneously calldelete_artifacton the sidecar's image — deleting an ECR object SAM did not push. - The actual SAM-managed image at index 1 is never cleaned up.
Select the container by ContainerName consistently with the build path (and with the fix to issue #1), or persist the chosen index via Metadata so delete can find it. Falling back to index 0 only when no ContainerName was set is the matching behavior.
This was raised in earlier review rounds and not addressed.
| if not service_arns: | ||
| continue | ||
| try: | ||
| svc_response = ecs_client.describe_services(cluster=cluster_arn, services=service_arns) |
There was a problem hiding this comment.
[BUG] describe_services(cluster=cluster_arn, services=service_arns) is called with every service ARN returned by a single list_services page. list_services returns up to 100 ARNs per page, but the ECS DescribeServices API rejects requests with more than 10 service identifiers (InvalidParameterException: Services cannot have more than 10 items). Any cluster with more than 10 services in a page will fail this call, the exception is swallowed by the bare except Exception: on line 201 at debug level, and no service in that cluster gets a forced redeployment — silently.
Batch service_arns into chunks of 10 and call describe_services per chunk. For example:
for i in range(0, len(service_arns), 10):
chunk = service_arns[i:i + 10]
svc_response = ecs_client.describe_services(cluster=cluster_arn, services=chunk)
...This was raised in earlier review rounds and not addressed.
| self.log_prefix, | ||
| svc["serviceName"], | ||
| ) | ||
| except Exception: |
There was a problem hiding this comment.
[ERROR_HANDLING] ,203
The two broad exception handlers in _force_ecs_deployment swallow every failure mode of the redeploy step:
except Exception:on line 201 catches anything fromdescribe_services/update_service(the 10-item limit in Fix AWS credential retrieval #3, IAM denials onecs:UpdateService, throttling, etc.) and logs only atdebuglevel.except ClientError:on line 203 catches IAM/throttling/network errors fromlist_clusters/list_services/describe_task_definitionatwarninglevel, but doesn't fail the sync.
The forced ECS deployment is the step that actually makes the new image take effect for running services. If it silently fails, the user sees a successful image push and a successful sync but the running task continues using the old image — exactly the failure mode sam sync is supposed to prevent. At minimum, surface these as warning-level messages that explicitly call out that running services were not redeployed and may continue serving the old image, so the user knows manual action is required. Better: let unexpected failures propagate so the sync flow is marked failed.
|
Thanks for the PR Eric, I'd suggest to apply this change to a fresh branch created from develop, currently this PR seems branched off from master has like changes from 3 years ago. |
Extends SAM CLI to build, package, and deploy container images for
AWS::ECS::TaskDefinition and AWS::BedrockAgentCore::Runtime resources
using the same Metadata convention as Lambda Image functions.
## Changes
- Add resource type constants and RESOURCES_WITH_IMAGE_COMPONENT entries
- Add ContainerBuildDefinition to build graph with Architecture support
- Add build_container_images() to ApplicationBuilder
- Create packageable resource classes for ECR push during package/deploy
- Create SamContainerServiceProvider to discover buildable resources
- Wire container builds into BuildContext.run()
- Create ECSContainerSyncFlow for sam sync support
- Register in SyncFlowFactory
- Extend sync_ecr_stack for auto ECR repo creation (--resolve-image-repos)
- Support ContainerName metadata for multi-container ECS TaskDefinitions
- Update sam build help text
## Template Usage
```yaml
Resources:
MyAgent:
Type: AWS::BedrockAgentCore::Runtime
Metadata:
Dockerfile: Dockerfile
DockerContext: ./agent
DockerTag: latest
Architecture: arm64
Properties:
AgentRuntimeArtifact:
ContainerConfiguration:
ContainerUri: placeholder
MyTask:
Type: AWS::ECS::TaskDefinition
Metadata:
Dockerfile: Dockerfile
DockerContext: ./app
ContainerName: web # targets specific container
Properties:
ContainerDefinitions:
- Name: web
Image: placeholder
```
## Testing
- 51 new unit tests (853 total passing)
- Integration test with Docker build
- End-to-end validated: build → ECR push → CloudFormation deploy
890ee38 to
a956895
Compare
|
|
||
| def _get_target_index(self, container_defs): | ||
| """Find the target container index using ContainerName from Metadata.""" | ||
| metadata = getattr(self, "resource_metadata", None) or {} |
There was a problem hiding this comment.
[BUG] The new _get_target_index helper is dead code: it always returns 0 because resource_metadata is never plumbed onto the exporter instance.
In Template.export() (samcli/lib/package/artifact_exporter.py:493-495) the exporter is constructed as exporter_class(self.uploaders, self.code_signer, cache) and only exporter.parent_parameter_values is set before exporter.export(full_path, resource_dict, self.template_dir) is called. resource_dict is resource.get("Properties", {}) — the Metadata block is no longer accessible by the time ECSTaskDefinitionImageResource.export runs. getattr(self, "resource_metadata", None) therefore always evaluates to None, target_name is always falsy, and the loop is skipped — every export and delete operates on container_defs[0], ignoring the ContainerName metadata.
Symptoms in multi-container task definitions where the SAM-managed container is not first:
sam package/sam deploywrites the uploaded ECR URL into the wrong container (or fails because that container'sImageis already a private ECR URL — seeis_ecr_urlguard at line 716).sam deletedeletes whichever image is at index 0 in ECR, which can be an unrelated user-managed sidecar (e.g., a private-ECRaws-otel-collector) rather than the SAM-managed image.
The build path in ApplicationBuilder._update_built_resource (samcli/lib/build/app_builder.py:420-430) correctly honors ContainerName because the full resource dict (with Metadata) is passed through; the export/delete paths need the same plumbing. Either pass the resource Metadata block into the exporter (e.g., extend Template.export to set exporter.resource_metadata = resource.get("Metadata")) or change resource_dict here to the full resource dict so Metadata is reachable.
This was raised in earlier reviews and the helper added to address it is non-functional without the call-site change.
| return | ||
| target_idx = self._get_target_index(container_defs) | ||
| remote_path = container_defs[target_idx].get("Image") | ||
| if isinstance(remote_path, str) and is_ecr_url(remote_path): |
There was a problem hiding this comment.
[BUG] ECSTaskDefinitionImageResource.delete only inspects container_defs[target_idx], but combined with the _get_target_index bug above this is always index 0. For any task definition whose first container is not the SAM-managed one (e.g., a sidecar at index 0), the deploy phase pushed the SAM image to a different index, so index 0 still holds an unrelated image. If that unrelated image happens to live in private ECR (a common pattern for sidecars pulled through pull-through cache or shared between workloads), is_ecr_url matches and self.uploader.delete_artifact deletes an image SAM didn't manage.
Once the resource_metadata plumbing in #1 is fixed, this becomes correct because target_idx will resolve to the SAM-managed container by name. Until then, sam delete can silently destroy user-owned sidecar images.
| for i in range(0, len(service_arns), 10): | ||
| batch = service_arns[i : i + 10] | ||
| try: | ||
| svc_response = ecs_client.describe_services(cluster=cluster_arn, services=batch) |
There was a problem hiding this comment.
[BUG] A ClientError from update_service aborts processing the rest of the current 10-service batch.
Inside the inner try: at line 187, both describe_services and the iteration that calls update_service live in the same try body, with one shared except ClientError. So if the second service in a 10-service batch fails to update (e.g., transient throttling, a service mid-deletion, IAM denial), the loop exits the for svc in ... iteration immediately and services 3–10 in that batch are silently skipped — the warning logged is generic ("Failed to update services in ") and gives no indication that other services were not even attempted.
Wrap just the update_service call in its own try/except, e.g.:
try:
svc_response = ecs_client.describe_services(cluster=cluster_arn, services=batch)
except ClientError:
LOG.warning("%sFailed to describe services in %s", self.log_prefix, cluster_arn, exc_info=True)
continue
for svc in svc_response.get("services", []):
svc_task_def = svc.get("taskDefinition", "")
deployed_family = svc_task_def.rsplit("/", 1)[-1].split(":", 1)[0]
if deployed_family != family:
continue
try:
ecs_client.update_service(
cluster=cluster_arn,
service=svc["serviceName"],
forceNewDeployment=True,
)
LOG.info("%sForced new deployment for service %s", self.log_prefix, svc["serviceName"])
except ClientError:
LOG.warning(
"%sFailed to force deployment for service %s in %s",
self.log_prefix, svc.get("serviceName"), cluster_arn, exc_info=True,
)This also surfaces which service failed instead of only the cluster.
| cluster_arn, | ||
| exc_info=True, | ||
| ) | ||
| except ClientError: |
There was a problem hiding this comment.
[ERROR_HANDLING] The outer except ClientError at line 210 swallows every failure of the redeploy step at WARNING level: describe_task_definition, the list_clusters/list_services paginators (which raise ClientError on IAM denial or throttling), etc. Combined with the per-batch warnings in #3, a user can see "Image pushed successfully" followed by a single line "Failed to force ECS deployment" and assume the new image is running — when in fact no service was updated.
For the step that actually makes the new image take effect, this should at minimum:
- Surface the failure to the user (e.g.,
click.sechoor raise aSyncFlowExceptionso the sync result reflects the failure), not just log at WARNING. - Include enough detail (operation name, error code) for the user to act, e.g.
"Failed to force ECS deployment for task definition %s: %s. Run with --debug for details.".
If silent best-effort behavior is intentional (e.g., the task definition has no associated service), document that explicitly so the user understands a missing redeploy is expected and not a failure mode.
Which issue(s) does this change fix?
Fixes #8933
Why is this change necessary?
SAM CLI provides an excellent developer experience for Lambda Image functions (
sam build && sam deploy), but users deploying containerized workloads to ECS (Fargate) or Bedrock AgentCore must manage their Docker build/push/deploy pipeline separately — even when these resources live in the same CloudFormation template. This creates a fragmented workflow requiring external tooling for an identical operation: build image → push to ECR → deploy.How does it address the issue?
Extends the existing Lambda Image build pipeline to recognize
AWS::ECS::TaskDefinitionandAWS::BedrockAgentCore::Runtimeresources with aMetadatablock containingDockerfileandDockerContext. No new commands —sam build,sam package,sam deploy, andsam syncgain awareness of these resource types.Template example:
Key implementation details:
_build_lambda_image()— same Docker build logic, buildkit support included--resolve-image-reposauto-creates ECR repos via companion stackContainerNamemetadata targets specific containers in multi-container TaskDefinitionsArchitecturemetadata sets--platform(e.g.,arm64for AgentCore)ARTIFACT_TYPE = ZIPto pass thePackageTypefilter (these resources don't havePackageType)Design document:
designs/container_image_builds_ecs_agentcore.mdWhat side effects does this change have?
sam buildlogs "Found N container service resource(s) to build" when applicable resources are present. No behavior change for templates without these resources.--resolve-image-reposcreates ECR repos for ECS/AgentCore in addition to Lambda Image functions._update_built_resourceadds an optionalmetadataparameter (backward compatible, defaults toNone).Mandatory Checklist
PRs will only be reviewed after checklist is complete
make prpassesmake update-reproducible-reqsif dependencies were changedBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.