feat(evaluation): add KMS, tags, online data source, and updated_at to batch eval - #533
Conversation
…o batch eval Surface batch-evaluation API parameters that StartBatchEvaluation/ GetBatchEvaluation support but the SDK did not expose: - BatchEvaluationRunConfig.kms_key_arn -> StartBatchEvaluation kmsKeyArn, for encrypting evaluation data at rest with a customer-managed key. - BatchEvaluationRunConfig.tags -> StartBatchEvaluation tags. - OnlineEvaluationDataSourceConfig: new DataSourceConfig that emits onlineEvaluationConfigSource (onlineEvaluationConfigArn + optional sessionFilterConfig) to evaluate sessions captured by an existing OnlineEvaluationConfig. - BatchEvaluationResult.kms_key_arn and .updated_at, populated from the GetBatchEvaluation response. Bump boto3/botocore floor to 1.43.31, the first release whose bedrock-agentcore model includes these parameters. Adds unit tests for each path and exports OnlineEvaluationDataSourceConfig from bedrock_agentcore.evaluation.
✅ No Breaking Changes DetectedNo public API breaking changes found in this PR. |
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
|
Claude Security Review: the review did not actually analyze this PR (model took 0 turns — the skill likely failed during setup). See the run for details; a later push or re-run is needed for a real review. |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
|
Claude Security Review: the review did not analyze this PR (model took 0 turns). See the run for details; a later push or re-run is needed. |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
|
Claude Security Review: the review did not analyze this PR (model took 0 turns). See the run for details; a later push or re-run is needed. |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
notgitika
left a comment
notgitika
left a comment
There was a problem hiding this comment.
looks good to me, it is backwards compatible and just config changes. curious to see if you think adding integ tests here would be useful
| "boto3>=1.43.0", | ||
| "botocore>=1.43.0", |
There was a problem hiding this comment.
can we remove the dependency bump from this PR?
There was a problem hiding this comment.
What's wrong with a dependency bump here? We need an updated boto3 so the new parameters shows up.
| polling_timeout_seconds: int = 1800 | ||
| polling_interval_seconds: int = 30 | ||
| simulation_config: Optional[SimulationConfig] = None | ||
| kms_key_arn: Optional[str] = None |
There was a problem hiding this comment.
maybe adding some client side validation on the kms key shape could be good
There was a problem hiding this comment.
Good Idea but if a customers put an invalid arn here, the service will return a ValidationException.
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
3 participants
What
Expose batch-evaluation API parameters that
StartBatchEvaluation/GetBatchEvaluationalready support but the SDK did not surface:BatchEvaluationRunConfig.kms_key_arn→StartBatchEvaluation.kmsKeyArn— encrypt evaluation data at rest with a customer-managed KMS key.BatchEvaluationRunConfig.tags→StartBatchEvaluation.tags.OnlineEvaluationDataSourceConfig— newDataSourceConfigsubclass emittingonlineEvaluationConfigSource(onlineEvaluationConfigArn+ optionalsessionFilterConfig), to evaluate sessions captured by an existingOnlineEvaluationConfig.BatchEvaluationResult.kms_key_arnand.updated_at— populated from theGetBatchEvaluationresponse.boto3/botocorefloor to 1.43.31, the first release whosebedrock-agentcoremodel includes these parameters.OnlineEvaluationDataSourceConfigfrombedrock_agentcore.evaluation.Why
These are existing, optional service parameters that the runner silently dropped. KMS support in particular is a common requirement for customers with data-at-rest encryption mandates. All additions are backward compatible: every new field is optional and omitted from the request when unset.
Testing
to_data_source_configwith/without time range, ARN validation,updated_atsurfacing). Full evaluation suite: 359 passed (2 pre-existing collection errors in optionalstrands-agents-evalsintegration tests, unrelated to this change).ruff check/ruff format: clean.bedrock-agentcore(us-west-2):updated_at— a realGetBatchEvaluationresponse was parsed by the result model;updated_atpopulated correctly. ✅kms_key_arn— a realStartBatchEvaluationcall accepted and acted onkmsKeyArn; it reached the service's KMS-encrypt step (request rejected only because the test key's policy lackskms:GenerateDataKeyfor the service role — i.e. the parameter is wired through and honored end-to-end). ✅botocoreparam validation — verified viabotocore.Stubberagainst the real service model. ✅