SageMaker Pipelines: AWS Lambda integration

[elvinx]

Describe the feature you'd like
I would like to be able to define AWS Lambda as a step in my SageMaker workflows.

How would this feature be used? Please describe.
This will allow us to integrate 3rd party services/APIs to our ML workflows.

Describe alternatives you've considered
Using Step Functions

[wangweixun]

Hi there, thanks for your interests in SageMaker Pipelines. Your feedback is very important to us.

Can I ask how long your "3rd party services/APIs" will take to execute as a pipeline step? Do you expect them to finish within Lambda's 15 min runtime limit?

[elvinx]

Hi @wangweixun sometimes it might take longer than that for asynchronous jobs. We solve this with Step Functions by creating trigger, wait, check type of loops.

[kirit93]

@elvinx - we launched support for a custom callback step. Using this, you can define a step to trigger a lambda function or any other AWS service you want.

https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-callback

[jerrypeng7773]

lambda step is supported now, check this out

https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-lambda

@elvinx