No response from invoke container for Lambda inside docker-compose

[SorsOps]

Description:

Related to #2492

When running a docker-compose with an image that is using sam to run a local api, the api can never get a response from the lambda container. Everything works correctly when run locally not in a container.

I have read through all the answers in 2492 and tried the various permutations suggested, nothing has worked. There appeared to be no subsequent documentation follow up for getting this setup to work with docker.

Steps to reproduce:

DockerFile

FROM python:alpine
RUN apk add --no-cache --virtual build-deps build-base=0.5-r2 gcc=10.2.1_pre1-r3 bash=5.1.0-r0 && \
    pip install aws-sam-cli==1.23.0 && \
    apk del build-deps
COPY ./sam_entrypoint.sh /bin/sam_entrypoint.sh
RUN chmod +x /bin/sam_entrypoint.sh

WORKDIR /app
COPY ./node_modules /app/node_modules
COPY ./package.json /app/package.json
COPY ./template.yml /app/template.yml
COPY ./dist /app/dist
EXPOSE 3000 
ENTRYPOINT ["sh","/bin/sam_entrypoint.sh"]

sam_entrypoint.sh

#!/bin/bash
set -o errexit
/usr/local/bin/sam local start-api --debug \
  --template ./template.yml \
  --host 0.0.0.0 \
  --container-host host.docker.internal \
  --container-host-interface  127.0.0.1 \
  --docker-network host

Template.yml

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: "Automated lambda setup for api gateway"
Resources:
  res0:
    Type: "AWS::Serverless::Function"
    Properties:
      CodeUri: "./"
      Handler: "dist/index.handler"
      Runtime: "nodejs12.x"
      Timeout: 10
      Events:
        CatchAll:
          Type: "Api"
          Properties:
            Path: "/"
            Method: ANY
  res1:
    Type: "AWS::Serverless::Function"
    Properties:
      CodeUri: "./"
      Handler: "dist/index.handler"
      Runtime: "nodejs12.x"
      Timeout: 10
      Events:
        CatchAll:
          Type: "Api"
          Properties:
            Path: "/{proxy+}"
            Method: ANY

Docker-compose

version: '3.6'
services:
  sam_app:
    build: 
      context: .
      dockerfile: Dockerfile.test
    ports:
      - "3000:3000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock  
    extra_hosts:
      - "host.docker.internal:host-gateway"

Observed result:

sam_app_1  | 
sam_app_1  |    SAM CLI now collects telemetry to better understand customer needs.
sam_app_1  |
sam_app_1  |    You can OPT OUT and disable telemetry collection by setting the
sam_app_1  |    environment variable SAM_CLI_TELEMETRY=0 in your shell.
sam_app_1  |    Thanks for your help!
sam_app_1  |
sam_app_1  |    Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html
sam_app_1  |
sam_app_1  | 2021-04-28 17:40:50,630 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
sam_app_1  | 2021-04-28 17:40:50,630 | Using config file: samconfig.toml, config environment: default
sam_app_1  | 2021-04-28 17:40:50,631 | Expand command line arguments to:
sam_app_1  | 2021-04-28 17:40:50,631 | --template_file=/app/template.yml --host=0.0.0.0 --port=3000 --static_dir=public --layer_cache_basedir=/root/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1      
sam_app_1  | 2021-04-28 17:40:50,704 | local start-api command is called
sam_app_1  | 2021-04-28 17:40:50,709 | No Parameters detected in the template
sam_app_1  | 2021-04-28 17:40:50,732 | 3 stacks found in the template
sam_app_1  | 2021-04-28 17:40:50,732 | No Parameters detected in the template
sam_app_1  | 2021-04-28 17:40:50,753 | 3 resources found in the stack
sam_app_1  | 2021-04-28 17:40:50,753 | No Parameters detected in the template
sam_app_1  | 2021-04-28 17:40:50,773 | Found Serverless function with name='res0' and CodeUri='./'
sam_app_1  | 2021-04-28 17:40:50,773 | --base-dir is not presented, adjusting uri ./ relative to /app/template.yml
sam_app_1  | 2021-04-28 17:40:50,773 | Found Serverless function with name='res1' and CodeUri='./'
sam_app_1  | 2021-04-28 17:40:50,773 | --base-dir is not presented, adjusting uri ./ relative to /app/template.yml
sam_app_1  | 2021-04-28 17:40:50,777 | No Parameters detected in the template
sam_app_1  | 2021-04-28 17:40:50,798 | No Parameters detected in the template
sam_app_1  | 2021-04-28 17:40:50,819 | Found '1' API Events in Serverless function with name 'res0'
sam_app_1  | 2021-04-28 17:40:50,819 | Found '1' API Events in Serverless function with name 'res1'
sam_app_1  | 2021-04-28 17:40:50,819 | Detected Inline Swagger definition
sam_app_1  | 2021-04-28 17:40:50,819 | Lambda function integration not found in Swagger document at path='/' method='x-amazon-apigateway-any-method'
sam_app_1  | 2021-04-28 17:40:50,819 | Lambda function integration not found in Swagger document at path='/{proxy+}' method='x-amazon-apigateway-any-method'
sam_app_1  | 2021-04-28 17:40:50,819 | Found '0' APIs in resource 'ServerlessRestApi'
sam_app_1  | 2021-04-28 17:40:50,819 | Removed duplicates from '0' Explicit APIs and '2' Implicit APIs to produce '2' APIs
sam_app_1  | 2021-04-28 17:40:50,819 | 2 APIs found in the template
sam_app_1  | 2021-04-28 17:40:50,825 | Mounting res0 at http://0.0.0.0:3000/ [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
sam_app_1  | 2021-04-28 17:40:50,825 | Mounting res1 at http://0.0.0.0:3000/{proxy+} [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
sam_app_1  | 2021-04-28 17:40:50,826 | You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You 
only need to restart SAM CLI if you update your AWS SAM template
sam_app_1  | 2021-04-28 17:40:50,826 | Localhost server is starting up. Multi-threading = True
sam_app_1  | 2021-04-28 17:40:50  * Running on http://0.0.0.0:3000/ (Press CTRL+C to quit)
sam_app_1  | 2021-04-28 17:41:20,791 | Constructed String representation of Event to invoke Lambda. Event: {"body": null, "headers": {"Accept": "*/*", "Host": "127.0.0.1:3000", "User-Agent": "insomnia/2021.2.2", "X-Forwarded-Port": "3000", "X-Forwarded-Proto": "http"}, "httpMethod": "GET", "isBase64Encoded": false, "multiValueHeaders": {"Accept": ["*/*"], "Host": ["127.0.0.1:3000"], "User-Agent": ["insomnia/2021.2.2"], "X-Forwarded-Port": ["3000"], "X-Forwarded-Proto": 
["http"]}, "multiValueQueryStringParameters": null, "path": "/v1/openapi.json", "pathParameters": {"proxy": "v1/openapi.json"}, "queryStringParameters": null, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "domainName": "127.0.0.1:3000", "extendedRequestId": null, "httpMethod": "GET", "identity": {"accountId": null, "apiKey": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityPoolId": null, "sourceIp": "172.24.0.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/{proxy+}", "protocol": "HTTP/1.1", "requestId": "d9c5fa20-2328-43f7-b4d2-5a35028dfab1", "requestTime": "28/Apr/2021:17:40:50 +0000", "requestTimeEpoch": 1619631650, "resourceId": "123456", "resourcePath": "/{proxy+}", "stage": "Prod"}, "resource": "/{proxy+}", "stageVariables": null, "version": "1.0"}
sam_app_1  | 2021-04-28 17:41:20,791 | Found one Lambda function with name 'res1'
sam_app_1  | 2021-04-28 17:41:20,791 | Invoking dist/index.handler (nodejs12.x)
sam_app_1  | 2021-04-28 17:41:20,791 | Environment variables overrides data is standard format
sam_app_1  | 2021-04-28 17:41:20,791 | Loading AWS credentials from session with profile 'None'
sam_app_1  | 2021-04-28 17:41:20,803 | Resolving code path. Cwd=/app, CodeUri=/app
sam_app_1  | 2021-04-28 17:41:20,803 | Resolved absolute path to code is /app
sam_app_1  | 2021-04-28 17:41:20,803 | Code /app is not a zip/jar file
sam_app_1  | 2021-04-28 17:41:20,817 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-nodejs12.x:rapid-1.23.0.
sam_app_1  |
sam_app_1  | 2021-04-28 17:41:20,817 | Mounting /app as /var/task:ro,delegated inside runtime container
sam_app_1  | 2021-04-28 17:41:21,124 | Starting a timer for 10 seconds for function 'res1'
sam_app_1  | 2021-04-28 17:41:22,075 | Cleaning all decompressed code dirs
sam_app_1  | 2021-04-28 17:41:22,075 | No response from invoke container for res1
sam_app_1  | 2021-04-28 17:41:22,075 | Invalid lambda response received: Lambda response must be valid json
sam_app_1  | 2021-04-28 17:41:22 172.24.0.1 - - [28/Apr/2021 17:41:22] "GET /v1/openapi.json HTTP/1.1" 502 -

Expected result:

That the lambda can be executed correctly and a response received.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10
2. sam --version: 1.23.0
3. AWS region: us-east-1

[hoffa]

Is your Docker daemon running locally? Could you try with --container-host-interface 0.0.0.0 (which will bind the container ports to all network interfaces) to ensure it's not an issue with port binding?

[SorsOps]

Hi @hoffa I've tested with --container-host-interface 0.0.0.0, at this point I can't even get a request to go through to the docker-compose container at that point. Trying

http://0.0.0.0:3000/v1/openapi.json ,
http://localhost:3000/v1/openapi.json
http:/127.0.0.1:3000/v1/openapi.json

from the host system is inaccessible

[xazhao]

Hi @Andrewx82 could you try the command without --docker-network option? The default bridge network should work for your case.

[SorsOps]

Hi @xazhao The call is now accessible and I see it being processed in the logs , however I see no response from the containers

sam_app_1  | 2021-05-04 11:43:05,155 | Found one Lambda function with name 'res1'
sam_app_1  | 2021-05-04 11:43:05,155 | Invoking dist/index.handler (nodejs12.x)
sam_app_1  | 2021-05-04 11:43:05,155 | Environment variables overrides data is standard format
sam_app_1  | 2021-05-04 11:43:05,155 | Loading AWS credentials from session with profile 'None'
sam_app_1  | 2021-05-04 11:43:05,168 | Resolving code path. Cwd=/app, CodeUri=/app
sam_app_1  | 2021-05-04 11:43:05,168 | Resolved absolute path to code is /app
sam_app_1  | 2021-05-04 11:43:05,169 | Code /app is not a zip/jar file
sam_app_1  | 2021-05-04 11:43:05,173 | Image was not found.
sam_app_1  | Building image...........................................................................................................................................................................................................................................................................................................................................................
sam_app_1  | 2021-05-04 11:44:41,985 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-nodejs12.x:rapid-1.23.0.
sam_app_1  |
sam_app_1  | 2021-05-04 11:44:41,985 | Mounting /app as /var/task:ro,delegated inside runtime container
sam_app_1  | 2021-05-04 11:44:42,334 | Starting a timer for 10 seconds for function 'res1'
sam_app_1  | 2021-05-04 11:44:43,246 | Cleaning all decompressed code dirs
sam_app_1  | 2021-05-04 11:44:43,246 | No response from invoke container for res1
sam_app_1  | 2021-05-04 11:44:43,247 | Invalid lambda response received: Lambda response must be valid json
sam_app_1  | 2021-05-04 11:44:43 172.20.0.1 - - [04/May/2021 11:44:43] "GET /v1/openapi.json HTTP/1.1" 502 -

[xazhao]

@Andrewx82 extra-host in your docker compose file is for Linux. Since your OS is Windows, I guess that is actually causing the issue. Can you try it without extra-host?
Btw in your case, it seems there is no need to use --container-host-interface option since the default value should work.

[SorsOps]

Hi @xazhao

I have changed the sam_entrypoint.sh to the following

set -o errexit
BASEDIR=$(pwd)
/usr/local/bin/sam local start-api --debug \
  --template ./template.yml \
  --host 0.0.0.0 \
  # --docker-volume-basedir "${BASEDIR}" \
  --container-host host.docker.internal \
  # --container-host-interface  0.0.0.0 \
  # --docker-network host

along with the docker-compose to

version: '3.6'
services:
  sam_app:
    build: 
      context: .
      dockerfile: Dockerfile.test
    ports:
      - "3000:3000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock  

Still appears to be no response

sam_app_1  | 2021-05-05 07:47:59,514 | Found one Lambda function with name 'res1'
sam_app_1  | 2021-05-05 07:47:59,515 | Invoking dist/index.handler (nodejs12.x)
sam_app_1  | 2021-05-05 07:47:59,515 | Environment variables overrides data is standard format
sam_app_1  | 2021-05-05 07:47:59,515 | Loading AWS credentials from session with profile 'None'
sam_app_1  | 2021-05-05 07:47:59,531 | Resolving code path. Cwd=/app, CodeUri=/app
sam_app_1  | 2021-05-05 07:47:59,531 | Resolved absolute path to code is /app
sam_app_1  | 2021-05-05 07:47:59,531 | Code /app is not a zip/jar file
sam_app_1  | 2021-05-05 07:47:59,547 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-nodejs12.x:rapid-1.23.0.
sam_app_1  |
sam_app_1  | 2021-05-05 07:47:59,547 | Mounting /app as /var/task:ro,delegated inside runtime container
sam_app_1  | 2021-05-05 07:47:59,823 | Starting a timer for 10 seconds for function 'res1'
sam_app_1  | 2021-05-05 07:48:00,764 | Cleaning all decompressed code dirs
sam_app_1  | 2021-05-05 07:48:00,765 | No response from invoke container for res1
sam_app_1  | 2021-05-05 07:48:00,765 | Invalid lambda response received: Lambda response must be valid json
sam_app_1  | 2021-05-05 07:48:00 172.20.0.1 - - [05/May/2021 07:48:00] "GET /v1/openapi.json HTTP/1.1" 502 -

[SorsOps]

@xazhao @hoffa

Would it be possible to just get a simple example of the SAM CLI working in docker using a volume from the host? There is an implicit understanding in the docs that its possible, but I don't see a working example anywhere for reference

[xazhao]

@Andrewx82 I was testing using configurations from this comment: #2436 (comment). After adding --container-host host.docker.internal it works. It looks pretty similar setup with your setup though.

[SorsOps]

@xazhao I can confirm that that does not work . See my bash entry script

#!/bin/bash
set -o errexit
BASEDIR=$(pwd)
/usr/local/bin/sam local start-api --debug \
  --template ./template.yml \
  -v $BASEDIR \
  --host 0.0.0.0 \
  --container-host host.docker.internal 

I also tried with and without the -v option

Can we just get an official example instead of saying that it is supposed to work ?

[xazhao]

@Andrewx82 This is what I'm currently using.
Dockerfile

FROM amazon/aws-cli

RUN yum -y update && \
    yum install -y python3 && \
    python3 -m pip install aws-sam-cli

ENTRYPOINT [ "" ]

docker-compose.yml

services:
  testcase:
    build: .
    working_dir: $PWD
    command: sam local start-api --host 0.0.0.0 --container-host host.docker.internal
    ports:
      - 3000:3000
    volumes:
      - '$PWD:$PWD'
      - '/var/run/docker.sock:/var/run/docker.sock'`

Then open another terminal and use the following command:
curl localhost:3000/hello

Please note: My development environment is macOS and I think mounting volume is different in Windows (Seems you already did properly). For your No response ... error, it usually happens when lambda function is not hosted on a container correctly.
One thing you might try is see if you can resolve host.docker.internal to an IP address. For me, host.docker.internal resolves to 192.168.65.2. If I use this ip instead of host.docker.internal, it will work too.

[SorsOps]

@xazhao I've tried to align my files to be similar to yours
aws-sam.zip

It appears that something is happening here as I'm getting information about interacting with the lambda, however it doesn't seem like its reading the values correctly. For clarity I don't want to use volumes for this as I want to bake a docker image with the lambda

[sbhvt]

I'm having a very similar issue. I think that the root cause is related to the code path it is attempting to mount.

In the output that @Andrewx82 posted earlier, you see this line: Mounting /app as /var/task:ro,delegated inside runtime container. '/app' is the WORKDIR and is the path within the container but it is not the path on the host, and the docker engine needs the path on host for mounting.

In order to properly evaluate paths for mounts, the docker engine needs the path on the host of the docker engine. In @xazhao 's examples, this is accomplished because of the working_dir: $PWD line in the docker-compose file. Using the sample code that @xaxhao shared, I am able to reproduce the same error that @Andrewx82 shared by commenting out the working_dir: $PWD line from docker-compose.

(Note that if using the exact code shared by @xazhao , the WORKDIR is "aws" instead of "app", but the issue is the same regardless.) Here's what you see logged when working_dir: $PWD is commented out:

sam_app_1  | 2021-05-20 20:26:52,932 | Resolving code path. Cwd=/aws, CodeUri=/aws/hello_world
sam_app_1  | 2021-05-20 20:26:52,932 | Resolved absolute path to code is /aws/hello_world
sam_app_1  | 2021-05-20 20:26:52,933 | Code /aws/hello_world is not a zip/jar file
...

In the use case of running from within VSCode devcontainers, which is one of the use cases mentioned in #2492, the option with working_dir: $PWD is not a feasible solution.

I also tried this (working_dir commented out but passing $PWD to the --docker-volume-basedir parameter) :

version: '3.6'
services:
  sam_app:
    #working_dir: $PWD
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    command: sam local start-api --host 0.0.0.0 --container-host host.docker.internal --debug --docker-volume-basedir $PWD
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - $PWD:$PWD

In that case, here is the the output:

sam_app_1  | 2021-05-20 20:30:33,580 | --base-dir is not presented, adjusting uri hello_world/ relative to /aws/template.yml
...
sam_app_1  | 2021-05-20 20:26:52,932 | Resolving code path. Cwd=<MY_LOCAL_PATH>, CodeUri=/aws/hello_world
sam_app_1  | 2021-05-20 20:26:52,932 | Resolved absolute path to code is /aws/hello_world
sam_app_1  | 2021-05-20 20:26:52,933 | Code /aws/hello_world is not a zip/jar file
...

I looked at the codebase related to that area of code: resolve_code_path in samcli/lib/utils/codeuri.py. The calling code already resolved the codeuri parameter to an absolute path before calling that method. Judging by the output, it looks like this happens in when it logs --base-dir is not presented. It looks like if the CodeUri were a relative path (the raw value in the template itself) instead of the absolute path when that method was called (i.e. Resolving code path. Cwd=<MY_LOCAL_PATH>, CodeUri=./hello_world), then it would properly resolve the absolute path to code and therefore work to use the --docker-volume-basedir $PWD option. This would allow for the functionality as it existed in prior versions of SAM CLI using that --docker-volume-basedir option and correspond to the documentation on the usage of that parameter which says:

If Docker is running on a remote machine, you must mount the path where the AWS SAM file exists on the Docker machine and modify this value to match the remote machine.

Requiring the configuration to use working_dir doesn't seem a viable resolution to #2492, especially in light of how it conflicts with the documentation around the --docker-volume-basedir parameter. It seems also related to: #963 and #1834.

The desired behavior being described in this issue, #2492, #963 and #1834 seem like they would be resolved if you used --base-dir with the value passed for --docker-volume-basedir or resolved the stack location from the outset using the --docker-volume-basedir parameter value.