Skip to content

Conversation

@monroeclinton
Copy link

@monroeclinton monroeclinton commented Jan 25, 2026

This PR is a continuation of the discussion from #6900, #510 (comment), and #749.

Which issue(s) does this change fix?

#6900

Why is this change necessary?

This is necessary to allow running Event invocation types locally to make development more pleasant.

How does it address the issue?

This PR uses ThreadPoolExecutor to create an executor that runs Event invocation types, allowing for non-blocking invocations.

What side effects does this change have?

  • Event invocations now return a response with a 202 status code.
  • Event invocations that have an invalid function return a 404 with ResourceNotFound.

Mandatory Checklist

PRs will only be reviewed after checklist is complete

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@monroeclinton monroeclinton requested a review from a team as a code owner January 25, 2026 17:58
@github-actions github-actions bot added area/local/start-api sam local start-api command area/local/invoke sam local invoke command area/local/start-invoke pr/external stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. labels Jan 25, 2026
@monroeclinton
Copy link
Author

@reedham-aws could you retry rerunning the failed tests? Much appreciated

@monroeclinton
Copy link
Author

@reedham-aws could I get workflow approval again? thanks

@monroeclinton
Copy link
Author

@reedham-aws @vicheey thank you, looks like tests are passing. Let me know if there's anything I can do to help get this reviewed, or any feedback I can address

Copy link
Contributor

@reedham-aws reedham-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic looks solid to me. I didn't think about it too hard, but one small concern I have is how we're testing some of this logic. I feel like we'd want to test that the threads are being executed and cleaned properly beyond the original return of the 202. I think this is difficult to test considering it's a different process, but I'm not too sure. Do you have any thoughts on this @monroeclinton?

Side note: I didn't have time today, but maybe tomorrow I will try it out using samdev to confirm it's doing as expected.

"",
{
"Content-Type": "application/json",
"X-Amz-Durable-Execution-Arn": "arn:aws:lambda:us-west-2:123456789012:function:test-function:$LATEST/durable-execution/test-123",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite follow what's happening here; why are we removing the durable execution ARN and changing the InvocationType to DryRun below?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this test case is asserting that the Event (async) invocation returns a 202 status code. This test case mocks the invoke function to return that X-Amz-Durable-Execution-Arn header. Since this PR now runs the invoke function on a thread, we are unable to get the invoke response before returning the response with a 202 status code. Prior to this PR, passing Event invocation type (like this test does) will cause an exception (as would this test, if not for the mock), so I'm not entirely sure why this test case was included in the durable execution PR. Ideally we could retrieve the ARN and return it, I tried to reverse engineer the aws-durable-execution-emulator-x86_64 binary to see how the ARN is computed, but since the internal durable execution library uses Python 3.13 I couldn't fully disassemble it with pycdc.

For changing the InvocationType to Event below, since Event is now supported we need to pass in a different unsupported invocation type to test the error message when an unsupported invocation type is passed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense.

I just realized that the below changing from Event to DryRun is a different method, which definitely cleared up some confusion 😆

Comment on lines +560 to +626
@patch("samcli.local.lambda_service.local_lambda_http_service.ThreadPoolExecutor")
@patch("samcli.local.lambda_service.local_lambda_http_service.LambdaErrorResponses")
def test_invoke_request_handler_async_invocation_function_not_found_returns_error(
self, lambda_error_responses_mock, executor_class_mock
):
# Test that async invocation (Event type) returns error when function doesn't exist
executor_mock = Mock()
executor_class_mock.return_value = executor_mock

request_mock = Mock()
request_mock.get_data.return_value = b"{}"
request_mock.args = {}
request_mock.headers = {"X-Amz-Invocation-Type": "Event"}
local_lambda_http_service.request = request_mock

lambda_runner_mock = Mock()
lambda_runner_mock.get_function.side_effect = FunctionNotFound
lambda_error_responses_mock.resource_not_found.return_value = "function not found response"

service = LocalLambdaHttpService(lambda_runner=lambda_runner_mock, port=3000, host="localhost")

response = service._invoke_request_handler(function_name="NonExistentFunction")

# Verify error response is returned
self.assertEqual(response, "function not found response")
# Verify get_function was called to validate
lambda_runner_mock.get_function.assert_called_once_with("NonExistentFunction")
# Verify executor was NOT called since validation failed
executor_mock.submit.assert_not_called()
# Verify error response uses normalized function name
lambda_error_responses_mock.resource_not_found.assert_called_once_with("NonExistentFunction")

@patch("samcli.local.lambda_service.local_lambda_http_service.LOG")
@patch("samcli.local.lambda_service.local_lambda_http_service.ThreadPoolExecutor")
@patch("samcli.local.lambda_service.local_lambda_http_service.LambdaErrorResponses")
def test_invoke_request_handler_async_invocation_invalid_function_name_returns_error(
self, lambda_error_responses_mock, executor_class_mock, log_mock
):
# Test that async invocation (Event type) returns error when function name is invalid
executor_mock = Mock()
executor_class_mock.return_value = executor_mock

request_mock = Mock()
request_mock.get_data.return_value = b"{}"
request_mock.args = {}
request_mock.headers = {"X-Amz-Invocation-Type": "Event"}
local_lambda_http_service.request = request_mock

lambda_runner_mock = Mock()
test_exception = InvalidFunctionNameException("Invalid function name format")
lambda_runner_mock.get_function.side_effect = test_exception
lambda_error_responses_mock.validation_exception.return_value = "validation error response"

service = LocalLambdaHttpService(lambda_runner=lambda_runner_mock, port=3000, host="localhost")

response = service._invoke_request_handler(function_name="Invalid@Function#Name")

# Verify error response is returned
self.assertEqual(response, "validation error response")
# Verify get_function was called to validate
lambda_runner_mock.get_function.assert_called_once_with("Invalid@Function#Name")
# Verify executor was NOT called since validation failed
executor_mock.submit.assert_not_called()
# Verify error was logged
log_mock.error.assert_called_once_with("Validation error: %s", "Invalid function name format")
# Verify validation exception was called with the error message
lambda_error_responses_mock.validation_exception.assert_called_once_with("Invalid function name format")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests feel superfluous considering we test the same things for synchronous events, but I think we can leave them to test if there's somehow an unintended divergence in the path of sync/async.

self.assertTrue(result)


class TestLambdaRuntime_clean_runtime_containers(TestCase):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these tests related to changes in the PR? If so, could you explain why because I'm not following.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reedham-aws I had to get the test coverage to 94%, after adding test coverage to my new lines it was around 93.96% if I recall correctly

@monroeclinton
Copy link
Author

@reedham-aws I think that's a great idea for additional testing, I'll look more into testing the threads tomorrow

@reedham-aws
Copy link
Contributor

Thank you very much for you contributions and responses @monroeclinton!

@monroeclinton
Copy link
Author

monroeclinton commented Jan 29, 2026

@reedham-aws I went with using threading.Event flags and assertions that the invoke function finishes executing (i.e. does not block) after the 202 response being returned. I'm more than happy to explore different ideas for asserting the Event invocation functionality

2664b99

I was thinking about how best to assert cleanup of any threads, but since the LocalLambdaService is blocking, and only exits with SIGINT/SIGTERM, I think it's pretty much guaranteed that the threads will be cleaned up due to the ThreadPoolExecutor implementation

https://github.com/python/cpython/blob/1848ce61f349533ae5892a8c24c2e0e3c364fc8a/Lib/concurrent/futures/thread.py#L23

I had to bump test coverage more with:

80729fb

Copy link
Contributor

@reedham-aws reedham-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the changes, had one really minor nit.

I also tested the feature using samdev local start-lambda to see how it handled async events. Everything works how I expected; the server properly returns 202, but still executes after being enqueued. I tested by sending a stream of requests and got the responses immediately, and saw the invocation results properly stream in server side.

@reedham-aws
Copy link
Contributor

reedham-aws commented Jan 29, 2026

I was thinking about how best to assert cleanup of any threads, but since the LocalLambdaService is blocking, and only exits with SIGINT/SIGTERM, I think it's pretty much guaranteed that the threads will be cleaned up due to the ThreadPoolExecutor implementation

https://github.com/python/cpython/blob/1848ce61f349533ae5892a8c24c2e0e3c364fc8a/Lib/concurrent/futures/thread.py#L23

Also, maybe we could add a comment to this effect where we do the self.executor.submit?

@monroeclinton
Copy link
Author

@reedham-aws Thanks for the feedback, updated the test assertions and added a comment about ThreadPoolExecutor

@reedham-aws
Copy link
Contributor

Thank you @monroeclinton for the attention to all my feedback! I will try to get another member of the team to review this PR so we can look at merging it.


@pytest.mark.flaky(reruns=3)
@pytest.mark.timeout(timeout=300, method="thread")
def test_invoke_with_event_invocation_type(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't really test that the function itself actually ran here. It would be nice to capture the stdout from the invocation in case something between where the executor queues the invocation and actually running the function breaks.

Comment on lines +311 to +312
f"invocation-type: {invocation_type} is not supported. "
"Event and RequestResponse are only supported."
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
f"invocation-type: {invocation_type} is not supported. "
"Event and RequestResponse are only supported."
f"invocation-type: {invocation_type} is not supported. "
"Only Event and RequestResponse are supported"

@@ -112,7 +112,7 @@ def test_invoke_with_log_type_not_None(self):
def test_invoke_with_invocation_type_not_RequestResponse(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated to this PR - this test feels like it would be more appropriate in just test_start_lambda.py. This file is intended to be CDK specific but has additional validation tests 🤔

)

@patch("samcli.local.services.base_local_service.BaseLocalService.service_response")
def test_validation_exception(self, service_response_mock):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you 🫡

try:
self._invoke_lambda(function_name=function_name, **kwargs)
except Exception as e:
LOG.error("Async invocation failed for function %s: %s", function_name, str(e), exc_info=True)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes sense to sink the exceptions like you're doing here, but how does this appear for the user? Do you have a screenshot of what it looks like when you try async invoking a function that has a syntax error (so the sandbox crashes)?

stdout_stream_bytes = io.BytesIO()
stdout_stream_writer = StreamWriter(stdout_stream_string, stdout_stream_bytes, auto_flush=True)

normalized_function_name = normalize_sam_function_identifier(function_name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we call the get_function() utility you added here instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/local/invoke sam local invoke command area/local/start-api sam local start-api command area/local/start-invoke pr/external stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants