-
Notifications
You must be signed in to change notification settings - Fork 1.2k
feat: support for local Event invocation type #8590
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Conversation
d46c1be to
d65bf60
Compare
|
@reedham-aws could you retry rerunning the failed tests? Much appreciated |
8a8842c to
4860576
Compare
|
@reedham-aws could I get workflow approval again? thanks |
|
@reedham-aws @vicheey thank you, looks like tests are passing. Let me know if there's anything I can do to help get this reviewed, or any feedback I can address |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The logic looks solid to me. I didn't think about it too hard, but one small concern I have is how we're testing some of this logic. I feel like we'd want to test that the threads are being executed and cleaned properly beyond the original return of the 202. I think this is difficult to test considering it's a different process, but I'm not too sure. Do you have any thoughts on this @monroeclinton?
Side note: I didn't have time today, but maybe tomorrow I will try it out using samdev to confirm it's doing as expected.
| "", | ||
| { | ||
| "Content-Type": "application/json", | ||
| "X-Amz-Durable-Execution-Arn": "arn:aws:lambda:us-west-2:123456789012:function:test-function:$LATEST/durable-execution/test-123", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't quite follow what's happening here; why are we removing the durable execution ARN and changing the InvocationType to DryRun below?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this test case is asserting that the Event (async) invocation returns a 202 status code. This test case mocks the invoke function to return that X-Amz-Durable-Execution-Arn header. Since this PR now runs the invoke function on a thread, we are unable to get the invoke response before returning the response with a 202 status code. Prior to this PR, passing Event invocation type (like this test does) will cause an exception (as would this test, if not for the mock), so I'm not entirely sure why this test case was included in the durable execution PR. Ideally we could retrieve the ARN and return it, I tried to reverse engineer the aws-durable-execution-emulator-x86_64 binary to see how the ARN is computed, but since the internal durable execution library uses Python 3.13 I couldn't fully disassemble it with pycdc.
For changing the InvocationType to Event below, since Event is now supported we need to pass in a different unsupported invocation type to test the error message when an unsupported invocation type is passed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense.
I just realized that the below changing from Event to DryRun is a different method, which definitely cleared up some confusion 😆
| @patch("samcli.local.lambda_service.local_lambda_http_service.ThreadPoolExecutor") | ||
| @patch("samcli.local.lambda_service.local_lambda_http_service.LambdaErrorResponses") | ||
| def test_invoke_request_handler_async_invocation_function_not_found_returns_error( | ||
| self, lambda_error_responses_mock, executor_class_mock | ||
| ): | ||
| # Test that async invocation (Event type) returns error when function doesn't exist | ||
| executor_mock = Mock() | ||
| executor_class_mock.return_value = executor_mock | ||
|
|
||
| request_mock = Mock() | ||
| request_mock.get_data.return_value = b"{}" | ||
| request_mock.args = {} | ||
| request_mock.headers = {"X-Amz-Invocation-Type": "Event"} | ||
| local_lambda_http_service.request = request_mock | ||
|
|
||
| lambda_runner_mock = Mock() | ||
| lambda_runner_mock.get_function.side_effect = FunctionNotFound | ||
| lambda_error_responses_mock.resource_not_found.return_value = "function not found response" | ||
|
|
||
| service = LocalLambdaHttpService(lambda_runner=lambda_runner_mock, port=3000, host="localhost") | ||
|
|
||
| response = service._invoke_request_handler(function_name="NonExistentFunction") | ||
|
|
||
| # Verify error response is returned | ||
| self.assertEqual(response, "function not found response") | ||
| # Verify get_function was called to validate | ||
| lambda_runner_mock.get_function.assert_called_once_with("NonExistentFunction") | ||
| # Verify executor was NOT called since validation failed | ||
| executor_mock.submit.assert_not_called() | ||
| # Verify error response uses normalized function name | ||
| lambda_error_responses_mock.resource_not_found.assert_called_once_with("NonExistentFunction") | ||
|
|
||
| @patch("samcli.local.lambda_service.local_lambda_http_service.LOG") | ||
| @patch("samcli.local.lambda_service.local_lambda_http_service.ThreadPoolExecutor") | ||
| @patch("samcli.local.lambda_service.local_lambda_http_service.LambdaErrorResponses") | ||
| def test_invoke_request_handler_async_invocation_invalid_function_name_returns_error( | ||
| self, lambda_error_responses_mock, executor_class_mock, log_mock | ||
| ): | ||
| # Test that async invocation (Event type) returns error when function name is invalid | ||
| executor_mock = Mock() | ||
| executor_class_mock.return_value = executor_mock | ||
|
|
||
| request_mock = Mock() | ||
| request_mock.get_data.return_value = b"{}" | ||
| request_mock.args = {} | ||
| request_mock.headers = {"X-Amz-Invocation-Type": "Event"} | ||
| local_lambda_http_service.request = request_mock | ||
|
|
||
| lambda_runner_mock = Mock() | ||
| test_exception = InvalidFunctionNameException("Invalid function name format") | ||
| lambda_runner_mock.get_function.side_effect = test_exception | ||
| lambda_error_responses_mock.validation_exception.return_value = "validation error response" | ||
|
|
||
| service = LocalLambdaHttpService(lambda_runner=lambda_runner_mock, port=3000, host="localhost") | ||
|
|
||
| response = service._invoke_request_handler(function_name="Invalid@Function#Name") | ||
|
|
||
| # Verify error response is returned | ||
| self.assertEqual(response, "validation error response") | ||
| # Verify get_function was called to validate | ||
| lambda_runner_mock.get_function.assert_called_once_with("Invalid@Function#Name") | ||
| # Verify executor was NOT called since validation failed | ||
| executor_mock.submit.assert_not_called() | ||
| # Verify error was logged | ||
| log_mock.error.assert_called_once_with("Validation error: %s", "Invalid function name format") | ||
| # Verify validation exception was called with the error message | ||
| lambda_error_responses_mock.validation_exception.assert_called_once_with("Invalid function name format") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These tests feel superfluous considering we test the same things for synchronous events, but I think we can leave them to test if there's somehow an unintended divergence in the path of sync/async.
| self.assertTrue(result) | ||
|
|
||
|
|
||
| class TestLambdaRuntime_clean_runtime_containers(TestCase): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these tests related to changes in the PR? If so, could you explain why because I'm not following.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@reedham-aws I had to get the test coverage to 94%, after adding test coverage to my new lines it was around 93.96% if I recall correctly
|
@reedham-aws I think that's a great idea for additional testing, I'll look more into testing the threads tomorrow |
|
Thank you very much for you contributions and responses @monroeclinton! |
|
@reedham-aws I went with using I was thinking about how best to assert cleanup of any threads, but since the I had to bump test coverage more with: |
reedham-aws
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the changes, had one really minor nit.
I also tested the feature using samdev local start-lambda to see how it handled async events. Everything works how I expected; the server properly returns 202, but still executes after being enqueued. I tested by sending a stream of requests and got the responses immediately, and saw the invocation results properly stream in server side.
tests/unit/local/lambda_service/test_local_lambda_http_service.py
Outdated
Show resolved
Hide resolved
Also, maybe we could add a comment to this effect where we do the |
80729fb to
c669cfd
Compare
c669cfd to
ddde6c7
Compare
|
@reedham-aws Thanks for the feedback, updated the test assertions and added a comment about |
|
Thank you @monroeclinton for the attention to all my feedback! I will try to get another member of the team to review this PR so we can look at merging it. |
|
|
||
| @pytest.mark.flaky(reruns=3) | ||
| @pytest.mark.timeout(timeout=300, method="thread") | ||
| def test_invoke_with_event_invocation_type(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't really test that the function itself actually ran here. It would be nice to capture the stdout from the invocation in case something between where the executor queues the invocation and actually running the function breaks.
| f"invocation-type: {invocation_type} is not supported. " | ||
| "Event and RequestResponse are only supported." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| f"invocation-type: {invocation_type} is not supported. " | |
| "Event and RequestResponse are only supported." | |
| f"invocation-type: {invocation_type} is not supported. " | |
| "Only Event and RequestResponse are supported" |
| @@ -112,7 +112,7 @@ def test_invoke_with_log_type_not_None(self): | |||
| def test_invoke_with_invocation_type_not_RequestResponse(self): | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unrelated to this PR - this test feels like it would be more appropriate in just test_start_lambda.py. This file is intended to be CDK specific but has additional validation tests 🤔
| ) | ||
|
|
||
| @patch("samcli.local.services.base_local_service.BaseLocalService.service_response") | ||
| def test_validation_exception(self, service_response_mock): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thank you 🫡
| try: | ||
| self._invoke_lambda(function_name=function_name, **kwargs) | ||
| except Exception as e: | ||
| LOG.error("Async invocation failed for function %s: %s", function_name, str(e), exc_info=True) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It makes sense to sink the exceptions like you're doing here, but how does this appear for the user? Do you have a screenshot of what it looks like when you try async invoking a function that has a syntax error (so the sandbox crashes)?
| stdout_stream_bytes = io.BytesIO() | ||
| stdout_stream_writer = StreamWriter(stdout_stream_string, stdout_stream_bytes, auto_flush=True) | ||
|
|
||
| normalized_function_name = normalize_sam_function_identifier(function_name) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we call the get_function() utility you added here instead?
This PR is a continuation of the discussion from #6900, #510 (comment), and #749.
Which issue(s) does this change fix?
#6900
Why is this change necessary?
This is necessary to allow running
Eventinvocation types locally to make development more pleasant.How does it address the issue?
This PR uses
ThreadPoolExecutorto create an executor that runsEventinvocation types, allowing for non-blocking invocations.What side effects does this change have?
Eventinvocations now return a response with a202status code.Eventinvocations that have an invalid function return a404withResourceNotFound.Mandatory Checklist
PRs will only be reviewed after checklist is complete
make prpassesmake update-reproducible-reqsif dependencies were changedBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.