Skip to content
This repository was archived by the owner on Feb 11, 2026. It is now read-only.

B 21571 memory handling enhancement#10

Merged
cameroncaci merged 14 commits into
mainfrom
b-21571-memory-handling-enhancement
Nov 1, 2024
Merged

B 21571 memory handling enhancement#10
cameroncaci merged 14 commits into
mainfrom
b-21571-memory-handling-enhancement

Conversation

@cameroncaci
Copy link
Copy Markdown
Contributor

@cameroncaci cameroncaci commented Nov 1, 2024

  • Resolves timeout by introducing multithreading.
  • Resolves memory issues by storing the temp file in a multi-part upload rather than memory.
  • Added some typing to the file and requirements.txt for improved IDE intellisense.

Tested successfully in loadtest and staging.

Python pool, which would've used less code, is not possible in AWS Lambda due to infrastructure differences (Pool requires a dedicated shared memory location). Pipe and Process were used for manual process management.
https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/

Inherited PR

@cameroncaci cameroncaci merged commit 5dd9abc into main Nov 1, 2024
@cameroncaci cameroncaci deleted the b-21571-memory-handling-enhancement branch November 1, 2024 17:35
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants