fix(batch): delete >10 messages in legacy sqs processor by whardier · Pull Request #818 · aws-powertools/powertools-lambda-python

whardier · 2021-11-12T23:35:25Z

No related issue

Description of changes:

SQS/Batch - SImple addition to process chunks of records that need removed in 10's. Counterintuitive and migrating this behaviour to the batch processor scope itself would be too disruptive.

Checklist

Meet tenets criteria
Update tests
Update docs
PR title follows conventional commit semantics

Breaking change checklist

No breaking changes. Introduces no new exception handling and would fix batch handling of more than 10 records.

No RFC:

Migration process documented
Implement warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…e than 10 messages were processed that would need deleted.

whardier · 2021-11-12T23:54:34Z

Working on updating the tests now.. overlooked it.

… as list

codecov-commenter · 2021-11-13T00:01:57Z

Codecov Report

Merging #818 (9333e14) into develop (6869155) will decrease coverage by 0.05%.
The diff coverage is 86.36%.

@@             Coverage Diff             @@
##           develop     #818      +/-   ##
===========================================
- Coverage    99.96%   99.90%   -0.06%     
===========================================
  Files          119      119              
  Lines         5347     5364      +17     
  Branches       610      612       +2     
===========================================
+ Hits          5345     5359      +14     
- Misses           0        3       +3     
  Partials         2        2

Impacted Files	Coverage Δ
aws_lambda_powertools/utilities/batch/sqs.py	`96.15% <86.36%> (-3.85%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6869155...9333e14. Read the comment docs.

heitorlessa · 2021-11-13T16:13:32Z

Hey @whardier could you create an issue first detailing the bug please?

For example, what configuration are you using so that your function receives more than 10 messages per invocation where this is necessary? e.g. large batch size, longer batch window?

If a single function receives a few thousand records with 50% partial failure, this has the side effect of increasing function duration and potentially timing out if one is not careful.

An issue helps us think about potential side effects like these, compromises, what information we need to provide in the docs to help other customers unfamiliar with this scenario, etc.

Thanks a lot!

whardier · 2021-11-15T19:16:30Z

#824

heitorlessa · 2022-01-28T14:54:44Z

@mploski before you tackle the redis one, could you help revamp this PR so we can merge it?

Ask. Move to concurrent futures and pass the client to make it thread safe as explained in the review

Background. This is the legacy batch we keep around that we're only deleting 10 messages instead of all failed messages in a batch (could be hundreds).

mploski · 2022-01-28T21:26:46Z

@mploski before you tackle the redis one, could you help revamp this PR so we can merge it?

Ask. Move to concurrent futures and pass the client to make it thread safe as explained in the review

Background. This is the legacy batch we keep around that we're only deleting 10 messages instead of all failed messages in a batch (could be hundreds).

Sure @heitorlessa! will look at this at the end of the next week.

mploski · 2022-02-07T10:23:39Z

@whardier @heitorlessa I will make changes on top of this PR's branch today.

mploski · 2022-02-07T20:40:37Z

@whardier Could you fetch latest changes from upstream into your repo clone? You're lagging behind a bit :-)

…h-needs-batched

mploski · 2022-02-08T21:33:33Z

@whardier @heitorlessa Applied suggestions, please review.

whardier · 2022-02-08T22:49:09Z

This looks clean. I can't test it right away however. I like it.

heitorlessa

Looks awesome, thank you so much @mploski and also @whardier for kick starting it.

We can now make moves to release several papercuts and the mypy official support now.

Thank you!

heitorlessa · 2022-02-09T08:40:24Z

        self.fail_messages.clear()

-    def _clean(self):
+    def _clean(self) -> Optional[List]:


niiiice one! Thanks

Delete to messages at a time. Before this problems would occur if mor…

41604c2

…e than 10 messages were processed that would need deleted.

boring-cyborg Bot added the area/utilities label Nov 12, 2021

pull-request-size Bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Nov 12, 2021

Updated test (maybe?) test_partial_sqs_clean to pass entries_to_clean…

7bcce12

… as list

pull-request-size Bot removed the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Nov 12, 2021

boring-cyborg Bot added the tests label Nov 12, 2021

pull-request-size Bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 12, 2021

heitorlessa added the need-issue PRs that are missing related issues label Nov 14, 2021

whardier mentioned this pull request Nov 15, 2021

"Batch" of SQS messages above 10 count causes problems during cleanup since boto3.client('sqs').delete_message_batch is limited to 10. #824

Closed

heitorlessa changed the title ~~fix(utilities/batch/sqs): Delete 10 messages at a time. Before this problems would occur if more than 10 messages were processed that would need deleted.~~ fix(batch): delete >10 successful messages in legacy sqs processor Dec 31, 2021

heitorlessa added help wanted Could use a second pair of eyes/hands and removed need-issue PRs that are missing related issues labels Jan 28, 2022

heitorlessa added the p1 label Jan 31, 2022

heitorlessa assigned mploski Feb 2, 2022

Merge branch 'develop' into fix/batch-sqs-handler-delete-message-batc…

b70d73d

…h-needs-batched

pull-request-size Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 8, 2022

Use threading for batch message delete

9333e14

mploski force-pushed the fix/batch-sqs-handler-delete-message-batch-needs-batched branch from 6a8fc66 to 9333e14 Compare February 8, 2022 21:28

mploski requested a review from heitorlessa February 8, 2022 21:32

Add unit test

5953114

heitorlessa approved these changes Feb 9, 2022

View reviewed changes

heitorlessa added the area/batch label Feb 9, 2022

github-actions Bot added the bug Something isn't working label Feb 9, 2022

heitorlessa changed the title ~~fix(batch): delete >10 successful messages in legacy sqs processor~~ fix(batch): delete >10 messages in legacy sqs processor Feb 9, 2022

heitorlessa merged commit 2cc1135 into aws-powertools:develop Feb 9, 2022

heitorlessa mentioned this pull request Feb 9, 2022

Papercuts priorization before we consider v2 #1009

Closed

35 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(batch): delete >10 messages in legacy sqs processor#818

fix(batch): delete >10 messages in legacy sqs processor#818
heitorlessa merged 5 commits into
aws-powertools:developfrom
whardier:fix/batch-sqs-handler-delete-message-batch-needs-batched

whardier commented Nov 12, 2021 •

edited by heitorlessa

Loading

Uh oh!

whardier commented Nov 12, 2021 •

edited

Loading

Uh oh!

codecov-commenter commented Nov 13, 2021 •

edited

Loading

Uh oh!

heitorlessa commented Nov 13, 2021

Uh oh!

whardier commented Nov 15, 2021

Uh oh!

heitorlessa commented Jan 28, 2022

Uh oh!

mploski commented Jan 28, 2022 •

edited

Loading

Uh oh!

mploski commented Feb 7, 2022

Uh oh!

mploski commented Feb 7, 2022 •

edited

Loading

Uh oh!

mploski commented Feb 8, 2022

Uh oh!

whardier commented Feb 8, 2022

Uh oh!

heitorlessa left a comment

Uh oh!

heitorlessa Feb 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

whardier commented Nov 12, 2021 • edited by heitorlessa Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes:

Breaking change checklist

Uh oh!

whardier commented Nov 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Nov 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

heitorlessa commented Nov 13, 2021

Uh oh!

whardier commented Nov 15, 2021

Uh oh!

heitorlessa commented Jan 28, 2022

Uh oh!

mploski commented Jan 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mploski commented Feb 7, 2022

Uh oh!

mploski commented Feb 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mploski commented Feb 8, 2022

Uh oh!

whardier commented Feb 8, 2022

Uh oh!

heitorlessa left a comment

Choose a reason for hiding this comment

Uh oh!

heitorlessa Feb 9, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

whardier commented Nov 12, 2021 •

edited by heitorlessa

Loading

whardier commented Nov 12, 2021 •

edited

Loading

codecov-commenter commented Nov 13, 2021 •

edited

Loading

mploski commented Jan 28, 2022 •

edited

Loading

mploski commented Feb 7, 2022 •

edited

Loading