Skip to content

store: quickly fetch needed file on S3 through Marker#897

Merged
ti-chi-bot[bot] merged 1 commit into
pingcap:mainfrom
YangKeao:impl-892
Sep 16, 2025
Merged

store: quickly fetch needed file on S3 through Marker#897
ti-chi-bot[bot] merged 1 commit into
pingcap:mainfrom
YangKeao:impl-892

Conversation

@YangKeao
Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close #892

Problem Summary:

What is changed and how it works:

This PR is based on the assumption that:

  1. The file name lexicographical order is the same with the time order.
    The file name format is similar to RFC3339 (ISO8601), and RFC3339 (ISO8601)'s order is compatible with lexicographical order if there is no timezone / time shift.
  2. The S3 API ListObjects returns the files in utf8 binary order (which is lexicographical order if all your file names are ascii).
image

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code
  1. Start a mock s3 server docker run -p 9091:9090 -p 9191:9191 -t adobe/s3mock:4.8.0.
  2. Create a bucket aws s3api create-bucket --bucket my-bucket --endpoint-url=http://localhost:9091
  3. Upload some audit log files to it: for f in *;do curl --request PUT --upload-file ./$f http://localhost:9091/my-bucket/$f; done
  4. Replay with a given time: ./replayer --command-start-time 2025-09-15T22:06:10.044+08:00 --format audit_log_plugin --input s3://my-bucket?endpoint=http://localhost:9091\&region=us-east-1\&access-key=a\&secret-access-key=b\&force-path-style=1

Notable changes

  • Has configuration change
  • Has HTTP API interfaces change
  • Has tiproxyctl change
  • Other user behavior changes

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot Bot requested review from bb7133 and djshow832 September 15, 2025 15:05
@YangKeao YangKeao changed the title store: quick fetch needed file on S3 through Marker store: quickly fetch needed file on S3 through Marker Sep 15, 2025
@ti-chi-bot ti-chi-bot Bot added the size/XXL label Sep 15, 2025
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Sep 15, 2025

Codecov Report

❌ Patch coverage is 60.78431% with 20 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@cc85182). Learn more about missing BASE report.

Files with missing lines Patch % Lines
pkg/sqlreplay/store/rotate.go 60.78% 13 Missing and 7 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #897   +/-   ##
=======================================
  Coverage        ?   67.37%           
=======================================
  Files           ?      135           
  Lines           ?    13100           
  Branches        ?        0           
=======================================
  Hits            ?     8826           
  Misses          ?     3663           
  Partials        ?      611           
Flag Coverage Δ
unit 67.37% <60.78%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Yang Keao <yangkeao@chunibyo.icu>
@YangKeao
Copy link
Copy Markdown
Member Author

/hold

There is a little timezone issue. I realized it while dreaming 😂. Let me fix it in the morning.

@YangKeao
Copy link
Copy Markdown
Member Author

YangKeao commented Sep 16, 2025

/unhold

No issue, bad dream 😠 .

@ti-chi-bot
Copy link
Copy Markdown

ti-chi-bot Bot commented Sep 16, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djshow832

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the lgtm label Sep 16, 2025
@ti-chi-bot
Copy link
Copy Markdown

ti-chi-bot Bot commented Sep 16, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-09-16 04:11:55.200695325 +0000 UTC m=+939381.760576832: ☑️ agreed by djshow832.

@ti-chi-bot ti-chi-bot Bot added the approved label Sep 16, 2025
@ti-chi-bot ti-chi-bot Bot merged commit ef2307c into pingcap:main Sep 16, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

List S3 file with prefix

3 participants