Skip to content

[AMORO-4215] Refactor data expiring via ProcessFactory plugin#4218

Merged
zhoujinsong merged 5 commits into
apache:masterfrom
zhangwl9:AMORO-Data-expire-refact-dev
May 19, 2026
Merged

[AMORO-4215] Refactor data expiring via ProcessFactory plugin#4218
zhoujinsong merged 5 commits into
apache:masterfrom
zhangwl9:AMORO-Data-expire-refact-dev

Conversation

@zhangwl9

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Close #4215.

Brief change log

Code Restructuring

  • Created DataExpiringProcess.java - encapsulates data expiration logic as a standard Process plugin
  • Removed DataExpiringExecutor.java and simplified InlineTableExecutors
  • Updated IcebergProcessFactory.java to handle expire-data process creation

Configuration Migration

  • Removed ams.data-expiration section from config.yaml
  • Moved to plugins/process-factories.yaml with expire-data.enabled and expire-data.interval
  • Added corresponding thread pool config pool.expire-data.thread-count in plugins/execute-engines.yaml

Tests & Documentation

  • Removed DATA_EXPIRING operation type handling from tests
  • Updated deployment and user guide docs

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions Bot added type:docs Improvements or additions to documentation module:ams-server Ams server module type:build module:common labels May 15, 2026
@zhangwl9 zhangwl9 changed the title [AMORO-4215] Refactor DataExpiringExecutor via ProcessFactory plugin [AMORO-4215] Refactor data expiring via ProcessFactory plugin May 15, 2026
@zhangwl9 zhangwl9 force-pushed the AMORO-Data-expire-refact-dev branch 3 times, most recently from 878254a to cdd8b49 Compare May 19, 2026 03:20
# Conflicts:
#	amoro-ams/src/main/java/org/apache/amoro/server/AmoroServiceContainer.java
#	amoro-ams/src/main/java/org/apache/amoro/server/process/iceberg/IcebergProcessFactory.java

# Conflicts:
#	amoro-ams/src/main/java/org/apache/amoro/server/AmoroServiceContainer.java
#	amoro-ams/src/main/java/org/apache/amoro/server/process/iceberg/IcebergProcessFactory.java
#	amoro-ams/src/main/java/org/apache/amoro/server/table/DefaultTableRuntime.java
#	amoro-ams/src/main/java/org/apache/amoro/server/table/cleanup/CleanupOperation.java
#	amoro-ams/src/test/java/org/apache/amoro/server/process/iceberg/TestIcebergProcessFactory.java
#	amoro-ams/src/test/java/org/apache/amoro/server/scheduler/inline/PeriodicTableSchedulerTestBase.java
#	amoro-ams/src/test/java/org/apache/amoro/server/scheduler/inline/TestPeriodicTableSchedulerCleanup.java
#	amoro-common/src/test/java/org/apache/amoro/process/TestLocalExecutionEngine.java
#	dist/src/main/amoro-bin/conf/plugins/execute-engines.yaml
#	dist/src/main/amoro-bin/conf/plugins/process-factories.yaml
#	docs/admin-guides/deployment.md
#	docs/configuration/ams-config.md
@zhangwl9 zhangwl9 force-pushed the AMORO-Data-expire-refact-dev branch from 371448c to 0fa0d21 Compare May 19, 2026 03:41
张文领 and others added 4 commits May 19, 2026 12:29

@zhoujinsong zhoujinsong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Thanks a lot for the work!

@zhoujinsong zhoujinsong merged commit 38ef381 into apache:master May 19, 2026
6 checks passed
czy006 pushed a commit that referenced this pull request May 20, 2026
* Refactor DataExpiringExecutor via ProcessFactory plugin

# Conflicts:
#	amoro-ams/src/main/java/org/apache/amoro/server/AmoroServiceContainer.java
#	amoro-ams/src/main/java/org/apache/amoro/server/process/iceberg/IcebergProcessFactory.java

# Conflicts:
#	amoro-ams/src/main/java/org/apache/amoro/server/AmoroServiceContainer.java
#	amoro-ams/src/main/java/org/apache/amoro/server/process/iceberg/IcebergProcessFactory.java
#	amoro-ams/src/main/java/org/apache/amoro/server/table/DefaultTableRuntime.java
#	amoro-ams/src/main/java/org/apache/amoro/server/table/cleanup/CleanupOperation.java
#	amoro-ams/src/test/java/org/apache/amoro/server/process/iceberg/TestIcebergProcessFactory.java
#	amoro-ams/src/test/java/org/apache/amoro/server/scheduler/inline/PeriodicTableSchedulerTestBase.java
#	amoro-ams/src/test/java/org/apache/amoro/server/scheduler/inline/TestPeriodicTableSchedulerCleanup.java
#	amoro-common/src/test/java/org/apache/amoro/process/TestLocalExecutionEngine.java
#	dist/src/main/amoro-bin/conf/plugins/execute-engines.yaml
#	dist/src/main/amoro-bin/conf/plugins/process-factories.yaml
#	docs/admin-guides/deployment.md
#	docs/configuration/ams-config.md

* Add recover logic

* Fix logging message formatting in DataExpiringProcess

* Fix data expiration property name in documentation

Corrected the property name from 'expire-data' to 'data-expiration' in the documentation.

* Update using-tables.md

---------

Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn>
(cherry picked from commit 38ef381)
@zhangwl9 zhangwl9 deleted the AMORO-Data-expire-refact-dev branch May 22, 2026 03:15
j1wonpark pushed a commit to j1wonpark/amoro that referenced this pull request Jun 4, 2026
upstream 11커밋 동기화: AMS startup crash fix(apache#4224 AMORO-4223),
snapshot/data/orphan/dangling cleaning ProcessFactory 리팩터(apache#4226/apache#4218/apache#4209/apache#4214),
JUnit5 마이그레이션(apache#4199/apache#4204) 등. 사내 고유 JP CI(ts-ci-jp.yml: JDK11, tag-base) 보존.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ams-server Ams server module module:common type:build type:docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Subtask]: Refactor DataExpiring via ProcessFactory plugin

2 participants