[AMORO-4216] Refactor dangling-delete-files-cleaning via ProcessFactory plugin

[zhangwl9]

Why are the changes needed?

Close #4216.

Brief change log

• Replaced the old DanglingDeleteFilesCleaningExecutor with DanglingDeleteFilesCleaningProcess, a process-based implementation using TableMaintainerFactory.

• Moved config from AmoroManagementConf/config.yaml to process-factories.yaml/execute-engines.yaml, and removed the DANGLING_DELETE_FILES_CLEANING enum value from CleanupOperation along with related cleanup methods in DefaultTableRuntime.

• Deleted old scheduler tests and invalid configuration.

• updated deployment.md with the new configuration.

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before making a pull request

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[zhangwl9]

@zhoujinsong cc

[zhoujinsong]

LocalExecutionEngine tracks process status via CompletableFuture.whenComplete: if run() throws, the process is marked FAILED; if it returns normally, it is marked SUCCESS.

By catching Throwable and swallowing it here, exceptions never propagate to the CompletableFuture, so the process status is always reported as SUCCESS even when the actual operation failed. This makes failure invisible to the framework and to operators.

The fix is to let exceptions propagate out of run(). The same issue exists in OrphanFilesCleaningProcess and should be fixed there as well.

@Override
public void run() {
  AmoroTable<?> amoroTable = tableRuntime.loadTable();
  TableMaintainer tableMaintainer = TableMaintainerFactory.create(amoroTable, tableRuntime);
  tableMaintainer.cleanDanglingDeleteFiles();
  tableRuntime.updateState(
      DefaultTableRuntime.CLEANUP_STATE_KEY,
      cleanUp -> cleanUp.setLastDanglingDeleteFilesCleanTime(System.currentTimeMillis()));
}

If a caught-and-logged pattern is intentional (e.g., to avoid flooding failure metrics for transient errors), consider wrapping in a RuntimeException so the framework still sees a failure:

} catch (Throwable t) {
  LOG.error("Unexpected dangling delete files cleaning error for table {}",
      tableRuntime.getTableIdentifier(), t);
  throw new RuntimeException(t);
}

[zhangwl9]

fixup

[codecov-commenter]

Codecov Report

❌ Patch coverage is 52.17391% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.86%. Comparing base (75dd657) to head (38a2ae1).

Files with missing lines	Patch %	Lines
...ss/iceberg/DanglingDeleteFilesCleaningProcess.java	21.05%	15 Missing ⚠️
.../server/process/iceberg/IcebergProcessFactory.java	86.36%	0 Missing and 3 partials ⚠️
...rver/process/iceberg/SnapshotsExpiringProcess.java	0.00%	2 Missing ⚠️
...er/process/iceberg/OrphanFilesCleaningProcess.java	0.00%	1 Missing ⚠️
...src/main/java/org/apache/amoro/IcebergActions.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #4214      +/-   ##
============================================
+ Coverage     29.80%   29.86%   +0.06%     
- Complexity     4284     4286       +2     
============================================
  Files           677      679       +2     
  Lines         54978    55054      +76     
  Branches       7013     7037      +24     
============================================
+ Hits          16385    16441      +56     
- Misses        37370    37386      +16     
- Partials       1223     1227       +4     

Flag	Coverage Δ
core	29.86% <52.17%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[zhoujinsong]

LGTM.