[core] Introduce metadata.stats-dense-store to reduce meta size for multiple columns table by JingsongLi · Pull Request #4322 · apache/paimon

JingsongLi · 2024-10-14T12:30:54Z

Purpose

At present, Paimon uses BinaryRow to store statistical information. Generally, it is not a problem, but some businesses have fields with over 3000 columns.

The BinaryRow structure has a characteristic that each field occupies a fixed 8 bytes, and for a 3000 column table, one BinaryRow has 25kb of storage. SimpleStats has 3 BinaryRow, then it will be 100 kb storage. 100000 files have GB level storage. This is unacceptable.

This PR：

Introduce metadata.stats-dense-store to a dense mode to store SimpleStats and valueStatsCols in DataFileMeta.
You can set metadata.stats-mode = none, then valueStatsCols will be empty list, and SimpleStats is empty.
You can also set fields.b.stats-mode = full to enable stats for specific columns to enable data skipping, the meta storage will only contain b column.

Tests

AppendOnlyFileStoreTableTest
PrimaryKeyFileStoreTableTest

API and Format

Documentation

…ultiple columns table fix

wwj6591812 · 2024-10-14T15:23:19Z

Good，we just encountered a similar problem today!

wwj6591812 · 2024-10-14T15:52:40Z

+                                                    + " none statistic mode is set.")
+                                    .linebreak()
+                                    .text(
+                                            "Note, When this mode is enabled, the sdk in reading engine requires at least"


1、Change the "When" to "when".
2、the Paimon sdk in reading engine requires at least version 0.9.1 or 1.0.0 or higher?

wwj6591812 · 2024-10-14T15:58:10Z

        return fieldId >= SYSTEM_FIELD_ID_START;
    }
+
+    public static boolean isSystemField(String field) {


Why not add KEY_FIELD_PREFIX to SYSTEM_FIELD_NAMES?

What to solve?

If KEY_FIELD_PREFIX is not in SYSTEM_FIELD_NAMES, then the funtion name "isSystemField" is inappropriate.

So add KEY_FIELD_PREFIX to SYSTEM_FIELD_NAMES, what problem can be solved? Can you write a ut demo?

A solution is using a SYSTEM_FIELD_PREFIXS, and always using starWith. But it is not good for performance.

Let it go now.

wwj6591812 · 2024-10-14T16:01:26Z

+ */
+public class ProjectedArray implements InternalArray {
+
+    protected final int[] indexMapping;


The indexMapping、array and ProjectedArray should be private?

wwj6591812 · 2024-10-15T00:25:01Z


-        this.keyStatsConverter = new SimpleStatsConverter(keyType);
-        this.valueStatsConverter = new SimpleStatsConverter(valueType);
+        this.keyStatsConverter = new SimpleStatsConverter(keyType, false);


this.keyStatsConverter = new SimpleStatsConverter(keyType);

wwj6591812 · 2024-10-15T01:07:08Z

+        private final InternalArray array;
+        private final long notFoundValue;
+
+        protected NullCountsEvoArray(int[] indexMapping, InternalArray array, long notFoundValue) {


Why protected?

wwj6591812 · 2024-10-15T01:14:59Z

+        private final InternalArray array;
+        private final long notFoundValue;
+
+        protected NullCountsEvoArray(int[] indexMapping, InternalArray array, long notFoundValue) {


Why protected?

wwj6591812 · 2024-10-15T01:17:06Z

+                            fileDeserializer.get(), fileDeserializer.get(), fileDeserializer.get()),
                    new IndexIncrement(
                            indexEntrySerializer.deserializeList(view),
                            indexEntrySerializer.deserializeList(view)));


Condition 'version <= 2' is always 'true'

wwj6591812 · 2024-10-15T01:25:33Z

        } else if (version == 2) {
+            DataFileMeta09Serializer serializer = new DataFileMeta09Serializer();
+            return serializer::deserialize;
+        } else if (version == 3) {


public static final int DATA_FILE_META_VERSION_1 = 1;
public static final int DATA_FILE_META_VERSION_2= 2;
public static final int DATA_FILE_META_VERSION_3= 3;

No need to do this.

wwj6591812 · 2024-10-15T01:26:15Z


    public static final ConfigOption<String> METADATA_STATS_MODE =
-            key("metadata." + STATS_MODE_SUFFIX)
+            key("metadata.stats-mode")


Why do this?

Can become more intuitive in the code.

wwj6591812 · 2024-10-15T01:29:12Z

+                        null);
+        List<DataFileMeta> dataFiles = Collections.singletonList(dataFile);
+
+        LinkedHashMap<String, Pair<Integer, Integer>> dvRanges = new LinkedHashMap<>();


Map<String, Pair<Integer, Integer>> dvRanges = new LinkedHashMap<>();

IndexFileMeta requires a LinkedHashMap

wwj6591812 · 2024-10-15T01:34:08Z

Please modify this doc : https://paimon.apache.org/docs/master/flink/sql-ddl/#specify-statistics-mode

wwj6591812 · 2024-10-15T01:39:57Z

A question :
If I have a paimon table with 1000 columns, and I just want field B to use dense stat, other columns close stats.
Is my configuration correct？

metadata.stats-dense-store=true
metadata.stats-mode=none
fields.b.stats-mode=truncate(16)

wwj6591812 · 2024-10-15T01:47:54Z

+    public static final ConfigOption<Boolean> METADATA_STATS_DENSE_STORE =
+            key("metadata.stats-dense-store")
+                    .booleanType()
+                    .defaultValue(false)


Why the default value is not true?
You are worry about that many users are using old versions of Paimon sdk in their reading engine?

Yes, it is true

JingsongLi · 2024-10-15T02:51:24Z

A question : If I have a paimon table with 1000 columns, and I just want field B to use dense stat, other columns close stats. Is my configuration correct？

metadata.stats-dense-store=true metadata.stats-mode=none fields.b.stats-mode=truncate(16)

Correct!

wwj6591812 · 2024-10-15T03:37:58Z

Looks good to me!
I can't wait to use this feature quickly to solve our online produce problems!

+1

LinMingQiang · 2025-02-10T08:37:22Z

It should be related to this change : #5035

JingsongLi added 4 commits October 14, 2024 20:08

[core] Introduce metadata.stats-dense-store to reduce meta size for m…

a2f7893

…ultiple columns table fix

fix

b72d17e

fix

3d2e2f0

fix

547fa56

wwj6591812 reviewed Oct 15, 2024

View reviewed changes

Fix comments

7939203

JingsongLi merged commit 11636ac into apache:master Oct 15, 2024

This was referenced Nov 1, 2024

[doc]update fileSource and valueStatsCols in manifest.md #4430

Closed

[doc]Update fileSource and valueStatsCols in manifest.md #4431

Merged

JingsongLi mentioned this pull request Dec 22, 2024

[Core]Add data file root location in DataFileMeta #4751

Closed

Conversation

JingsongLi commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

API and Format

Documentation

Uh oh!

wwj6591812 commented Oct 14, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wwj6591812 commented Oct 15, 2024

Uh oh!

wwj6591812 commented Oct 15, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingsongLi commented Oct 15, 2024

Uh oh!

wwj6591812 commented Oct 15, 2024

Uh oh!

LinMingQiang commented Feb 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JingsongLi commented Oct 14, 2024 •

edited

Loading