[Issue pixelsdb#437] redesign the schema of metadata. (pixelsdb#474)

Support the new features in pixels metadata:
1. user management and role-based authentication;
2. range-based table partitioning;
3. schema versioning;
4. path-granular catalog management.

Currently, only the RPCs for catalog management are fully implemented.
In addition, the metadata of tables and layouts are optimized.
I also fixed a bug in layout permission conversion and a bug in layout dao's update method.

Author: bianhq

docs/TPC-H.md

@@ -12,20 +12,16 @@ The file(s) of each table are stored in a separate directory named by the table
 
 ## Create TPC-H Database
 Log in trino-cli and use the SQL statements in `scripts/sql/tpch_schema.sql` to create the TPC-H database in Pixels.
-Change the value of the `storage` table property in the create-table statement to `hdfs` if HDFS is used as the
-underlying storage system instead of S3.
-> Note that trino-cli can execute only one SQL statement at each time.
+In each `CREATE TABLE` statement, the table property `storage` defines the type of storage system used to store the
+files in this table, whereas the table property `paths` defines the URIs of the paths in which the table files are stored.
+Multiple URIs can be listed one after another, seperated by semicolon, in `paths`.
+> Note that the URIs in `paths` can have no storage scheme or their storage scheme must be consistent with `storage`. 
 
 Then, use `SHOW SCHEMAS` and `SHOW TABLES` statements to check if the tpch database has been
 created successfully.
 
-Connect to MySQL using the user `pixels`, and execute the SQL statements in `scripts/sql/tpch_layouts.sql`
-to create the table layouts for the tables in the TPC-H database in Pixels. Actually, these layouts
-should be created by the storage layout optimizer ([Rainbow](https://ieeexplore.ieee.org/document/8509421)).
-However, we directly load the layouts here for simplicity.
-
 Create the container to store the tables in S3. The container name is the same as the hostname
-(e.g., `pixels-tpch`) in the `LAYOUT_ORDER_PATH` and `LAYOUT_COMPACT_PATH` of each table layout.
+(e.g., `pixels-tpch`) in the `paths` of each table.
 Change the bucket name if it already exists.
 
 During data loading, Pixels will automatically create the folders in the bucket to store the files in each table.
@@ -94,6 +90,10 @@ The last parameter `-c` of `COMPACT` command is the maximum number
 of threads used for data compaction. For large tables such as `lineitem`, you can increase `-c` to
 improve the compaction performance. Compaction is normally faster than loading with same number of threads.
 
+> `compact.factor` in `$PIXELS_HOME/pixels.properties` determines how many row groups are compacted into a single
+> file. The default value is 32, which is appropriate in most conditions. An experimental evaluation of the effects
+> of compact factor on AWS S3 can be found in our [ICDE'22](https://ieeexplore.ieee.org/document/9835615) paper.
+
 To avoid scanning the small files in the ordered path during query execution,
 create an empty bucket in S3 and change the ordered path in the metadata database
 to the empty bucket.
@@ -112,7 +112,7 @@ STAT -s tpch -t partsupp -o false -c true
 STAT -s tpch -t orders -o false -c true
 STAT -s tpch -t lineitem -o false -c true
 ```
-After it is finished, statistics of each tpch column can be found in the `pixels_metadata.COLS` metadata table.
+When it is finished, statistics of each tpch column can be found in the `pixels_metadata.COLS` metadata table.
 Finally, manually update the row count for each tpch table in `pixels_metadata.TBLS.TBL_ROW_COUNT`.
 
 Set `splits.index.type=cost_based` and restart Trino to benefit from cost-based query optimization.

…o/pixelsdb/pixels/cache/ColumnletId.java …pixelsdb/pixels/cache/ColumnChunkId.javapixels-cache/src/main/java/io/pixelsdb/pixels/cache/ColumnletId.java renamed to pixels-cache/src/main/java/io/pixelsdb/pixels/cache/ColumnChunkId.java pixels-cache/src/main/java/io/pixelsdb/pixels/cache/ColumnletId.java renamed to pixels-cache/src/main/java/io/pixelsdb/pixels/cache/ColumnChunkId.java

@@ -30,20 +30,20 @@
  * @author guodong
  * @author hank
  */
-public class ColumnletId
+public class ColumnChunkId
 {
     public short rowGroupId;
     public short columnId;
     public boolean direct;
 
-    public ColumnletId(short rowGroupId, short columnId, boolean direct)
+    public ColumnChunkId(short rowGroupId, short columnId, boolean direct)
     {
         this.rowGroupId = rowGroupId;
         this.columnId = columnId;
         this.direct = direct;
     }
 
-    public ColumnletId() {}
+    public ColumnChunkId() {}
 
     @Override
     public boolean equals(Object o)
@@ -56,7 +56,7 @@ public boolean equals(Object o)
         {
             return false;
         }
-        ColumnletId other = (ColumnletId) o;
+        ColumnChunkId other = (ColumnChunkId) o;
         return Objects.equals(rowGroupId, other.rowGroupId) &&
                 Objects.equals(columnId, other.columnId);
     }

pixels-cache/src/main/java/io/pixelsdb/pixels/cache/PixelsCacheReader.java

@@ -188,7 +188,7 @@ public ByteBuffer get(long blockId, short rowGroupId, short columnId, boolean di
         return null;
     }
 
-    public void batchGet(List<ColumnletId> columnletIds, byte[][] container)
+    public void batchGet(List<ColumnChunkId> columnChunkIds, byte[][] container)
     {
         // TODO batch get cache items. merge cache accesses to reduce the number of jni invocation.
     }

pixels-cache/src/main/java/io/pixelsdb/pixels/cache/PixelsCacheWriter.java

@@ -165,21 +165,21 @@ public PixelsCacheWriter build()
             MemoryMappedFile indexFile = new MemoryMappedFile(builderIndexLocation, builderIndexSize);
             PixelsRadix radix;
             // check if cache and index exists.
-            Set<String> cachedColumnlets = new HashSet<>();
+            Set<String> cachedColumnChunks = new HashSet<>();
             // if overwrite is not true, and cache and index file already exists, reconstruct radix from existing index.
             if (!builderOverwrite && PixelsCacheUtil.checkMagic(indexFile) && PixelsCacheUtil.checkMagic(cacheFile))
             {
                 // cache exists in local cache file and index, reload the index.
                 radix = PixelsCacheUtil.loadRadixIndex(indexFile);
-                // build cachedColumnlets for PixelsCacheWriter.
+                // build cachedColumnChunks for PixelsCacheWriter.
                 int cachedVersion = PixelsCacheUtil.getIndexVersion(indexFile);
                 MetadataService metadataService = new MetadataService(
                         cacheConfig.getMetaHost(), cacheConfig.getMetaPort());
                 Layout cachedLayout = metadataService.getLayout(
                         cacheConfig.getSchema(), cacheConfig.getTable(), cachedVersion);
-                Compact compact = cachedLayout.getCompactObject();
+                Compact compact = cachedLayout.getCompact();
                 int cacheBorder = compact.getCacheBorder();
-                cachedColumnlets.addAll(compact.getColumnletOrder().subList(0, cacheBorder));
+                cachedColumnChunks.addAll(compact.getColumnChunkOrder().subList(0, cacheBorder));
                 metadataService.shutdown();
             }
             //   else, create a new radix tree, and initialize the index and cache file.
@@ -193,7 +193,7 @@ public PixelsCacheWriter build()
             Storage storage = StorageFactory.Instance().getStorage(cacheConfig.getStorageScheme());
 
             return new PixelsCacheWriter(cacheFile, indexFile, storage, radix,
-                    cachedColumnlets, etcdUtil, builderHostName);
+                    cachedColumnChunks, etcdUtil, builderHostName);
         }
     }
 
@@ -321,9 +321,9 @@ private int internalUpdateAll(int version, Layout layout, String[] files)
     {
         int status = 0;
         // get the new caching layout
-        Compact compact = layout.getCompactObject();
+        Compact compact = layout.getCompact();
         int cacheBorder = compact.getCacheBorder();
-        List<String> cacheColumnletOrders = compact.getColumnletOrder().subList(0, cacheBorder);
+        List<String> cacheColumnletOrders = compact.getColumnChunkOrder().subList(0, cacheBorder);
         // set rwFlag as write
         logger.debug("Set index rwFlag as write");
         try
@@ -429,9 +429,9 @@ private int internalUpdateIncremental(int version, Layout layout, String[] files
         /**
          * Get the new caching layout.
          */
-        Compact compact = layout.getCompactObject();
+        Compact compact = layout.getCompact();
         int cacheBorder = compact.getCacheBorder();
-        List<String> nextVersionCached = compact.getColumnletOrder().subList(0, cacheBorder);
+        List<String> nextVersionCached = compact.getColumnChunkOrder().subList(0, cacheBorder);
         /**
          * Prepare structures for the survived and new coming cache elements.
          */

pixels-cache/src/main/java/io/pixelsdb/pixels/cache/PixelsPartitionCacheWriter.java

@@ -373,10 +373,10 @@ public int updateAll(int version, Layout layout)
             }
             String fileStr = keyValue.getValue().toString(StandardCharsets.UTF_8);
             String[] files = fileStr.split(";");
-            Compact compact = layout.getCompactObject();
+            Compact compact = layout.getCompact();
             int cacheBorder = compact.getCacheBorder();
-            List<String> cacheColumnletOrders = compact.getColumnletOrder().subList(0, cacheBorder);
-            return internalUpdateAll(version, cacheColumnletOrders, files);
+            List<String> cacheColumnChunkOrders = compact.getColumnChunkOrder().subList(0, cacheBorder);
+            return internalUpdateAll(version, cacheColumnChunkOrders, files);
         }
         catch (IOException | InterruptedException e)
         {
@@ -399,10 +399,10 @@ public int updateIncremental(int version, Layout layout)
             }
             String fileStr = keyValue.getValue().toString(StandardCharsets.UTF_8);
             String[] files = fileStr.split(";");
-            Compact compact = layout.getCompactObject();
+            Compact compact = layout.getCompact();
             int cacheBorder = compact.getCacheBorder();
-            List<String> cacheColumnletOrders = compact.getColumnletOrder().subList(0, cacheBorder);
-            return internalUpdateIncremental(version, cacheColumnletOrders, files);
+            List<String> cacheColumnChunkOrders = compact.getColumnChunkOrder().subList(0, cacheBorder);
+            return internalUpdateIncremental(version, cacheColumnChunkOrders, files);
         }
         catch (IOException | InterruptedException e)
         {

pixels-cli/src/main/java/io/pixelsdb/pixels/cli/Main.java

@@ -124,7 +124,7 @@ public static void main(String args[])
                         .help("specify the number of consumer threads used for data generation");
                 argumentParser.addArgument("-e", "--enable_encoding").setDefault(true)
                         .help("specify the option of enabling encoding or not");
-                argumentParser.addArgument("-l", "--loading_data_path")
+                argumentParser.addArgument("-l", "--loading_data_paths")
                         .help("specify the path of loading data");
 
                 Namespace ns = null;
@@ -330,12 +330,11 @@ public static long executeSQL(String jdbcUrl, Properties jdbcProperties, String
 
     /**
      * Check if the order or compact path from pixels metadata is valid.
-     * @param path the order or compact path from pixels metadata.
+     * @param paths the order or compact pathw from pixels metadata.
      */
-    public static void validateOrderOrCompactPath(String path)
+    public static void validateOrderOrCompactPath(String[] paths)
     {
-        requireNonNull(path, "path is null");
-        String[] paths = path.split(";");
+        requireNonNull(paths, "paths is null");
         checkArgument(paths.length > 0, "path must contain at least one valid directory");
         try
         {

pixels-cli/src/main/java/io/pixelsdb/pixels/cli/executor/CompactExecutor.java

@@ -19,6 +19,7 @@
  */
 package io.pixelsdb.pixels.cli.executor;
 
+import com.google.common.base.Joiner;
 import io.pixelsdb.pixels.common.metadata.MetadataService;
 import io.pixelsdb.pixels.common.metadata.domain.Compact;
 import io.pixelsdb.pixels.common.metadata.domain.Layout;
@@ -76,8 +77,8 @@ public void execute(Namespace ns, String command) throws Exception
 
         requireNonNull(layout, String.format("writable layout is not found for table '%s.%s'.",
                 schemaName, tableName));
-        Compact compact = layout.getCompactObject();
-        int numRowGroupInBlock = compact.getNumRowGroupInBlock();
+        Compact compact = layout.getCompact();
+        int numRowGroupInBlock = compact.getNumRowGroupInFile();
         int numColumn = compact.getNumColumn();
         CompactLayout compactLayout;
         if (naive.equalsIgnoreCase("yes") || naive.equalsIgnoreCase("y"))
@@ -91,15 +92,15 @@ public void execute(Namespace ns, String command) throws Exception
 
         // get input file paths
         ConfigFactory configFactory = ConfigFactory.Instance();
-        validateOrderOrCompactPath(layout.getOrderPath());
-        validateOrderOrCompactPath(layout.getCompactPath());
+        validateOrderOrCompactPath(layout.getOrderedPathUris());
+        validateOrderOrCompactPath(layout.getCompactPathUris());
         // PIXELS-399: it is not a problem if the order or compact path contains multiple directories
-        Storage orderStorage = StorageFactory.Instance().getStorage(layout.getOrderPath());
-        Storage compactStorage = StorageFactory.Instance().getStorage(layout.getCompactPath());
+        Storage orderStorage = StorageFactory.Instance().getStorage(layout.getOrderedPathUris()[0]);
+        Storage compactStorage = StorageFactory.Instance().getStorage(layout.getCompactPathUris()[0]);
         long blockSize = Long.parseLong(configFactory.getProperty("block.size"));
         short replication = Short.parseShort(configFactory.getProperty("block.replication"));
-        List<Status> statuses = orderStorage.listStatus(layout.getOrderPath());
-        String[] targetPaths = layout.getCompactPath().split(";");
+        List<Status> statuses = orderStorage.listStatus(layout.getOrderedPathUris());
+        String[] targetPaths = layout.getCompactPathUris();
         int targetPathId = 0;
 
         // compact
@@ -186,8 +187,8 @@ public void execute(Namespace ns, String command) throws Exception
         while (!compactExecutor.awaitTermination(100, TimeUnit.SECONDS));
 
         long endTime = System.currentTimeMillis();
-        System.out.println("Pixels files in '" + layout.getOrderPath() + "' are compacted into '" +
-                layout.getCompactPath() + "' by " + threadNum + " threads in " +
+        System.out.println("Pixels files in '" + Joiner.on(";").join(layout.getOrderedPathUris()) + "' are compacted into '" +
+                Joiner.on(";").join(layout.getCompactPathUris()) + "' by " + threadNum + " threads in " +
                 (endTime - startTime) / 1000 + "s.");
     }
 }

pixels-cli/src/main/java/io/pixelsdb/pixels/cli/executor/LoadExecutor.java

@@ -33,6 +33,7 @@
 import java.util.concurrent.LinkedBlockingQueue;
 
 import static io.pixelsdb.pixels.cli.Main.validateOrderOrCompactPath;
+import static java.util.Objects.requireNonNull;
 
 /**
  * @author hank
@@ -48,13 +49,14 @@ public void execute(Namespace ns, String command) throws Exception
         String origin = ns.getString("original_data_path");
         int rowNum = Integer.parseInt(ns.getString("row_num"));
         String regex = ns.getString("row_regex");
-        String loadingDataPath = ns.getString("loading_data_path");
+        String[] loadingDataPaths = requireNonNull(
+                ns.getString("loading_data_paths"), "paths is null").split(";");
         int threadNum = Integer.parseInt(ns.getString("consumer_thread_num"));
         boolean enableEncoding = Boolean.parseBoolean(ns.getString("enable_encoding"));
         System.out.println("enable encoding: " + enableEncoding);
-        if (loadingDataPath != null && !loadingDataPath.isEmpty())
+        if (loadingDataPaths.length > 0)
         {
-            validateOrderOrCompactPath(loadingDataPath);
+            validateOrderOrCompactPath(loadingDataPaths);
         }
 
         if (!origin.endsWith("/"))
@@ -64,7 +66,7 @@ public void execute(Namespace ns, String command) throws Exception
 
         Storage storage = StorageFactory.Instance().getStorage(origin);
 
-        Parameters parameters = new Parameters(schemaName, tableName, rowNum, regex, enableEncoding, loadingDataPath);
+        Parameters parameters = new Parameters(schemaName, tableName, rowNum, regex, enableEncoding, loadingDataPaths);
 
         // source already exist, producer option is false, add list of source to the queue
         List<String> fileList = storage.listPaths(origin);

pixels-cli/src/main/java/io/pixelsdb/pixels/cli/executor/StatExecutor.java

@@ -64,17 +64,17 @@ public void execute(Namespace ns, String command) throws Exception
             {
                 if (orderedEnabled)
                 {
-                    String orderedPath = layout.getOrderPath();
-                    validateOrderOrCompactPath(orderedPath);
-                    Storage storage = StorageFactory.Instance().getStorage(orderedPath);
-                    files.addAll(storage.listPaths(orderedPath));
+                    String[] orderedPaths = layout.getOrderedPathUris();
+                    validateOrderOrCompactPath(orderedPaths);
+                    Storage storage = StorageFactory.Instance().getStorage(orderedPaths[0]);
+                    files.addAll(storage.listPaths(orderedPaths));
                 }
                 if (compactEnabled)
                 {
-                    String compactPath = layout.getCompactPath();
-                    validateOrderOrCompactPath(compactPath);
-                    Storage storage = StorageFactory.Instance().getStorage(compactPath);
-                    files.addAll(storage.listPaths(compactPath));
+                    String[] compactPaths = layout.getCompactPathUris();
+                    validateOrderOrCompactPath(compactPaths);
+                    Storage storage = StorageFactory.Instance().getStorage(compactPaths[0]);
+                    files.addAll(storage.listPaths(compactPaths));
                 }
             }
         }