Add basic versioning (open-edge-platform#238)

* Add project versioning capabilities

* Update changelog

Co-authored-by: Roman Donchenko <roman.donchenko@intel.com>

Author: Maxim Zhiltsov

.github/workflows/health_check.yml

@@ -19,7 +19,7 @@ jobs:
       - name: Installing dependencies
         run: |
           pip install tensorflow pytest pytest-cov
-          pip install -e ./
+          pip install -e .[default]
       - name: Code instrumentation
         run: |
           pytest -v --cov --cov-report xml:coverage.xml

.github/workflows/pr_checks.yml

@@ -31,7 +31,7 @@ jobs:
       - name: Installing dependencies
         run: |
           pip install tensorflow pytest
-          pip install -e ./
+          pip install -e .[default]
       - name: Unit testing
         run: |
           pytest -v

CHANGELOG.md

@@ -8,20 +8,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## \[Unreleased\]
 ### Added
-- TBD
+- A new installation target: `pip install datumaro[default]`, which should
+  be used by default. The simple `datumaro` is supposed for library users.
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
+- Dataset and project versioning capabilities (Git-like)
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
+- "dataset revpath" concept in CLI, allowing to pass a dataset path with
+  the dataset format in `diff`, `merge`, `explain` and `info` CLI commands
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
+- `add`, `remove`, `commit`, `checkout`, `log`, `status`, `info` CLI commands
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
 
 ### Changed
+- A project can contain and manage multiple datasets instead of a single one.
+  CLI operations can be applied to the whole project, or to separate datasets.
+  Datasets are modified inplace, by default
+  (<https://github.com/openvinotoolkit/datumaro/issues/328>)
+- CLI help for builtin plugins doesn't require project
+  (<https://github.com/openvinotoolkit/datumaro/issues/328>)
 - Annotation-related classes were moved into a new module,
   `datumaro.components.annotation`
   (<https://github.com/openvinotoolkit/datumaro/pull/439>)
 - Rollback utilities replaced with Scope utilities
   (<https://github.com/openvinotoolkit/datumaro/pull/444>)
+- The `Project` class from `datumaro.components` is changed completely
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
+- `diff` and `ediff` are joined into a single `diff` CLI command
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
+- Projects use new file layout, incompatible with old projects.
+  An old project can be updated with `datum project migrate`
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
+- Inheriting `CliPlugin` is not required in plugin classes
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
+- `Importer`s do not create `Project`s anymore and just return a list of
+  extractor configurations
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
 
 ### Deprecated
 - TBD
 
 ### Removed
-- TBD
+- `import`, `project merge` CLI commands
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
+- Support for project hierarchies. A project cannot be a source anymore
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
+- Project cannot have independent internal dataset anymore. All the project
+  data must be stored in the project data sources
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
+- `datumaro_project` format
+  (<https://github.com/openvinotoolkit/datumaro/pull/238>)
 
 ### Fixed
 - Deprecation warning in `open_images_format.py`
@@ -71,7 +106,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Calling `ProjectDataset.transform()` with a string argument (<https://github.com/openvinotoolkit/datumaro/issues/402>)
 - Attributes casting for CVAT format (<https://github.com/openvinotoolkit/datumaro/pull/403>)
 - Loading of custom project plugins (<https://github.com/openvinotoolkit/datumaro/issues/404>)
-- Reading, writing anno file and saving name of the subset for test subset 
+- Reading, writing anno file and saving name of the subset for test subset
   (<https://github.com/openvinotoolkit/datumaro/pull/447>)
 
 ### Security

MANIFEST.in

@@ -1,2 +1,3 @@
 include README.md
-include requirements-core.txt
+include requirements-core.txt
+include requirements-default.txt

datumaro/cli/__main__.py

@@ -9,7 +9,8 @@
 
 from ..version import VERSION
 from . import commands, contexts
-from .util import CliException, add_subparser
+from .util import add_subparser
+from .util.errors import CliException
 
 _log_levels = {
     'debug': log.DEBUG,
@@ -59,22 +60,31 @@ def make_parser():
     _LogManager._define_loglevel_option(parser)
 
     known_contexts = [
-        ('project', contexts.project, "Actions with project (deprecated)"),
+        ('project', contexts.project, "Actions with project"),
         ('source', contexts.source, "Actions with data sources"),
         ('model', contexts.model, "Actions with models"),
     ]
     known_commands = [
-        ('create', commands.create, "Create project"),
-        ('import', commands.import_, "Create project from existing dataset"),
+        ("Project modification:", None, ''),
+        ('create', commands.create, "Create empty project"),
         ('add', commands.add, "Add data source to project"),
         ('remove', commands.remove, "Remove data source from project"),
+
+        ("", None, ''),
+        ("Project versioning:", None, ''),
+        ('checkout', commands.checkout, "Switch to another branch or revision"),
+        ('commit', commands.commit, "Commit changes in tracked files"),
+        ('log', commands.log, "List history"),
+        ('status', commands.status, "Display current status"),
+
+        ("", None, ''),
+        ("Dataset and project operations:", None, ''),
         ('export', commands.export, "Export project in some format"),
-        ('filter', commands.filter, "Filter project"),
-        ('transform', commands.transform, "Transform project"),
-        ('merge', commands.merge, "Merge projects"),
-        ('convert', commands.convert, "Convert dataset into another format"),
-        ('diff', commands.diff, "Compare projects with intersection"),
-        ('ediff', commands.ediff, "Compare projects for equality"),
+        ('filter', commands.filter, "Filter project items"),
+        ('transform', commands.transform, "Modify project items"),
+        ('merge', commands.merge, "Merge datasets"),
+        ('convert', commands.convert, "Convert dataset between formats"),
+        ('diff', commands.diff, "Compare datasets"),
         ('stats', commands.stats, "Compute project statistics"),
         ('info', commands.info, "Print project info"),
         ('explain', commands.explain, "Run Explainable AI algorithm for model"),
@@ -105,7 +115,8 @@ def make_parser():
     subcommands = parser.add_subparsers(title=subcommands_desc,
         description="", help=argparse.SUPPRESS)
     for command_name, command, _ in known_contexts + known_commands:
-        add_subparser(subcommands, command_name, command.build_parser)
+        if command is not None:
+            add_subparser(subcommands, command_name, command.build_parser)
 
     return parser
 
@@ -121,7 +132,10 @@ def main(args=None):
         return 1
 
     try:
-        return args.command(args)
+        retcode = args.command(args)
+        if retcode is None:
+            retcode = 0
+        return retcode
     except CliException as e:
         log.error(e)
         return 1

datumaro/cli/commands/__init__.py

@@ -5,6 +5,6 @@
 # pylint: disable=redefined-builtin
 
 from . import (
-    add, convert, create, diff, ediff, explain, export, filter, import_, info,
-    merge, remove, stats, transform, validate,
+    add, checkout, commit, convert, create, diff, explain, export, filter, info,
+    log, merge, remove, stats, status, transform, validate,
 )

datumaro/cli/commands/checkout.py

@@ -0,0 +1,70 @@
+# Copyright (C) 2021 Intel Corporation
+#
+# SPDX-License-Identifier: MIT
+
+import argparse
+
+from datumaro.util.scope import scope_add, scoped
+
+from ..util import MultilineFormatter
+from ..util.project import load_project
+
+
+def build_parser(parser_ctor=argparse.ArgumentParser):
+    parser = parser_ctor(help="Navigate to a revision",
+        description="""
+        Command forms:|n
+        1) %(prog)s <revision>|n
+        2) %(prog)s [--] <source1> ...|n
+        3) %(prog)s <revision> [--] <source1> <source2> ...|n
+        |n
+        1 - Restores a revision and all the sources in the working directory.|n
+        2, 3 - Restores only specified sources from the specified revision.|n
+        |s|sThe current revision is used, when not set.|n
+        |s|s"--" is optionally used to separate source names and revisions.|n
+        |n
+        Examples:|n
+        - Restore the previous revision:|n
+        |s|s%(prog)s HEAD~1 |n |n
+        - Restore the saved version of a source in the working tree|n
+        |s|s%(prog)s -- source-1 |n |n
+        - Restore a previous version of a source|n
+        |s|s%(prog)s 33fbfbe my-source
+        """, formatter_class=MultilineFormatter)
+
+    parser.add_argument('_positionals', nargs=argparse.REMAINDER,
+        help=argparse.SUPPRESS) # workaround for -- eaten by positionals
+    parser.add_argument('rev', nargs='?',
+        help="Commit or tag (default: current)")
+    parser.add_argument('sources', nargs='*',
+        help="Sources to restore (default: all)")
+    parser.add_argument('-f', '--force', action='store_true',
+        help="Allows to overwrite unsaved changes in case of conflicts "
+            "(default: %(default)s)")
+    parser.add_argument('-p', '--project', dest='project_dir', default='.',
+        help="Directory of the project to operate on (default: current dir)")
+    parser.set_defaults(command=checkout_command)
+
+    return parser
+
+@scoped
+def checkout_command(args):
+    has_sep = '--' in args._positionals
+    if has_sep:
+        pos = args._positionals.index('--')
+        if 1 < pos:
+            raise argparse.ArgumentError(None,
+                message="Expected no more than 1 revision argument")
+    else:
+        pos = 1
+    args.rev = (args._positionals[:pos] or [None])[0]
+    args.sources = args._positionals[pos + has_sep:]
+    if has_sep and not args.sources:
+        raise argparse.ArgumentError('sources', message="When '--' is used, "
+            "at least 1 source name must be specified")
+
+    project = scope_add(load_project(args.project_dir))
+
+    project.checkout(rev=args.rev, sources=args.sources, force=args.force)
+
+    return 0

datumaro/cli/commands/commit.py

@@ -0,0 +1,54 @@
+# Copyright (C) 2020-2021 Intel Corporation
+#
+# SPDX-License-Identifier: MIT
+
+import argparse
+
+from datumaro.util.scope import scope_add, scoped
+
+from ..util import MultilineFormatter
+from ..util.project import load_project
+
+
+def build_parser(parser_ctor=argparse.ArgumentParser):
+    parser = parser_ctor(help="Create a revision",
+        description="""
+        Creates a new revision from the current state of the working directory.|n
+        |n
+        Examples:|n
+        - Create a revision:|n
+        |s|s%(prog)s -m "Added data"
+        """, formatter_class=MultilineFormatter)
+
+    parser.add_argument('-m', '--message', required=True, help="Commit message")
+    parser.add_argument('--allow-empty', action='store_true',
+        help="Allow commits with no changes (default: %(default)s)")
+    parser.add_argument('--allow-foreign', action='store_true',
+        help="Allow commits with non-Datumaro changes (default: %(default)s)")
+    parser.add_argument('--no-cache', action='store_true',
+        help="Don't put committed datasets into cache, "
+            "save only metainfo (default: %(default)s)")
+    parser.add_argument('-p', '--project', dest='project_dir', default='.',
+        help="Directory of the project to operate on (default: current dir)")
+    parser.set_defaults(command=commit_command)
+
+    return parser
+
+@scoped
+def commit_command(args):
+    project = scope_add(load_project(args.project_dir))
+
+    old_tree = project.head
+
+    new_commit = project.commit(args.message, allow_empty=args.allow_empty,
+        allow_foreign=args.allow_foreign, no_cache=args.no_cache)
+
+    new_tree = project.working_tree
+    diff = project.diff(old_tree, new_tree)
+
+    print("Moved to commit '%s' %s" % (new_commit, args.message))
+    print(" %s targets changed" % len(diff))
+    for t, s in diff.items():
+        print(" %s %s" % (s.name, t))
+
+    return 0

datumaro/cli/commands/convert.py

@@ -12,34 +12,41 @@
 from datumaro.util.os_util import make_file_name
 
 from ..contexts.project import FilterModes
-from ..util import CliException, MultilineFormatter
+from ..util import MultilineFormatter
+from ..util.errors import CliException
 from ..util.project import generate_next_file_name
 
 
 def build_parser(parser_ctor=argparse.ArgumentParser):
-    builtin_importers = sorted(Environment().importers.items)
-    builtin_converters = sorted(Environment().converters.items)
+    builtin_readers = sorted(
+        set(Environment().importers) | set(Environment().extractors))
+    builtin_writers = sorted(Environment().converters)
 
     parser = parser_ctor(help="Convert an existing dataset to another format",
         description="""
-            Converts a dataset from one format to another.
-            You can add your own formats using a project.|n
-            |n
-            Supported input formats: %s|n
-            |n
-            Supported output formats: %s|n
-            |n
-            Examples:|n
-            - Export a dataset as a PASCAL VOC dataset, include images:|n
-            |s|sconvert -i src/path -f voc -- --save-images|n
-            |n
-            - Export a dataset as a COCO dataset to a specific directory:|n
-            |s|sconvert -i src/path -f coco -o path/I/like/
-        """ % (', '.join(builtin_importers), ', '.join(builtin_converters)),
+        Converts a dataset from one format to another.
+        You can add your own formats and do many more by creating a
+        Datumaro project.|n
+        |n
+        This command serves as an alias for the "create", "add", and
+        "export" commands, allowing to obtain the same results simpler
+        and faster. Check descriptions of these commands for more info.|n
+        |n
+        Supported input formats: {}|n
+        |n
+        Supported output formats: {}|n
+        |n
+        Examples:|n
+        - Export a dataset as a PASCAL VOC dataset, include images:|n
+        |s|s%(prog)s -i src/path -f voc -- --save-images|n
+        |n
+        - Export a dataset as a COCO dataset to a specific directory:|n
+        |s|s%(prog)s -i src/path -f coco -o path/I/like/
+        """.format(', '.join(builtin_readers), ', '.join(builtin_writers)),
         formatter_class=MultilineFormatter)
 
     parser.add_argument('-i', '--input-path', default='.', dest='source',
-        help="Path to look for a dataset")
+        help="Input dataset path (default: current dir)")
     parser.add_argument('-if', '--input-format',
         help="Input dataset format. Will try to detect, if not specified.")
     parser.add_argument('-f', '--output-format', required=True,
@@ -49,13 +56,15 @@ def build_parser(parser_ctor=argparse.ArgumentParser):
     parser.add_argument('--overwrite', action='store_true',
         help="Overwrite existing files in the save directory")
     parser.add_argument('-e', '--filter',
-        help="Filter expression for dataset items")
+        help="XML XPath filter expression for dataset items. Read \"filter\" "
+            "command docs for more info")
     parser.add_argument('--filter-mode', default=FilterModes.i.name,
         type=FilterModes.parse,
-        help="Filter mode (options: %s; default: %s)" % \
+        help="Filter mode, one of %s (default: %s)" % \
             (', '.join(FilterModes.list_options()) , '%(default)s'))
     parser.add_argument('extra_args', nargs=argparse.REMAINDER,
-        help="Additional arguments for output format (pass '-- -h' for help)")
+        help="Additional arguments for output format (pass '-- -h' for help). "
+            "Must be specified after the main command arguments")
     parser.set_defaults(command=convert_command)
 
     return parser