feat: add import taxonomy endpoint by rpenido · Pull Request #33663 · openedx/openedx-platform

rpenido · 2023-11-06T22:19:22Z

Description

Add a rest api endpoint to import taxonomies

Supporting Information

Part of [Tagging] Import a Taxonomy ("Bare Bones") modular-learning#115
Depends on:
- feat: add import taxonomy endpoint openedx-core#112

Testing instructions

Please ensure that the tests cover the expected behavior of the view

Private-ref: FAL-3532

openedx-webhooks · 2023-11-06T22:19:27Z

Thanks for the pull request, @rpenido! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

supporting documentation
Open edX discussion forum threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

open-craft-grove · 2023-11-07T18:49:48Z

Sandbox deployment started.

open-craft-grove · 2023-11-07T18:49:54Z

Sandbox deployment started.

open-craft-grove · 2023-11-07T19:17:46Z

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

open-craft-grove · 2023-11-08T15:49:33Z

Sandbox deployment started.

open-craft-grove · 2023-11-08T16:24:32Z

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

…date tests

open-craft-grove · 2023-11-08T17:19:32Z

Sandbox deployment started.

open-craft-grove · 2023-11-08T17:47:45Z

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

open-craft-grove · 2023-11-08T19:30:34Z

Sandbox update request received. Deployment will start soon.

open-craft-grove · 2023-11-08T19:34:59Z

Sandbox deployment started.

open-craft-grove · 2023-11-08T20:04:17Z

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

pomegranited

My apologies @rpenido , I neglected to send this review! My comments are mainly about aligning this to your work on openedx/openedx-core#112 (review).

pomegranited · 2023-11-07T23:27:35Z

+    @action(detail=False, url_path="import", methods=["post"])
+    def create_import(self, request: Request, **_kwargs) -> HttpResponse:
+        """
+        Creates a new taxonomy and imports the tags from the uploaded file.
+        """
+        perm = "oel_tagging.import_taxonomy"
+        if not request.user.has_perm(perm):
+            raise PermissionDenied("You do not have permission to import taxonomies")
+
+        body = TaxonomyImportNewBodySerializer(data=request.data)
+        body.is_valid(raise_exception=True)
+
+        taxonomy_name = body.validated_data["taxonomy_name"]
+        taxonomy_description = body.validated_data["taxonomy_description"]
+        file = body.validated_data["file"].file
+        parser_format = body.validated_data["parser_format"]
+
+        # ToDo: This code is temporary
+        # In the future, the orgs parameter will be defined in the request body from the frontend
+        # See: https://github.com/openedx/modular-learning/issues/116
+        if oel_tagging_rules.is_taxonomy_admin(request.user):
+            orgs = None
+        else:
+            orgs = get_admin_orgs(request.user)
+
+        taxonomy = create_taxonomy(taxonomy_name, taxonomy_description, orgs=orgs)
+        try:
+            import_success = import_tags(taxonomy, file, parser_format)
+
+            if import_success:
+                return HttpResponse(status=200)
+            else:
+                import_error = get_last_import_log(taxonomy)
+                taxonomy.delete()
+                return HttpResponseBadRequest(import_error)
+        except ValueError as e:
+            return HttpResponseBadRequest(e)


Now that the parent TaxonomyView returns the created taxonomy, we can do this with less duplicated code:

Suggested change

@action(detail=False, url_path="import", methods=["post"])

def create_import(self, request: Request, **_kwargs) -> HttpResponse:

"""

Creates a new taxonomy and imports the tags from the uploaded file.

"""

perm = "oel_tagging.import_taxonomy"

if not request.user.has_perm(perm):

raise PermissionDenied("You do not have permission to import taxonomies")

body = TaxonomyImportNewBodySerializer(data=request.data)

body.is_valid(raise_exception=True)

taxonomy_name = body.validated_data["taxonomy_name"]

taxonomy_description = body.validated_data["taxonomy_description"]

file = body.validated_data["file"].file

parser_format = body.validated_data["parser_format"]

# ToDo: This code is temporary

# In the future, the orgs parameter will be defined in the request body from the frontend

# See: https://github.com/openedx/modular-learning/issues/116

if oel_tagging_rules.is_taxonomy_admin(request.user):

orgs = None

else:

orgs = get_admin_orgs(request.user)

taxonomy = create_taxonomy(taxonomy_name, taxonomy_description, orgs=orgs)

try:

import_success = import_tags(taxonomy, file, parser_format)

if import_success:

return HttpResponse(status=200)

else:

import_error = get_last_import_log(taxonomy)

taxonomy.delete()

return HttpResponseBadRequest(import_error)

except ValueError as e:

return HttpResponseBadRequest(e)

@action(detail=False, url_path="import", methods=["post"])

def create_import(self, request: Request, **kwargs) -> HttpResponse:

"""

Creates a new taxonomy with the given orgs and imports the tags from the uploaded file.

"""

response = super().create_import(request, **kwargs)

# If creation was successful, set the orgs for the new taxonomy

if status.is_success(response.status_code):

# ToDo: This code is temporary

# In the future, the orgs parameter will be defined in the request body from the frontend

# See: https://github.com/openedx/modular-learning/issues/116

if oel_tagging_rules.is_taxonomy_admin(request.user):

orgs = None

else:

orgs = get_admin_orgs(request.user)

taxonomy = get_taxonomy(response.data['id']) # or wherever it is

set_taxonomy_orgs(taxonomy, all_orgs=False, orgs=orgs)

# might need to re-serialize the response to include the orgs?

return response

I'm not sure about this one @pomegranited.

We are preventing some code duplication in the View but duplicating the logic of the edx-platform/create_taxonomy.

It will be a pain if our create_taxonomy gets complicated. Or we will get in trouble if we change some behavior from the oel_tagging/create_taxonomy.

Do you think it is ok to leave the duplicated create_import?

@rpenido I don't think we should be writing/maintaining more code than we need..

@bradenmacdonald , could we get a 2nd opinion here?

I like @pomegranited's suggestion; it re-uses more code. If we're going to have a separate "create_import" function at all, we should aim to keep it as small as possible, just calling the other create and import functions.

The content_tagging version of create_taxonomy is only two lines of code so it's fine to duplicate it here. Just make sure to add comments explaining everything and any concerns about that.

Thank you for the discussion! 😄

Done here: 1b65728

pomegranited · 2023-11-07T23:36:15Z

+
+    @action(detail=True, url_path="tags/import", methods=["put"])
+    def update_import(self, request: Request, **_kwargs) -> HttpResponse:
+        """
+        Creates a new taxonomy and imports the tags from the uploaded file.
+        """
+        perm = "oel_tagging.import_taxonomy"
+        if not request.user.has_perm(perm):
+            raise PermissionDenied("You do not have permission to import taxonomies")
+
+        body = TaxonomyImportBodySerializer(data=request.data)
+        body.is_valid(raise_exception=True)
+
+        file = body.validated_data["file"].file
+        parser_format = body.validated_data["parser_format"]
+
+        # ToDo: This code is temporary
+        # In the future, the orgs parameter will be defined in the request body from the frontend
+        # See: https://github.com/openedx/modular-learning/issues/116
+        if oel_tagging_rules.is_taxonomy_admin(request.user):
+            orgs = None
+        else:
+            orgs = get_admin_orgs(request.user)
+
+        taxonomy = self.get_object()
+        try:
+            import_success = import_tags(taxonomy, file, parser_format)
+
+            if import_success:
+                return HttpResponse(status=200)
+            else:
+                import_error = get_last_import_log(taxonomy)
+                return HttpResponseBadRequest(import_error)
+        except ValueError as e:
+            return HttpResponseBadRequest(e)
+


Can this whole method can go away? I think the parent TaxonomyView now handles all that's needed here.

Suggested change

@action(detail=True, url_path="tags/import", methods=["put"])

def update_import(self, request: Request, **_kwargs) -> HttpResponse:

"""

Creates a new taxonomy and imports the tags from the uploaded file.

"""

perm = "oel_tagging.import_taxonomy"

if not request.user.has_perm(perm):

raise PermissionDenied("You do not have permission to import taxonomies")

body = TaxonomyImportBodySerializer(data=request.data)

body.is_valid(raise_exception=True)

file = body.validated_data["file"].file

parser_format = body.validated_data["parser_format"]

# ToDo: This code is temporary

# In the future, the orgs parameter will be defined in the request body from the frontend

# See: https://github.com/openedx/modular-learning/issues/116

if oel_tagging_rules.is_taxonomy_admin(request.user):

orgs = None

else:

orgs = get_admin_orgs(request.user)

taxonomy = self.get_object()

try:

import_success = import_tags(taxonomy, file, parser_format)

if import_success:

return HttpResponse(status=200)

else:

import_error = get_last_import_log(taxonomy)

return HttpResponseBadRequest(import_error)

except ValueError as e:

return HttpResponseBadRequest(e)

Done ea67de1!

open-craft-grove · 2023-11-10T12:49:29Z

Sandbox deployment started.

open-craft-grove · 2023-11-10T13:19:33Z

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

open-craft-grove · 2023-11-10T13:48:29Z

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

open-craft-grove · 2023-11-10T13:48:44Z

Sandbox deployment started.

open-craft-grove · 2023-11-10T14:18:41Z

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

rpenido · 2023-11-10T16:39:51Z

Hi @bradenmacdonald!

This is ready for a final review and merge.

I added 3c1a9d0 to fix the constraint update in our lint upstream (openedx/edx-lint#376) that was preventing the tests from passing.

If you checkout master and run make compile-requirements you can see the following diff:

diff --git a/requirements/common_constraints.txt b/requirements/common_constraints.txt
index d1be149855..53dff4a22d 100644
--- a/requirements/common_constraints.txt
+++ b/requirements/common_constraints.txt
@@ -24,7 +24,3 @@ Django<4.0
 elasticsearch<7.14.0
 
 # django-simple-history>3.0.0 adds indexing and causes a lot of migrations to be affected
-
-# tox>4.0.0 isn't yet compatible with many tox plugins, causing CI failures in almost all repos.
-# Details can be found in this discussion: https://github.com/tox-dev/tox/discussions/1810
-tox<4.0.0
diff --git a/requirements/edx/development.txt b/requirements/edx/development.txt
index 59637f095e..6b3f653618 100644
--- a/requirements/edx/development.txt
+++ b/requirements/edx/development.txt
@@ -2037,7 +2037,6 @@ tomlkit==0.12.1
     #   snowflake-connector-python
 tox==3.28.0
     # via
-    #   -c requirements/edx/../common_constraints.txt
     #   -r requirements/edx/testing.txt
     #   tox-battery
 tox-battery==0.6.2
diff --git a/requirements/edx/testing.txt b/requirements/edx/testing.txt
index 1fbf98b980..8f62ae7b7f 100644
--- a/requirements/edx/testing.txt
+++ b/requirements/edx/testing.txt
@@ -1502,7 +1502,6 @@ tomlkit==0.12.1
     #   snowflake-connector-python
 tox==3.28.0
     # via
-    #   -c requirements/edx/../common_constraints.txt
     #   -r requirements/edx/testing.in
     #   tox-battery
 tox-battery==0.6.2

This seems to be handled here:

Unpin tox, tox-battery and pytest-metadata #33532

open-craft-grove · 2023-11-10T16:45:36Z

Sandbox update request received. Deployment will start soon.

open-craft-grove · 2023-11-10T16:49:27Z

Sandbox deployment started.

open-craft-grove · 2023-11-10T17:19:31Z

Sandbox deployment successful.

Sandbox LMS is available at pr-33663-139931.staging.do.opencraft.hosting
Sandbox Studio is available at studio.pr-33663-139931.staging.do.opencraft.hosting

bradenmacdonald

LGTM! I can merge on Monday.

bradenmacdonald · 2023-11-13T19:49:31Z

@rpenido I was going to merge this now but it will need a rebase first (or merge master into it).

openedx-webhooks · 2023-11-14T19:12:22Z

@rpenido 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

edx-pipeline-bot · 2023-11-14T20:15:52Z

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

edx-pipeline-bot · 2023-11-14T20:37:10Z

2U Release Notice: This PR has been deployed to the edX production environment.

open-craft-grove · 2023-11-15T10:45:25Z

Sandbox update request received. Deployment will start soon.

open-craft-grove · 2023-11-15T11:00:25Z

Sandbox update request received. Deployment will start soon.

open-craft-grove · 2023-11-15T11:15:31Z

Sandbox update request received. Deployment will start soon.

open-craft-grove · 2023-11-15T11:30:29Z

Sandbox update request received. Deployment will start soon.

open-craft-grove · 2023-11-15T11:45:25Z

Sandbox update request received. Deployment will start soon.

open-craft-grove · 2023-11-15T12:00:37Z

Sandbox update request received. Deployment will start soon.

open-craft-grove · 2023-11-15T12:15:41Z

Sandbox update request received. Deployment will start soon.

open-craft-grove · 2023-11-15T13:00:54Z

Sandbox update request received. Deployment will start soon.

feat: add import taxonomy endpoint (edx-platform)

dad3503

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Nov 6, 2023

rpenido changed the title ~~feat: add import taxonomy endpoint (edx-platform)~~ feat: add import taxonomy endpoint Nov 6, 2023

rpenido added 2 commits November 7, 2023 15:36

refactor: add import taxonomy endpoint

13c7ecc

refactor: cleanup

4eb0ad2

Merge branch 'master' into rpenido/fal-3532-import-taxonomy-bare-bones

553ada0

refactor: remove overlaped rules, return Taxonomy after import and up…

a123c21

…date tests

style: fix pylint

3a2e3f8

pomegranited requested changes Nov 8, 2023

View reviewed changes

rpenido added 4 commits November 9, 2023 20:59

chore: bump openedx-learning constrain

805e76d

refactor: cleaning update_import override

ea67de1

fix: remove unused import

ad20000

refactor: remove duplicate code in create_import

1b65728

rpenido force-pushed the rpenido/fal-3532-import-taxonomy-bare-bones branch from 484af49 to 1b65728 Compare November 10, 2023 12:42

rpenido added 2 commits November 10, 2023 09:53

chore: update openedx-learning version

a8d0c9e

Merge branch 'master' into rpenido/fal-3532-import-taxonomy-bare-bones

9913e78

chore: fix dependencies constraint

3c1a9d0

rpenido marked this pull request as ready for review November 10, 2023 16:40

rpenido mentioned this pull request Nov 10, 2023

feat: add import taxonomy feature [FC-0036] openedx/frontend-app-authoring#675

Merged

bradenmacdonald approved these changes Nov 10, 2023

View reviewed changes

Merge branch 'master' into rpenido/fal-3532-import-taxonomy-bare-bones

7c30a56

bradenmacdonald merged commit 034d632 into openedx:master Nov 14, 2023

bradenmacdonald deleted the rpenido/fal-3532-import-taxonomy-bare-bones branch November 14, 2023 19:12

Uh oh!

Conversation

rpenido commented Nov 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Supporting Information

Testing instructions

Uh oh!

openedx-webhooks commented Nov 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

open-craft-grove commented Nov 7, 2023

Uh oh!

open-craft-grove commented Nov 7, 2023

Uh oh!

open-craft-grove commented Nov 7, 2023

Uh oh!

open-craft-grove commented Nov 8, 2023

Uh oh!

open-craft-grove commented Nov 8, 2023

Uh oh!

open-craft-grove commented Nov 8, 2023

Uh oh!

open-craft-grove commented Nov 8, 2023

Uh oh!

open-craft-grove commented Nov 8, 2023

Uh oh!

open-craft-grove commented Nov 8, 2023

Uh oh!

open-craft-grove commented Nov 8, 2023

Uh oh!

pomegranited left a comment

Choose a reason for hiding this comment

Uh oh!

pomegranited Nov 7, 2023

Choose a reason for hiding this comment

Uh oh!

rpenido Nov 10, 2023

Choose a reason for hiding this comment

Uh oh!

pomegranited Nov 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bradenmacdonald Nov 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rpenido Nov 10, 2023

Choose a reason for hiding this comment

Uh oh!

pomegranited Nov 7, 2023

Choose a reason for hiding this comment

Uh oh!

rpenido Nov 10, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

open-craft-grove commented Nov 10, 2023

Uh oh!

open-craft-grove commented Nov 10, 2023

Uh oh!

open-craft-grove commented Nov 10, 2023

Uh oh!

open-craft-grove commented Nov 10, 2023

Uh oh!

open-craft-grove commented Nov 10, 2023

Uh oh!

rpenido commented Nov 10, 2023

Uh oh!

open-craft-grove commented Nov 10, 2023

Uh oh!

open-craft-grove commented Nov 10, 2023

Uh oh!

open-craft-grove commented Nov 10, 2023

Uh oh!

bradenmacdonald left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

rpenido commented Nov 6, 2023 •

edited

Loading

openedx-webhooks commented Nov 6, 2023 •

edited

Loading

pomegranited Nov 10, 2023 •

edited

Loading

bradenmacdonald Nov 10, 2023 •

edited

Loading

bradenmacdonald left a comment •

edited

Loading