Add workspace import_dir command to the CLI by shreyas-goenka · Pull Request #418 · databricks/cli

shreyas-goenka · 2023-06-01T09:24:03Z

TODO: add integration tests for functionality and progress events

shreyas-goenka · 2023-06-01T09:24:59Z

outputs:

shreyas.goenka@THW32HFW6T vandaley-industries % cli workspace import_dir . /Users/shreyas.goenka@databricks.com/new-vandaley
Import started
Error: Request failed for POST /Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/c.py: Path (/Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/c) already exists.

shreyas.goenka@THW32HFW6T vandaley-industries % cli workspace import_dir . /Users/shreyas.goenka@databricks.com/new-vandaley --overwrite
Import started
Uploaded foo/c.py -> /Users/shreyas.goenka@databricks.com/new-vandaley/foo/c
Uploaded foo/b.py -> /Users/shreyas.goenka@databricks.com/new-vandaley/foo/b
Uploaded exported/foo/b.py -> /Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/b
Uploaded exported/foo/c.py -> /Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/c
Uploaded myDir/b.txt -> /Users/shreyas.goenka@databricks.com/new-vandaley/myDir/b.txt
Uploaded exported/my-notebook.py.py -> /Users/shreyas.goenka@databricks.com/new-vandaley/exported/my-notebook.py
Uploaded .gitignore -> /Users/shreyas.goenka@databricks.com/new-vandaley/.gitignore
Uploaded myDir/a.txt -> /Users/shreyas.goenka@databricks.com/new-vandaley/myDir/a.txt
Import completed

shreyas.goenka@THW32HFW6T vandaley-industries % cli workspace import_dir . /Users/shreyas.goenka@databricks.com/new-vandaley --overwrite --output json
{
  "source_path":".",
  "target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley",
  "type":"IMPORT_STARTED"
}
{
  "source_path":"foo/b.py",
  "target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/foo/b",
  "type":"UPLOAD_COMPLETE"
}
{
  "source_path":"exported/my-notebook.py.py",
  "target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/exported/my-notebook.py",
  "type":"UPLOAD_COMPLETE"
}
{
  "source_path":"exported/foo/c.py",
  "target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/c",
  "type":"UPLOAD_COMPLETE"
}
{
  "source_path":"exported/foo/b.py",
  "target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/exported/foo/b",
  "type":"UPLOAD_COMPLETE"
}
{
  "source_path":"myDir/b.txt",
  "target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/myDir/b.txt",
  "type":"UPLOAD_COMPLETE"
}
{
  "source_path":"sync-snapshots/0bf79f861fa32456.json",
  "target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/sync-snapshots/0bf79f861fa32456.json",
  "type":"UPLOAD_COMPLETE"
}
{
  "source_path":".gitignore",
  "target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/.gitignore",
  "type":"UPLOAD_COMPLETE"
}
{
  "source_path":"foo/c.py",
  "target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/foo/c",
  "type":"UPLOAD_COMPLETE"
}
{
  "source_path":"myDir/a.txt",
  "target_path":"/Users/shreyas.goenka@databricks.com/new-vandaley/myDir/a.txt",
  "type":"UPLOAD_COMPLETE"
}
{
  "type":"IMPORT_COMPLETE"
}

cmd/workspace/workspace/import_dir.go

pietern · 2023-06-01T09:59:36Z

cmd/workspace/workspace/import_dir.go

+
+		// Initialize error wait group, and spawn the progress event emitter inside
+		// the error wait group
+		group, ctx := errgroup.WithContext(ctx)


What's the behavior of the returned context?

The returned context is cancelled if there's an error in any of the subroutines. see: https://pkg.go.dev/golang.org/x/sync/errgroup#WithContext

cmd/workspace/workspace/import_events.go

pietern · 2023-06-01T10:10:30Z

libs/cmdio/render.go

+	if err != nil {
+		return err
+	}
+	_, err = w.Write([]byte("\n"))


Can you split this into a separate PR? Might affect other functionality.

Done in this PR: #443

libs/sync/sync.go

pietern · 2023-06-01T10:12:12Z

libs/sync/sync.go

+		return "", fmt.Errorf("could not find remote path for %s", localPath)
+	}
+	return s.repoFiles.RemotePath(relativePath)
+}


Can you add a comment here why this is a public function? Also this needs a mutex because the snapshot code is accessing it from a separate goroutine.

A better approach is to include the remote name in the events, then you don't need this function or the mutex.

Yeah, having this in the events is then you have to use reflection to read the remote name values, which is OK maybe

The type switch is there already right?

Ah right, using type assertions should be enough here

shreyas-goenka · 2023-06-06T14:50:16Z

Closing because we decided to move forward with just a simple walk dir

Add workspace import_dir command to the CLI

9f4bf12

shreyas-goenka requested review from andrewnester, nfx and pietern June 1, 2023 09:25

Merge remote-tracking branch 'origin' into import_dir

1244b3a

pietern reviewed Jun 1, 2023

View reviewed changes

shreyas-goenka added 23 commits June 2, 2023 00:04

sync options redefine better

2303f20

-

0382c74

-

289dd0d

helper wsfs function

fb5ea16

add check for isdir in readdir

931fae1

added some intgegration test for repofiles and moved to filer

57a9578

repofiles integration test

dc0ff89

wip import-dir acc tests

2be6b85

testdata

c1ccf82

-

efc72b5

Merge remote-tracking branch 'origin' into import_dir

7099874

remove underscore

0f1cb67

set template in annotation

49f7969

undo remote path public

298fed2

address some comments

6cc2571

-

bb17546

-

ae09abf

-

14b63d0

-

5fdd8a2

-

9629032

removes not a directory error

e9a309e

-

74cb87f

Merge remote-tracking branch 'origin' into import_dir

59081f7

shreyas-goenka added 6 commits June 6, 2023 01:01

some cleanup

58acf64

Merge remote-tracking branch 'origin' into import_dir

f1e99d3

Merge remote-tracking branch 'origin' into import_dir

764d9e9

revert changes to reposfile

8166aed

-

f0f75ff

made events work and moved validation outside sync object

43aadba

shreyas-goenka closed this Jun 6, 2023

Conversation

shreyas-goenka commented Jun 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shreyas-goenka commented Jun 1, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka commented Jun 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shreyas-goenka commented Jun 1, 2023 •

edited

Loading