Skip to content

Add dbfs scheme prefix to paths in cp command output#516

Merged
shreyas-goenka merged 3 commits intomainfrom
full-cp-paths
Jun 27, 2023
Merged

Add dbfs scheme prefix to paths in cp command output#516
shreyas-goenka merged 3 commits intomainfrom
full-cp-paths

Conversation

@shreyas-goenka
Copy link
Copy Markdown
Contributor

Changes

Adds the dbfs:/ prefix to paths output by the cp command so they can be used with the CLI

Tests

Manually

Currently there are no integration tests for command output, I'll add them in separately

@shreyas-goenka
Copy link
Copy Markdown
Contributor Author

Output:

shreyas.goenka@THW32HFW6T fs-cp % cli fs cp file:/Users/shreyas.goenka/projects/fs-cp/source-dir dbfs:/Users/shreyas.goenka@databricks.com/target-dir --recursive
Users/shreyas.goenka/projects/fs-cp/source-dir/bar.txt -> dbfs:/Users/shreyas.goenka@databricks.com/target-dir/bar.txt
Users/shreyas.goenka/projects/fs-cp/source-dir/foo.txt -> dbfs:/Users/shreyas.goenka@databricks.com/target-dir/foo.txt

shreyas.goenka@THW32HFW6T fs-cp % cli fs cp file:/Users/shreyas.goenka/projects/fs-cp/source-dir dbfs:/Users/shreyas.goenka@databricks.com/target-dir --recursive --output=json
{
  "source_path":"Users/shreyas.goenka/projects/fs-cp/source-dir/bar.txt",
  "target_path":"dbfs:/Users/shreyas.goenka@databricks.com/target-dir/bar.txt",
  "type":"FILE_SKIPPED"
}
{
  "source_path":"Users/shreyas.goenka/projects/fs-cp/source-dir/foo.txt",
  "target_path":"dbfs:/Users/shreyas.goenka@databricks.com/target-dir/foo.txt",
  "type":"FILE_SKIPPED"
}

fullTargetPath := targetPath
if c.targetScheme != "" {
fullTargetPath = path.Join(c.targetScheme+":", targetPath)
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The duplication of this block is a bit of a smell.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, Not sure how to refactor it better

@shreyas-goenka shreyas-goenka merged commit 1f5cdfd into main Jun 27, 2023
@shreyas-goenka shreyas-goenka deleted the full-cp-paths branch June 27, 2023 12:42
@pietern pietern mentioned this pull request Jun 28, 2023
pietern added a commit that referenced this pull request Jun 28, 2023
## Changes

CLI:
* Add --absolute flag for ls command
([#508](#508)).
* Add dbfs scheme prefix to paths in cp command output
([#516](#516)).
* Add provider detection to the repos create command
([#528](#528)).
* Added configure-cluster flag for auth login
([#500](#500)).
* Added prompts for Databricks profile for auth login command
([#502](#502)).
* Allow specifying repo by path for repos commands
([#526](#526)).
* Decode contents by default in workspace export command
([#531](#531)).
* Fixed jobs create command to only accept JSON payload
([#498](#498)).
* Make local files default for fs commands
([#506](#506)).
* Remove \r from new line print statments
([#509](#509)).
* Remove extra call to filer.Stat in dbfs filer.Read
([#515](#515)).
* Update alerts command integration test
([#512](#512)).
* Update variable regex to support hyphens
([#503](#503)).

Bundles:
* Add DATABRICKS_BUNDLE_TMP env variable
([#462](#462)).
* Update Terraform provider schema structs
([#504](#504)).

Dependencies:
* Bump github.com/databricks/databricks-sdk-go from
0.9.1-0.20230614092458-b5bbc1c8dabb to 0.10.0
([#497](#497)).

Internal:
* Use direct download for workspace filer read
([#514](#514)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants