Skip to content

Minor template tweaks #832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lennartkats-db merged 2 commits into from
Oct 4, 2023
Merged

Minor template tweaks #832

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
303cafc
Minor template tweaks
lennartkats-db Oct 3, 2023
d55e3c9
Remove setting
lennartkats-db Oct 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion libs/template/templates/default-python/databricks_template_schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
"type": "string",
"default": "yes",
"enum": ["yes", "no"],
"description": "Include a stub (sample) Python package 'my_project/src'",
"description": "Include a stub (sample) Python package in '{{.project_name}}/src'",
"order": 4
}
}
Expand Down
1 change: 1 addition & 0 deletions libs/template/templates/default-python/template/__preamble.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ This file only template directives; it is skipped for the actual output.
{{skip "{{.project_name}}/tests/main_test.py"}}
{{skip "{{.project_name}}/setup.py"}}
{{skip "{{.project_name}}/pytest.ini"}}
{{skip "{{.project_name}}/requirements-dev.txt"}}
{{end}}

{{if $notDLT}}
Expand Down
3 changes: 0 additions & 3 deletions ...s/default-python/template/{{.project_name}}/resources/{{.project_name}}_pipeline.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,3 @@ resources:
libraries:
- notebook:
path: ../src/dlt_pipeline.ipynb

configuration:
bundle.sourcePath: /Workspace/${workspace.file_path}/src
1 change: 0 additions & 1 deletion .../template/templates/default-python/template/{{.project_name}}/src/dlt_pipeline.ipynb.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@
"# Import DLT and src/{{.project_name}}\n",
"import dlt\n",
"import sys\n",
"sys.path.append(spark.conf.get(\"bundle.sourcePath\", \".\"))\n",
Copy link
Copy Markdown
Contributor Author

@lennartkats-db lennartkats-db Oct 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to verify the PYTHONPATH fix for DLT has been rolled out everywhere before making this change.

Copy link
Copy Markdown
Contributor Author

@lennartkats-db lennartkats-db Oct 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @justintang-db

Copy link
Copy Markdown

@justintang-db justintang-db Oct 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. DLT side change is in dlt release 37 and has been rolled out to all stages yesterday.

"from pyspark.sql.functions import expr\n",
"from {{.project_name}} import main"
{{else}}
Expand Down