Skip to content

feat: adding sparkml notebook#99

Merged
hyunuk merged 42 commits intoGoogleCloudPlatform:mainfrom
hyunuk:serverless-spark-neos
Jan 23, 2024
Merged

feat: adding sparkml notebook#99
hyunuk merged 42 commits intoGoogleCloudPlatform:mainfrom
hyunuk:serverless-spark-neos

Conversation

@hyunuk
Copy link
Copy Markdown
Collaborator

@hyunuk hyunuk commented Dec 4, 2023 •
edited
Loading

This PR enables Neos for SparkML notebook functionality.

Key additions:

  • Vertex AI Workbench Instance:
    • Hosts the notebook environment.
    • Initially stopped for cost optimization.
    • Executes a post-startup script upon activation:
      • Downloads the notebook file (ipynb) to the instance.
      • Imports a Dataproc session template for running Dataproc Serverless interactive sessions.
  • Storage Bucket: Provides storage for ML models.
  • Workbench Service Account (SA): Dedicated SA for the workbench instance.
  • IAM Role for Workbench SA: Grants necessary permissions to the SA for function execution.

@hyunuk hyunuk requested review from a team, bradmiro and davenportjw as code owners December 4, 2023 16:45
@review-notebook-app
Copy link
Copy Markdown

review-notebook-app Bot commented Dec 4, 2023

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

bradmiro
bradmiro reviewed Dec 7, 2023
View reviewed changes
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
hyunuk and others added 13 commits December 10, 2023 09:42
bradmiro
bradmiro reviewed Jan 4, 2024
View reviewed changes
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
@hyunuk
Copy link
Copy Markdown
Collaborator Author

hyunuk commented Jan 10, 2024

/gcbrun

@hyunuk hyunuk requested a review from bradmiro January 12, 2024 22:56
bradmiro
bradmiro reviewed Jan 16, 2024
View reviewed changes
Comment thread main.tf Outdated
Comment thread metadata.display.yaml
Comment thread metadata.yaml
Comment thread metadata.yaml
Comment thread versions.tf Outdated
Comment thread versions.tf Outdated
bradmiro
bradmiro reviewed Jan 16, 2024
View reviewed changes
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
Comment thread src/ipynb/spark_ml.ipynb
bradmiro
bradmiro reviewed Jan 16, 2024
View reviewed changes
Comment thread test/integration/analytics_lakehouse/analytics_lakehouse_test.go Outdated
@hyunuk hyunuk merged commit 4b2169a into GoogleCloudPlatform:main Jan 23, 2024
@release-please release-please Bot mentioned this pull request Jan 16, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@apeabody apeabody apeabody left review comments

@bradmiro bradmiro bradmiro approved these changes

@davenportjw davenportjw Awaiting requested review from davenportjw davenportjw is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hyunuk @apeabody @bradmiro