From 9aff082cb6bfdf33f5222f28654fd80b61f25bde Mon Sep 17 00:00:00 2001 From: cojenco Date: Tue, 19 Oct 2021 11:45:54 -0700 Subject: [PATCH 001/172] samples: add cloud client samples from python-docs-samples (#626) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Renaming storage gcloud samples folder. (#418) * Add gcloud-based storage usage samples. (#419) * Refactor cloud client storage samples. (#421) * Add more storage samples for the cloud client libraries. (#432) * Auto-update dependencies. (#456) * Fix import order lint errors Change-Id: Ieaf7237fc6f925daec46a07d2e81a452b841198a * Add storage acl samples Change-Id: Ib44f9bb42bf0c0607e64905a26369f06ea5fb231 * Address review comments Change-Id: I94973a839f38ef3d1ec657c3c79f666eca56728b * Fix lint issue Change-Id: Ie9cf585303931f200a763d691906ad56221105fd * Auto-update dependencies. (#540) * Auto-update dependencies. (#542) * Move to google-cloud (#544) * Add new "quickstart" samples (#547) * Quickstart tests (#569) * Add tests for quickstarts * Update secrets * Add basic readme generator (#580) * Generate readmes for most service samples (#599) * Update samples to support latest Google Cloud Python (#656) * Edited upload/download to perform encryption properly (#667) * Storage Encryption Key Rotation Sample using Veneer + Tests (#672) * Auto-update dependencies. (#715) * Adds storage Pub/Sub notification polling tutorial (#875) * Remove cloud config fixture (#887) * Remove cloud config fixture * Fix client secrets * Fix bigtable instance * Auto-update dependencies. (#914) * Auto-update dependencies. * xfail the error reporting test * Fix lint * Re-generate all readmes * Add bucket-level IAM samples (#919) * Add bucket-level IAM samples * Address review comments * Auto-update dependencies. (#927) * Fix README rst links (#962) * Fix README rst links * Update all READMEs * Auto-update dependencies. (#1004) * Auto-update dependencies. * Fix natural language samples * Fix pubsub iam samples * Fix language samples * Fix bigquery samples * Add bucket label samples (#1045) * Auto-update dependencies. (#1055) * Auto-update dependencies. * Explicitly use latest bigtable client Change-Id: Id71e9e768f020730e4ca9514a0d7ebaa794e7d9e * Revert language update for now Change-Id: I8867f154e9a5aae00d0047c9caf880e5e8f50c53 * Remove pdb. smh Change-Id: I5ff905fadc026eebbcd45512d4e76e003e3b2b43 * Auto-update dependencies. (#1057) * Auto-update dependencies. (#1073) * Auto-update dependencies. (#1093) * Auto-update dependencies. * Fix storage notification poll sample Change-Id: I6afbc79d15e050531555e4c8e51066996717a0f3 * Fix spanner samples Change-Id: I40069222c60d57e8f3d3878167591af9130895cb * Drop coverage because it's not useful Change-Id: Iae399a7083d7866c3c7b9162d0de244fbff8b522 * Try again to fix flaky logging test Change-Id: I6225c074701970c17c426677ef1935bb6d7e36b4 * Auto-update dependencies. (#1097) * Update all generated readme auth instructions (#1121) Change-Id: I03b5eaef8b17ac3dc3c0339fd2c7447bd3e11bd2 * Fix TypeError when running Storage notification polling exmaple. (#1135) * Adds storage Pub/Sub notification polling tutorial * Fix formatting and add some tests * Auto-generate README * Simplify implementation, remove classes * Simplified example, removed de-duping * regenerate README * Remove explicit project parameter. * Fix notification TypeError on start. * Fix linter error. * Fix ordered list ordinals. * Rerun nox readmegen. * Add support for overwrite attributes (#1142) * Add support for overwrite attributes, bug fixes * Lint fix for overwrite line * Switch variable to snake_case * Handle case where attribute not set (#1143) * Added Link to Python Setup Guide (#1158) * Update Readme.rst to add Python setup guide As requested in b/64770713. This sample is linked in documentation https://cloud.google.com/bigtable/docs/scaling, and it would make more sense to update the guide here than in the documentation. * Update README.rst * Update README.rst * Update README.rst * Update README.rst * Update README.rst * Update install_deps.tmpl.rst * Updated readmegen scripts and re-generated related README files * Fixed the lint error * Auto-update dependencies. (#1138) * storage requester pays samples (#1122) * storage requester pays samples * Added tests and fixed linting issues * google-cloud-storage version update * changed get_bucket to bucket for downloading * small change * Auto-update dependencies. (#1186) * Auto-update dependencies. (#1234) * Auto-update dependencies. * Drop pytest-logcapture as it's no longer needed Change-Id: Ia8b9e8aaf248e9770db6bc4842a4532df8383893 * Auto-update dependencies. (#1239) * Added "Open in Cloud Shell" buttons to README files (#1254) * Auto-update dependencies. (#1263) * Auto-update dependencies. (#1272) * Auto-update dependencies. * Update requirements.txt * Auto-update dependencies. (#1282) * Auto-update dependencies. * Fix storage acl sample Change-Id: I413bea899fdde4c4859e4070a9da25845b81f7cf * Auto-update dependencies. (#1320) * Auto-update dependencies. (#1359) * Auto-update dependencies. (#1377) * Auto-update dependencies. * Update requirements.txt * Auto-update dependencies. (#1389) * Auto-update dependencies. * Regenerate the README files and fix the Open in Cloud Shell link for some samples (#1441) * Update READMEs to fix numbering and add git clone (#1464) * Fix typo. (#1509) Fixes https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1485 * Storage: add KMS samples (#1510) * Storage: add KMS samples * Add CLOUD_KMS_KEY environment variable * [Storage] Update kms samples (#1517) * Storage: add KMS samples * Add CLOUD_KMS_KEY environment variable * Add region tags around samples * Add more testing * Fix tests and lint * Remove leftover merge conflict. (#1657) * Add region tag to upload_blob snippet (#1671) * Bucket lock samples (#1588) * [Storage] Add spacing in sample code. (#1735) * Add spacing in sample code. * remove whitespace * Auto-update dependencies. (#1846) ACK, merging. * Update requirements.txt (#1944) * Update requirements.txt * Adding some rate limiting * Auto-update dependencies. (#1980) * Auto-update dependencies. * Update requirements.txt * Update requirements.txt * storage: bucket policy only samples (#1976) * humble beginnings * Verified integration tests and updated README.rst * Updating samples to reflect fixed surface * Use release 1.14.0 * Add sleep to avoid bucket rate limit (#2136) * feat(storage): Add snippets for v4 signed URLs (#2142) * feat(storage): Add snippets for v4 signed URLs * lint * fix .format() * add v4 command to switch statement * fix region tag * change if => elif to try to make func less complex * move main to a function * storage: add list buckets (#2149) * Add list_buckets sample * Allow for more if conditions * Drop xfail for passing test_remove_bucket_label (#2173) The Python client was fixed in https://github.com/googleapis/google-cloud-python/issues/3711 so the test now passes. * Update string reported in snippet and update test * Update list blobs to use new client.list_blobs() method. (#2296) * Update list blob samples * Update requirements.txt * Fix lint issues * Use latest storage client * [Storage] Add comment to clarify which package version is necessary (#2315) * Add comment to clarify which package version * Lint and add another comment to related sample * Storage: HMAC key samples (#2372) Add samples for HMAC key functionality: list, create, get, activate, deactivate, delete. Includes tests and version bump for client library. * Remove required argument from list buckets sample (#2394) * Remove required argument from list buckets sample * Remove required argument from list buckets sample * Fixup sample for list_hmac_keys (#2404) Correct printed metadata to match canonical samples. * Bucket metadata sample (#2414) * Remove required argument from list buckets sample * Bucket metadata sample * Bucket metadata sample * Adds updates for samples profiler ... vision (#2439) * fix: add bucket-name as required arg to v4 snippets (#2502) * [Storage] Support rename of BPO to UniformBucketLevelAccess (#2335) * Update BPO -> UBLA * Update BPO -> UBLA * Fix region tag (#2515) * Update documentation for prefix, delimiter search (#2537) * Update documentation for prefix, delimiter search * Remove whitespace. * [Storage] Split samples (#2602) * split bucket_lock samples and lint * split samples * blacken * fix typos * Add missing tests and lint * lint * fix typos * fix typo * typo * remove README * Auto-update dependencies. (#2005) * Auto-update dependencies. * Revert update of appengine/flexible/datastore. * revert update of appengine/flexible/scipy * revert update of bigquery/bqml * revert update of bigquery/cloud-client * revert update of bigquery/datalab-migration * revert update of bigtable/quickstart * revert update of compute/api * revert update of container_registry/container_analysis * revert update of dataflow/run_template * revert update of datastore/cloud-ndb * revert update of dialogflow/cloud-client * revert update of dlp * revert update of functions/imagemagick * revert update of functions/ocr/app * revert update of healthcare/api-client/fhir * revert update of iam/api-client * revert update of iot/api-client/gcs_file_to_device * revert update of iot/api-client/mqtt_example * revert update of language/automl * revert update of run/image-processing * revert update of vision/automl * revert update testing/requirements.txt * revert update of vision/cloud-client/detect * revert update of vision/cloud-client/product_search * revert update of jobs/v2/api_client * revert update of jobs/v3/api_client * revert update of opencensus * revert update of translate/cloud-client * revert update to speech/cloud-client Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> Co-authored-by: Doug Mahugh * samples(storage): IAM conditions samples (#2730) * docs(storage): use policy.bindings in Storage/IAM samples * update view Bucket IAM to use policy.bindings * update remove Bucket IAM to use policy.bindings * blacken * add IAM condition sample * add conditional iam binding sample * bump storage requirement to 1.25.0 * fix tests * remove unused imports * fix: Use unique resources for storage snippets. (#3029) * fix: use unique buckets and blobs for acl tests * fix: use unique buckets and blobs for snippets tests * fix: reuse test_bucket within module to avoid exhausting quota * fix: Due to retention policy, don't reuse fixture for bucket lock * fix: randomize blob names to disperse file edits * fix: Reuse HMAC key as we have a limit of 5 (#3037) * fix: Reuse HMAC key as we have a limit of 5 * fix: harden storage test fixtures (#3039) * fix: improve UBLA test fixtures * fix: improve IAM test fixtures * storage: Fix docs for signed URL generation (#3008) Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Co-authored-by: Christopher Wilcox * chore(deps): update dependency google-cloud-storage to v1.26.0 (#3046) * chore(deps): update dependency google-cloud-storage to v1.26.0 * chore(deps): specify dependencies by python version * chore: up other deps to try to remove errors Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: Leah Cole * feat: add remove conditional binding sample (#3107) * feat: add remove conditional binding sample * fix iam test fixture * fix silly mistake of removing all bindings * fix ubla test * address feedback * revert changes to tests * Simplify noxfile setup. (#2806) * chore(deps): update dependency requests to v2.23.0 * Simplify noxfile and add version control. * Configure appengine/standard to only test Python 2.7. * Update Kokokro configs to match noxfile. * Add requirements-test to each folder. * Remove Py2 versions from everything execept appengine/standard. * Remove conftest.py. * Remove appengine/standard/conftest.py * Remove 'no-sucess-flaky-report' from pytest.ini. * Add GAE SDK back to appengine/standard tests. * Fix typo. * Roll pytest to python 2 version. * Add a bunch of testing requirements. * Remove typo. * Add appengine lib directory back in. * Add some additional requirements. * Fix issue with flake8 args. * Even more requirements. * Readd appengine conftest.py. * Add a few more requirements. * Even more Appengine requirements. * Add webtest for appengine/standard/mailgun. * Add some additional requirements. * Add workaround for issue with mailjet-rest. * Add responses for appengine/standard/mailjet. Co-authored-by: Renovate Bot * [storage] feat: add post policy sample (#3231) * feat: add post policy sample * use 1.27.0 * fix * simplify iterator Co-authored-by: Jonathan Lui * Update dependency google-cloud-pubsub to v1.4.2 in Storage and Pub/Sub (#3343) * Remove name attribute from the input (#3569) If name='submit' is specified for the input type='submit' the endpoint returns the following error: InvalidPolicyDocument The content of the form does not meet the conditions specified in the policy document.
Policy did not reference these fields: submit
 Co-authored-by: Takashi Matsuo * [storage] fix: use unique blob name (#3568) * [storage] fix: use unique blob name fixes #3567 * add some comments * chore(deps): update dependency google-cloud-storage to v1.28.0 (#3260) Co-authored-by: Takashi Matsuo * [storage] fix: use a different bucket for requester_pays_test (#3655) * [storage] fix: use a different bucket for requester_pays_test fixes #3654 * rename to README.md, added the envvar to the template * add REQUESTER_PAYS_TEST_BUCKET env var * just use REQUESTER_PAYS_TEST_BUCKET * docs(storage): add samples for lifer cycle and versioning (#3578) * docs(storage): add samples for lifer cycle and versioning * docs(storage): nits * docs(storage): lint fix Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore: some lint fixes (#3750) * chore(deps): update dependency google-cloud-pubsub to v1.4.3 (#3725) Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Co-authored-by: Takashi Matsuo * docs(storage): add samples (#3687) * chore(deps): update dependency google-cloud-storage to v1.28.1 (#3785) * chore(deps): update dependency google-cloud-storage to v1.28.1 * [asset] testing: use uuid instead of time Co-authored-by: Takashi Matsuo * docs(storage): add samples for file archive generation and cors configuration (#3794) * chore(deps): update dependency google-cloud-pubsub to v1.5.0 (#3781) Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> * Replace GCLOUD_PROJECT with GOOGLE_CLOUD_PROJECT. (#4022) * [storage] testing: use multiple projects (#4048) * [storage] testing: use multiple projects We still need to use the old project for some tests. fixes #4033 fixes #4029 * remove print * use uuid instead of time.time() * lint fix * chore(deps): update dependency google-cloud-storage to v1.29.0 (#4040) * Update dependency google-cloud-pubsub to v1.6.0 (#4039) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-pubsub](https://togithub.com/googleapis/python-pubsub) | minor | `==1.5.0` -> `==1.6.0` | --- ### Release Notes
googleapis/python-pubsub ### [`v1.6.0`](https://togithub.com/googleapis/python-pubsub/blob/master/CHANGELOG.md#​160-httpswwwgithubcomgoogleapispython-pubsubcomparev150v160-2020-06-09) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v1.5.0...v1.6.0) ##### Features - Add flow control for message publishing ([#​96](https://www.github.com/googleapis/python-pubsub/issues/96)) ([06085c4](https://www.github.com/googleapis/python-pubsub/commit/06085c4083b9dccdd50383257799904510bbf3a0)) ##### Bug Fixes - Fix PubSub incompatibility with api-core 1.17.0+ ([#​103](https://www.github.com/googleapis/python-pubsub/issues/103)) ([c02060f](https://www.github.com/googleapis/python-pubsub/commit/c02060fbbe6e2ca4664bee08d2de10665d41dc0b)) ##### Documentation - Clarify that Schedulers shouldn't be used with multiple SubscriberClients ([#​100](https://togithub.com/googleapis/python-pubsub/pull/100)) ([cf9e87c](https://togithub.com/googleapis/python-pubsub/commit/cf9e87c80c0771f3fa6ef784a8d76cb760ad37ef)) - Fix update subscription/snapshot/topic samples ([#​113](https://togithub.com/googleapis/python-pubsub/pull/113)) ([e62c38b](https://togithub.com/googleapis/python-pubsub/commit/e62c38bb33de2434e32f866979de769382dea34a)) ##### Internal / Testing Changes - Re-generated service implementaton using synth: removed experimental notes from the RetryPolicy and filtering features in anticipation of GA, added DetachSubscription (experimental) ([#​114](https://togithub.com/googleapis/python-pubsub/pull/114)) ([0132a46](https://togithub.com/googleapis/python-pubsub/commit/0132a4680e0727ce45d5e27d98ffc9f3541a0962)) - Incorporate will_accept() checks into publish() ([#​108](https://togithub.com/googleapis/python-pubsub/pull/108)) ([6c7677e](https://togithub.com/googleapis/python-pubsub/commit/6c7677ecb259672bbb9b6f7646919e602c698570))
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-cloud-pubsub to v1.6.1 (#4242) Co-authored-by: gcf-merge-on-green[bot] <60162190+gcf-merge-on-green[bot]@users.noreply.github.com> * chore(deps): update dependency pytest to v5.4.3 (#4279) * chore(deps): update dependency pytest to v5.4.3 * specify pytest for python 2 in appengine Co-authored-by: Leah Cole * chore(deps): update dependency mock to v4 (#4287) * chore(deps): update dependency mock to v4 * specify mock version for appengine python 2 Co-authored-by: Leah Cole * chore(deps): update dependency google-cloud-pubsub to v1.7.0 (#4290) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-pubsub](https://togithub.com/googleapis/python-pubsub) | minor | `==1.6.1` -> `==1.7.0` | --- ### Release Notes
googleapis/python-pubsub ### [`v1.7.0`](https://togithub.com/googleapis/python-pubsub/blob/master/CHANGELOG.md#​170-httpswwwgithubcomgoogleapispython-pubsubcomparev161v170-2020-07-13) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v1.6.1...v1.7.0) ##### New Features - Add support for server-side flow control. ([#​143](https://togithub.com/googleapis/python-pubsub/pull/143)) ([04e261c](https://www.github.com/googleapis/python-pubsub/commit/04e261c602a2919cc75b3efa3dab099fb2cf704c)) ##### Dependencies - Update samples dependency `google-cloud-pubsub` to `v1.6.1`. ([#​144](https://togithub.com/googleapis/python-pubsub/pull/144)) ([1cb6746](https://togithub.com/googleapis/python-pubsub/commit/1cb6746b00ebb23dbf1663bae301b32c3fc65a88)) ##### Documentation - Add pubsub/cloud-client samples from the common samples repo (with commit history). ([#​151](https://togithub.com/googleapis/python-pubsub/pull/151)) - Add flow control section to publish overview. ([#​129](https://togithub.com/googleapis/python-pubsub/pull/129)) ([acc19eb](https://www.github.com/googleapis/python-pubsub/commit/acc19eb048eef067d9818ef3e310b165d9c6307e)) - Add a link to Pub/Sub filtering language public documentation to `pubsub.proto`. ([#​121](https://togithub.com/googleapis/python-pubsub/pull/121)) ([8802d81](https://www.github.com/googleapis/python-pubsub/commit/8802d8126247f22e26057e68a42f5b5a82dcbf0d))
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * Fix mismatched storage region tags (#4194) * Update dependency google-cloud-storage to v1.30.0 * Update dependency pytest to v6 (#4390) * chore(deps): update dependency google-cloud-storage to v1.31.0 (#4564) Co-authored-by: Takashi Matsuo * chore: fix some more unmatched region tags (#4585) fixes #4549 Co-authored-by: Dina Graves Portman * Update storage_get_metadata.py (#4615) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency google-cloud-storage to v1.31.1 (#4714) * chore(deps): update dependency google-cloud-storage to v1.31.2 (#4750) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | patch | `==1.31.1` -> `==1.31.2` | --- ### Release Notes
googleapis/python-storage ### [`v1.31.2`](https://togithub.com/googleapis/python-storage/blob/master/CHANGELOG.md#​1312-httpswwwgithubcomgoogleapispython-storagecomparev1311v1312-2020-09-23) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v1.31.1...v1.31.2)
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency pytest to v6.1.1 (#4761) * chore(deps): update dependency google-cloud-storage to v1.32.0 (#4871) * chore(deps): update dependency pytest to v6.1.2 (#4921) Co-authored-by: Charles Engelke * change pprint to print. (#4856) * change pprint to print. Line 57 had pprint.pprint.. changing it to print. * Update storage_get_bucket_metadata.py Removing pprint import Co-authored-by: Dina Graves Portman Co-authored-by: Charles Engelke * chore(deps): update dependency google-cloud-storage to v1.33.0 (#4990) * Add patch call (#5013) I believe a call to `blob.patch()` is necessary to actually save the metadata back to GCS. * fix: add a comment to draw attention to using get_blob, not blob (#5052) * fix: add a comment to draw attention to using get_blob, not blob * docs: further elaboration * docs: add clarifying doc string to download file * Update storage_download_file.py * Update storage_download_file.py * chore(deps): update dependency mock to v4.0.3 (#5062) * fix(storage): Update comment, prefix should include delimiter (#5064) * chore(deps): update dependency google-cloud-storage to v1.35.0 (#5074) * chore(deps): update dependency pytest to v6.2.1 (#5076) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [pytest](https://docs.pytest.org/en/latest/) ([source](https://togithub.com/pytest-dev/pytest)) | minor | `==6.1.2` -> `==6.2.1` | --- ### Release Notes
pytest-dev/pytest ### [`v6.2.1`](https://togithub.com/pytest-dev/pytest/releases/6.2.1) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/6.2.0...6.2.1) # pytest 6.2.1 (2020-12-15) ## Bug Fixes - [#​7678](https://togithub.com/pytest-dev/pytest/issues/7678): Fixed bug where `ImportPathMismatchError` would be raised for files compiled in the host and loaded later from an UNC mounted path (Windows). - [#​8132](https://togithub.com/pytest-dev/pytest/issues/8132): Fixed regression in `approx`: in 6.2.0 `approx` no longer raises `TypeError` when dealing with non-numeric types, falling back to normal comparison. Before 6.2.0, array types like tf.DeviceArray fell through to the scalar case, and happened to compare correctly to a scalar if they had only one element. After 6.2.0, these types began failing, because they inherited neither from standard Python number hierarchy nor from `numpy.ndarray`. `approx` now converts arguments to `numpy.ndarray` if they expose the array protocol and are not scalars. This treats array-like objects like numpy arrays, regardless of size. ### [`v6.2.0`](https://togithub.com/pytest-dev/pytest/releases/6.2.0) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/6.1.2...6.2.0) # pytest 6.2.0 (2020-12-12) ## Breaking Changes - [#​7808](https://togithub.com/pytest-dev/pytest/issues/7808): pytest now supports python3.6+ only. ## Deprecations - [#​7469](https://togithub.com/pytest-dev/pytest/issues/7469): Directly constructing/calling the following classes/functions is now deprecated: - `_pytest.cacheprovider.Cache` - `_pytest.cacheprovider.Cache.for_config()` - `_pytest.cacheprovider.Cache.clear_cache()` - `_pytest.cacheprovider.Cache.cache_dir_from_config()` - `_pytest.capture.CaptureFixture` - `_pytest.fixtures.FixtureRequest` - `_pytest.fixtures.SubRequest` - `_pytest.logging.LogCaptureFixture` - `_pytest.pytester.Pytester` - `_pytest.pytester.Testdir` - `_pytest.recwarn.WarningsRecorder` - `_pytest.recwarn.WarningsChecker` - `_pytest.tmpdir.TempPathFactory` - `_pytest.tmpdir.TempdirFactory` These have always been considered private, but now issue a deprecation warning, which may become a hard error in pytest 7.0.0. - [#​7530](https://togithub.com/pytest-dev/pytest/issues/7530): The `--strict` command-line option has been deprecated, use `--strict-markers` instead. We have plans to maybe in the future to reintroduce `--strict` and make it an encompassing flag for all strictness related options (`--strict-markers` and `--strict-config` at the moment, more might be introduced in the future). - [#​7988](https://togithub.com/pytest-dev/pytest/issues/7988): The `@pytest.yield_fixture` decorator/function is now deprecated. Use pytest.fixture instead. `yield_fixture` has been an alias for `fixture` for a very long time, so can be search/replaced safely. ## Features - [#​5299](https://togithub.com/pytest-dev/pytest/issues/5299): pytest now warns about unraisable exceptions and unhandled thread exceptions that occur in tests on Python>=3.8. See unraisable for more information. - [#​7425](https://togithub.com/pytest-dev/pytest/issues/7425): New pytester fixture, which is identical to testdir but its methods return pathlib.Path when appropriate instead of `py.path.local`. This is part of the movement to use pathlib.Path objects internally, in order to remove the dependency to `py` in the future. Internally, the old Testdir <\_pytest.pytester.Testdir> is now a thin wrapper around Pytester <\_pytest.pytester.Pytester>, preserving the old interface. - [#​7695](https://togithub.com/pytest-dev/pytest/issues/7695): A new hook was added, pytest_markeval_namespace which should return a dictionary. This dictionary will be used to augment the "global" variables available to evaluate skipif/xfail/xpass markers. Pseudo example `conftest.py`: ```{.sourceCode .python} def pytest_markeval_namespace(): return {"color": "red"} ``` `test_func.py`: ```{.sourceCode .python} @​pytest.mark.skipif("color == 'blue'", reason="Color is not red") def test_func(): assert False ``` - [#​8006](https://togithub.com/pytest-dev/pytest/issues/8006): It is now possible to construct a ~pytest.MonkeyPatch object directly as `pytest.MonkeyPatch()`, in cases when the monkeypatch fixture cannot be used. Previously some users imported it from the private \_pytest.monkeypatch.MonkeyPatch namespace. Additionally, MonkeyPatch.context <pytest.MonkeyPatch.context> is now a classmethod, and can be used as `with MonkeyPatch.context() as mp: ...`. This is the recommended way to use `MonkeyPatch` directly, since unlike the `monkeypatch` fixture, an instance created directly is not `undo()`-ed automatically. ## Improvements - [#​1265](https://togithub.com/pytest-dev/pytest/issues/1265): Added an `__str__` implementation to the ~pytest.pytester.LineMatcher class which is returned from `pytester.run_pytest().stdout` and similar. It returns the entire output, like the existing `str()` method. - [#​2044](https://togithub.com/pytest-dev/pytest/issues/2044): Verbose mode now shows the reason that a test was skipped in the test's terminal line after the "SKIPPED", "XFAIL" or "XPASS". - [#​7469](https://togithub.com/pytest-dev/pytest/issues/7469) The types of builtin pytest fixtures are now exported so they may be used in type annotations of test functions. The newly-exported types are: - `pytest.FixtureRequest` for the request fixture. - `pytest.Cache` for the cache fixture. - `pytest.CaptureFixture[str]` for the capfd and capsys fixtures. - `pytest.CaptureFixture[bytes]` for the capfdbinary and capsysbinary fixtures. - `pytest.LogCaptureFixture` for the caplog fixture. - `pytest.Pytester` for the pytester fixture. - `pytest.Testdir` for the testdir fixture. - `pytest.TempdirFactory` for the tmpdir_factory fixture. - `pytest.TempPathFactory` for the tmp_path_factory fixture. - `pytest.MonkeyPatch` for the monkeypatch fixture. - `pytest.WarningsRecorder` for the recwarn fixture. Constructing them is not supported (except for MonkeyPatch); they are only meant for use in type annotations. Doing so will emit a deprecation warning, and may become a hard-error in pytest 7.0. Subclassing them is also not supported. This is not currently enforced at runtime, but is detected by type-checkers such as mypy. - [#​7527](https://togithub.com/pytest-dev/pytest/issues/7527): When a comparison between namedtuple <collections.namedtuple> instances of the same type fails, pytest now shows the differing field names (possibly nested) instead of their indexes. - [#​7615](https://togithub.com/pytest-dev/pytest/issues/7615): Node.warn <\_pytest.nodes.Node.warn> now permits any subclass of Warning, not just PytestWarning <pytest.PytestWarning>. - [#​7701](https://togithub.com/pytest-dev/pytest/issues/7701): Improved reporting when using `--collected-only`. It will now show the number of collected tests in the summary stats. - [#​7710](https://togithub.com/pytest-dev/pytest/issues/7710): Use strict equality comparison for non-numeric types in pytest.approx instead of raising TypeError. This was the undocumented behavior before 3.7, but is now officially a supported feature. - [#​7938](https://togithub.com/pytest-dev/pytest/issues/7938): New `--sw-skip` argument which is a shorthand for `--stepwise-skip`. - [#​8023](https://togithub.com/pytest-dev/pytest/issues/8023): Added `'node_modules'` to default value for norecursedirs. - [#​8032](https://togithub.com/pytest-dev/pytest/issues/8032): doClassCleanups <unittest.TestCase.doClassCleanups> (introduced in unittest in Python and 3.8) is now called appropriately. ## Bug Fixes - [#​4824](https://togithub.com/pytest-dev/pytest/issues/4824): Fixed quadratic behavior and improved performance of collection of items using autouse fixtures and xunit fixtures. - [#​7758](https://togithub.com/pytest-dev/pytest/issues/7758): Fixed an issue where some files in packages are getting lost from `--lf` even though they contain tests that failed. Regressed in pytest 5.4.0. - [#​7911](https://togithub.com/pytest-dev/pytest/issues/7911): Directories created by by tmp_path and tmpdir are now considered stale after 3 days without modification (previous value was 3 hours) to avoid deleting directories still in use in long running test suites. - [#​7913](https://togithub.com/pytest-dev/pytest/issues/7913): Fixed a crash or hang in pytester.spawn <\_pytest.pytester.Pytester.spawn> when the readline module is involved. - [#​7951](https://togithub.com/pytest-dev/pytest/issues/7951): Fixed handling of recursive symlinks when collecting tests. - [#​7981](https://togithub.com/pytest-dev/pytest/issues/7981): Fixed symlinked directories not being followed during collection. Regressed in pytest 6.1.0. - [#​8016](https://togithub.com/pytest-dev/pytest/issues/8016): Fixed only one doctest being collected when using `pytest --doctest-modules path/to/an/__init__.py`. ## Improved Documentation - [#​7429](https://togithub.com/pytest-dev/pytest/issues/7429): Add more information and use cases about skipping doctests. - [#​7780](https://togithub.com/pytest-dev/pytest/issues/7780): Classes which should not be inherited from are now marked `final class` in the API reference. - [#​7872](https://togithub.com/pytest-dev/pytest/issues/7872): `_pytest.config.argparsing.Parser.addini()` accepts explicit `None` and `"string"`. - [#​7878](https://togithub.com/pytest-dev/pytest/issues/7878): In pull request section, ask to commit after editing changelog and authors file. ## Trivial/Internal Changes - [#​7802](https://togithub.com/pytest-dev/pytest/issues/7802): The `attrs` dependency requirement is now >=19.2.0 instead of >=17.4.0. - [#​8014](https://togithub.com/pytest-dev/pytest/issues/8014): .pyc files created by pytest's assertion rewriting now conform to the newer PEP-552 format on Python>=3.7. (These files are internal and only interpreted by pytest itself.)
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-cloud-pubsub to v2.2.0 (#4673) * chore(deps): update dependency google-cloud-pubsub to v2.2.0 * run pubsub script on healthcare/api-client/v1/dicom * iot pubsub fixes, fix lint * revert some changes pubsub script made * try using return_value for mock * undo previous change * try adding publish_time in mock * move publish_time param * make publish_time a float * make publish_time a datetime * try using object instead of datetime * another attempt * undo the black stuff that messed up lint Co-authored-by: Leah Cole Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(Dockerfile): add Python 3.9 (#4968) * chore(Dockerfile): add Python 3.9 * Add py3.9 kokoro dir * fix typo * Add GPG keys * Add 3.9 to noxfiles * Update composer dep to avoid deprecation spam * fix(storage): add py-3.9 specific key * update psycopg2-binary, only run test in py-3.9 build * add libmemcached-dev to the Dockerfile * disable appengine standard test in py-3.9 build * disable py-3.9 build for appengine cloud_debugger * skip py-3.9 build for composer/workflows * skip tests with pyarrow for py-3.9 build * avoid ReferenceError in iot builds * skip some tests due to pip error * add a temporary statement for debugging * fix lint * use correct constant * disable 2.7 builds * disable builds due to pip conflict The conflict is between google-cloud-monitoring==2.0.0 and opencensus-ext-stackdriver. * remove temporary debugging statement * really skip py-3.9 build for pubsub/streaming-analytics * copyright year fix * fix(storage): explicitly use the test project for the test bucket * fix(storage): use correct cloud project * fix: disable py-3.9 builds - appengine/standard_python3/bigquery - data-science-onramp/data-ingestion * disable py-3.9 build - dataflow/encryption-keys - dataflow/flex-templates/streaming_beam * disable type hint checks Co-authored-by: Takashi Matsuo * fix(storage): list all versions (#5325) ## Description Add the `versions=True` variable to the `list_file_archived_generations function` to actually list all the versions instead of the last one only. Fixes the incongruency between python and the other languages in the [Listing noncurrent object versions code samples](https://cloud.google.com/storage/docs/using-object-versioning#list). ## Checklist - [x] I have followed [Sample Guidelines from AUTHORING_GUIDE.MD](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md) - [x] README is updated to include [all relevant information](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#readme-file) - [x] **Tests** pass: `nox -s py-3.6` (see [Test Environment Setup](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#test-environment-setup)) - [x] **Lint** pass: `nox -s lint` (see [Test Environment Setup](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#test-environment-setup)) - [ ] These samples need a new **API enabled** in testing projects to pass (let us know which ones) - [ ] These samples need a new/updated **env vars** in testing projects set to pass (let us know which ones) - [x] Please **merge** this PR for me once it is approved. - [ ] This sample adds a new sample directory, and I updated the [CODEOWNERS file](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/.github/CODEOWNERS) with the codeowners for this sample * docs: address sample feedback issues (#5329) ## Description Fixes #5180, captures work from #5181 authored by @keegan2149, thank you! ## Checklist - [x] I have followed [Sample Guidelines from AUTHORING_GUIDE.MD](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md) - [x] README is updated to include [all relevant information](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#readme-file) - [x] **Tests** pass: `nox -s py-3.6` (see [Test Environment Setup](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#test-environment-setup)) - [x] **Lint** pass: `nox -s lint` (see [Test Environment Setup](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#test-environment-setup)) - [x] Please **merge** this PR for me once it is approved. * chore(deps): update dependency google-cloud-pubsub to v2.3.0 (#5347) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-pubsub](https://togithub.com/googleapis/python-pubsub) | `==2.2.0` -> `==2.3.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.3.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.3.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.3.0/compatibility-slim/2.2.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.3.0/confidence-slim/2.2.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/python-pubsub ### [`v2.3.0`](https://togithub.com/googleapis/python-pubsub/blob/master/CHANGELOG.md#​230-httpswwwgithubcomgoogleapispython-pubsubcomparev220v230-2021-02-08) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.2.0...v2.3.0) ##### Features - surface SchemaServiceClient in google.cloud.pubsub ([#​281](https://www.github.com/googleapis/python-pubsub/issues/281)) ([8751bcc](https://www.github.com/googleapis/python-pubsub/commit/8751bcc5eb782df55769b48253629a3bde3d4661)) ##### Bug Fixes - client version missing from the user agent header ([#​275](https://www.github.com/googleapis/python-pubsub/issues/275)) ([b112f4f](https://www.github.com/googleapis/python-pubsub/commit/b112f4fcbf6f2bce8dcf37871bdc540b11f54fe3)) - Don't open the google.cloud package by adding pubsub.py ([#​269](https://www.github.com/googleapis/python-pubsub/issues/269)) ([542d79d](https://www.github.com/googleapis/python-pubsub/commit/542d79d7c5fb7403016150ba477485756cd4097b)) - flaky samples tests ([#​263](https://www.github.com/googleapis/python-pubsub/issues/263)) ([3d6a29d](https://www.github.com/googleapis/python-pubsub/commit/3d6a29de07cc09be663c90a3333f4cd33633994f)) - Modify synth.py to update grpc transport options ([#​266](https://www.github.com/googleapis/python-pubsub/issues/266)) ([41dcd30](https://www.github.com/googleapis/python-pubsub/commit/41dcd30636168f3dd1248f1d99170d531fc9bcb8)) - pass anonymous credentials for emulator ([#​250](https://www.github.com/googleapis/python-pubsub/issues/250)) ([8eed8e1](https://www.github.com/googleapis/python-pubsub/commit/8eed8e16019510dc8b20fb6b009d61a7ac532d26)) - remove grpc send/recieve limits ([#​259](https://www.github.com/googleapis/python-pubsub/issues/259)) ([fd2840c](https://www.github.com/googleapis/python-pubsub/commit/fd2840c10f92b03da7f4b40ac69c602220757c0a))
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-cloud-storage to v1.35.1 (#5321) * chore(deps): update dependency google-cloud-pubsub to v2.4.0 (#5399) * chore(deps): update dependency google-cloud-storage to v1.36.1 (#5353) * chore(deps): update dependency google-cloud-storage to v1.36.1 * moving media transcoder separately Co-authored-by: Leah Cole Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency google-cloud-storage to v1.36.2 (#5520) * chore(deps): update dependency google-cloud-storage to v1.37.0 (#5580) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | `==1.36.2` -> `==1.37.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.37.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.37.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.37.0/compatibility-slim/1.36.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.37.0/confidence-slim/1.36.2)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/python-storage ### [`v1.37.0`](https://togithub.com/googleapis/python-storage/blob/master/CHANGELOG.md#​1370-httpswwwgithubcomgoogleapispython-storagecomparev1362v1370-2021-03-24) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v1.36.2...v1.37.0) ##### Features - add blob.open() for file-like I/O ([#​385](https://www.github.com/googleapis/python-storage/issues/385)) ([440a0a4](https://www.github.com/googleapis/python-storage/commit/440a0a4ffe00b1f7c562b0e9c1e47dbadeca33e1)), closes [#​29](https://www.github.com/googleapis/python-storage/issues/29) ##### Bug Fixes - update user_project usage and documentation in bucket/client class methods ([#​396](https://www.github.com/googleapis/python-storage/issues/396)) ([1a2734b](https://www.github.com/googleapis/python-storage/commit/1a2734ba6d316ce51e4e141571331e86196462b9)) ##### [1.36.2](https://www.github.com/googleapis/python-storage/compare/v1.36.1...v1.36.2) (2021-03-09) ##### Bug Fixes - update batch connection to request api endpoint info from client ([#​392](https://www.github.com/googleapis/python-storage/issues/392)) ([91fc6d9](https://www.github.com/googleapis/python-storage/commit/91fc6d9870a36308b15a827ed6a691e5b4669b62)) ##### [1.36.1](https://www.github.com/googleapis/python-storage/compare/v1.36.0...v1.36.1) (2021-02-19) ##### Bug Fixes - allow metadata keys to be cleared ([#​383](https://www.github.com/googleapis/python-storage/issues/383)) ([79d27da](https://www.github.com/googleapis/python-storage/commit/79d27da9fe842e44a9091076ea0ef52c5ef5ff72)), closes [#​381](https://www.github.com/googleapis/python-storage/issues/381) - allow signed url version v4 without signed credentials ([#​356](https://www.github.com/googleapis/python-storage/issues/356)) ([3e69bf9](https://www.github.com/googleapis/python-storage/commit/3e69bf92496616c5de28094dd42260b35c3bf982)) - correctly encode bytes for V2 signature ([#​382](https://www.github.com/googleapis/python-storage/issues/382)) ([f44212b](https://www.github.com/googleapis/python-storage/commit/f44212b7b91a67ca661898400fe632f9fb3ec8f6))
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-cloud-pubsub to v2.4.1 (#5610) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-pubsub](https://togithub.com/googleapis/python-pubsub) | `==2.4.0` -> `==2.4.1` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.4.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.4.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.4.1/compatibility-slim/2.4.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.4.1/confidence-slim/2.4.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/python-pubsub ### [`v2.4.1`](https://togithub.com/googleapis/python-pubsub/blob/master/CHANGELOG.md#​241-httpswwwgithubcomgoogleapispython-pubsubcomparev240v241-2021-03-30) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.4.0...2.4.1) ##### Bug Fixes - Move `await_msg_callbacks` flag to `subscribe()` method, fixing a regression in Pub/Sub Lite client. ([#​320](https://www.github.com/googleapis/python-pubsub/issues/320)) ([d40d027](https://www.github.com/googleapis/python-pubsub/commit/d40d02713c8c189937ae5c21d099b88a3131a59f)) - SSL error when using the client with the emulator. ([#​297](https://www.github.com/googleapis/python-pubsub/issues/297)) ([83db672](https://www.github.com/googleapis/python-pubsub/commit/83db67239d3521457138699109f766d574a0a2c4)) ##### Implementation Changes - (samples) Bump the max_time to 10 minutes for a flaky test. ([#​311](https://www.github.com/googleapis/python-pubsub/issues/311)) ([e2678d4](https://www.github.com/googleapis/python-pubsub/commit/e2678d47c08e6b03782d2d744a4e630b933fdd51)), closes [#​291](https://www.github.com/googleapis/python-pubsub/issues/291) - (samples) Mark delivery attempts test as flaky. ([#​326](https://www.github.com/googleapis/python-pubsub/issues/326)) ([5a97ef1](https://www.github.com/googleapis/python-pubsub/commit/5a97ef1bb7512fe814a8f72a43b3e9698434cd8d)) - (samples) Mitigate flakiness in subscriber_tests. ([#​304](https://www.github.com/googleapis/python-pubsub/issues/304)) ([271a385](https://www.github.com/googleapis/python-pubsub/commit/271a3856d835967f18f6becdae5ad53d585d0ccf)) - (samples) Retry `InternalServerError` in dead letter policy test. ([#​329](https://www.github.com/googleapis/python-pubsub/issues/329)) ([34c9b11](https://www.github.com/googleapis/python-pubsub/commit/34c9b11ae697c280f32642c3101b7f7da971f589)), closes [#​321](https://www.github.com/googleapis/python-pubsub/issues/321) ##### Documentation - Remove EXPERIMENTAL tag for ordering keys in `types.py`. ([#​323](https://www.github.com/googleapis/python-pubsub/issues/323)) ([659cd7a](https://www.github.com/googleapis/python-pubsub/commit/659cd7ae2784245d4217fbc722dac04bd3222d32)) - Remove EXPERIMENTAL tag from `Schema` service (via synth). ([#​307](https://www.github.com/googleapis/python-pubsub/issues/307)) ([ad85202](https://www.github.com/googleapis/python-pubsub/commit/ad852028836520db779c5cc33689ffd7e5458a7d))
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * docs: update description of parameters in storage_upload_file (#5707) * following Java's example https://github.com/googleapis/google-cloud-java/blob/b36db6a957bcfb7b6ccdb77fb12b4cc7fa22b807/google-cloud-examples/src/main/java/com/google/cloud/examples/storage/objects/UploadObject.java#L33-L40 * samples(storage): update storage_set_bucket_public_iam to explicitly set role and member (#5708) * chore: fix typo on noxfile (#5739) * chore: add noxfile config * chore: fix typo on noxfile * Remove "chore: add noxfile config" This reverts commit 61972125cbbf110941da1227afed53f169bad3a6. * chore: fix the base noxfile_config * fix(storage): retry flaky test (#5744) Fixes #5684 * chore(deps): update dependency google-api-python-client to v2.3.0 (#5689) * Update storage_list_files_with_prefix.py (#5747) * chore(deps): update dependency google-cloud-storage to v1.38.0 (#5640) Test failures are unrelated * chore(deps): update dependency pytest to v6.2.4 (#5787) Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com> * chore(deps): update dependency google-cloud-pubsub to v2.4.2 (#5810) Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com> * chore(deps): update dependency google-cloud-pubsub to v2.5.0 (#5845) * chore(deps): update dependency google-api-python-client to v2.4.0 (#5820) * chore(deps): update dependency google-api-python-client to v2.5.0 (#5857) Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com> * chore(deps): update dependency google-api-python-client to v2.6.0 (#5890) Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com> * chore(deps): update dependency google-api-python-client to v2.7.0 (#6062) * chore(deps): update dependency google-cloud-pubsub to v2.6.0 (#6233) * public access prevention samples & tests (#4971) * public access prevention samples & tests * linted files * respnded to PR comments * updated docstring * updated docstring * refactored fixture code * renamed samples * updated location for constants * updated location for constants * updated samples to conform to sample guidelines * added license * updated headers * Updating requirements * used f strings * linted files * f string suggestions from code review Co-authored-by: Dina Graves Portman Co-authored-by: Dina Graves Portman * chore(deps): update dependency google-api-python-client to v2.11.0 (#6101) * chore(deps): update dependency google-cloud-pubsub to v2.6.1 (#6284) * chore(deps): update dependency backoff to v1.11.0 (#6285) Co-authored-by: Dina Graves Portman * chore(deps): update dependency google-cloud-storage to v1.41.0 (#6197) * chore(deps): update dependency google-cloud-storage to v1.41.0 * revert dataflow flex templates * revert all dataflow changes * correct my mistake with dataflow stuff * restore dataflow file Co-authored-by: Leah Cole * chore(deps): update dependency google-api-python-client to v2.12.0 (#6269) * chore(deps): update dependency google-api-python-client to v2.12.0 * revert dataflow * revert dataflow Co-authored-by: Leah Cole * chore(deps): update dependency google-cloud-pubsub to v2.7.0 (#6486) Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com> Co-authored-by: Anthonios Partheniou * fix(storage): update service account email for acl tests (#6529) * fix: update test email for acl tests. previous email was deleted in the project * update to service account without project editor owner permissions * update test email to avoid creating new service accounts * docs(storage): update description in storage_download_file (#6553) * Add storage move_blob sample and fix confusion with rename (#6554) * Add storage move_blob sample and fix confusion with rename * fix license heading issues * Add descriptive comments to parameters * Update storage/cloud-client/storage_move_file.py * Apply suggestions from code review Add print statement in except clause Co-authored-by: cojenco Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency backoff to v1.11.1 (#6571) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency google-api-python-client to v2.15.0 (#6574) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-api-python-client](https://togithub.com/googleapis/google-api-python-client) | `==2.12.0` -> `==2.15.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/compatibility-slim/2.12.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/confidence-slim/2.12.0)](https://docs.renovatebot.com/merge-confidence/) | | [google-api-python-client](https://togithub.com/googleapis/google-api-python-client) | `==2.11.0` -> `==2.15.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/compatibility-slim/2.11.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/confidence-slim/2.11.0)](https://docs.renovatebot.com/merge-confidence/) | | [google-api-python-client](https://togithub.com/googleapis/google-api-python-client) | `==2.1.0` -> `==2.15.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/compatibility-slim/2.1.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.15.0/confidence-slim/2.1.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/google-api-python-client ### [`v2.15.0`](https://togithub.com/googleapis/google-api-python-client/blob/master/CHANGELOG.md#​2150-httpswwwgithubcomgoogleapisgoogle-api-python-clientcomparev2141v2150-2021-07-27) [Compare Source](https://togithub.com/googleapis/google-api-python-client/compare/v2.14.1...v2.15.0) ##### Features - **alertcenter:** update the api https://github.com/googleapis/google-api-python-client/commit/70810a52c85c6d0d6f00d7afb41c8608261eaebc ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **chat:** update the api https://github.com/googleapis/google-api-python-client/commit/a577cd0b71951176bbf849c1f7f139127205da54 ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **cloudbuild:** update the api https://github.com/googleapis/google-api-python-client/commit/9066056a8b106d441fb7686fe84359484d0d58bc ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **content:** update the api https://github.com/googleapis/google-api-python-client/commit/b123349da33c11c0172a8efb3fadef685a30e6e1 ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **displayvideo:** update the api https://github.com/googleapis/google-api-python-client/commit/c525d726ee6cffdd4bc7afd69080d5e52bae83a0 ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **dns:** update the api https://github.com/googleapis/google-api-python-client/commit/13436ccd2b835fda5cb86952ac4ea991ee8651d8 ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **eventarc:** update the api https://github.com/googleapis/google-api-python-client/commit/6be3394a64a5eb509f68ef779680fd36837708ee ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **file:** update the api https://github.com/googleapis/google-api-python-client/commit/817a0e636771445a988ef479bd52740f754b901a ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **monitoring:** update the api https://github.com/googleapis/google-api-python-client/commit/bd32149f308467f0f659119587afc77dcec65b14 ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **people:** update the api https://github.com/googleapis/google-api-python-client/commit/aa6b47df40c5289f33aef6fb6aa007df2d038e20 ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **retail:** update the api https://github.com/googleapis/google-api-python-client/commit/d39f06e2d77034bc837604a41dd52c577f158bf2 ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **securitycenter:** update the api https://github.com/googleapis/google-api-python-client/commit/999fab5178208639c9eef289f9f441052ed832fc ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **speech:** update the api https://github.com/googleapis/google-api-python-client/commit/3b2c0fa62b2a0c86bba1e97f1b18f93250dbd551 ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) - **sqladmin:** update the api https://github.com/googleapis/google-api-python-client/commit/cef24d829ab5be71563a2b668b8f6cf5dda2c8e4 ([a36e3b1](https://www.github.com/googleapis/google-api-python-client/commit/a36e3b130d609dfdc5b3ac0a70ff1b014c4bc75f)) ##### Documentation - update license to be Apache-2.0 compliant ([#​1461](https://www.togithub.com/googleapis/google-api-python-client/issues/1461)) ([882844c](https://www.github.com/googleapis/google-api-python-client/commit/882844c7b6a15493d0fb8693cd5e9159e3a12535)) ##### [2.14.1](https://www.github.com/googleapis/google-api-python-client/compare/v2.14.0...v2.14.1) (2021-07-25) ##### Bug Fixes - drop six dependency ([#​1452](https://www.togithub.com/googleapis/google-api-python-client/issues/1452)) ([9f7b410](https://www.github.com/googleapis/google-api-python-client/commit/9f7b4109b370e89c29db6c58c6bd2e09002c8d42)) ### [`v2.14.1`](https://togithub.com/googleapis/google-api-python-client/blob/master/CHANGELOG.md#​2141-httpswwwgithubcomgoogleapisgoogle-api-python-clientcomparev2140v2141-2021-07-25) [Compare Source](https://togithub.com/googleapis/google-api-python-client/compare/v2.14.0...v2.14.1) ### [`v2.14.0`](https://togithub.com/googleapis/google-api-python-client/blob/master/CHANGELOG.md#​2140-httpswwwgithubcomgoogleapisgoogle-api-python-clientcomparev2130v2140-2021-07-20) [Compare Source](https://togithub.com/googleapis/google-api-python-client/compare/v2.13.0...v2.14.0) ##### Features - **analyticsadmin:** update the api https://github.com/googleapis/google-api-python-client/commit/a2e2d768e5412072ef11891ae7fb9145e2c4693d ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **androiddeviceprovisioning:** update the api https://github.com/googleapis/google-api-python-client/commit/83151f4ebd2992a53f815133304d8cb2c72d50c5 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **chat:** update the api https://github.com/googleapis/google-api-python-client/commit/8e39e1ef5482735fbaaed3be74ee472cf44cd941 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **cloudasset:** update the api https://github.com/googleapis/google-api-python-client/commit/ebd9b97ec74f0f257ccb4833f747f88d02075926 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **cloudfunctions:** update the api https://github.com/googleapis/google-api-python-client/commit/06332af99b1b1a9894bf4f553e014936225761de ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **cloudsearch:** update the api https://github.com/googleapis/google-api-python-client/commit/4aab6137bb350cb841a6b48fd37df67a209ba031 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **content:** update the api https://github.com/googleapis/google-api-python-client/commit/c65f297a775687fbfcbae827f892fc996a3d1ab1 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **datacatalog:** update the api https://github.com/googleapis/google-api-python-client/commit/af28eef0b37a5d0bb3a299f9fd9740b63f9e23bd ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **dns:** update the api https://github.com/googleapis/google-api-python-client/commit/e2ba913fc51f78ce4b9fb6f9de97f61bd35cd356 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **documentai:** update the api https://github.com/googleapis/google-api-python-client/commit/d1b9df7ee0a041d4cf632a77a626764c37e72889 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **file:** update the api https://github.com/googleapis/google-api-python-client/commit/0cd6277980d02363e3d609901d12d62b594adc92 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **firebaseappcheck:** update the api https://github.com/googleapis/google-api-python-client/commit/f8c39017aa392c0930ab79cdf7f828fe1e97e313 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **firebasestorage:** update the api https://github.com/googleapis/google-api-python-client/commit/66b6961871fea5b1a41a5b8359d7f76d6e390386 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **gameservices:** update the api https://github.com/googleapis/google-api-python-client/commit/31fd4dc22bd1e615caeafc22482caad65bbd55e9 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **gkehub:** update the api https://github.com/googleapis/google-api-python-client/commit/58ae34d8dfb4a7827b4f56e99fd48dedc64b4364 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **ml:** update the api https://github.com/googleapis/google-api-python-client/commit/15e0de32f2ea94d6ed3e0c18cd6e59cc239b37e7 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **monitoring:** update the api https://github.com/googleapis/google-api-python-client/commit/2b52d9ff5341caec20577538c0c4eaf83a896651 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **notebooks:** update the api https://github.com/googleapis/google-api-python-client/commit/c4698a84e526ab47710d2bde22827b337f2f480c ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **people:** update the api https://github.com/googleapis/google-api-python-client/commit/a646e56d40f2c7df40f48d42442c1941fc1c6674 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **recommender:** update the api https://github.com/googleapis/google-api-python-client/commit/ef997b0293c0e075208c7af15fa4e9bd6f29e883 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **secretmanager:** update the api https://github.com/googleapis/google-api-python-client/commit/489541e760eae9745724eb8cad74007903dd4f5b ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **spanner:** update the api https://github.com/googleapis/google-api-python-client/commit/acdb8fccfbb9f243f06dfff68d61cee2e58c9e45 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) - **testing:** update the api https://github.com/googleapis/google-api-python-client/commit/e2bde192a3e20ebd00995185cd92b47a086be8d9 ([0770807](https://www.github.com/googleapis/google-api-python-client/commit/0770807d690618cb51196d2c1ef812a8e0c03115)) ##### Bug Fixes - **deps:** pin 'google-{api,cloud}-core', 'google-auth' to allow 2.x versions ([#​1449](https://www.togithub.com/googleapis/google-api-python-client/issues/1449)) ([96d2bb0](https://www.github.com/googleapis/google-api-python-client/commit/96d2bb04f99d57f0fff2b81e8f4e792782deb712)) ### [`v2.13.0`](https://togithub.com/googleapis/google-api-python-client/blob/master/CHANGELOG.md#​2130-httpswwwgithubcomgoogleapisgoogle-api-python-clientcomparev2120v2130-2021-07-13) [Compare Source](https://togithub.com/googleapis/google-api-python-client/compare/v2.12.0...v2.13.0) ##### Features - **analyticsadmin:** update the api https://github.com/googleapis/google-api-python-client/commit/96675a8d9158ec13353fe241f858201fc51b784d ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **composer:** update the api https://github.com/googleapis/google-api-python-client/commit/add2fbdc3afb6696537eb087bc1d79df9194a37a ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **container:** update the api https://github.com/googleapis/google-api-python-client/commit/f8fae98db6d1943411b1a6c0f5a65dea336569f6 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **content:** update the api https://github.com/googleapis/google-api-python-client/commit/0814e009a4a11800db5b4afd7b6260e504c98047 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **datacatalog:** update the api https://github.com/googleapis/google-api-python-client/commit/99706059e58bb3d616253a1af2cd162b5a0b0279 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **dataflow:** update the api https://github.com/googleapis/google-api-python-client/commit/d5f09ef30392532bcfdd82901148bdd3ac6eec01 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **docs:** update the api https://github.com/googleapis/google-api-python-client/commit/dc66f4cafba86baff6149b2f6e59ae1888006911 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **file:** update the api https://github.com/googleapis/google-api-python-client/commit/523fc5c900f53489d56400deb650f6586c9681a0 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **firebasehosting:** update the api https://github.com/googleapis/google-api-python-client/commit/c83ac386b65f82e7ba29851d56b496b09a29cf98 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **healthcare:** update the api https://github.com/googleapis/google-api-python-client/commit/a407471b14349b8c08018196041568f2a35f8d4f ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **ideahub:** update the api https://github.com/googleapis/google-api-python-client/commit/c6b0d83940f238b1330896240492e8db397dcd15 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **managedidentities:** update the api https://github.com/googleapis/google-api-python-client/commit/863b333da7848029fd1614fd48b46cfbe12afcd5 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **memcache:** update the api https://github.com/googleapis/google-api-python-client/commit/17dc001e4649f54944066ce153e3c552c850a146 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **metastore:** update the api https://github.com/googleapis/google-api-python-client/commit/f3a76c9359babc48cc0b76ce7e3be0711ba028ae ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **slides:** update the api https://github.com/googleapis/google-api-python-client/commit/314d61b9ef8c5c30f9756462504dc0df92284cb2 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **sqladmin:** update the api https://github.com/googleapis/google-api-python-client/commit/62784e0b1b5752b480afe1ddd77dcf412bb35dbb ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **tpu:** update the api https://github.com/googleapis/google-api-python-client/commit/16bf712cca4a393d96e4135de3d02e5005051b6d ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) - **youtube:** update the api https://github.com/googleapis/google-api-python-client/commit/ec21dff96d9538ad6c7f9b318eca88178533aa95 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) ##### Bug Fixes - **keep:** update the api https://github.com/googleapis/google-api-python-client/commit/08fee732e96d3220e624c8fca7b8a9b0c0bcb146 ([1a4514d](https://www.github.com/googleapis/google-api-python-client/commit/1a4514d2862f81fc97e424cd550c286cda0fc859)) ##### Documentation - add recommendation to use v2.x and static discovery artifacts ([#​1434](https://www.togithub.com/googleapis/google-api-python-client/issues/1434)) ([ca7328c](https://www.github.com/googleapis/google-api-python-client/commit/ca7328cb5340ea282a3d98782926a0b6881a33ed))
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-cloud-storage to v1.42.0 (#6576) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | `==1.41.0` -> `==1.42.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.0/compatibility-slim/1.41.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.0/confidence-slim/1.41.0)](https://docs.renovatebot.com/merge-confidence/) | | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | `==1.38.0` -> `==1.42.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.0/compatibility-slim/1.38.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.0/confidence-slim/1.38.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/python-storage ### [`v1.42.0`](https://togithub.com/googleapis/python-storage/blob/master/CHANGELOG.md#​1420-httpswwwgithubcomgoogleapispython-storagecomparev1411v1420-2021-08-05) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v1.41.1...v1.42.0) ##### Features - add 'page_size' parameter to 'Bucket.list_blobs, list_buckets ([#​520](https://www.togithub.com/googleapis/python-storage/issues/520)) ([c5f4ad8](https://www.github.com/googleapis/python-storage/commit/c5f4ad8fddd1849a4229b0126c4c022bccb90128)) ##### Bug Fixes - **deps:** add explicit ranges for 'google-api-core' and 'google-auth' ([#​530](https://www.togithub.com/googleapis/python-storage/issues/530)) ([310f207](https://www.github.com/googleapis/python-storage/commit/310f207411da0382af310172344f19c644c14e6a)) - downloading no longer marks metadata fields as 'changed' ([#​523](https://www.togithub.com/googleapis/python-storage/issues/523)) ([160d1ec](https://www.github.com/googleapis/python-storage/commit/160d1ecb41f1f269b25cb68b2d2f7daf418bf01c)) - make 'requests.exceptions.ChunkedEncodingError retryable by default ([#​526](https://www.togithub.com/googleapis/python-storage/issues/526)) ([4abb403](https://www.github.com/googleapis/python-storage/commit/4abb40310eca7ec45afc4bc5e4dfafbe083e74d2)) ##### Documentation - update supported / removed Python versions in README ([#​519](https://www.togithub.com/googleapis/python-storage/issues/519)) ([1f1b138](https://www.github.com/googleapis/python-storage/commit/1f1b138865fb171535ee0cf768aff1987ed58914)) ##### [1.41.1](https://www.github.com/googleapis/python-storage/compare/v1.41.0...v1.41.1) (2021-07-20) ##### Bug Fixes - **deps:** pin `{api,cloud}-core`, `auth` to allow 2.x versions on Python 3 ([#​512](https://www.togithub.com/googleapis/python-storage/issues/512)) ([4d7500e](https://www.github.com/googleapis/python-storage/commit/4d7500e39c51efd817b8363b69c88be040f3edb8)) - remove trailing commas from error message constants ([#​505](https://www.togithub.com/googleapis/python-storage/issues/505)) ([d4a86ce](https://www.github.com/googleapis/python-storage/commit/d4a86ceb7a7c5e00ba7bae37c7078d52478040ff)), closes [#​501](https://www.togithub.com/googleapis/python-storage/issues/501) ##### Documentation - replace usage of deprecated function `download_as_string` in docs ([#​508](https://www.togithub.com/googleapis/python-storage/issues/508)) ([8dfa4d4](https://www.github.com/googleapis/python-storage/commit/8dfa4d429dce94b671dc3e3755e52ab82733f61a)) ### [`v1.41.1`](https://togithub.com/googleapis/python-storage/blob/master/CHANGELOG.md#​1411-httpswwwgithubcomgoogleapispython-storagecomparev1410v1411-2021-07-20) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v1.41.0...v1.41.1)
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-api-python-client to v2.17.0 (#6586) * chore: Review the language fixes. (#6591) * Update main.py Fixed formatting changes. * Update storage_create_hmac_key.py * Update storage_add_bucket_iam_member.py Fixed minor language issues. * chore(deps): update dependency google-cloud-pubsub to v2.7.1 (#6598) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-pubsub](https://togithub.com/googleapis/python-pubsub) | `==2.7.0` -> `==2.7.1` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.7.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.7.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.7.1/compatibility-slim/2.7.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-pubsub/2.7.1/confidence-slim/2.7.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/python-pubsub ### [`v2.7.1`](https://togithub.com/googleapis/python-pubsub/blob/master/CHANGELOG.md#​271-httpswwwgithubcomgoogleapispython-pubsubcomparev270v271-2021-08-13) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.7.0...v2.7.1)
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-api-python-client to v2.18.0 (#6606) * chore(deps): update dependency google-api-python-client to v2.19.0 (#6642) * chore(deps): update dependency google-cloud-pubsub to v2.8.0 (#6664) * chore(deps): update dependency google-api-python-client to v2.19.1 (#6656) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-api-python-client](https://togithub.com/googleapis/google-api-python-client) | `==2.19.0` -> `==2.19.1` | [![age](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.19.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.19.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.19.1/compatibility-slim/2.19.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.19.1/confidence-slim/2.19.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/google-api-python-client ### [`v2.19.1`](https://togithub.com/googleapis/google-api-python-client/blob/master/CHANGELOG.md#​2191-httpswwwgithubcomgoogleapisgoogle-api-python-clientcomparev2190v2191-2021-09-02) [Compare Source](https://togithub.com/googleapis/google-api-python-client/compare/v2.19.0...v2.19.1)
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-cloud-storage to v1.42.1 (#6677) * chore(deps): update dependency google-api-python-client to v2.20.0 (#6675) * chore(deps): update dependency google-cloud-storage to v1.42.2 (#6700) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | `==1.42.1` -> `==1.42.2` | [![age](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.2/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.2/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.2/compatibility-slim/1.42.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-cloud-storage/1.42.2/confidence-slim/1.42.1)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/python-storage ### [`v1.42.2`](https://togithub.com/googleapis/python-storage/blob/master/CHANGELOG.md#​1422-httpswwwgithubcomgoogleapispython-storagecomparev1421v1422-2021-09-16) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v1.42.1...v1.42.2)
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-api-python-client to v2.21.0 (#6689) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-api-python-client](https://togithub.com/googleapis/google-api-python-client) | `==2.20.0` -> `==2.21.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.21.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.21.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.21.0/compatibility-slim/2.20.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.21.0/confidence-slim/2.20.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/google-api-python-client ### [`v2.21.0`](https://togithub.com/googleapis/google-api-python-client/blob/master/CHANGELOG.md#​2210-httpswwwgithubcomgoogleapisgoogle-api-python-clientcomparev2200v2210-2021-09-14) [Compare Source](https://togithub.com/googleapis/google-api-python-client/compare/v2.20.0...v2.21.0) ##### Features - **apigee:** update the api https://github.com/googleapis/google-api-python-client/commit/0e4fed7f1e08a616cbc81243c24391bc20ce5edb ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **bigquery:** update the api https://github.com/googleapis/google-api-python-client/commit/04e112ce89d6ddb3aeaae889c2de36070d6c2814 ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **bigtableadmin:** update the api https://github.com/googleapis/google-api-python-client/commit/6b77931c3c9aba59d5b326c570a2080252c8beb1 ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **cloudprofiler:** update the api https://github.com/googleapis/google-api-python-client/commit/3009ee3c238ae1fa51c529e9f187ec26693138a4 ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **container:** update the api https://github.com/googleapis/google-api-python-client/commit/e5d01ecee51da0c7a2543b833a1395a94c27bef6 ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **dataproc:** update the api https://github.com/googleapis/google-api-python-client/commit/fec73562a93b5a532bce6c91f0d30ec4fbd54ddb ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **displayvideo:** update the api https://github.com/googleapis/google-api-python-client/commit/22caa4f2f8ecb0f0ad6cfac547f9deb76fdcbebb ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **documentai:** update the api https://github.com/googleapis/google-api-python-client/commit/444836b9afe8d3eb8d52a1431652bfda1ad3288b ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **healthcare:** update the api https://github.com/googleapis/google-api-python-client/commit/2f3173aa4b4d154c909eea853a0c4c306834e0ab ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **ideahub:** update the api https://github.com/googleapis/google-api-python-client/commit/8ebf9d2bd419561d5eacb78823aa1fc519fe2710 ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **memcache:** update the api https://github.com/googleapis/google-api-python-client/commit/393dce7a3e584ad6be58c832ec826fe3b44e353b ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **mybusinesslodging:** update the api https://github.com/googleapis/google-api-python-client/commit/c51a0d15e634c2ab1c7762533f33d59f10b01875 ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **speech:** update the api https://github.com/googleapis/google-api-python-client/commit/bf6e86f6ee8c3985e1ce6f0475ef4f8685b52060 ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **streetviewpublish:** update the api https://github.com/googleapis/google-api-python-client/commit/c8cf30cd67f5588d7cbe60631e42f0a49ea6c307 ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea)) - **youtube:** update the api https://github.com/googleapis/google-api-python-client/commit/855cbfea1f6d46af07c4b80ab26fc30ca46370b7 ([e5e87b1](https://www.github.com/googleapis/google-api-python-client/commit/e5e87b1ca5fb6e81f6d83d970c3e4f683ecdcdea))
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-api-python-client to v2.22.0 (#6716) * fix(storage): skip test temporarily before pap changes release (#6750) * fix(storage): add time for bucket patch changes to propagate (#6752) * fix(storage): add backoff to retry flaky test (#6745) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency google-api-python-client to v2.23.0 (#6754) Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> * fix: update samples for pap, unspecified -> inherited (#6757) * fix: update samples for pap, unspecified->inherited * restore original unspecified while adding new inherited * woops, missed a place * remove test skip #6750 * lint fix * oops * Update storage/cloud-client/storage_set_public_access_prevention_inherited.py Co-authored-by: cojenco * add top level comment * move comment above regionalization tag * update version * change over unspecified test to inherited * this one too * skip inconsistent tests Co-authored-by: cojenco * chore(deps): update dependency google-api-python-client to v2.24.0 (#6795) * fix(storage): add py-3.10 configs to noxfile config (#6903) * fix(storage): update noxfile_config to add py-3.10 configs * update service account email configs * chore(deps): update dependency google-api-python-client to v2.25.0 (#6901) [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [google-api-python-client](https://togithub.com/googleapis/google-api-python-client) | `==2.24.0` -> `==2.25.0` | [![age](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.25.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.25.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.25.0/compatibility-slim/2.24.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-api-python-client/2.25.0/confidence-slim/2.24.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
googleapis/google-api-python-client ### [`v2.25.0`](https://togithub.com/googleapis/google-api-python-client/blob/master/CHANGELOG.md#​2250-httpswwwgithubcomgoogleapisgoogle-api-python-clientcomparev2240v2250-2021-10-09) [Compare Source](https://togithub.com/googleapis/google-api-python-client/compare/v2.24.0...v2.25.0) ##### Features - enable self signed jwt for service account credentials ([#​1553](https://www.togithub.com/googleapis/google-api-python-client/issues/1553)) ([1fb3c8e](https://www.github.com/googleapis/google-api-python-client/commit/1fb3c8ec61295adc876fa449e92fe5d682f33cbd))
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples). * feat(storage): add retry configurations sample and test (#6900) * storage: add configure_retries sample * add test for storage_configure_retries sample * revise comments per discussion with TW * update comment * address comments add configs demo * samples: add cloud client samples from python-docs-samples * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * add noxfile * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * remove invalid unused region tags Co-authored-by: Jon Wayne Parrott Co-authored-by: DPE bot Co-authored-by: Jason Dobry Co-authored-by: ryanmats Co-authored-by: BrandonY Co-authored-by: Bill Prin Co-authored-by: michaelawyu Co-authored-by: Frank Natividad Co-authored-by: Jeffrey Rennie Co-authored-by: Chris Broadfoot Co-authored-by: michaelawyu Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> Co-authored-by: Alix Hamilton Co-authored-by: Billy Jacobson Co-authored-by: Charles Engelke Co-authored-by: Charles Engelke Co-authored-by: Jonathan Lui Co-authored-by: John Whitlock Co-authored-by: Gus Class Co-authored-by: Chris Cotter Co-authored-by: JesseLovelace <43148100+JesseLovelace@users.noreply.github.com> Co-authored-by: Doug Mahugh Co-authored-by: Christopher Wilcox Co-authored-by: Jake Stambaugh Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Co-authored-by: WhiteSource Renovate Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: Leah Cole Co-authored-by: Antonio Matarrese Co-authored-by: Takashi Matsuo Co-authored-by: HemangChothani <50404902+HemangChothani@users.noreply.github.com> Co-authored-by: gcf-merge-on-green[bot] <60162190+gcf-merge-on-green[bot]@users.noreply.github.com> Co-authored-by: Ace Nassri Co-authored-by: Dina Graves Portman Co-authored-by: Shivaji Dutta Co-authored-by: Sarah Spikes Co-authored-by: Aldo D'Aquino Co-authored-by: BenWhitehead Co-authored-by: Dan Lee <71398022+dandhlee@users.noreply.github.com> Co-authored-by: Sameena Shaffeeullah Co-authored-by: Anthonios Partheniou Co-authored-by: Aaron Gabriel Neyer Co-authored-by: pallabiwrites <87546424+pallabiwrites@users.noreply.github.com> Co-authored-by: Owl Bot --- storage/samples/snippets/README.md | 10 + storage/samples/snippets/acl_test.py | 172 ++++++ storage/samples/snippets/bucket_lock_test.py | 176 ++++++ storage/samples/snippets/conftest.py | 41 ++ storage/samples/snippets/encryption_test.py | 125 +++++ storage/samples/snippets/hmac_samples_test.py | 121 +++++ storage/samples/snippets/iam_test.py | 146 +++++ .../samples/snippets/notification_polling.py | 137 +++++ .../snippets/notification_polling_test.py | 55 ++ storage/samples/snippets/noxfile.py | 270 +++++++++ storage/samples/snippets/noxfile_config.py | 96 ++++ .../snippets/public_access_prevention_test.py | 50 ++ storage/samples/snippets/quickstart.py | 37 ++ storage/samples/snippets/quickstart_test.py | 28 + .../samples/snippets/requester_pays_test.py | 65 +++ .../samples/snippets/requirements-test.txt | 3 + storage/samples/snippets/requirements.txt | 3 + storage/samples/snippets/snippets_test.py | 511 ++++++++++++++++++ .../snippets/storage_activate_hmac_key.py | 53 ++ ...rage_add_bucket_conditional_iam_binding.py | 78 +++ .../storage_add_bucket_default_owner.py | 51 ++ .../snippets/storage_add_bucket_iam_member.py | 45 ++ .../snippets/storage_add_bucket_label.py | 46 ++ .../snippets/storage_add_bucket_owner.py | 52 ++ .../snippets/storage_add_file_owner.py | 54 ++ .../storage_bucket_delete_default_kms_key.py | 40 ++ .../storage_change_default_storage_class.py | 41 ++ .../storage_change_file_storage_class.py | 46 ++ .../samples/snippets/storage_compose_file.py | 55 ++ .../snippets/storage_configure_retries.py | 63 +++ storage/samples/snippets/storage_copy_file.py | 60 ++ .../storage_copy_file_archived_generation.py | 63 +++ .../snippets/storage_cors_configuration.py | 48 ++ .../samples/snippets/storage_create_bucket.py | 37 ++ .../storage_create_bucket_class_location.py | 44 ++ .../snippets/storage_create_hmac_key.py | 53 ++ .../snippets/storage_deactivate_hmac_key.py | 54 ++ ...age_define_bucket_website_configuration.py | 50 ++ .../samples/snippets/storage_delete_bucket.py | 38 ++ .../samples/snippets/storage_delete_file.py | 40 ++ ...storage_delete_file_archived_generation.py | 48 ++ .../snippets/storage_delete_hmac_key.py | 47 ++ ...age_disable_bucket_lifecycle_management.py | 41 ++ ...torage_disable_default_event_based_hold.py | 40 ++ .../storage_disable_requester_pays.py | 40 ++ ...age_disable_uniform_bucket_level_access.py | 41 ++ .../snippets/storage_disable_versioning.py | 40 ++ .../storage_download_encrypted_file.py | 69 +++ .../samples/snippets/storage_download_file.py | 59 ++ .../storage_download_file_requester_pays.py | 53 ++ .../snippets/storage_download_public_file.py | 49 ++ ...rage_enable_bucket_lifecycle_management.py | 45 ++ ...storage_enable_default_event_based_hold.py | 40 ++ .../snippets/storage_enable_requester_pays.py | 39 ++ ...rage_enable_uniform_bucket_level_access.py | 41 ++ .../snippets/storage_enable_versioning.py | 40 ++ .../storage_generate_encryption_key.py | 39 ++ .../storage_generate_signed_post_policy_v4.py | 65 +++ .../storage_generate_signed_url_v2.py | 54 ++ .../storage_generate_signed_url_v4.py | 61 +++ .../storage_generate_upload_signed_url_v4.py | 66 +++ .../snippets/storage_get_bucket_labels.py | 41 ++ .../snippets/storage_get_bucket_metadata.py | 57 ++ .../storage_get_default_event_based_hold.py | 45 ++ .../samples/snippets/storage_get_hmac_key.py | 51 ++ .../samples/snippets/storage_get_metadata.py | 72 +++ .../storage_get_public_access_prevention.py | 40 ++ .../storage_get_requester_pays_status.py | 40 ++ .../snippets/storage_get_retention_policy.py | 46 ++ .../snippets/storage_get_service_account.py | 37 ++ ...storage_get_uniform_bucket_level_access.py | 53 ++ .../samples/snippets/storage_list_buckets.py | 35 ++ .../storage_list_file_archived_generations.py | 39 ++ .../samples/snippets/storage_list_files.py | 40 ++ .../storage_list_files_with_prefix.py | 71 +++ .../snippets/storage_list_hmac_keys.py | 43 ++ .../snippets/storage_lock_retention_policy.py | 48 ++ .../samples/snippets/storage_make_public.py | 44 ++ storage/samples/snippets/storage_move_file.py | 63 +++ .../snippets/storage_object_csek_to_cmek.py | 59 ++ .../snippets/storage_object_get_kms_key.py | 42 ++ .../snippets/storage_print_bucket_acl.py | 36 ++ .../storage_print_bucket_acl_for_user.py | 41 ++ .../snippets/storage_print_file_acl.py | 37 ++ .../storage_print_file_acl_for_user.py | 45 ++ .../storage_release_event_based_hold.py | 43 ++ .../storage_release_temporary_hold.py | 43 ++ ...e_remove_bucket_conditional_iam_binding.py | 67 +++ .../storage_remove_bucket_default_owner.py | 54 ++ .../storage_remove_bucket_iam_member.py | 49 ++ .../snippets/storage_remove_bucket_label.py | 49 ++ .../snippets/storage_remove_bucket_owner.py | 47 ++ .../storage_remove_cors_configuration.py | 39 ++ .../snippets/storage_remove_file_owner.py | 53 ++ .../storage_remove_retention_policy.py | 47 ++ .../samples/snippets/storage_rename_file.py | 44 ++ .../snippets/storage_rotate_encryption_key.py | 66 +++ .../storage_set_bucket_default_kms_key.py | 43 ++ .../snippets/storage_set_bucket_public_iam.py | 45 ++ .../snippets/storage_set_event_based_hold.py | 42 ++ .../samples/snippets/storage_set_metadata.py | 41 ++ ...e_set_public_access_prevention_enforced.py | 43 ++ ..._set_public_access_prevention_inherited.py | 50 ++ ...et_public_access_prevention_unspecified.py | 43 ++ .../snippets/storage_set_retention_policy.py | 45 ++ .../snippets/storage_set_temporary_hold.py | 42 ++ .../snippets/storage_upload_encrypted_file.py | 68 +++ .../samples/snippets/storage_upload_file.py | 52 ++ .../snippets/storage_upload_with_kms_key.py | 52 ++ .../storage_view_bucket_iam_members.py | 40 ++ .../uniform_bucket_level_access_test.py | 52 ++ 111 files changed, 6558 insertions(+) create mode 100644 storage/samples/snippets/README.md create mode 100644 storage/samples/snippets/acl_test.py create mode 100644 storage/samples/snippets/bucket_lock_test.py create mode 100644 storage/samples/snippets/conftest.py create mode 100644 storage/samples/snippets/encryption_test.py create mode 100644 storage/samples/snippets/hmac_samples_test.py create mode 100644 storage/samples/snippets/iam_test.py create mode 100644 storage/samples/snippets/notification_polling.py create mode 100644 storage/samples/snippets/notification_polling_test.py create mode 100644 storage/samples/snippets/noxfile.py create mode 100644 storage/samples/snippets/noxfile_config.py create mode 100644 storage/samples/snippets/public_access_prevention_test.py create mode 100644 storage/samples/snippets/quickstart.py create mode 100644 storage/samples/snippets/quickstart_test.py create mode 100644 storage/samples/snippets/requester_pays_test.py create mode 100644 storage/samples/snippets/requirements-test.txt create mode 100644 storage/samples/snippets/requirements.txt create mode 100644 storage/samples/snippets/snippets_test.py create mode 100644 storage/samples/snippets/storage_activate_hmac_key.py create mode 100644 storage/samples/snippets/storage_add_bucket_conditional_iam_binding.py create mode 100644 storage/samples/snippets/storage_add_bucket_default_owner.py create mode 100644 storage/samples/snippets/storage_add_bucket_iam_member.py create mode 100644 storage/samples/snippets/storage_add_bucket_label.py create mode 100644 storage/samples/snippets/storage_add_bucket_owner.py create mode 100644 storage/samples/snippets/storage_add_file_owner.py create mode 100644 storage/samples/snippets/storage_bucket_delete_default_kms_key.py create mode 100644 storage/samples/snippets/storage_change_default_storage_class.py create mode 100644 storage/samples/snippets/storage_change_file_storage_class.py create mode 100644 storage/samples/snippets/storage_compose_file.py create mode 100644 storage/samples/snippets/storage_configure_retries.py create mode 100644 storage/samples/snippets/storage_copy_file.py create mode 100644 storage/samples/snippets/storage_copy_file_archived_generation.py create mode 100644 storage/samples/snippets/storage_cors_configuration.py create mode 100644 storage/samples/snippets/storage_create_bucket.py create mode 100644 storage/samples/snippets/storage_create_bucket_class_location.py create mode 100644 storage/samples/snippets/storage_create_hmac_key.py create mode 100644 storage/samples/snippets/storage_deactivate_hmac_key.py create mode 100644 storage/samples/snippets/storage_define_bucket_website_configuration.py create mode 100644 storage/samples/snippets/storage_delete_bucket.py create mode 100644 storage/samples/snippets/storage_delete_file.py create mode 100644 storage/samples/snippets/storage_delete_file_archived_generation.py create mode 100644 storage/samples/snippets/storage_delete_hmac_key.py create mode 100644 storage/samples/snippets/storage_disable_bucket_lifecycle_management.py create mode 100644 storage/samples/snippets/storage_disable_default_event_based_hold.py create mode 100644 storage/samples/snippets/storage_disable_requester_pays.py create mode 100644 storage/samples/snippets/storage_disable_uniform_bucket_level_access.py create mode 100644 storage/samples/snippets/storage_disable_versioning.py create mode 100644 storage/samples/snippets/storage_download_encrypted_file.py create mode 100644 storage/samples/snippets/storage_download_file.py create mode 100644 storage/samples/snippets/storage_download_file_requester_pays.py create mode 100644 storage/samples/snippets/storage_download_public_file.py create mode 100644 storage/samples/snippets/storage_enable_bucket_lifecycle_management.py create mode 100644 storage/samples/snippets/storage_enable_default_event_based_hold.py create mode 100644 storage/samples/snippets/storage_enable_requester_pays.py create mode 100644 storage/samples/snippets/storage_enable_uniform_bucket_level_access.py create mode 100644 storage/samples/snippets/storage_enable_versioning.py create mode 100644 storage/samples/snippets/storage_generate_encryption_key.py create mode 100644 storage/samples/snippets/storage_generate_signed_post_policy_v4.py create mode 100644 storage/samples/snippets/storage_generate_signed_url_v2.py create mode 100644 storage/samples/snippets/storage_generate_signed_url_v4.py create mode 100644 storage/samples/snippets/storage_generate_upload_signed_url_v4.py create mode 100644 storage/samples/snippets/storage_get_bucket_labels.py create mode 100644 storage/samples/snippets/storage_get_bucket_metadata.py create mode 100644 storage/samples/snippets/storage_get_default_event_based_hold.py create mode 100644 storage/samples/snippets/storage_get_hmac_key.py create mode 100644 storage/samples/snippets/storage_get_metadata.py create mode 100644 storage/samples/snippets/storage_get_public_access_prevention.py create mode 100644 storage/samples/snippets/storage_get_requester_pays_status.py create mode 100644 storage/samples/snippets/storage_get_retention_policy.py create mode 100644 storage/samples/snippets/storage_get_service_account.py create mode 100644 storage/samples/snippets/storage_get_uniform_bucket_level_access.py create mode 100644 storage/samples/snippets/storage_list_buckets.py create mode 100644 storage/samples/snippets/storage_list_file_archived_generations.py create mode 100644 storage/samples/snippets/storage_list_files.py create mode 100644 storage/samples/snippets/storage_list_files_with_prefix.py create mode 100644 storage/samples/snippets/storage_list_hmac_keys.py create mode 100644 storage/samples/snippets/storage_lock_retention_policy.py create mode 100644 storage/samples/snippets/storage_make_public.py create mode 100644 storage/samples/snippets/storage_move_file.py create mode 100644 storage/samples/snippets/storage_object_csek_to_cmek.py create mode 100644 storage/samples/snippets/storage_object_get_kms_key.py create mode 100644 storage/samples/snippets/storage_print_bucket_acl.py create mode 100644 storage/samples/snippets/storage_print_bucket_acl_for_user.py create mode 100644 storage/samples/snippets/storage_print_file_acl.py create mode 100644 storage/samples/snippets/storage_print_file_acl_for_user.py create mode 100644 storage/samples/snippets/storage_release_event_based_hold.py create mode 100644 storage/samples/snippets/storage_release_temporary_hold.py create mode 100644 storage/samples/snippets/storage_remove_bucket_conditional_iam_binding.py create mode 100644 storage/samples/snippets/storage_remove_bucket_default_owner.py create mode 100644 storage/samples/snippets/storage_remove_bucket_iam_member.py create mode 100644 storage/samples/snippets/storage_remove_bucket_label.py create mode 100644 storage/samples/snippets/storage_remove_bucket_owner.py create mode 100644 storage/samples/snippets/storage_remove_cors_configuration.py create mode 100644 storage/samples/snippets/storage_remove_file_owner.py create mode 100644 storage/samples/snippets/storage_remove_retention_policy.py create mode 100644 storage/samples/snippets/storage_rename_file.py create mode 100644 storage/samples/snippets/storage_rotate_encryption_key.py create mode 100644 storage/samples/snippets/storage_set_bucket_default_kms_key.py create mode 100644 storage/samples/snippets/storage_set_bucket_public_iam.py create mode 100644 storage/samples/snippets/storage_set_event_based_hold.py create mode 100644 storage/samples/snippets/storage_set_metadata.py create mode 100644 storage/samples/snippets/storage_set_public_access_prevention_enforced.py create mode 100644 storage/samples/snippets/storage_set_public_access_prevention_inherited.py create mode 100644 storage/samples/snippets/storage_set_public_access_prevention_unspecified.py create mode 100644 storage/samples/snippets/storage_set_retention_policy.py create mode 100644 storage/samples/snippets/storage_set_temporary_hold.py create mode 100644 storage/samples/snippets/storage_upload_encrypted_file.py create mode 100644 storage/samples/snippets/storage_upload_file.py create mode 100644 storage/samples/snippets/storage_upload_with_kms_key.py create mode 100644 storage/samples/snippets/storage_view_bucket_iam_members.py create mode 100644 storage/samples/snippets/uniform_bucket_level_access_test.py diff --git a/storage/samples/snippets/README.md b/storage/samples/snippets/README.md new file mode 100644 index 00000000000..3d7e3664f58 --- /dev/null +++ b/storage/samples/snippets/README.md @@ -0,0 +1,10 @@ + +For requester_pays_test.py, we need to use a different Storage bucket. + +The test looks for an environment variable `REQUESTER_PAYS_TEST_BUCKET`. + +Also, the service account for the test needs to have `Billing Project +Manager` role in order to make changes on buckets with requester pays +enabled. + +We added that role to the test service account. diff --git a/storage/samples/snippets/acl_test.py b/storage/samples/snippets/acl_test.py new file mode 100644 index 00000000000..fd2088ad638 --- /dev/null +++ b/storage/samples/snippets/acl_test.py @@ -0,0 +1,172 @@ +# Copyright 2016 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import uuid + +import backoff +from google.cloud import storage +from googleapiclient.errors import HttpError +import pytest + +import storage_add_bucket_default_owner +import storage_add_bucket_owner +import storage_add_file_owner +import storage_print_bucket_acl +import storage_print_bucket_acl_for_user +import storage_print_file_acl +import storage_print_file_acl_for_user +import storage_remove_bucket_default_owner +import storage_remove_bucket_owner +import storage_remove_file_owner + +# Typically we'd use a @example.com address, but GCS requires a real Google +# account. Retrieve a service account email with storage admin permissions. +TEST_EMAIL = ( + "py38-storage-test" + "@python-docs-samples-tests.iam.gserviceaccount.com" +) + + +@pytest.fixture(scope="module") +def test_bucket(): + """Yields a bucket that is deleted after the test completes.""" + + # The new projects have uniform bucket-level access and our tests don't + # pass with those buckets. We need to use the old main project for now. + original_value = os.environ['GOOGLE_CLOUD_PROJECT'] + os.environ['GOOGLE_CLOUD_PROJECT'] = os.environ['MAIN_GOOGLE_CLOUD_PROJECT'] + bucket = None + while bucket is None or bucket.exists(): + bucket_name = "acl-test-{}".format(uuid.uuid4()) + bucket = storage.Client().bucket(bucket_name) + bucket.create() + yield bucket + bucket.delete(force=True) + # Set the value back. + os.environ['GOOGLE_CLOUD_PROJECT'] = original_value + + +@pytest.fixture +def test_blob(test_bucket): + """Yields a blob that is deleted after the test completes.""" + bucket = test_bucket + blob = bucket.blob("storage_acl_test_sigil-{}".format(uuid.uuid4())) + blob.upload_from_string("Hello, is it me you're looking for?") + yield blob + + +def test_print_bucket_acl(test_bucket, capsys): + storage_print_bucket_acl.print_bucket_acl(test_bucket.name) + out, _ = capsys.readouterr() + assert out + + +def test_print_bucket_acl_for_user(test_bucket, capsys): + test_bucket.acl.user(TEST_EMAIL).grant_owner() + test_bucket.acl.save() + + storage_print_bucket_acl_for_user.print_bucket_acl_for_user( + test_bucket.name, TEST_EMAIL + ) + + out, _ = capsys.readouterr() + assert "OWNER" in out + + +@backoff.on_exception(backoff.expo, HttpError, max_time=60) +def test_add_bucket_owner(test_bucket): + storage_add_bucket_owner.add_bucket_owner(test_bucket.name, TEST_EMAIL) + + test_bucket.acl.reload() + assert "OWNER" in test_bucket.acl.user(TEST_EMAIL).get_roles() + + +@backoff.on_exception(backoff.expo, HttpError, max_time=60) +def test_remove_bucket_owner(test_bucket): + test_bucket.acl.user(TEST_EMAIL).grant_owner() + test_bucket.acl.save() + + storage_remove_bucket_owner.remove_bucket_owner( + test_bucket.name, TEST_EMAIL) + + test_bucket.acl.reload() + assert "OWNER" not in test_bucket.acl.user(TEST_EMAIL).get_roles() + + +@backoff.on_exception(backoff.expo, HttpError, max_time=60) +def test_add_bucket_default_owner(test_bucket): + storage_add_bucket_default_owner.add_bucket_default_owner( + test_bucket.name, TEST_EMAIL + ) + + test_bucket.default_object_acl.reload() + roles = test_bucket.default_object_acl.user(TEST_EMAIL).get_roles() + assert "OWNER" in roles + + +@backoff.on_exception(backoff.expo, HttpError, max_time=60) +def test_remove_bucket_default_owner(test_bucket): + test_bucket.acl.user(TEST_EMAIL).grant_owner() + test_bucket.acl.save() + + storage_remove_bucket_default_owner.remove_bucket_default_owner( + test_bucket.name, TEST_EMAIL + ) + + test_bucket.default_object_acl.reload() + roles = test_bucket.default_object_acl.user(TEST_EMAIL).get_roles() + assert "OWNER" not in roles + + +def test_print_blob_acl(test_blob, capsys): + storage_print_file_acl.print_blob_acl( + test_blob.bucket.name, test_blob.name) + out, _ = capsys.readouterr() + assert out + + +@backoff.on_exception(backoff.expo, HttpError, max_time=60) +def test_print_blob_acl_for_user(test_blob, capsys): + test_blob.acl.user(TEST_EMAIL).grant_owner() + test_blob.acl.save() + + storage_print_file_acl_for_user.print_blob_acl_for_user( + test_blob.bucket.name, test_blob.name, TEST_EMAIL + ) + + out, _ = capsys.readouterr() + assert "OWNER" in out + + +@backoff.on_exception(backoff.expo, HttpError, max_time=60) +def test_add_blob_owner(test_blob): + storage_add_file_owner.add_blob_owner( + test_blob.bucket.name, test_blob.name, TEST_EMAIL) + + test_blob.acl.reload() + assert "OWNER" in test_blob.acl.user(TEST_EMAIL).get_roles() + + +@backoff.on_exception(backoff.expo, HttpError, max_time=60) +def test_remove_blob_owner(test_blob): + test_blob.acl.user(TEST_EMAIL).grant_owner() + test_blob.acl.save() + + storage_remove_file_owner.remove_blob_owner( + test_blob.bucket.name, test_blob.name, TEST_EMAIL + ) + + test_blob.acl.reload() + assert "OWNER" not in test_blob.acl.user(TEST_EMAIL).get_roles() diff --git a/storage/samples/snippets/bucket_lock_test.py b/storage/samples/snippets/bucket_lock_test.py new file mode 100644 index 00000000000..67d4ec6853a --- /dev/null +++ b/storage/samples/snippets/bucket_lock_test.py @@ -0,0 +1,176 @@ +# Copyright 2018 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import time +import uuid + +from google.cloud import storage +import pytest + +import storage_disable_default_event_based_hold +import storage_enable_default_event_based_hold +import storage_get_default_event_based_hold +import storage_get_retention_policy +import storage_lock_retention_policy +import storage_release_event_based_hold +import storage_release_temporary_hold +import storage_remove_retention_policy +import storage_set_event_based_hold +import storage_set_retention_policy +import storage_set_temporary_hold + + +BLOB_NAME = "storage_snippets_test_sigil" +BLOB_CONTENT = "Hello, is it me you're looking for?" +# Retention policy for 5 seconds +RETENTION_POLICY = 5 + + +@pytest.fixture +def bucket(): + """Yields a bucket that is deleted after the test completes.""" + bucket = None + while bucket is None or bucket.exists(): + bucket_name = "bucket-lock-{}".format(uuid.uuid4()) + bucket = storage.Client().bucket(bucket_name) + bucket.create() + yield bucket + bucket.delete(force=True) + + +def test_retention_policy_no_lock(bucket, capsys): + storage_set_retention_policy.set_retention_policy( + bucket.name, RETENTION_POLICY + ) + bucket.reload() + + assert bucket.retention_period is RETENTION_POLICY + assert bucket.retention_policy_effective_time is not None + assert bucket.retention_policy_locked is None + + storage_get_retention_policy.get_retention_policy(bucket.name) + out, _ = capsys.readouterr() + assert "Retention Policy for {}".format(bucket.name) in out + assert "Retention Period: 5" in out + assert "Effective Time: " in out + assert "Retention Policy is locked" not in out + + blob = bucket.blob(BLOB_NAME) + blob.upload_from_string(BLOB_CONTENT) + + assert blob.retention_expiration_time is not None + + storage_remove_retention_policy.remove_retention_policy(bucket.name) + bucket.reload() + assert bucket.retention_period is None + + time.sleep(RETENTION_POLICY) + + +def test_retention_policy_lock(bucket, capsys): + storage_set_retention_policy.set_retention_policy( + bucket.name, RETENTION_POLICY + ) + bucket.reload() + assert bucket.retention_policy_locked is None + + storage_lock_retention_policy.lock_retention_policy(bucket.name) + bucket.reload() + assert bucket.retention_policy_locked is True + + storage_get_retention_policy.get_retention_policy(bucket.name) + out, _ = capsys.readouterr() + assert "Retention Policy is locked" in out + + +def test_enable_disable_bucket_default_event_based_hold(bucket, capsys): + storage_get_default_event_based_hold.get_default_event_based_hold( + bucket.name + ) + out, _ = capsys.readouterr() + assert ( + "Default event-based hold is not enabled for {}".format(bucket.name) + in out + ) + assert ( + "Default event-based hold is enabled for {}".format(bucket.name) + not in out + ) + + storage_enable_default_event_based_hold.enable_default_event_based_hold( + bucket.name + ) + bucket.reload() + + assert bucket.default_event_based_hold is True + + storage_get_default_event_based_hold.get_default_event_based_hold( + bucket.name + ) + out, _ = capsys.readouterr() + assert ( + "Default event-based hold is enabled for {}".format(bucket.name) in out + ) + + # Changes to the bucket will be readable immediately after writing, + # but configuration changes may take time to propagate. + time.sleep(10) + + blob = bucket.blob(BLOB_NAME) + blob.upload_from_string(BLOB_CONTENT) + assert blob.event_based_hold is True + + storage_release_event_based_hold.release_event_based_hold( + bucket.name, blob.name + ) + blob.reload() + assert blob.event_based_hold is False + + storage_disable_default_event_based_hold.disable_default_event_based_hold( + bucket.name + ) + bucket.reload() + assert bucket.default_event_based_hold is False + + +def test_enable_disable_temporary_hold(bucket): + blob = bucket.blob(BLOB_NAME) + blob.upload_from_string(BLOB_CONTENT) + assert blob.temporary_hold is None + + storage_set_temporary_hold.set_temporary_hold(bucket.name, blob.name) + blob.reload() + assert blob.temporary_hold is True + + storage_release_temporary_hold.release_temporary_hold( + bucket.name, blob.name + ) + blob.reload() + assert blob.temporary_hold is False + + +def test_enable_disable_event_based_hold(bucket): + blob = bucket.blob(BLOB_NAME) + blob.upload_from_string(BLOB_CONTENT) + assert blob.event_based_hold is None + + storage_set_event_based_hold.set_event_based_hold(bucket.name, blob.name) + blob.reload() + assert blob.event_based_hold is True + + storage_release_event_based_hold.release_event_based_hold( + bucket.name, blob.name + ) + blob.reload() + assert blob.event_based_hold is False diff --git a/storage/samples/snippets/conftest.py b/storage/samples/snippets/conftest.py new file mode 100644 index 00000000000..b0db57561d8 --- /dev/null +++ b/storage/samples/snippets/conftest.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import time +import uuid + +from google.cloud import storage +import pytest + + +@pytest.fixture(scope="function") +def bucket(): + """Yields a bucket that is deleted after the test completes.""" + # The new projects enforces uniform bucket level access, so + # we need to use the old main project for now. + original_value = os.environ['GOOGLE_CLOUD_PROJECT'] + os.environ['GOOGLE_CLOUD_PROJECT'] = os.environ['MAIN_GOOGLE_CLOUD_PROJECT'] + bucket = None + while bucket is None or bucket.exists(): + bucket_name = f"uniform-bucket-level-access-{uuid.uuid4().hex}" + bucket = storage.Client().bucket(bucket_name) + bucket.create() + yield bucket + time.sleep(3) + bucket.delete(force=True) + # Set the value back. + os.environ['GOOGLE_CLOUD_PROJECT'] = original_value diff --git a/storage/samples/snippets/encryption_test.py b/storage/samples/snippets/encryption_test.py new file mode 100644 index 00000000000..6c2377e0fe6 --- /dev/null +++ b/storage/samples/snippets/encryption_test.py @@ -0,0 +1,125 @@ +# Copyright 2016 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import base64 +import os +import tempfile +import uuid + +from google.api_core.exceptions import NotFound +from google.cloud import storage +from google.cloud.storage import Blob +import pytest + +import storage_download_encrypted_file +import storage_generate_encryption_key +import storage_object_csek_to_cmek +import storage_rotate_encryption_key +import storage_upload_encrypted_file + +BUCKET = os.environ["CLOUD_STORAGE_BUCKET"] +KMS_KEY = os.environ["CLOUD_KMS_KEY"] + +TEST_ENCRYPTION_KEY = "brtJUWneL92g5q0N2gyDSnlPSYAiIVZ/cWgjyZNeMy0=" +TEST_ENCRYPTION_KEY_DECODED = base64.b64decode(TEST_ENCRYPTION_KEY) + +TEST_ENCRYPTION_KEY_2 = "o4OD7SWCaPjfeEGhAY+YCgMdY9UW+OJ8mvfWD9lNtO4=" +TEST_ENCRYPTION_KEY_2_DECODED = base64.b64decode(TEST_ENCRYPTION_KEY_2) + + +def test_generate_encryption_key(capsys): + storage_generate_encryption_key.generate_encryption_key() + out, _ = capsys.readouterr() + encoded_key = out.split(":", 1).pop().strip() + key = base64.b64decode(encoded_key) + assert len(key) == 32, "Returned key should be 32 bytes" + + +def test_upload_encrypted_blob(): + with tempfile.NamedTemporaryFile() as source_file: + source_file.write(b"test") + + storage_upload_encrypted_file.upload_encrypted_blob( + BUCKET, + source_file.name, + "test_encrypted_upload_blob", + TEST_ENCRYPTION_KEY, + ) + + +@pytest.fixture(scope="module") +def test_blob(): + """Provides a pre-existing blob in the test bucket.""" + bucket = storage.Client().bucket(BUCKET) + blob_name = "test_blob_{}".format(uuid.uuid4().hex) + blob = Blob( + blob_name, + bucket, + encryption_key=TEST_ENCRYPTION_KEY_DECODED, + ) + content = "Hello, is it me you're looking for?" + blob.upload_from_string(content) + + yield blob.name, content + + # To delete an encrypted blob, you have to provide the same key + # used for the blob. When you provide a wrong key, you'll get + # NotFound. + try: + # Clean up for the case that the rotation didn't occur. + blob.delete() + except NotFound as e: + # For the case that the rotation succeeded. + print("Ignoring 404, detail: {}".format(e)) + blob = Blob( + blob_name, + bucket, + encryption_key=TEST_ENCRYPTION_KEY_2_DECODED + ) + blob.delete() + + +def test_download_blob(test_blob): + test_blob_name, test_blob_content = test_blob + with tempfile.NamedTemporaryFile() as dest_file: + storage_download_encrypted_file.download_encrypted_blob( + BUCKET, test_blob_name, dest_file.name, TEST_ENCRYPTION_KEY + ) + + downloaded_content = dest_file.read().decode("utf-8") + assert downloaded_content == test_blob_content + + +def test_rotate_encryption_key(test_blob): + test_blob_name, test_blob_content = test_blob + storage_rotate_encryption_key.rotate_encryption_key( + BUCKET, test_blob_name, TEST_ENCRYPTION_KEY, TEST_ENCRYPTION_KEY_2 + ) + + with tempfile.NamedTemporaryFile() as dest_file: + storage_download_encrypted_file.download_encrypted_blob( + BUCKET, test_blob_name, dest_file.name, TEST_ENCRYPTION_KEY_2 + ) + + downloaded_content = dest_file.read().decode("utf-8") + assert downloaded_content == test_blob_content + + +def test_object_csek_to_cmek(test_blob): + test_blob_name, test_blob_content = test_blob + cmek_blob = storage_object_csek_to_cmek.object_csek_to_cmek( + BUCKET, test_blob_name, TEST_ENCRYPTION_KEY_2, KMS_KEY + ) + + assert cmek_blob.download_as_string(), test_blob_content diff --git a/storage/samples/snippets/hmac_samples_test.py b/storage/samples/snippets/hmac_samples_test.py new file mode 100644 index 00000000000..60eba240184 --- /dev/null +++ b/storage/samples/snippets/hmac_samples_test.py @@ -0,0 +1,121 @@ +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Tests for hmac.py. Requires GOOGLE_CLOUD_PROJECT (valid project) and +HMAC_KEY_TEST_SERVICE_ACCOUNT (valid service account email) env variables to be +set in order to run. +""" + + +import os + +import google.api_core.exceptions +from google.cloud import storage +import pytest + +import storage_activate_hmac_key +import storage_create_hmac_key +import storage_deactivate_hmac_key +import storage_delete_hmac_key +import storage_get_hmac_key +import storage_list_hmac_keys + +# We are reaching maximum number of HMAC keys on the service account. +# We change the service account based on the value of +# RUN_TESTS_SESSION in noxfile_config.py. +# The reason we can not use multiple project is that our new projects +# are enforced to have +# 'constraints/iam.disableServiceAccountKeyCreation' policy. + +PROJECT_ID = os.environ["MAIN_GOOGLE_CLOUD_PROJECT"] +SERVICE_ACCOUNT_EMAIL = os.environ["HMAC_KEY_TEST_SERVICE_ACCOUNT"] +STORAGE_CLIENT = storage.Client(project=PROJECT_ID) + + +@pytest.fixture(scope="module") +def new_hmac_key(): + """ + Fixture to create a new HMAC key, and to guarantee all keys are deleted at + the end of the module. + + NOTE: Due to the module scope, test order in this file is significant + """ + hmac_key, secret = STORAGE_CLIENT.create_hmac_key( + service_account_email=SERVICE_ACCOUNT_EMAIL, project_id=PROJECT_ID + ) + yield hmac_key + # Re-fetch the key metadata in case state has changed during the test. + hmac_key = STORAGE_CLIENT.get_hmac_key_metadata( + hmac_key.access_id, project_id=PROJECT_ID + ) + if hmac_key.state == "DELETED": + return + if not hmac_key.state == "INACTIVE": + hmac_key.state = "INACTIVE" + hmac_key.update() + hmac_key.delete() + + +def test_list_keys(capsys, new_hmac_key): + hmac_keys = storage_list_hmac_keys.list_keys(PROJECT_ID) + assert "HMAC Keys:" in capsys.readouterr().out + assert hmac_keys.num_results >= 1 + + +def test_create_key(capsys): + hmac_key = storage_create_hmac_key.create_key( + PROJECT_ID, SERVICE_ACCOUNT_EMAIL + ) + hmac_key.state = "INACTIVE" + hmac_key.update() + hmac_key.delete() + assert "Key ID:" in capsys.readouterr().out + assert hmac_key.access_id + + +def test_get_key(capsys, new_hmac_key): + hmac_key = storage_get_hmac_key.get_key(new_hmac_key.access_id, PROJECT_ID) + assert "HMAC key metadata" in capsys.readouterr().out + assert hmac_key.access_id == new_hmac_key.access_id + + +def test_activate_key(capsys, new_hmac_key): + new_hmac_key.state = "INACTIVE" + new_hmac_key.update() + hmac_key = storage_activate_hmac_key.activate_key( + new_hmac_key.access_id, PROJECT_ID + ) + assert "State: ACTIVE" in capsys.readouterr().out + assert hmac_key.state == "ACTIVE" + + +def test_deactivate_key(capsys, new_hmac_key): + hmac_key = storage_deactivate_hmac_key.deactivate_key( + new_hmac_key.access_id, PROJECT_ID + ) + assert "State: INACTIVE" in capsys.readouterr().out + assert hmac_key.state == "INACTIVE" + + +def test_delete_key(capsys, new_hmac_key): + # Due to reuse of the HMAC key for each test function, the previous + # test has deactivated the key already. + try: + new_hmac_key.state = "INACTIVE" + new_hmac_key.update() + except google.api_core.exceptions.BadRequest: + pass + + storage_delete_hmac_key.delete_key(new_hmac_key.access_id, PROJECT_ID) + assert "The key is deleted" in capsys.readouterr().out diff --git a/storage/samples/snippets/iam_test.py b/storage/samples/snippets/iam_test.py new file mode 100644 index 00000000000..eb7638de5f4 --- /dev/null +++ b/storage/samples/snippets/iam_test.py @@ -0,0 +1,146 @@ +# Copyright 2017 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import re +import time +import uuid + +from google.cloud import storage +import pytest + +import storage_add_bucket_conditional_iam_binding +import storage_add_bucket_iam_member +import storage_remove_bucket_conditional_iam_binding +import storage_remove_bucket_iam_member +import storage_set_bucket_public_iam +import storage_view_bucket_iam_members + +MEMBER = "group:dpebot@google.com" +ROLE = "roles/storage.legacyBucketReader" + +CONDITION_TITLE = "match-prefix" +CONDITION_DESCRIPTION = "Applies to objects matching a prefix" +CONDITION_EXPRESSION = ( + 'resource.name.startsWith("projects/_/buckets/bucket-name/objects/prefix-a-")' +) + + +@pytest.fixture(scope="module") +def bucket(): + bucket = None + while bucket is None or bucket.exists(): + storage_client = storage.Client() + bucket_name = "test-iam-{}".format(uuid.uuid4()) + bucket = storage_client.bucket(bucket_name) + bucket.iam_configuration.uniform_bucket_level_access_enabled = True + storage_client.create_bucket(bucket) + yield bucket + time.sleep(3) + bucket.delete(force=True) + + +@pytest.fixture(scope="function") +def public_bucket(): + # The new projects don't allow to make a bucket available to public, so + # we need to use the old main project for now. + original_value = os.environ['GOOGLE_CLOUD_PROJECT'] + os.environ['GOOGLE_CLOUD_PROJECT'] = os.environ['MAIN_GOOGLE_CLOUD_PROJECT'] + bucket = None + while bucket is None or bucket.exists(): + storage_client = storage.Client() + bucket_name = "test-iam-{}".format(uuid.uuid4()) + bucket = storage_client.bucket(bucket_name) + bucket.iam_configuration.uniform_bucket_level_access_enabled = True + storage_client.create_bucket(bucket) + yield bucket + time.sleep(3) + bucket.delete(force=True) + # Set the value back. + os.environ['GOOGLE_CLOUD_PROJECT'] = original_value + + +def test_view_bucket_iam_members(capsys, bucket): + storage_view_bucket_iam_members.view_bucket_iam_members(bucket.name) + assert re.match("Role: .*, Members: .*", capsys.readouterr().out) + + +def test_add_bucket_iam_member(bucket): + storage_add_bucket_iam_member.add_bucket_iam_member(bucket.name, ROLE, MEMBER) + policy = bucket.get_iam_policy(requested_policy_version=3) + assert any( + binding["role"] == ROLE and MEMBER in binding["members"] + for binding in policy.bindings + ) + + +def test_add_bucket_conditional_iam_binding(bucket): + storage_add_bucket_conditional_iam_binding.add_bucket_conditional_iam_binding( + bucket.name, + ROLE, + CONDITION_TITLE, + CONDITION_DESCRIPTION, + CONDITION_EXPRESSION, + {MEMBER}, + ) + policy = bucket.get_iam_policy(requested_policy_version=3) + assert any( + binding["role"] == ROLE + and binding["members"] == {MEMBER} + and binding["condition"] + == { + "title": CONDITION_TITLE, + "description": CONDITION_DESCRIPTION, + "expression": CONDITION_EXPRESSION, + } + for binding in policy.bindings + ) + + +def test_remove_bucket_iam_member(public_bucket): + storage_remove_bucket_iam_member.remove_bucket_iam_member( + public_bucket.name, ROLE, MEMBER) + + policy = public_bucket.get_iam_policy(requested_policy_version=3) + assert not any( + binding["role"] == ROLE and MEMBER in binding["members"] + for binding in policy.bindings + ) + + +def test_remove_bucket_conditional_iam_binding(bucket): + storage_remove_bucket_conditional_iam_binding.remove_bucket_conditional_iam_binding( + bucket.name, ROLE, CONDITION_TITLE, CONDITION_DESCRIPTION, CONDITION_EXPRESSION + ) + + policy = bucket.get_iam_policy(requested_policy_version=3) + condition = { + "title": CONDITION_TITLE, + "description": CONDITION_DESCRIPTION, + "expression": CONDITION_EXPRESSION, + } + assert not any( + (binding["role"] == ROLE and binding.get("condition") == condition) + for binding in policy.bindings + ) + + +def test_set_bucket_public_iam(public_bucket): + storage_set_bucket_public_iam.set_bucket_public_iam(public_bucket.name) + policy = public_bucket.get_iam_policy(requested_policy_version=3) + assert any( + binding["role"] == "roles/storage.objectViewer" + and "allUsers" in binding["members"] + for binding in policy.bindings + ) diff --git a/storage/samples/snippets/notification_polling.py b/storage/samples/snippets/notification_polling.py new file mode 100644 index 00000000000..3182db6da34 --- /dev/null +++ b/storage/samples/snippets/notification_polling.py @@ -0,0 +1,137 @@ +#!/usr/bin/env python + +# Copyright 2017 Google Inc. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""This application demonstrates how to poll for GCS notifications from a +Cloud Pub/Sub subscription, parse the incoming message, and acknowledge the +successful processing of the message. + +This application will work with any subscription configured for pull rather +than push notifications. If you do not already have notifications configured, +you may consult the docs at +https://cloud.google.com/storage/docs/reporting-changes or follow the steps +below: + +1. First, follow the common setup steps for these snippets, specically + configuring auth and installing dependencies. See the README's "Setup" + section. + +2. Activate the Google Cloud Pub/Sub API, if you have not already done so. + https://console.cloud.google.com/flows/enableapi?apiid=pubsub + +3. Create a Google Cloud Storage bucket: + $ gsutil mb gs://testbucket + +4. Create a Cloud Pub/Sub topic and publish bucket notifications there: + $ gsutil notification create -f json -t testtopic gs://testbucket + +5. Create a subscription for your new topic: + $ gcloud beta pubsub subscriptions create testsubscription --topic=testtopic + +6. Run this program: + $ python notification_polling.py my-project-id testsubscription + +7. While the program is running, upload and delete some files in the testbucket + bucket (you could use the console or gsutil) and watch as changes scroll by + in the app. +""" + +import argparse +import json +import time + +from google.cloud import pubsub_v1 + + +def summarize(message): + data = message.data.decode("utf-8") + attributes = message.attributes + + event_type = attributes["eventType"] + bucket_id = attributes["bucketId"] + object_id = attributes["objectId"] + generation = attributes["objectGeneration"] + description = ( + "\tEvent type: {event_type}\n" + "\tBucket ID: {bucket_id}\n" + "\tObject ID: {object_id}\n" + "\tGeneration: {generation}\n" + ).format( + event_type=event_type, + bucket_id=bucket_id, + object_id=object_id, + generation=generation, + ) + + if "overwroteGeneration" in attributes: + description += "\tOverwrote generation: %s\n" % ( + attributes["overwroteGeneration"] + ) + if "overwrittenByGeneration" in attributes: + description += "\tOverwritten by generation: %s\n" % ( + attributes["overwrittenByGeneration"] + ) + + payload_format = attributes["payloadFormat"] + if payload_format == "JSON_API_V1": + object_metadata = json.loads(data) + size = object_metadata["size"] + content_type = object_metadata["contentType"] + metageneration = object_metadata["metageneration"] + description += ( + "\tContent type: {content_type}\n" + "\tSize: {object_size}\n" + "\tMetageneration: {metageneration}\n" + ).format( + content_type=content_type, + object_size=size, + metageneration=metageneration, + ) + return description + + +def poll_notifications(project, subscription_name): + """Polls a Cloud Pub/Sub subscription for new GCS events for display.""" + subscriber = pubsub_v1.SubscriberClient() + subscription_path = subscriber.subscription_path( + project, subscription_name + ) + + def callback(message): + print("Received message:\n{}".format(summarize(message))) + message.ack() + + subscriber.subscribe(subscription_path, callback=callback) + + # The subscriber is non-blocking, so we must keep the main thread from + # exiting to allow it to process messages in the background. + print("Listening for messages on {}".format(subscription_path)) + while True: + time.sleep(60) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "project", help="The ID of the project that owns the subscription" + ) + parser.add_argument( + "subscription", help="The ID of the Pub/Sub subscription" + ) + args = parser.parse_args() + poll_notifications(args.project, args.subscription) diff --git a/storage/samples/snippets/notification_polling_test.py b/storage/samples/snippets/notification_polling_test.py new file mode 100644 index 00000000000..dfb241b842d --- /dev/null +++ b/storage/samples/snippets/notification_polling_test.py @@ -0,0 +1,55 @@ +# Copyright 2017 Google Inc. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from google.cloud.pubsub_v1.subscriber.message import Message +import mock + +from notification_polling import summarize + + +MESSAGE_ID = 12345 + + +def test_parse_json_message(): + attributes = { + "eventType": "OBJECT_FINALIZE", + "bucketId": "mybucket", + "objectId": "myobject", + "objectGeneration": 1234567, + "resource": "projects/_/buckets/mybucket/objects/myobject#1234567", + "notificationConfig": ( + "projects/_/buckets/mybucket/" "notificationConfigs/5" + ), + "payloadFormat": "JSON_API_V1", + } + data = ( + b"{" + b' "size": 12345,' + b' "contentType": "text/html",' + b' "metageneration": 1' + b"}" + ) + message = Message( + mock.Mock(data=data, attributes=attributes, publish_time=mock.Mock(seconds=0.0, nanos=0.0)), MESSAGE_ID, delivery_attempt=0, request_queue=mock.Mock() + ) + assert summarize(message) == ( + "\tEvent type: OBJECT_FINALIZE\n" + "\tBucket ID: mybucket\n" + "\tObject ID: myobject\n" + "\tGeneration: 1234567\n" + "\tContent type: text/html\n" + "\tSize: 12345\n" + "\tMetageneration: 1\n" + ) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py new file mode 100644 index 00000000000..93a9122cc45 --- /dev/null +++ b/storage/samples/snippets/noxfile.py @@ -0,0 +1,270 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import print_function + +import os +from pathlib import Path +import sys +from typing import Callable, Dict, List, Optional + +import nox + + +# WARNING - WARNING - WARNING - WARNING - WARNING +# WARNING - WARNING - WARNING - WARNING - WARNING +# DO NOT EDIT THIS FILE EVER! +# WARNING - WARNING - WARNING - WARNING - WARNING +# WARNING - WARNING - WARNING - WARNING - WARNING + +BLACK_VERSION = "black==19.10b0" + +# Copy `noxfile_config.py` to your directory and modify it instead. + +# `TEST_CONFIG` dict is a configuration hook that allows users to +# modify the test configurations. The values here should be in sync +# with `noxfile_config.py`. Users will copy `noxfile_config.py` into +# their directory and modify it. + +TEST_CONFIG = { + # You can opt out from the test for specific Python versions. + "ignored_versions": [], + # Old samples are opted out of enforcing Python type hints + # All new samples should feature them + "enforce_type_hints": False, + # An envvar key for determining the project id to use. Change it + # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a + # build specific Cloud project. You can also use your own string + # to use your own Cloud project. + "gcloud_project_env": "GOOGLE_CLOUD_PROJECT", + # 'gcloud_project_env': 'BUILD_SPECIFIC_GCLOUD_PROJECT', + # If you need to use a specific version of pip, + # change pip_version_override to the string representation + # of the version number, for example, "20.2.4" + "pip_version_override": None, + # A dictionary you want to inject into your test. Don't put any + # secrets here. These values will override predefined values. + "envs": {}, +} + + +try: + # Ensure we can import noxfile_config in the project's directory. + sys.path.append(".") + from noxfile_config import TEST_CONFIG_OVERRIDE +except ImportError as e: + print("No user noxfile_config found: detail: {}".format(e)) + TEST_CONFIG_OVERRIDE = {} + +# Update the TEST_CONFIG with the user supplied values. +TEST_CONFIG.update(TEST_CONFIG_OVERRIDE) + + +def get_pytest_env_vars() -> Dict[str, str]: + """Returns a dict for pytest invocation.""" + ret = {} + + # Override the GCLOUD_PROJECT and the alias. + env_key = TEST_CONFIG["gcloud_project_env"] + # This should error out if not set. + ret["GOOGLE_CLOUD_PROJECT"] = os.environ[env_key] + + # Apply user supplied envs. + ret.update(TEST_CONFIG["envs"]) + return ret + + +# DO NOT EDIT - automatically generated. +# All versions used to test samples. +ALL_VERSIONS = ["3.6", "3.7", "3.8", "3.9", "3.10"] + +# Any default versions that should be ignored. +IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] + +TESTED_VERSIONS = sorted([v for v in ALL_VERSIONS if v not in IGNORED_VERSIONS]) + +INSTALL_LIBRARY_FROM_SOURCE = os.environ.get("INSTALL_LIBRARY_FROM_SOURCE", False) in ( + "True", + "true", +) + +# Error if a python version is missing +nox.options.error_on_missing_interpreters = True + +# +# Style Checks +# + + +def _determine_local_import_names(start_dir: str) -> List[str]: + """Determines all import names that should be considered "local". + + This is used when running the linter to insure that import order is + properly checked. + """ + file_ext_pairs = [os.path.splitext(path) for path in os.listdir(start_dir)] + return [ + basename + for basename, extension in file_ext_pairs + if extension == ".py" + or os.path.isdir(os.path.join(start_dir, basename)) + and basename not in ("__pycache__") + ] + + +# Linting with flake8. +# +# We ignore the following rules: +# E203: whitespace before ‘:’ +# E266: too many leading ‘#’ for block comment +# E501: line too long +# I202: Additional newline in a section of imports +# +# We also need to specify the rules which are ignored by default: +# ['E226', 'W504', 'E126', 'E123', 'W503', 'E24', 'E704', 'E121'] +FLAKE8_COMMON_ARGS = [ + "--show-source", + "--builtin=gettext", + "--max-complexity=20", + "--import-order-style=google", + "--exclude=.nox,.cache,env,lib,generated_pb2,*_pb2.py,*_pb2_grpc.py", + "--ignore=E121,E123,E126,E203,E226,E24,E266,E501,E704,W503,W504,I202", + "--max-line-length=88", +] + + +@nox.session +def lint(session: nox.sessions.Session) -> None: + if not TEST_CONFIG["enforce_type_hints"]: + session.install("flake8", "flake8-import-order") + else: + session.install("flake8", "flake8-import-order", "flake8-annotations") + + local_names = _determine_local_import_names(".") + args = FLAKE8_COMMON_ARGS + [ + "--application-import-names", + ",".join(local_names), + ".", + ] + session.run("flake8", *args) + + +# +# Black +# + + +@nox.session +def blacken(session: nox.sessions.Session) -> None: + session.install(BLACK_VERSION) + python_files = [path for path in os.listdir(".") if path.endswith(".py")] + + session.run("black", *python_files) + + +# +# Sample Tests +# + + +PYTEST_COMMON_ARGS = ["--junitxml=sponge_log.xml"] + + +def _session_tests( + session: nox.sessions.Session, post_install: Callable = None +) -> None: + if TEST_CONFIG["pip_version_override"]: + pip_version = TEST_CONFIG["pip_version_override"] + session.install(f"pip=={pip_version}") + """Runs py.test for a particular project.""" + if os.path.exists("requirements.txt"): + if os.path.exists("constraints.txt"): + session.install("-r", "requirements.txt", "-c", "constraints.txt") + else: + session.install("-r", "requirements.txt") + + if os.path.exists("requirements-test.txt"): + if os.path.exists("constraints-test.txt"): + session.install("-r", "requirements-test.txt", "-c", "constraints-test.txt") + else: + session.install("-r", "requirements-test.txt") + + if INSTALL_LIBRARY_FROM_SOURCE: + session.install("-e", _get_repo_root()) + + if post_install: + post_install(session) + + session.run( + "pytest", + *(PYTEST_COMMON_ARGS + session.posargs), + # Pytest will return 5 when no tests are collected. This can happen + # on travis where slow and flaky tests are excluded. + # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html + success_codes=[0, 5], + env=get_pytest_env_vars(), + ) + + +@nox.session(python=ALL_VERSIONS) +def py(session: nox.sessions.Session) -> None: + """Runs py.test for a sample using the specified version of Python.""" + if session.python in TESTED_VERSIONS: + _session_tests(session) + else: + session.skip( + "SKIPPED: {} tests are disabled for this sample.".format(session.python) + ) + + +# +# Readmegen +# + + +def _get_repo_root() -> Optional[str]: + """ Returns the root folder of the project. """ + # Get root of this repository. Assume we don't have directories nested deeper than 10 items. + p = Path(os.getcwd()) + for i in range(10): + if p is None: + break + if Path(p / ".git").exists(): + return str(p) + # .git is not available in repos cloned via Cloud Build + # setup.py is always in the library's root, so use that instead + # https://github.com/googleapis/synthtool/issues/792 + if Path(p / "setup.py").exists(): + return str(p) + p = p.parent + raise Exception("Unable to detect repository root.") + + +GENERATED_READMES = sorted([x for x in Path(".").rglob("*.rst.in")]) + + +@nox.session +@nox.parametrize("path", GENERATED_READMES) +def readmegen(session: nox.sessions.Session, path: str) -> None: + """(Re-)generates the readme for a sample.""" + session.install("jinja2", "pyyaml") + dir_ = os.path.dirname(path) + + if os.path.exists(os.path.join(dir_, "requirements.txt")): + session.install("-r", os.path.join(dir_, "requirements.txt")) + + in_file = os.path.join(dir_, "README.rst.in") + session.run( + "python", _get_repo_root() + "/scripts/readme-gen/readme_gen.py", in_file + ) diff --git a/storage/samples/snippets/noxfile_config.py b/storage/samples/snippets/noxfile_config.py new file mode 100644 index 00000000000..463da97de55 --- /dev/null +++ b/storage/samples/snippets/noxfile_config.py @@ -0,0 +1,96 @@ +# Copyright 2020 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Default TEST_CONFIG_OVERRIDE for python repos. + +# You can copy this file into your directory, then it will be imported from +# the noxfile.py. + +# The source of truth: +# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/noxfile_config.py + +import os + + +# We are reaching maximum number of HMAC keys on the service account. +# We change the service account based on the value of +# RUN_TESTS_SESSION. The reason we can not use multiple project is +# that our new projects are enforced to have +# 'constraints/iam.disableServiceAccountKeyCreation' policy. +def get_service_account_email(): + session = os.environ.get('RUN_TESTS_SESSION') + if session == 'py-3.6': + return ('py36-storage-test@' + 'python-docs-samples-tests.iam.gserviceaccount.com') + if session == 'py-3.7': + return ('py37-storage-test@' + 'python-docs-samples-tests.iam.gserviceaccount.com') + if session == 'py-3.8': + return ('py38-storage-test@' + 'python-docs-samples-tests.iam.gserviceaccount.com') + if session == 'py-3.9': + return ('py39-storage-test@' + 'python-docs-samples-tests.iam.gserviceaccount.com') + if session == 'py-3.10': + return ('py310-storage-test@' + 'python-docs-samples-tests.iam.gserviceaccount.com') + return os.environ['HMAC_KEY_TEST_SERVICE_ACCOUNT'] + + +# We change the value of CLOUD_KMS_KEY based on the value of +# RUN_TESTS_SESSION. +def get_cloud_kms_key(): + session = os.environ.get('RUN_TESTS_SESSION') + if session == 'py-3.6': + return ('projects/python-docs-samples-tests-py36/locations/us/' + 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') + if session == 'py-3.7': + return ('projects/python-docs-samples-tests-py37/locations/us/' + 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') + if session == 'py-3.8': + return ('projects/python-docs-samples-tests-py38/locations/us/' + 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') + if session == 'py-3.9': + return ('projects/python-docs-samples-tests-py39/locations/us/' + 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') + if session == 'py-3.10': + return ('projects/python-docs-samples-tests-310/locations/us/' + 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') + return os.environ['CLOUD_KMS_KEY'] + + +TEST_CONFIG_OVERRIDE = { + # You can opt out from the test for specific Python versions. + 'ignored_versions': ["2.7"], + + # An envvar key for determining the project id to use. Change it + # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a + # build specific Cloud project. You can also use your own string + # to use your own Cloud project. + # 'gcloud_project_env': 'GOOGLE_CLOUD_PROJECT', + 'gcloud_project_env': 'BUILD_SPECIFIC_GCLOUD_PROJECT', + + # A dictionary you want to inject into your test. Don't put any + # secrets here. These values will override predefined values. + 'envs': { + 'HMAC_KEY_TEST_SERVICE_ACCOUNT': get_service_account_email(), + 'CLOUD_KMS_KEY': get_cloud_kms_key(), + # Some tests can not use multiple projects because of several reasons: + # 1. The new projects is enforced to have the + # 'constraints/iam.disableServiceAccountKeyCreation' policy. + # 2. The new projects buckets need to have universal permission model. + # For those tests, we'll use the original project. + 'MAIN_GOOGLE_CLOUD_PROJECT': 'python-docs-samples-tests' + }, +} diff --git a/storage/samples/snippets/public_access_prevention_test.py b/storage/samples/snippets/public_access_prevention_test.py new file mode 100644 index 00000000000..40d3924b296 --- /dev/null +++ b/storage/samples/snippets/public_access_prevention_test.py @@ -0,0 +1,50 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import pytest + +import storage_get_public_access_prevention +import storage_set_public_access_prevention_enforced +import storage_set_public_access_prevention_inherited +import storage_set_public_access_prevention_unspecified + + +@pytest.mark.skip(reason="Inconsistent due to unspecified->inherited change") +def test_get_public_access_prevention(bucket, capsys): + short_name = storage_get_public_access_prevention + short_name.get_public_access_prevention(bucket.name) + out, _ = capsys.readouterr() + assert f"Public access prevention is inherited for {bucket.name}." in out + + +def test_set_public_access_prevention_enforced(bucket, capsys): + short_name = storage_set_public_access_prevention_enforced + short_name.set_public_access_prevention_enforced(bucket.name) + out, _ = capsys.readouterr() + assert f"Public access prevention is set to enforced for {bucket.name}." in out + + +@pytest.mark.skip(reason="Inconsistent due to unspecified->inherited change") +def test_set_public_access_prevention_unspecified(bucket, capsys): + short_name = storage_set_public_access_prevention_unspecified + short_name.set_public_access_prevention_unspecified(bucket.name) + out, _ = capsys.readouterr() + assert f"Public access prevention is 'unspecified' for {bucket.name}." in out + + +def test_set_public_access_prevention_inherited(bucket, capsys): + short_name = storage_set_public_access_prevention_inherited + short_name.set_public_access_prevention_inherited(bucket.name) + out, _ = capsys.readouterr() + assert f"Public access prevention is 'inherited' for {bucket.name}." in out diff --git a/storage/samples/snippets/quickstart.py b/storage/samples/snippets/quickstart.py new file mode 100644 index 00000000000..578e50753f0 --- /dev/null +++ b/storage/samples/snippets/quickstart.py @@ -0,0 +1,37 @@ +#!/usr/bin/env python + +# Copyright 2016 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +def run_quickstart(): + # [START storage_quickstart] + # Imports the Google Cloud client library + from google.cloud import storage + + # Instantiates a client + storage_client = storage.Client() + + # The name for the new bucket + bucket_name = "my-new-bucket" + + # Creates the new bucket + bucket = storage_client.create_bucket(bucket_name) + + print("Bucket {} created.".format(bucket.name)) + # [END storage_quickstart] + + +if __name__ == "__main__": + run_quickstart() diff --git a/storage/samples/snippets/quickstart_test.py b/storage/samples/snippets/quickstart_test.py new file mode 100644 index 00000000000..f6e06ad93e8 --- /dev/null +++ b/storage/samples/snippets/quickstart_test.py @@ -0,0 +1,28 @@ +# Copyright 2016 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import mock + +import quickstart + + +@mock.patch("google.cloud.storage.client.Client.create_bucket") +def test_quickstart(create_bucket_mock, capsys): + # Unlike other quickstart tests, this one mocks out the creation + # because buckets are expensive, globally-namespaced object. + create_bucket_mock.return_value = mock.sentinel.bucket + + quickstart.run_quickstart() + + create_bucket_mock.assert_called_with("my-new-bucket") diff --git a/storage/samples/snippets/requester_pays_test.py b/storage/samples/snippets/requester_pays_test.py new file mode 100644 index 00000000000..9f85c6bdb20 --- /dev/null +++ b/storage/samples/snippets/requester_pays_test.py @@ -0,0 +1,65 @@ +# Copyright 2017 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import tempfile + +from google.cloud import storage +import pytest + +import storage_disable_requester_pays +import storage_download_file_requester_pays +import storage_enable_requester_pays +import storage_get_requester_pays_status + + +# We use a different bucket from other tests. +BUCKET = os.environ["REQUESTER_PAYS_TEST_BUCKET"] +PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"] + + +def test_enable_requester_pays(capsys): + storage_enable_requester_pays.enable_requester_pays(BUCKET) + out, _ = capsys.readouterr() + assert "Requester Pays has been enabled for {}".format(BUCKET) in out + + +def test_disable_requester_pays(capsys): + storage_disable_requester_pays.disable_requester_pays(BUCKET) + out, _ = capsys.readouterr() + assert "Requester Pays has been disabled for {}".format(BUCKET) in out + + +def test_get_requester_pays_status(capsys): + storage_get_requester_pays_status.get_requester_pays_status(BUCKET) + out, _ = capsys.readouterr() + assert "Requester Pays is disabled for {}".format(BUCKET) in out + + +@pytest.fixture +def test_blob(): + """Provides a pre-existing blob in the test bucket.""" + bucket = storage.Client().bucket(BUCKET) + blob = bucket.blob("storage_snippets_test_sigil") + blob.upload_from_string("Hello, is it me you're looking for?") + return blob + + +def test_download_file_requester_pays(test_blob, capsys): + with tempfile.NamedTemporaryFile() as dest_file: + storage_download_file_requester_pays.download_file_requester_pays( + BUCKET, PROJECT, test_blob.name, dest_file.name + ) + + assert dest_file.read() diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt new file mode 100644 index 00000000000..2b550f467fa --- /dev/null +++ b/storage/samples/snippets/requirements-test.txt @@ -0,0 +1,3 @@ +pytest==6.2.4 +mock==4.0.3 +backoff==1.11.1 \ No newline at end of file diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt new file mode 100644 index 00000000000..76ac6ee7c57 --- /dev/null +++ b/storage/samples/snippets/requirements.txt @@ -0,0 +1,3 @@ +google-cloud-pubsub==2.8.0 +google-cloud-storage==1.42.3 +google-api-python-client==2.25.0 diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py new file mode 100644 index 00000000000..dd8e6aeaf1e --- /dev/null +++ b/storage/samples/snippets/snippets_test.py @@ -0,0 +1,511 @@ +# Copyright 2016 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import tempfile +import time +import uuid + +from google.cloud import storage +import google.cloud.exceptions +import pytest +import requests + +import storage_add_bucket_label +import storage_bucket_delete_default_kms_key +import storage_change_default_storage_class +import storage_change_file_storage_class +import storage_compose_file +import storage_configure_retries +import storage_copy_file +import storage_copy_file_archived_generation +import storage_cors_configuration +import storage_create_bucket_class_location +import storage_define_bucket_website_configuration +import storage_delete_file +import storage_delete_file_archived_generation +import storage_disable_bucket_lifecycle_management +import storage_disable_versioning +import storage_download_file +import storage_download_public_file +import storage_enable_bucket_lifecycle_management +import storage_enable_versioning +import storage_generate_signed_post_policy_v4 +import storage_generate_signed_url_v2 +import storage_generate_signed_url_v4 +import storage_generate_upload_signed_url_v4 +import storage_get_bucket_labels +import storage_get_bucket_metadata +import storage_get_metadata +import storage_get_service_account +import storage_list_buckets +import storage_list_file_archived_generations +import storage_list_files +import storage_list_files_with_prefix +import storage_make_public +import storage_move_file +import storage_object_get_kms_key +import storage_remove_bucket_label +import storage_remove_cors_configuration +import storage_rename_file +import storage_set_bucket_default_kms_key +import storage_set_metadata +import storage_upload_file +import storage_upload_with_kms_key + +KMS_KEY = os.environ["CLOUD_KMS_KEY"] + + +def test_enable_default_kms_key(test_bucket): + storage_set_bucket_default_kms_key.enable_default_kms_key( + bucket_name=test_bucket.name, kms_key_name=KMS_KEY + ) + time.sleep(2) # Let change propagate as needed + bucket = storage.Client().get_bucket(test_bucket.name) + assert bucket.default_kms_key_name.startswith(KMS_KEY) + bucket.default_kms_key_name = None + bucket.patch() + + +def test_get_bucket_labels(test_bucket): + storage_get_bucket_labels.get_bucket_labels(test_bucket.name) + + +def test_add_bucket_label(test_bucket, capsys): + storage_add_bucket_label.add_bucket_label(test_bucket.name) + out, _ = capsys.readouterr() + assert "example" in out + + +def test_remove_bucket_label(test_bucket, capsys): + storage_add_bucket_label.add_bucket_label(test_bucket.name) + storage_remove_bucket_label.remove_bucket_label(test_bucket.name) + out, _ = capsys.readouterr() + assert "Removed labels" in out + + +@pytest.fixture(scope="module") +def test_bucket(): + """Yields a bucket that is deleted after the test completes.""" + bucket = None + while bucket is None or bucket.exists(): + bucket_name = "storage-snippets-test-{}".format(uuid.uuid4()) + bucket = storage.Client().bucket(bucket_name) + bucket.create() + yield bucket + bucket.delete(force=True) + + +@pytest.fixture(scope="function") +def test_public_bucket(): + # The new projects don't allow to make a bucket available to public, so + # for some tests we need to use the old main project for now. + original_value = os.environ['GOOGLE_CLOUD_PROJECT'] + os.environ['GOOGLE_CLOUD_PROJECT'] = os.environ['MAIN_GOOGLE_CLOUD_PROJECT'] + bucket = None + while bucket is None or bucket.exists(): + storage_client = storage.Client() + bucket_name = "storage-snippets-test-{}".format(uuid.uuid4()) + bucket = storage_client.bucket(bucket_name) + storage_client.create_bucket(bucket) + yield bucket + bucket.delete(force=True) + # Set the value back. + os.environ['GOOGLE_CLOUD_PROJECT'] = original_value + + +@pytest.fixture +def test_blob(test_bucket): + """Yields a blob that is deleted after the test completes.""" + bucket = test_bucket + blob = bucket.blob("storage_snippets_test_sigil-{}".format(uuid.uuid4())) + blob.upload_from_string("Hello, is it me you're looking for?") + yield blob + + +@pytest.fixture(scope="function") +def test_public_blob(test_public_bucket): + """Yields a blob that is deleted after the test completes.""" + bucket = test_public_bucket + blob = bucket.blob("storage_snippets_test_sigil-{}".format(uuid.uuid4())) + blob.upload_from_string("Hello, is it me you're looking for?") + yield blob + + +@pytest.fixture +def test_bucket_create(): + """Yields a bucket object that is deleted after the test completes.""" + bucket = None + while bucket is None or bucket.exists(): + bucket_name = "storage-snippets-test-{}".format(uuid.uuid4()) + bucket = storage.Client().bucket(bucket_name) + yield bucket + bucket.delete(force=True) + + +def test_list_buckets(test_bucket, capsys): + storage_list_buckets.list_buckets() + out, _ = capsys.readouterr() + assert test_bucket.name in out + + +def test_list_blobs(test_blob, capsys): + storage_list_files.list_blobs(test_blob.bucket.name) + out, _ = capsys.readouterr() + assert test_blob.name in out + + +def test_bucket_metadata(test_bucket, capsys): + storage_get_bucket_metadata.bucket_metadata(test_bucket.name) + out, _ = capsys.readouterr() + assert test_bucket.name in out + + +def test_list_blobs_with_prefix(test_blob, capsys): + storage_list_files_with_prefix.list_blobs_with_prefix( + test_blob.bucket.name, prefix="storage_snippets" + ) + out, _ = capsys.readouterr() + assert test_blob.name in out + + +def test_upload_blob(test_bucket): + with tempfile.NamedTemporaryFile() as source_file: + source_file.write(b"test") + + storage_upload_file.upload_blob( + test_bucket.name, source_file.name, "test_upload_blob" + ) + + +def test_upload_blob_with_kms(test_bucket): + with tempfile.NamedTemporaryFile() as source_file: + source_file.write(b"test") + storage_upload_with_kms_key.upload_blob_with_kms( + test_bucket.name, source_file.name, "test_upload_blob_encrypted", KMS_KEY + ) + bucket = storage.Client().bucket(test_bucket.name) + kms_blob = bucket.get_blob("test_upload_blob_encrypted") + assert kms_blob.kms_key_name.startswith(KMS_KEY) + + +def test_download_blob(test_blob): + with tempfile.NamedTemporaryFile() as dest_file: + storage_download_file.download_blob( + test_blob.bucket.name, test_blob.name, dest_file.name + ) + + assert dest_file.read() + + +def test_blob_metadata(test_blob, capsys): + storage_get_metadata.blob_metadata(test_blob.bucket.name, test_blob.name) + out, _ = capsys.readouterr() + assert test_blob.name in out + + +def test_set_blob_metadata(test_blob, capsys): + storage_set_metadata.set_blob_metadata(test_blob.bucket.name, test_blob.name) + out, _ = capsys.readouterr() + assert test_blob.name in out + + +def test_delete_blob(test_blob): + storage_delete_file.delete_blob(test_blob.bucket.name, test_blob.name) + + +def test_make_blob_public(test_public_blob): + storage_make_public.make_blob_public( + test_public_blob.bucket.name, test_public_blob.name) + + r = requests.get(test_public_blob.public_url) + assert r.text == "Hello, is it me you're looking for?" + + +def test_generate_signed_url(test_blob, capsys): + url = storage_generate_signed_url_v2.generate_signed_url( + test_blob.bucket.name, test_blob.name + ) + + r = requests.get(url) + assert r.text == "Hello, is it me you're looking for?" + + +def test_generate_download_signed_url_v4(test_blob, capsys): + url = storage_generate_signed_url_v4.generate_download_signed_url_v4( + test_blob.bucket.name, test_blob.name + ) + + r = requests.get(url) + assert r.text == "Hello, is it me you're looking for?" + + +def test_generate_upload_signed_url_v4(test_bucket, capsys): + blob_name = "storage_snippets_test_upload" + content = b"Uploaded via v4 signed url" + url = storage_generate_upload_signed_url_v4.generate_upload_signed_url_v4( + test_bucket.name, blob_name + ) + + requests.put( + url, data=content, headers={"content-type": "application/octet-stream"}, + ) + + bucket = storage.Client().bucket(test_bucket.name) + blob = bucket.blob(blob_name) + assert blob.download_as_string() == content + + +def test_generate_signed_policy_v4(test_bucket, capsys): + blob_name = "storage_snippets_test_form" + short_name = storage_generate_signed_post_policy_v4 + form = short_name.generate_signed_post_policy_v4(test_bucket.name, blob_name) + assert "name='key' value='{}'".format(blob_name) in form + assert "name='x-goog-signature'" in form + assert "name='x-goog-date'" in form + assert "name='x-goog-credential'" in form + assert "name='x-goog-algorithm' value='GOOG4-RSA-SHA256'" in form + assert "name='policy'" in form + assert "name='x-goog-meta-test' value='data'" in form + assert "type='file' name='file'/>" in form + + +def test_rename_blob(test_blob): + bucket = storage.Client().bucket(test_blob.bucket.name) + + try: + bucket.delete_blob("test_rename_blob") + except google.cloud.exceptions.exceptions.NotFound: + print("test_rename_blob not found in bucket {}".format(bucket.name)) + + storage_rename_file.rename_blob(bucket.name, test_blob.name, "test_rename_blob") + + assert bucket.get_blob("test_rename_blob") is not None + assert bucket.get_blob(test_blob.name) is None + + +def test_move_blob(test_bucket_create, test_blob): + bucket = test_blob.bucket + storage.Client().create_bucket(test_bucket_create) + + try: + test_bucket_create.delete_blob("test_move_blob") + except google.cloud.exceptions.NotFound: + print("test_move_blob not found in bucket {}".format(test_bucket_create.name)) + + storage_move_file.move_blob( + bucket.name, test_blob.name, test_bucket_create.name, "test_move_blob" + ) + + assert test_bucket_create.get_blob("test_move_blob") is not None + assert bucket.get_blob(test_blob.name) is None + + +def test_copy_blob(test_blob): + bucket = storage.Client().bucket(test_blob.bucket.name) + + try: + bucket.delete_blob("test_copy_blob") + except google.cloud.exceptions.NotFound: + pass + + storage_copy_file.copy_blob( + bucket.name, test_blob.name, bucket.name, "test_copy_blob" + ) + + assert bucket.get_blob("test_copy_blob") is not None + assert bucket.get_blob(test_blob.name) is not None + + +def test_versioning(test_bucket, capsys): + bucket = storage_enable_versioning.enable_versioning(test_bucket) + out, _ = capsys.readouterr() + assert "Versioning was enabled for bucket" in out + assert bucket.versioning_enabled is True + + bucket = storage_disable_versioning.disable_versioning(test_bucket) + out, _ = capsys.readouterr() + assert "Versioning was disabled for bucket" in out + assert bucket.versioning_enabled is False + + +def test_bucket_lifecycle_management(test_bucket, capsys): + bucket = storage_enable_bucket_lifecycle_management.enable_bucket_lifecycle_management( + test_bucket + ) + out, _ = capsys.readouterr() + assert "[]" in out + assert "Lifecycle management is enable" in out + assert len(list(bucket.lifecycle_rules)) > 0 + + bucket = storage_disable_bucket_lifecycle_management.disable_bucket_lifecycle_management( + test_bucket + ) + out, _ = capsys.readouterr() + assert "[]" in out + assert len(list(bucket.lifecycle_rules)) == 0 + + +def test_create_bucket_class_location(test_bucket_create): + bucket = storage_create_bucket_class_location.create_bucket_class_location( + test_bucket_create.name + ) + + assert bucket.location == "US" + assert bucket.storage_class == "COLDLINE" + + +def test_bucket_delete_default_kms_key(test_bucket, capsys): + test_bucket.default_kms_key_name = KMS_KEY + test_bucket.patch() + + assert test_bucket.default_kms_key_name == KMS_KEY + + bucket = storage_bucket_delete_default_kms_key.bucket_delete_default_kms_key( + test_bucket.name + ) + + out, _ = capsys.readouterr() + assert bucket.default_kms_key_name is None + assert bucket.name in out + + +def test_get_service_account(capsys): + storage_get_service_account.get_service_account() + + out, _ = capsys.readouterr() + + assert "@gs-project-accounts.iam.gserviceaccount.com" in out + + +def test_download_public_file(test_public_blob): + storage_make_public.make_blob_public( + test_public_blob.bucket.name, test_public_blob.name) + with tempfile.NamedTemporaryFile() as dest_file: + storage_download_public_file.download_public_file( + test_public_blob.bucket.name, test_public_blob.name, dest_file.name + ) + + assert dest_file.read() == b"Hello, is it me you're looking for?" + + +def test_define_bucket_website_configuration(test_bucket): + bucket = storage_define_bucket_website_configuration.define_bucket_website_configuration( + test_bucket.name, "index.html", "404.html" + ) + + website_val = {"mainPageSuffix": "index.html", "notFoundPage": "404.html"} + + assert bucket._properties["website"] == website_val + + +def test_object_get_kms_key(test_bucket): + with tempfile.NamedTemporaryFile() as source_file: + storage_upload_with_kms_key.upload_blob_with_kms( + test_bucket.name, source_file.name, "test_upload_blob_encrypted", KMS_KEY + ) + kms_key = storage_object_get_kms_key.object_get_kms_key( + test_bucket.name, "test_upload_blob_encrypted" + ) + + assert kms_key.startswith(KMS_KEY) + + +def test_storage_compose_file(test_bucket): + source_files = ["test_upload_blob_1", "test_upload_blob_2"] + for source in source_files: + blob = test_bucket.blob(source) + blob.upload_from_string(source) + + with tempfile.NamedTemporaryFile() as dest_file: + destination = storage_compose_file.compose_file( + test_bucket.name, source_files[0], source_files[1], dest_file.name + ) + composed = destination.download_as_string() + + assert composed.decode("utf-8") == source_files[0] + source_files[1] + + +def test_cors_configuration(test_bucket, capsys): + bucket = storage_cors_configuration.cors_configuration(test_bucket) + out, _ = capsys.readouterr() + assert "Set CORS policies for bucket" in out + assert len(bucket.cors) > 0 + + bucket = storage_remove_cors_configuration.remove_cors_configuration(test_bucket) + out, _ = capsys.readouterr() + assert "Remove CORS policies for bucket" in out + assert len(bucket.cors) == 0 + + +def test_delete_blobs_archived_generation(test_blob, capsys): + storage_delete_file_archived_generation.delete_file_archived_generation( + test_blob.bucket.name, test_blob.name, test_blob.generation + ) + out, _ = capsys.readouterr() + assert "blob " + test_blob.name + " was deleted" in out + blob = test_blob.bucket.get_blob(test_blob.name, generation=test_blob.generation) + assert blob is None + + +def test_change_default_storage_class(test_bucket, capsys): + bucket = storage_change_default_storage_class.change_default_storage_class( + test_bucket + ) + out, _ = capsys.readouterr() + assert "Default storage class for bucket" in out + assert bucket.storage_class == 'COLDLINE' + + +def test_change_file_storage_class(test_blob, capsys): + blob = storage_change_file_storage_class.change_file_storage_class( + test_blob.bucket.name, test_blob.name + ) + out, _ = capsys.readouterr() + assert "Blob {} in bucket {}". format(blob.name, blob.bucket.name) in out + assert blob.storage_class == 'NEARLINE' + + +def test_copy_file_archived_generation(test_blob): + bucket = storage.Client().bucket(test_blob.bucket.name) + + try: + bucket.delete_blob("test_copy_blob") + except google.cloud.exceptions.NotFound: + pass + + storage_copy_file_archived_generation.copy_file_archived_generation( + bucket.name, test_blob.name, bucket.name, "test_copy_blob", test_blob.generation + ) + + assert bucket.get_blob("test_copy_blob") is not None + assert bucket.get_blob(test_blob.name) is not None + + +def test_list_blobs_archived_generation(test_blob, capsys): + storage_list_file_archived_generations.list_file_archived_generations( + test_blob.bucket.name + ) + out, _ = capsys.readouterr() + assert str(test_blob.generation) in out + + +def test_storage_configure_retries(test_blob, capsys): + storage_configure_retries.configure_retries(test_blob.bucket.name, test_blob.name) + + # This simply checks if the retry configurations were set and printed as intended. + out, _ = capsys.readouterr() + assert "The following library method is customized to be retried" in out + assert "_should_retry" in out + assert "initial=1.5, maximum=45.0, multiplier=1.2, deadline=500.0" in out diff --git a/storage/samples/snippets/storage_activate_hmac_key.py b/storage/samples/snippets/storage_activate_hmac_key.py new file mode 100644 index 00000000000..e77cd80665d --- /dev/null +++ b/storage/samples/snippets/storage_activate_hmac_key.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_activate_hmac_key] +from google.cloud import storage + + +def activate_key(access_id, project_id): + """ + Activate the HMAC key with the given access ID. + """ + # project_id = "Your Google Cloud project ID" + # access_id = "ID of an inactive HMAC key" + + storage_client = storage.Client(project=project_id) + + hmac_key = storage_client.get_hmac_key_metadata( + access_id, project_id=project_id + ) + hmac_key.state = "ACTIVE" + hmac_key.update() + + print("The HMAC key metadata is:") + print("Service Account Email: {}".format(hmac_key.service_account_email)) + print("Key ID: {}".format(hmac_key.id)) + print("Access ID: {}".format(hmac_key.access_id)) + print("Project ID: {}".format(hmac_key.project)) + print("State: {}".format(hmac_key.state)) + print("Created At: {}".format(hmac_key.time_created)) + print("Updated At: {}".format(hmac_key.updated)) + print("Etag: {}".format(hmac_key.etag)) + return hmac_key + + +# [END storage_activate_hmac_key] + +if __name__ == "__main__": + activate_key(access_id=sys.argv[1], project_id=sys.argv[2]) diff --git a/storage/samples/snippets/storage_add_bucket_conditional_iam_binding.py b/storage/samples/snippets/storage_add_bucket_conditional_iam_binding.py new file mode 100644 index 00000000000..ddc0fc0286d --- /dev/null +++ b/storage/samples/snippets/storage_add_bucket_conditional_iam_binding.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_add_bucket_conditional_iam_binding] +from google.cloud import storage + + +def add_bucket_conditional_iam_binding( + bucket_name, role, title, description, expression, members +): + """Add a conditional IAM binding to a bucket's IAM policy.""" + # bucket_name = "your-bucket-name" + # role = "IAM role, e.g. roles/storage.objectViewer" + # members = {"IAM identity, e.g. user: name@example.com}" + # title = "Condition title." + # description = "Condition description." + # expression = "Condition expression." + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + policy = bucket.get_iam_policy(requested_policy_version=3) + + # Set the policy's version to 3 to use condition in bindings. + policy.version = 3 + + policy.bindings.append( + { + "role": role, + "members": members, + "condition": { + "title": title, + "description": description, + "expression": expression, + }, + } + ) + + bucket.set_iam_policy(policy) + + print("Added the following member(s) with role {} to {}:".format(role, bucket_name)) + + for member in members: + print(" {}".format(member)) + + print("with condition:") + print(" Title: {}".format(title)) + print(" Description: {}".format(description)) + print(" Expression: {}".format(expression)) + + +# [END storage_add_bucket_conditional_iam_binding] + + +if __name__ == "__main__": + add_bucket_conditional_iam_binding( + bucket_name=sys.argv[1], + role=sys.argv[2], + title=sys.argv[3], + description=sys.argv[4], + expression=sys.argv[5], + members=set(sys.argv[6::]), + ) diff --git a/storage/samples/snippets/storage_add_bucket_default_owner.py b/storage/samples/snippets/storage_add_bucket_default_owner.py new file mode 100644 index 00000000000..932b1328f3f --- /dev/null +++ b/storage/samples/snippets/storage_add_bucket_default_owner.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_add_bucket_default_owner] +from google.cloud import storage + + +def add_bucket_default_owner(bucket_name, user_email): + """Adds a user as an owner in the given bucket's default object access + control list.""" + # bucket_name = "your-bucket-name" + # user_email = "name@example.com" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + # Reload fetches the current ACL from Cloud Storage. + bucket.acl.reload() + + # You can also use `group`, `domain`, `all_authenticated` and `all` to + # grant access to different types of entities. You can also use + # `grant_read` or `grant_write` to grant different roles. + bucket.default_object_acl.user(user_email).grant_owner() + bucket.default_object_acl.save() + + print( + "Added user {} as an owner in the default acl on bucket {}.".format( + user_email, bucket_name + ) + ) + + +# [END storage_add_bucket_default_owner] + +if __name__ == "__main__": + add_bucket_default_owner(bucket_name=sys.argv[1], user_email=sys.argv[2]) diff --git a/storage/samples/snippets/storage_add_bucket_iam_member.py b/storage/samples/snippets/storage_add_bucket_iam_member.py new file mode 100644 index 00000000000..727f1848325 --- /dev/null +++ b/storage/samples/snippets/storage_add_bucket_iam_member.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_add_bucket_iam_member] +from google.cloud import storage + + +def add_bucket_iam_member(bucket_name, role, member): + """Add a new member to an IAM Policy""" + # bucket_name = "your-bucket-name" + # role = "IAM role, e.g., roles/storage.objectViewer" + # member = "IAM identity, e.g., user: name@example.com" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + policy = bucket.get_iam_policy(requested_policy_version=3) + + policy.bindings.append({"role": role, "members": {member}}) + + bucket.set_iam_policy(policy) + + print("Added {} with role {} to {}.".format(member, role, bucket_name)) + + +# [END storage_add_bucket_iam_member] + + +if __name__ == "__main__": + add_bucket_iam_member(bucket_name=sys.argv[1], role=sys.argv[2], member=sys.argv[3]) diff --git a/storage/samples/snippets/storage_add_bucket_label.py b/storage/samples/snippets/storage_add_bucket_label.py new file mode 100644 index 00000000000..8ae8fe1f407 --- /dev/null +++ b/storage/samples/snippets/storage_add_bucket_label.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# [START storage_add_bucket_label] +import pprint +# [END storage_add_bucket_label] +import sys +# [START storage_add_bucket_label] + +from google.cloud import storage + + +def add_bucket_label(bucket_name): + """Add a label to a bucket.""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + labels = bucket.labels + labels["example"] = "label" + bucket.labels = labels + bucket.patch() + + print("Updated labels on {}.".format(bucket.name)) + pprint.pprint(bucket.labels) + + +# [END storage_add_bucket_label] + +if __name__ == "__main__": + add_bucket_label(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_add_bucket_owner.py b/storage/samples/snippets/storage_add_bucket_owner.py new file mode 100644 index 00000000000..acdb60dc5a2 --- /dev/null +++ b/storage/samples/snippets/storage_add_bucket_owner.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_add_bucket_owner] +from google.cloud import storage + + +def add_bucket_owner(bucket_name, user_email): + """Adds a user as an owner on the given bucket.""" + # bucket_name = "your-bucket-name" + # user_email = "name@example.com" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + + # Reload fetches the current ACL from Cloud Storage. + bucket.acl.reload() + + # You can also use `group()`, `domain()`, `all_authenticated()` and `all()` + # to grant access to different types of entities. + # You can also use `grant_read()` or `grant_write()` to grant different + # roles. + bucket.acl.user(user_email).grant_owner() + bucket.acl.save() + + print( + "Added user {} as an owner on bucket {}.".format( + user_email, bucket_name + ) + ) + + +# [END storage_add_bucket_owner] + +if __name__ == "__main__": + add_bucket_owner(bucket_name=sys.argv[1], user_email=sys.argv[2]) diff --git a/storage/samples/snippets/storage_add_file_owner.py b/storage/samples/snippets/storage_add_file_owner.py new file mode 100644 index 00000000000..9e9342590c4 --- /dev/null +++ b/storage/samples/snippets/storage_add_file_owner.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_add_file_owner] +from google.cloud import storage + + +def add_blob_owner(bucket_name, blob_name, user_email): + """Adds a user as an owner on the given blob.""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + # user_email = "name@example.com" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + # Reload fetches the current ACL from Cloud Storage. + blob.acl.reload() + + # You can also use `group`, `domain`, `all_authenticated` and `all` to + # grant access to different types of entities. You can also use + # `grant_read` or `grant_write` to grant different roles. + blob.acl.user(user_email).grant_owner() + blob.acl.save() + + print( + "Added user {} as an owner on blob {} in bucket {}.".format( + user_email, blob_name, bucket_name + ) + ) + + +# [END storage_add_file_owner] + +if __name__ == "__main__": + add_blob_owner( + bucket_name=sys.argv[1], blob_name=sys.argv[2], user_email=sys.argv[3], + ) diff --git a/storage/samples/snippets/storage_bucket_delete_default_kms_key.py b/storage/samples/snippets/storage_bucket_delete_default_kms_key.py new file mode 100644 index 00000000000..3df23767df0 --- /dev/null +++ b/storage/samples/snippets/storage_bucket_delete_default_kms_key.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_bucket_delete_default_kms_key] +from google.cloud import storage + + +def bucket_delete_default_kms_key(bucket_name): + """Delete a default KMS key of bucket""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.default_kms_key_name = None + bucket.patch() + + print("Default KMS key was removed from {}".format(bucket.name)) + return bucket + + +# [END storage_bucket_delete_default_kms_key] + +if __name__ == "__main__": + bucket_delete_default_kms_key(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_change_default_storage_class.py b/storage/samples/snippets/storage_change_default_storage_class.py new file mode 100644 index 00000000000..8a72719bad3 --- /dev/null +++ b/storage/samples/snippets/storage_change_default_storage_class.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_change_default_storage_class] +from google.cloud import storage +from google.cloud.storage import constants + + +def change_default_storage_class(bucket_name): + """Change the default storage class of the bucket""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.storage_class = constants.COLDLINE_STORAGE_CLASS + bucket.patch() + + print("Default storage class for bucket {} has been set to {}".format(bucket_name, bucket.storage_class)) + return bucket + + +# [END storage_change_default_storage_class] + +if __name__ == "__main__": + change_default_storage_class(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_change_file_storage_class.py b/storage/samples/snippets/storage_change_file_storage_class.py new file mode 100644 index 00000000000..d5dda56a709 --- /dev/null +++ b/storage/samples/snippets/storage_change_file_storage_class.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_change_file_storage_class] +from google.cloud import storage + + +def change_file_storage_class(bucket_name, blob_name): + """Change the default storage class of the blob""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + blob = bucket.get_blob(blob_name) + blob.update_storage_class("NEARLINE") + + print( + "Blob {} in bucket {} had its storage class set to {}".format( + blob_name, + bucket_name, + blob.storage_class + ) + ) + return blob +# [END storage_change_file_storage_class] + + +if __name__ == "__main__": + change_file_storage_class(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_compose_file.py b/storage/samples/snippets/storage_compose_file.py new file mode 100644 index 00000000000..2c1443f22f4 --- /dev/null +++ b/storage/samples/snippets/storage_compose_file.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_compose_file] +from google.cloud import storage + + +def compose_file(bucket_name, first_blob_name, second_blob_name, destination_blob_name): + """Concatenate source blobs into destination blob.""" + # bucket_name = "your-bucket-name" + # first_blob_name = "first-object-name" + # second_blob_name = "second-blob-name" + # destination_blob_name = "destination-object-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + destination = bucket.blob(destination_blob_name) + destination.content_type = "text/plain" + + # sources is a list of Blob instances, up to the max of 32 instances per request + sources = [bucket.get_blob(first_blob_name), bucket.get_blob(second_blob_name)] + destination.compose(sources) + + print( + "New composite object {} in the bucket {} was created by combining {} and {}".format( + destination_blob_name, bucket_name, first_blob_name, second_blob_name + ) + ) + return destination + + +# [END storage_compose_file] + +if __name__ == "__main__": + compose_file( + bucket_name=sys.argv[1], + first_blob_name=sys.argv[2], + second_blob_name=sys.argv[3], + destination_blob_name=sys.argv[4], + ) diff --git a/storage/samples/snippets/storage_configure_retries.py b/storage/samples/snippets/storage_configure_retries.py new file mode 100644 index 00000000000..9543111b3a7 --- /dev/null +++ b/storage/samples/snippets/storage_configure_retries.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that configures retries on an operation call. +This sample is used on this page: + https://cloud.google.com/storage/docs/retry-strategy +For more information, see README.md. +""" + +# [START storage_configure_retries] +from google.cloud import storage +from google.cloud.storage.retry import DEFAULT_RETRY + + +def configure_retries(bucket_name, blob_name): + """Configures retries with customizations.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + # The ID of your GCS object + # blob_name = "your-object-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + # Customize retry with a deadline of 500 seconds (default=120 seconds). + modified_retry = DEFAULT_RETRY.with_deadline(500.0) + # Customize retry with an initial wait time of 1.5 (default=1.0). + # Customize retry with a wait time multiplier per iteration of 1.2 (default=2.0). + # Customize retry with a maximum wait time of 45.0 (default=60.0). + modified_retry = modified_retry.with_delay(initial=1.5, multiplier=1.2, maximum=45.0) + + # blob.delete() uses DEFAULT_RETRY_IF_GENERATION_SPECIFIED by default. + # Override with modified_retry so the function retries even if the generation + # number is not specified. + print( + f"The following library method is customized to be retried according to the following configurations: {modified_retry}" + ) + + blob.delete(retry=modified_retry) + print("Blob {} deleted with a customized retry strategy.".format(blob_name)) + + +# [END storage_configure_retries] + + +if __name__ == "__main__": + configure_retries(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_copy_file.py b/storage/samples/snippets/storage_copy_file.py new file mode 100644 index 00000000000..5d36aa94b44 --- /dev/null +++ b/storage/samples/snippets/storage_copy_file.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_copy_file] +from google.cloud import storage + + +def copy_blob( + bucket_name, blob_name, destination_bucket_name, destination_blob_name +): + """Copies a blob from one bucket to another with a new name.""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + # destination_bucket_name = "destination-bucket-name" + # destination_blob_name = "destination-object-name" + + storage_client = storage.Client() + + source_bucket = storage_client.bucket(bucket_name) + source_blob = source_bucket.blob(blob_name) + destination_bucket = storage_client.bucket(destination_bucket_name) + + blob_copy = source_bucket.copy_blob( + source_blob, destination_bucket, destination_blob_name + ) + + print( + "Blob {} in bucket {} copied to blob {} in bucket {}.".format( + source_blob.name, + source_bucket.name, + blob_copy.name, + destination_bucket.name, + ) + ) + + +# [END storage_copy_file] + +if __name__ == "__main__": + copy_blob( + bucket_name=sys.argv[1], + blob_name=sys.argv[2], + destination_bucket_name=sys.argv[3], + destination_blob_name=sys.argv[4], + ) diff --git a/storage/samples/snippets/storage_copy_file_archived_generation.py b/storage/samples/snippets/storage_copy_file_archived_generation.py new file mode 100644 index 00000000000..988ebcbebd8 --- /dev/null +++ b/storage/samples/snippets/storage_copy_file_archived_generation.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_copy_file_archived_generation] +from google.cloud import storage + + +def copy_file_archived_generation( + bucket_name, blob_name, destination_bucket_name, destination_blob_name, generation +): + """Copies a blob from one bucket to another with a new name with the same generation.""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + # destination_bucket_name = "destination-bucket-name" + # destination_blob_name = "destination-object-name" + # generation = 1579287380533984 + + storage_client = storage.Client() + + source_bucket = storage_client.bucket(bucket_name) + source_blob = source_bucket.blob(blob_name) + destination_bucket = storage_client.bucket(destination_bucket_name) + + blob_copy = source_bucket.copy_blob( + source_blob, destination_bucket, destination_blob_name, source_generation=generation + ) + + print( + "Generation {} of the blob {} in bucket {} copied to blob {} in bucket {}.".format( + source_blob.generation, + source_blob.name, + source_bucket.name, + blob_copy.name, + destination_bucket.name, + ) + ) + + +# [END storage_copy_file_archived_generation] + +if __name__ == "__main__": + copy_file_archived_generation( + bucket_name=sys.argv[1], + blob_name=sys.argv[2], + destination_bucket_name=sys.argv[3], + destination_blob_name=sys.argv[4], + generation=sys.argv[5] + ) diff --git a/storage/samples/snippets/storage_cors_configuration.py b/storage/samples/snippets/storage_cors_configuration.py new file mode 100644 index 00000000000..3d2595a9ddb --- /dev/null +++ b/storage/samples/snippets/storage_cors_configuration.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_cors_configuration] +from google.cloud import storage + + +def cors_configuration(bucket_name): + """Set a bucket's CORS policies configuration.""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + bucket.cors = [ + { + "origin": ["*"], + "responseHeader": [ + "Content-Type", + "x-goog-resumable"], + "method": ['PUT', 'POST'], + "maxAgeSeconds": 3600 + } + ] + bucket.patch() + + print("Set CORS policies for bucket {} is {}".format(bucket.name, bucket.cors)) + return bucket + + +# [END storage_cors_configuration] + +if __name__ == "__main__": + cors_configuration(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_create_bucket.py b/storage/samples/snippets/storage_create_bucket.py new file mode 100644 index 00000000000..aaee9e23485 --- /dev/null +++ b/storage/samples/snippets/storage_create_bucket.py @@ -0,0 +1,37 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_create_bucket] +from google.cloud import storage + + +def create_bucket(bucket_name): + """Creates a new bucket.""" + # bucket_name = "your-new-bucket-name" + + storage_client = storage.Client() + + bucket = storage_client.create_bucket(bucket_name) + + print("Bucket {} created".format(bucket.name)) + + +# [END storage_create_bucket] + +if __name__ == "__main__": + create_bucket(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_create_bucket_class_location.py b/storage/samples/snippets/storage_create_bucket_class_location.py new file mode 100644 index 00000000000..64c2652d77a --- /dev/null +++ b/storage/samples/snippets/storage_create_bucket_class_location.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_create_bucket_class_location] +from google.cloud import storage + + +def create_bucket_class_location(bucket_name): + """Create a new bucket in specific location with storage class""" + # bucket_name = "your-new-bucket-name" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + bucket.storage_class = "COLDLINE" + new_bucket = storage_client.create_bucket(bucket, location="us") + + print( + "Created bucket {} in {} with storage class {}".format( + new_bucket.name, new_bucket.location, new_bucket.storage_class + ) + ) + return new_bucket + + +# [END storage_create_bucket_class_location] + +if __name__ == "__main__": + create_bucket_class_location(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_create_hmac_key.py b/storage/samples/snippets/storage_create_hmac_key.py new file mode 100644 index 00000000000..27a418c39bd --- /dev/null +++ b/storage/samples/snippets/storage_create_hmac_key.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_create_hmac_key] +from google.cloud import storage + + +def create_key(project_id, service_account_email): + """ + Create a new HMAC key using the given project and service account. + """ + # project_id = 'Your Google Cloud project ID' + # service_account_email = 'Service account used to generate the HMAC key' + + storage_client = storage.Client(project=project_id) + + hmac_key, secret = storage_client.create_hmac_key( + service_account_email=service_account_email, project_id=project_id + ) + + print("The base64 encoded secret is {}".format(secret)) + print("Do not miss that secret, there is no API to recover it.") + print("The HMAC key metadata is:") + print("Service Account Email: {}".format(hmac_key.service_account_email)) + print("Key ID: {}".format(hmac_key.id)) + print("Access ID: {}".format(hmac_key.access_id)) + print("Project ID: {}".format(hmac_key.project)) + print("State: {}".format(hmac_key.state)) + print("Created At: {}".format(hmac_key.time_created)) + print("Updated At: {}".format(hmac_key.updated)) + print("Etag: {}".format(hmac_key.etag)) + return hmac_key + + +# [END storage_create_hmac_key] + +if __name__ == "__main__": + create_key(project_id=sys.argv[1], service_account_email=sys.argv[2]) diff --git a/storage/samples/snippets/storage_deactivate_hmac_key.py b/storage/samples/snippets/storage_deactivate_hmac_key.py new file mode 100644 index 00000000000..389efb998d5 --- /dev/null +++ b/storage/samples/snippets/storage_deactivate_hmac_key.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_deactivate_hmac_key] +from google.cloud import storage + + +def deactivate_key(access_id, project_id): + """ + Deactivate the HMAC key with the given access ID. + """ + # project_id = "Your Google Cloud project ID" + # access_id = "ID of an active HMAC key" + + storage_client = storage.Client(project=project_id) + + hmac_key = storage_client.get_hmac_key_metadata( + access_id, project_id=project_id + ) + hmac_key.state = "INACTIVE" + hmac_key.update() + + print("The HMAC key is now inactive.") + print("The HMAC key metadata is:") + print("Service Account Email: {}".format(hmac_key.service_account_email)) + print("Key ID: {}".format(hmac_key.id)) + print("Access ID: {}".format(hmac_key.access_id)) + print("Project ID: {}".format(hmac_key.project)) + print("State: {}".format(hmac_key.state)) + print("Created At: {}".format(hmac_key.time_created)) + print("Updated At: {}".format(hmac_key.updated)) + print("Etag: {}".format(hmac_key.etag)) + return hmac_key + + +# [END storage_deactivate_hmac_key] + +if __name__ == "__main__": + deactivate_key(access_id=sys.argv[1], project_id=sys.argv[2]) diff --git a/storage/samples/snippets/storage_define_bucket_website_configuration.py b/storage/samples/snippets/storage_define_bucket_website_configuration.py new file mode 100644 index 00000000000..ce6c7e66cdb --- /dev/null +++ b/storage/samples/snippets/storage_define_bucket_website_configuration.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_define_bucket_website_configuration] +from google.cloud import storage + + +def define_bucket_website_configuration(bucket_name, main_page_suffix, not_found_page): + """Configure website-related properties of bucket""" + # bucket_name = "your-bucket-name" + # main_page_suffix = "index.html" + # not_found_page = "404.html" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.configure_website(main_page_suffix, not_found_page) + bucket.patch() + + print( + "Static website bucket {} is set up to use {} as the index page and {} as the 404 page".format( + bucket.name, main_page_suffix, not_found_page + ) + ) + return bucket + + +# [END storage_define_bucket_website_configuration] + +if __name__ == "__main__": + define_bucket_website_configuration( + bucket_name=sys.argv[1], + main_page_suffix=sys.argv[2], + not_found_page=sys.argv[3], + ) diff --git a/storage/samples/snippets/storage_delete_bucket.py b/storage/samples/snippets/storage_delete_bucket.py new file mode 100644 index 00000000000..b3e264c74e7 --- /dev/null +++ b/storage/samples/snippets/storage_delete_bucket.py @@ -0,0 +1,38 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_delete_bucket] +from google.cloud import storage + + +def delete_bucket(bucket_name): + """Deletes a bucket. The bucket must be empty.""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.delete() + + print("Bucket {} deleted".format(bucket.name)) + + +# [END storage_delete_bucket] + +if __name__ == "__main__": + delete_bucket(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_delete_file.py b/storage/samples/snippets/storage_delete_file.py new file mode 100644 index 00000000000..1105f3725a1 --- /dev/null +++ b/storage/samples/snippets/storage_delete_file.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_delete_file] +from google.cloud import storage + + +def delete_blob(bucket_name, blob_name): + """Deletes a blob from the bucket.""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + blob.delete() + + print("Blob {} deleted.".format(blob_name)) + + +# [END storage_delete_file] + +if __name__ == "__main__": + delete_blob(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_delete_file_archived_generation.py b/storage/samples/snippets/storage_delete_file_archived_generation.py new file mode 100644 index 00000000000..4e490900100 --- /dev/null +++ b/storage/samples/snippets/storage_delete_file_archived_generation.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_delete_file_archived_generation] +from google.cloud import storage + + +def delete_file_archived_generation(bucket_name, blob_name, generation): + """Delete a blob in the bucket with the given generation.""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + # generation = 1579287380533984 + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.delete_blob(blob_name, generation=generation) + print( + "Generation {} of blob {} was deleted from {}".format( + generation, blob_name, bucket_name + ) + ) + + +# [END storage_delete_file_archived_generation] + + +if __name__ == "__main__": + delete_file_archived_generation( + bucket_name=sys.argv[1], + blob_name=sys.argv[2], + generation=sys.argv[3] + ) diff --git a/storage/samples/snippets/storage_delete_hmac_key.py b/storage/samples/snippets/storage_delete_hmac_key.py new file mode 100644 index 00000000000..403dc193b22 --- /dev/null +++ b/storage/samples/snippets/storage_delete_hmac_key.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_delete_hmac_key] +from google.cloud import storage + + +def delete_key(access_id, project_id): + """ + Delete the HMAC key with the given access ID. Key must have state INACTIVE + in order to succeed. + """ + # project_id = "Your Google Cloud project ID" + # access_id = "ID of an HMAC key (must be in INACTIVE state)" + + storage_client = storage.Client(project=project_id) + + hmac_key = storage_client.get_hmac_key_metadata( + access_id, project_id=project_id + ) + hmac_key.delete() + + print( + "The key is deleted, though it may still appear in list_hmac_keys()" + " results." + ) + + +# [END storage_delete_hmac_key] + +if __name__ == "__main__": + delete_key(access_id=sys.argv[1], project_id=sys.argv[2]) diff --git a/storage/samples/snippets/storage_disable_bucket_lifecycle_management.py b/storage/samples/snippets/storage_disable_bucket_lifecycle_management.py new file mode 100644 index 00000000000..9ef6971fb10 --- /dev/null +++ b/storage/samples/snippets/storage_disable_bucket_lifecycle_management.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2020 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_disable_bucket_lifecycle_management] +from google.cloud import storage + + +def disable_bucket_lifecycle_management(bucket_name): + """Disable lifecycle management for a bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.clear_lifecyle_rules() + bucket.patch() + rules = bucket.lifecycle_rules + + print("Lifecycle management is disable for bucket {} and the rules are {}".format(bucket_name, list(rules))) + return bucket + + +# [END storage_disable_bucket_lifecycle_management] + +if __name__ == "__main__": + disable_bucket_lifecycle_management(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_disable_default_event_based_hold.py b/storage/samples/snippets/storage_disable_default_event_based_hold.py new file mode 100644 index 00000000000..dff3ed3c129 --- /dev/null +++ b/storage/samples/snippets/storage_disable_default_event_based_hold.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_disable_default_event_based_hold] +from google.cloud import storage + + +def disable_default_event_based_hold(bucket_name): + """Disables the default event based hold on a given bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.default_event_based_hold = False + bucket.patch() + + print("Default event based hold was disabled for {}".format(bucket_name)) + + +# [END storage_disable_default_event_based_hold] + + +if __name__ == "__main__": + disable_default_event_based_hold(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_disable_requester_pays.py b/storage/samples/snippets/storage_disable_requester_pays.py new file mode 100644 index 00000000000..c49cc28eaab --- /dev/null +++ b/storage/samples/snippets/storage_disable_requester_pays.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_disable_requester_pays] +from google.cloud import storage + + +def disable_requester_pays(bucket_name): + """Disable a bucket's requesterpays metadata""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.requester_pays = False + bucket.patch() + + print("Requester Pays has been disabled for {}".format(bucket_name)) + + +# [END storage_disable_requester_pays] + + +if __name__ == "__main__": + disable_requester_pays(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_disable_uniform_bucket_level_access.py b/storage/samples/snippets/storage_disable_uniform_bucket_level_access.py new file mode 100644 index 00000000000..4f4691611b9 --- /dev/null +++ b/storage/samples/snippets/storage_disable_uniform_bucket_level_access.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_disable_uniform_bucket_level_access] +from google.cloud import storage + + +def disable_uniform_bucket_level_access(bucket_name): + """Disable uniform bucket-level access for a bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + bucket.iam_configuration.uniform_bucket_level_access_enabled = False + bucket.patch() + + print( + "Uniform bucket-level access was disabled for {}.".format(bucket.name) + ) + + +# [END storage_disable_uniform_bucket_level_access] + +if __name__ == "__main__": + disable_uniform_bucket_level_access(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_disable_versioning.py b/storage/samples/snippets/storage_disable_versioning.py new file mode 100644 index 00000000000..98832ba6856 --- /dev/null +++ b/storage/samples/snippets/storage_disable_versioning.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2020 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_disable_versioning] +from google.cloud import storage + + +def disable_versioning(bucket_name): + """Disable versioning for this bucket.""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.versioning_enabled = False + bucket.patch() + + print("Versioning was disabled for bucket {}".format(bucket)) + return bucket + + +# [END storage_disable_versioning] + +if __name__ == "__main__": + disable_versioning(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_download_encrypted_file.py b/storage/samples/snippets/storage_download_encrypted_file.py new file mode 100644 index 00000000000..ac7071fbefb --- /dev/null +++ b/storage/samples/snippets/storage_download_encrypted_file.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_download_encrypted_file] +import base64 +# [END storage_download_encrypted_file] +import sys +# [START storage_download_encrypted_file] + +from google.cloud import storage + + +def download_encrypted_blob( + bucket_name, + source_blob_name, + destination_file_name, + base64_encryption_key, +): + """Downloads a previously-encrypted blob from Google Cloud Storage. + + The encryption key provided must be the same key provided when uploading + the blob. + """ + # bucket_name = "your-bucket-name" + # source_blob_name = "storage-object-name" + # destination_file_name = "local/path/to/file" + # base64_encryption_key = "base64-encoded-encryption-key" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + + # Encryption key must be an AES256 key represented as a bytestring with + # 32 bytes. Since it's passed in as a base64 encoded string, it needs + # to be decoded. + encryption_key = base64.b64decode(base64_encryption_key) + blob = bucket.blob(source_blob_name, encryption_key=encryption_key) + + blob.download_to_filename(destination_file_name) + + print( + "Blob {} downloaded to {}.".format( + source_blob_name, destination_file_name + ) + ) + + +# [END storage_download_encrypted_file] + +if __name__ == "__main__": + download_encrypted_blob( + bucket_name=sys.argv[1], + source_blob_name=sys.argv[2], + destination_file_name=sys.argv[3], + base64_encryption_key=sys.argv[4], + ) diff --git a/storage/samples/snippets/storage_download_file.py b/storage/samples/snippets/storage_download_file.py new file mode 100644 index 00000000000..f8a1c93c83c --- /dev/null +++ b/storage/samples/snippets/storage_download_file.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_download_file] +from google.cloud import storage + + +def download_blob(bucket_name, source_blob_name, destination_file_name): + """Downloads a blob from the bucket.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The ID of your GCS object + # source_blob_name = "storage-object-name" + + # The path to which the file should be downloaded + # destination_file_name = "local/path/to/file" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + + # Construct a client side representation of a blob. + # Note `Bucket.blob` differs from `Bucket.get_blob` as it doesn't retrieve + # any content from Google Cloud Storage. As we don't need additional data, + # using `Bucket.blob` is preferred here. + blob = bucket.blob(source_blob_name) + blob.download_to_filename(destination_file_name) + + print( + "Downloaded storage object {} from bucket {} to local file {}.".format( + source_blob_name, bucket_name, destination_file_name + ) + ) + + +# [END storage_download_file] + +if __name__ == "__main__": + download_blob( + bucket_name=sys.argv[1], + source_blob_name=sys.argv[2], + destination_file_name=sys.argv[3], + ) diff --git a/storage/samples/snippets/storage_download_file_requester_pays.py b/storage/samples/snippets/storage_download_file_requester_pays.py new file mode 100644 index 00000000000..babbafda7c2 --- /dev/null +++ b/storage/samples/snippets/storage_download_file_requester_pays.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_download_file_requester_pays] +from google.cloud import storage + + +def download_file_requester_pays( + bucket_name, project_id, source_blob_name, destination_file_name +): + """Download file using specified project as the requester""" + # bucket_name = "your-bucket-name" + # project_id = "your-project-id" + # source_blob_name = "source-blob-name" + # destination_file_name = "local-destination-file-name" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name, user_project=project_id) + blob = bucket.blob(source_blob_name) + blob.download_to_filename(destination_file_name) + + print( + "Blob {} downloaded to {} using a requester-pays request.".format( + source_blob_name, destination_file_name + ) + ) + + +# [END storage_download_file_requester_pays] + +if __name__ == "__main__": + download_file_requester_pays( + bucket_name=sys.argv[1], + project_id=sys.argv[2], + source_blob_name=sys.argv[3], + destination_file_name=sys.argv[4], + ) diff --git a/storage/samples/snippets/storage_download_public_file.py b/storage/samples/snippets/storage_download_public_file.py new file mode 100644 index 00000000000..8fbb68405af --- /dev/null +++ b/storage/samples/snippets/storage_download_public_file.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_download_public_file] +from google.cloud import storage + + +def download_public_file(bucket_name, source_blob_name, destination_file_name): + """Downloads a public blob from the bucket.""" + # bucket_name = "your-bucket-name" + # source_blob_name = "storage-object-name" + # destination_file_name = "local/path/to/file" + + storage_client = storage.Client.create_anonymous_client() + + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(source_blob_name) + blob.download_to_filename(destination_file_name) + + print( + "Downloaded public blob {} from bucket {} to {}.".format( + source_blob_name, bucket.name, destination_file_name + ) + ) + + +# [END storage_download_public_file] + +if __name__ == "__main__": + download_public_file( + bucket_name=sys.argv[1], + source_blob_name=sys.argv[2], + destination_file_name=sys.argv[3], + ) diff --git a/storage/samples/snippets/storage_enable_bucket_lifecycle_management.py b/storage/samples/snippets/storage_enable_bucket_lifecycle_management.py new file mode 100644 index 00000000000..61c7d7b20d9 --- /dev/null +++ b/storage/samples/snippets/storage_enable_bucket_lifecycle_management.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python + +# Copyright 2020 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_enable_bucket_lifecycle_management] +from google.cloud import storage + + +def enable_bucket_lifecycle_management(bucket_name): + """Enable lifecycle management for a bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + rules = bucket.lifecycle_rules + + print("Lifecycle management rules for bucket {} are {}".format(bucket_name, list(rules))) + bucket.add_lifecycle_delete_rule(age=2) + bucket.patch() + + rules = bucket.lifecycle_rules + print("Lifecycle management is enable for bucket {} and the rules are {}".format(bucket_name, list(rules))) + + return bucket + + +# [END storage_enable_bucket_lifecycle_management] + +if __name__ == "__main__": + enable_bucket_lifecycle_management(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_enable_default_event_based_hold.py b/storage/samples/snippets/storage_enable_default_event_based_hold.py new file mode 100644 index 00000000000..a535390c913 --- /dev/null +++ b/storage/samples/snippets/storage_enable_default_event_based_hold.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_enable_default_event_based_hold] +from google.cloud import storage + + +def enable_default_event_based_hold(bucket_name): + """Enables the default event based hold on a given bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + bucket.default_event_based_hold = True + bucket.patch() + + print("Default event based hold was enabled for {}".format(bucket_name)) + + +# [END storage_enable_default_event_based_hold] + + +if __name__ == "__main__": + enable_default_event_based_hold(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_enable_requester_pays.py b/storage/samples/snippets/storage_enable_requester_pays.py new file mode 100644 index 00000000000..9787008ddcf --- /dev/null +++ b/storage/samples/snippets/storage_enable_requester_pays.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_enable_requester_pays] +from google.cloud import storage + + +def enable_requester_pays(bucket_name): + """Enable a bucket's requesterpays metadata""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.requester_pays = True + bucket.patch() + + print("Requester Pays has been enabled for {}".format(bucket_name)) + + +# [END storage_enable_requester_pays] + +if __name__ == "__main__": + enable_requester_pays(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_enable_uniform_bucket_level_access.py b/storage/samples/snippets/storage_enable_uniform_bucket_level_access.py new file mode 100644 index 00000000000..c689bb735c6 --- /dev/null +++ b/storage/samples/snippets/storage_enable_uniform_bucket_level_access.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_enable_uniform_bucket_level_access] +from google.cloud import storage + + +def enable_uniform_bucket_level_access(bucket_name): + """Enable uniform bucket-level access for a bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + bucket.iam_configuration.uniform_bucket_level_access_enabled = True + bucket.patch() + + print( + "Uniform bucket-level access was enabled for {}.".format(bucket.name) + ) + + +# [END storage_enable_uniform_bucket_level_access] + +if __name__ == "__main__": + enable_uniform_bucket_level_access(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_enable_versioning.py b/storage/samples/snippets/storage_enable_versioning.py new file mode 100644 index 00000000000..89693e42656 --- /dev/null +++ b/storage/samples/snippets/storage_enable_versioning.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2020 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_enable_versioning] +from google.cloud import storage + + +def enable_versioning(bucket_name): + """Enable versioning for this bucket.""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + bucket.versioning_enabled = True + bucket.patch() + + print("Versioning was enabled for bucket {}".format(bucket.name)) + return bucket + + +# [END storage_enable_versioning] + +if __name__ == "__main__": + enable_versioning(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_generate_encryption_key.py b/storage/samples/snippets/storage_generate_encryption_key.py new file mode 100644 index 00000000000..a973418a611 --- /dev/null +++ b/storage/samples/snippets/storage_generate_encryption_key.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_generate_encryption_key] +import base64 +import os + + +def generate_encryption_key(): + """Generates a 256 bit (32 byte) AES encryption key and prints the + base64 representation. + + This is included for demonstration purposes. You should generate your own + key. Please remember that encryption keys should be handled with a + comprehensive security policy. + """ + key = os.urandom(32) + encoded_key = base64.b64encode(key).decode("utf-8") + + print("Base 64 encoded encryption key: {}".format(encoded_key)) + + +# [END storage_generate_encryption_key] + +if __name__ == "__main__": + generate_encryption_key() diff --git a/storage/samples/snippets/storage_generate_signed_post_policy_v4.py b/storage/samples/snippets/storage_generate_signed_post_policy_v4.py new file mode 100644 index 00000000000..8217714e2ed --- /dev/null +++ b/storage/samples/snippets/storage_generate_signed_post_policy_v4.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python + +# Copyright 2020 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# [START storage_generate_signed_post_policy_v4] +import datetime +# [END storage_generate_signed_post_policy_v4] +import sys +# [START storage_generate_signed_post_policy_v4] + +from google.cloud import storage + + +def generate_signed_post_policy_v4(bucket_name, blob_name): + """Generates a v4 POST Policy and prints an HTML form.""" + # bucket_name = 'your-bucket-name' + # blob_name = 'your-object-name' + + storage_client = storage.Client() + + policy = storage_client.generate_signed_post_policy_v4( + bucket_name, + blob_name, + expiration=datetime.timedelta(minutes=10), + fields={ + 'x-goog-meta-test': 'data' + } + ) + + # Create an HTML form with the provided policy + header = "
\n" + form = header.format(policy["url"]) + + # Include all fields returned in the HTML form as they're required + for key, value in policy["fields"].items(): + form += " \n".format(key, value) + + form += "
\n" + form += "
\n" + form += "
" + + print(form) + + return form + + +# [END storage_generate_signed_post_policy_v4] + +if __name__ == "__main__": + generate_signed_post_policy_v4( + bucket_name=sys.argv[1], blob_name=sys.argv[2] + ) diff --git a/storage/samples/snippets/storage_generate_signed_url_v2.py b/storage/samples/snippets/storage_generate_signed_url_v2.py new file mode 100644 index 00000000000..abea3dd540b --- /dev/null +++ b/storage/samples/snippets/storage_generate_signed_url_v2.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_generate_signed_url_v2] +import datetime +# [END storage_generate_signed_url_v2] +import sys +# [START storage_generate_signed_url_v2] + +from google.cloud import storage + + +def generate_signed_url(bucket_name, blob_name): + """Generates a v2 signed URL for downloading a blob. + + Note that this method requires a service account key file. You can not use + this if you are using Application Default Credentials from Google Compute + Engine or from the Google Cloud SDK. + """ + # bucket_name = 'your-bucket-name' + # blob_name = 'your-object-name' + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + url = blob.generate_signed_url( + # This URL is valid for 1 hour + expiration=datetime.timedelta(hours=1), + # Allow GET requests using this URL. + method="GET", + ) + + print("The signed url for {} is {}".format(blob.name, url)) + return url + + +# [END storage_generate_signed_url_v2] + +if __name__ == "__main__": + generate_signed_url(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_generate_signed_url_v4.py b/storage/samples/snippets/storage_generate_signed_url_v4.py new file mode 100644 index 00000000000..2a45b23e9be --- /dev/null +++ b/storage/samples/snippets/storage_generate_signed_url_v4.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# [START storage_generate_signed_url_v4] +import datetime +# [END storage_generate_signed_url_v4] +import sys +# [START storage_generate_signed_url_v4] + +from google.cloud import storage + + +def generate_download_signed_url_v4(bucket_name, blob_name): + """Generates a v4 signed URL for downloading a blob. + + Note that this method requires a service account key file. You can not use + this if you are using Application Default Credentials from Google Compute + Engine or from the Google Cloud SDK. + """ + # bucket_name = 'your-bucket-name' + # blob_name = 'your-object-name' + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + url = blob.generate_signed_url( + version="v4", + # This URL is valid for 15 minutes + expiration=datetime.timedelta(minutes=15), + # Allow GET requests using this URL. + method="GET", + ) + + print("Generated GET signed URL:") + print(url) + print("You can use this URL with any user agent, for example:") + print("curl '{}'".format(url)) + return url + + +# [END storage_generate_signed_url_v4] + +if __name__ == "__main__": + generate_download_signed_url_v4( + bucket_name=sys.argv[1], blob_name=sys.argv[2] + ) diff --git a/storage/samples/snippets/storage_generate_upload_signed_url_v4.py b/storage/samples/snippets/storage_generate_upload_signed_url_v4.py new file mode 100644 index 00000000000..dc1da88647f --- /dev/null +++ b/storage/samples/snippets/storage_generate_upload_signed_url_v4.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# [START storage_generate_upload_signed_url_v4] +import datetime +# [END storage_generate_upload_signed_url_v4] +import sys +# [START storage_generate_upload_signed_url_v4] + +from google.cloud import storage + + +def generate_upload_signed_url_v4(bucket_name, blob_name): + """Generates a v4 signed URL for uploading a blob using HTTP PUT. + + Note that this method requires a service account key file. You can not use + this if you are using Application Default Credentials from Google Compute + Engine or from the Google Cloud SDK. + """ + # bucket_name = 'your-bucket-name' + # blob_name = 'your-object-name' + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + url = blob.generate_signed_url( + version="v4", + # This URL is valid for 15 minutes + expiration=datetime.timedelta(minutes=15), + # Allow PUT requests using this URL. + method="PUT", + content_type="application/octet-stream", + ) + + print("Generated PUT signed URL:") + print(url) + print("You can use this URL with any user agent, for example:") + print( + "curl -X PUT -H 'Content-Type: application/octet-stream' " + "--upload-file my-file '{}'".format(url) + ) + return url + + +# [END storage_generate_upload_signed_url_v4] + + +if __name__ == "__main__": + generate_upload_signed_url_v4( + bucket_name=sys.argv[1], blob_name=sys.argv[2] + ) diff --git a/storage/samples/snippets/storage_get_bucket_labels.py b/storage/samples/snippets/storage_get_bucket_labels.py new file mode 100644 index 00000000000..b3bcd6208b8 --- /dev/null +++ b/storage/samples/snippets/storage_get_bucket_labels.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# [START storage_get_bucket_labels] +import pprint +# [END storage_get_bucket_labels] +import sys +# [START storage_get_bucket_labels] + +from google.cloud import storage + + +def get_bucket_labels(bucket_name): + """Prints out a bucket's labels.""" + # bucket_name = 'your-bucket-name' + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + + labels = bucket.labels + pprint.pprint(labels) + + +# [END storage_get_bucket_labels] + +if __name__ == "__main__": + get_bucket_labels(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_get_bucket_metadata.py b/storage/samples/snippets/storage_get_bucket_metadata.py new file mode 100644 index 00000000000..87cd5eddc4e --- /dev/null +++ b/storage/samples/snippets/storage_get_bucket_metadata.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import sys + +# [START storage_get_bucket_metadata] + +from google.cloud import storage + + +def bucket_metadata(bucket_name): + """Prints out a bucket's metadata.""" + # bucket_name = 'your-bucket-name' + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + print(f"ID: {bucket.id}") + print(f"Name: {bucket.name}") + print(f"Storage Class: {bucket.storage_class}") + print(f"Location: {bucket.location}") + print(f"Location Type: {bucket.location_type}") + print(f"Cors: {bucket.cors}") + print(f"Default Event Based Hold: {bucket.default_event_based_hold}") + print(f"Default KMS Key Name: {bucket.default_kms_key_name}") + print(f"Metageneration: {bucket.metageneration}") + print( + f"Public Access Prevention: {bucket.iam_configuration.public_access_prevention}" + ) + print(f"Retention Effective Time: {bucket.retention_policy_effective_time}") + print(f"Retention Period: {bucket.retention_period}") + print(f"Retention Policy Locked: {bucket.retention_policy_locked}") + print(f"Requester Pays: {bucket.requester_pays}") + print(f"Self Link: {bucket.self_link}") + print(f"Time Created: {bucket.time_created}") + print(f"Versioning Enabled: {bucket.versioning_enabled}") + print(f"Labels: {bucket.labels}") + + +# [END storage_get_bucket_metadata] + +if __name__ == "__main__": + bucket_metadata(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_get_default_event_based_hold.py b/storage/samples/snippets/storage_get_default_event_based_hold.py new file mode 100644 index 00000000000..4cf13914d8e --- /dev/null +++ b/storage/samples/snippets/storage_get_default_event_based_hold.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_get_default_event_based_hold] +from google.cloud import storage + + +def get_default_event_based_hold(bucket_name): + """Gets the default event based hold on a given bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + + if bucket.default_event_based_hold: + print("Default event-based hold is enabled for {}".format(bucket_name)) + else: + print( + "Default event-based hold is not enabled for {}".format( + bucket_name + ) + ) + + +# [END storage_get_default_event_based_hold] + + +if __name__ == "__main__": + get_default_event_based_hold(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_get_hmac_key.py b/storage/samples/snippets/storage_get_hmac_key.py new file mode 100644 index 00000000000..4dc52240d27 --- /dev/null +++ b/storage/samples/snippets/storage_get_hmac_key.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_get_hmac_key] +from google.cloud import storage + + +def get_key(access_id, project_id): + """ + Retrieve the HMACKeyMetadata with the given access id. + """ + # project_id = "Your Google Cloud project ID" + # access_id = "ID of an HMAC key" + + storage_client = storage.Client(project=project_id) + + hmac_key = storage_client.get_hmac_key_metadata( + access_id, project_id=project_id + ) + + print("The HMAC key metadata is:") + print("Service Account Email: {}".format(hmac_key.service_account_email)) + print("Key ID: {}".format(hmac_key.id)) + print("Access ID: {}".format(hmac_key.access_id)) + print("Project ID: {}".format(hmac_key.project)) + print("State: {}".format(hmac_key.state)) + print("Created At: {}".format(hmac_key.time_created)) + print("Updated At: {}".format(hmac_key.updated)) + print("Etag: {}".format(hmac_key.etag)) + return hmac_key + + +# [END storage_get_hmac_key] + +if __name__ == "__main__": + get_key(access_id=sys.argv[1], project_id=sys.argv[2]) diff --git a/storage/samples/snippets/storage_get_metadata.py b/storage/samples/snippets/storage_get_metadata.py new file mode 100644 index 00000000000..c5ef0b4ccc5 --- /dev/null +++ b/storage/samples/snippets/storage_get_metadata.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_get_metadata] +from google.cloud import storage + + +def blob_metadata(bucket_name, blob_name): + """Prints out a blob's metadata.""" + # bucket_name = 'your-bucket-name' + # blob_name = 'your-object-name' + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + # Retrieve a blob, and its metadata, from Google Cloud Storage. + # Note that `get_blob` differs from `Bucket.blob`, which does not + # make an HTTP request. + blob = bucket.get_blob(blob_name) + + print("Blob: {}".format(blob.name)) + print("Bucket: {}".format(blob.bucket.name)) + print("Storage class: {}".format(blob.storage_class)) + print("ID: {}".format(blob.id)) + print("Size: {} bytes".format(blob.size)) + print("Updated: {}".format(blob.updated)) + print("Generation: {}".format(blob.generation)) + print("Metageneration: {}".format(blob.metageneration)) + print("Etag: {}".format(blob.etag)) + print("Owner: {}".format(blob.owner)) + print("Component count: {}".format(blob.component_count)) + print("Crc32c: {}".format(blob.crc32c)) + print("md5_hash: {}".format(blob.md5_hash)) + print("Cache-control: {}".format(blob.cache_control)) + print("Content-type: {}".format(blob.content_type)) + print("Content-disposition: {}".format(blob.content_disposition)) + print("Content-encoding: {}".format(blob.content_encoding)) + print("Content-language: {}".format(blob.content_language)) + print("Metadata: {}".format(blob.metadata)) + print("Custom Time: {}".format(blob.custom_time)) + print("Temporary hold: ", "enabled" if blob.temporary_hold else "disabled") + print( + "Event based hold: ", + "enabled" if blob.event_based_hold else "disabled", + ) + if blob.retention_expiration_time: + print( + "retentionExpirationTime: {}".format( + blob.retention_expiration_time + ) + ) + + +# [END storage_get_metadata] + +if __name__ == "__main__": + blob_metadata(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_get_public_access_prevention.py b/storage/samples/snippets/storage_get_public_access_prevention.py new file mode 100644 index 00000000000..275b84e3553 --- /dev/null +++ b/storage/samples/snippets/storage_get_public_access_prevention.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_get_public_access_prevention] +from google.cloud import storage + + +def get_public_access_prevention(bucket_name): + """Gets the public access prevention setting (either 'inherited' or 'enforced') for a bucket.""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + iam_configuration = bucket.iam_configuration + + print( + f"Public access prevention is {iam_configuration.public_access_prevention} for {bucket.name}." + ) + + +# [END storage_get_public_access_prevention] + +if __name__ == "__main__": + get_public_access_prevention(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_get_requester_pays_status.py b/storage/samples/snippets/storage_get_requester_pays_status.py new file mode 100644 index 00000000000..2014d654c0b --- /dev/null +++ b/storage/samples/snippets/storage_get_requester_pays_status.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_get_requester_pays_status] +from google.cloud import storage + + +def get_requester_pays_status(bucket_name): + """Get a bucket's requester pays metadata""" + # bucket_name = "my-bucket" + storage_client = storage.Client() + + bucket = storage_client.get_bucket(bucket_name) + requester_pays_status = bucket.requester_pays + + if requester_pays_status: + print("Requester Pays is enabled for {}".format(bucket_name)) + else: + print("Requester Pays is disabled for {}".format(bucket_name)) + + +# [END storage_get_requester_pays_status] + +if __name__ == "__main__": + get_requester_pays_status(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_get_retention_policy.py b/storage/samples/snippets/storage_get_retention_policy.py new file mode 100644 index 00000000000..f2ca26d2630 --- /dev/null +++ b/storage/samples/snippets/storage_get_retention_policy.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_get_retention_policy] +from google.cloud import storage + + +def get_retention_policy(bucket_name): + """Gets the retention policy on a given bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + bucket.reload() + + print("Retention Policy for {}".format(bucket_name)) + print("Retention Period: {}".format(bucket.retention_period)) + if bucket.retention_policy_locked: + print("Retention Policy is locked") + + if bucket.retention_policy_effective_time: + print( + "Effective Time: {}".format(bucket.retention_policy_effective_time) + ) + + +# [END storage_get_retention_policy] + + +if __name__ == "__main__": + get_retention_policy(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_get_service_account.py b/storage/samples/snippets/storage_get_service_account.py new file mode 100644 index 00000000000..58ababb91cb --- /dev/null +++ b/storage/samples/snippets/storage_get_service_account.py @@ -0,0 +1,37 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# [START storage_get_service_account] +from google.cloud import storage + + +def get_service_account(): + """Get the service account email""" + storage_client = storage.Client() + + email = storage_client.get_service_account_email() + print( + "The GCS service account for project {} is: {} ".format( + storage_client.project, email + ) + ) + + +# [END storage_get_service_account] + +if __name__ == "__main__": + get_service_account() diff --git a/storage/samples/snippets/storage_get_uniform_bucket_level_access.py b/storage/samples/snippets/storage_get_uniform_bucket_level_access.py new file mode 100644 index 00000000000..eddb8bc1ac8 --- /dev/null +++ b/storage/samples/snippets/storage_get_uniform_bucket_level_access.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_get_uniform_bucket_level_access] +from google.cloud import storage + + +def get_uniform_bucket_level_access(bucket_name): + """Get uniform bucket-level access for a bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + iam_configuration = bucket.iam_configuration + + if iam_configuration.uniform_bucket_level_access_enabled: + print( + "Uniform bucket-level access is enabled for {}.".format( + bucket.name + ) + ) + print( + "Bucket will be locked on {}.".format( + iam_configuration.uniform_bucket_level_locked_time + ) + ) + else: + print( + "Uniform bucket-level access is disabled for {}.".format( + bucket.name + ) + ) + + +# [END storage_get_uniform_bucket_level_access] + +if __name__ == "__main__": + get_uniform_bucket_level_access(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_list_buckets.py b/storage/samples/snippets/storage_list_buckets.py new file mode 100644 index 00000000000..f5897e47a42 --- /dev/null +++ b/storage/samples/snippets/storage_list_buckets.py @@ -0,0 +1,35 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_list_buckets] +from google.cloud import storage + + +def list_buckets(): + """Lists all buckets.""" + + storage_client = storage.Client() + buckets = storage_client.list_buckets() + + for bucket in buckets: + print(bucket.name) + + +# [END storage_list_buckets] + + +if __name__ == "__main__": + list_buckets() diff --git a/storage/samples/snippets/storage_list_file_archived_generations.py b/storage/samples/snippets/storage_list_file_archived_generations.py new file mode 100644 index 00000000000..dc2f5eaf5c2 --- /dev/null +++ b/storage/samples/snippets/storage_list_file_archived_generations.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_list_file_archived_generations] +from google.cloud import storage + + +def list_file_archived_generations(bucket_name): + """Lists all the blobs in the bucket with generation.""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + + blobs = storage_client.list_blobs(bucket_name, versions=True) + + for blob in blobs: + print("{},{}".format(blob.name, blob.generation)) + + +# [END storage_list_file_archived_generations] + + +if __name__ == "__main__": + list_file_archived_generations(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_list_files.py b/storage/samples/snippets/storage_list_files.py new file mode 100644 index 00000000000..c6a80d9fadb --- /dev/null +++ b/storage/samples/snippets/storage_list_files.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_list_files] +from google.cloud import storage + + +def list_blobs(bucket_name): + """Lists all the blobs in the bucket.""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + + # Note: Client.list_blobs requires at least package version 1.17.0. + blobs = storage_client.list_blobs(bucket_name) + + for blob in blobs: + print(blob.name) + + +# [END storage_list_files] + + +if __name__ == "__main__": + list_blobs(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_list_files_with_prefix.py b/storage/samples/snippets/storage_list_files_with_prefix.py new file mode 100644 index 00000000000..f79413fb6f1 --- /dev/null +++ b/storage/samples/snippets/storage_list_files_with_prefix.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_list_files_with_prefix] +from google.cloud import storage + + +def list_blobs_with_prefix(bucket_name, prefix, delimiter=None): + """Lists all the blobs in the bucket that begin with the prefix. + + This can be used to list all blobs in a "folder", e.g. "public/". + + The delimiter argument can be used to restrict the results to only the + "files" in the given "folder". Without the delimiter, the entire tree under + the prefix is returned. For example, given these blobs: + + a/1.txt + a/b/2.txt + + If you specify prefix ='a/', without a delimiter, you'll get back: + + a/1.txt + a/b/2.txt + + However, if you specify prefix='a/' and delimiter='/', you'll get back + only the file directly under 'a/': + + a/1.txt + + As part of the response, you'll also get back a blobs.prefixes entity + that lists the "subfolders" under `a/`: + + a/b/ + """ + + storage_client = storage.Client() + + # Note: Client.list_blobs requires at least package version 1.17.0. + blobs = storage_client.list_blobs(bucket_name, prefix=prefix, delimiter=delimiter) + + print("Blobs:") + for blob in blobs: + print(blob.name) + + if delimiter: + print("Prefixes:") + for prefix in blobs.prefixes: + print(prefix) + + +# [END storage_list_files_with_prefix] + +if __name__ == "__main__": + list_blobs_with_prefix( + bucket_name=sys.argv[1], prefix=sys.argv[2], delimiter=sys.argv[3] + ) diff --git a/storage/samples/snippets/storage_list_hmac_keys.py b/storage/samples/snippets/storage_list_hmac_keys.py new file mode 100644 index 00000000000..8e5c53b589d --- /dev/null +++ b/storage/samples/snippets/storage_list_hmac_keys.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_list_hmac_keys] +from google.cloud import storage + + +def list_keys(project_id): + """ + List all HMAC keys associated with the project. + """ + # project_id = "Your Google Cloud project ID" + + storage_client = storage.Client(project=project_id) + hmac_keys = storage_client.list_hmac_keys(project_id=project_id) + print("HMAC Keys:") + for hmac_key in hmac_keys: + print( + "Service Account Email: {}".format(hmac_key.service_account_email) + ) + print("Access ID: {}".format(hmac_key.access_id)) + return hmac_keys + + +# [END storage_list_hmac_keys] + +if __name__ == "__main__": + list_keys(project_id=sys.argv[1]) diff --git a/storage/samples/snippets/storage_lock_retention_policy.py b/storage/samples/snippets/storage_lock_retention_policy.py new file mode 100644 index 00000000000..d59572f5dad --- /dev/null +++ b/storage/samples/snippets/storage_lock_retention_policy.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_lock_retention_policy] +from google.cloud import storage + + +def lock_retention_policy(bucket_name): + """Locks the retention policy on a given bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + # get_bucket gets the current metageneration value for the bucket, + # required by lock_retention_policy. + bucket = storage_client.get_bucket(bucket_name) + + # Warning: Once a retention policy is locked it cannot be unlocked + # and retention period can only be increased. + bucket.lock_retention_policy() + + print("Retention policy for {} is now locked".format(bucket_name)) + print( + "Retention policy effective as of {}".format( + bucket.retention_policy_effective_time + ) + ) + + +# [END storage_lock_retention_policy] + + +if __name__ == "__main__": + lock_retention_policy(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_make_public.py b/storage/samples/snippets/storage_make_public.py new file mode 100644 index 00000000000..79ae40d123b --- /dev/null +++ b/storage/samples/snippets/storage_make_public.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_make_public] +from google.cloud import storage + + +def make_blob_public(bucket_name, blob_name): + """Makes a blob publicly accessible.""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + blob.make_public() + + print( + "Blob {} is publicly accessible at {}".format( + blob.name, blob.public_url + ) + ) + + +# [END storage_make_public] + +if __name__ == "__main__": + make_blob_public(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_move_file.py b/storage/samples/snippets/storage_move_file.py new file mode 100644 index 00000000000..a881a38bade --- /dev/null +++ b/storage/samples/snippets/storage_move_file.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python + +# Copyright 2019 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_move_file] +from google.cloud import storage + + +def move_blob(bucket_name, blob_name, destination_bucket_name, destination_blob_name): + """Moves a blob from one bucket to another with a new name.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + # The ID of your GCS object + # blob_name = "your-object-name" + # The ID of the bucket to move the object to + # destination_bucket_name = "destination-bucket-name" + # The ID of your new GCS object (optional) + # destination_blob_name = "destination-object-name" + + storage_client = storage.Client() + + source_bucket = storage_client.bucket(bucket_name) + source_blob = source_bucket.blob(blob_name) + destination_bucket = storage_client.bucket(destination_bucket_name) + + blob_copy = source_bucket.copy_blob( + source_blob, destination_bucket, destination_blob_name + ) + source_bucket.delete_blob(blob_name) + + print( + "Blob {} in bucket {} moved to blob {} in bucket {}.".format( + source_blob.name, + source_bucket.name, + blob_copy.name, + destination_bucket.name, + ) + ) + + +# [END storage_move_file] + +if __name__ == "__main__": + move_blob( + bucket_name=sys.argv[1], + blob_name=sys.argv[2], + destination_bucket_name=sys.argv[3], + destination_blob_name=sys.argv[4], + ) diff --git a/storage/samples/snippets/storage_object_csek_to_cmek.py b/storage/samples/snippets/storage_object_csek_to_cmek.py new file mode 100644 index 00000000000..9d4d710bf50 --- /dev/null +++ b/storage/samples/snippets/storage_object_csek_to_cmek.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import base64 +import sys + +# [START storage_object_csek_to_cmek] +from google.cloud import storage + + +def object_csek_to_cmek(bucket_name, blob_name, encryption_key, kms_key_name): + """Change a blob's customer-supplied encryption key to KMS key""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + # encryption_key = "TIbv/fjexq+VmtXzAlc63J4z5kFmWJ6NdAPQulQBT7g=" + # kms_key_name = "projects/PROJ/locations/LOC/keyRings/RING/cryptoKey/KEY" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + current_encryption_key = base64.b64decode(encryption_key) + source_blob = bucket.blob(blob_name, encryption_key=current_encryption_key) + + destination_blob = bucket.blob(blob_name, kms_key_name=kms_key_name) + token, rewritten, total = destination_blob.rewrite(source_blob) + + while token is not None: + token, rewritten, total = destination_blob.rewrite(source_blob, token=token) + + print( + "Blob {} in bucket {} is now managed by the KMS key {} instead of a customer-supplied encryption key".format( + blob_name, bucket_name, kms_key_name + ) + ) + return destination_blob + + +# [END storage_object_csek_to_cmek] + +if __name__ == "__main__": + object_csek_to_cmek( + bucket_name=sys.argv[1], + blob_name=sys.argv[2], + encryption_key=sys.argv[3], + kms_key_name=sys.argv[4], + ) diff --git a/storage/samples/snippets/storage_object_get_kms_key.py b/storage/samples/snippets/storage_object_get_kms_key.py new file mode 100644 index 00000000000..dddfc9151b8 --- /dev/null +++ b/storage/samples/snippets/storage_object_get_kms_key.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_object_get_kms_key] +from google.cloud import storage + + +def object_get_kms_key(bucket_name, blob_name): + """Retrieve the KMS key of a blob""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + blob = bucket.get_blob(blob_name) + + kms_key = blob.kms_key_name + + print("The KMS key of a blob is {}".format(blob.kms_key_name)) + return kms_key + + +# [END storage_object_get_kms_key] + +if __name__ == "__main__": + object_get_kms_key(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_print_bucket_acl.py b/storage/samples/snippets/storage_print_bucket_acl.py new file mode 100644 index 00000000000..0804f7a9a89 --- /dev/null +++ b/storage/samples/snippets/storage_print_bucket_acl.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_print_bucket_acl] +from google.cloud import storage + + +def print_bucket_acl(bucket_name): + """Prints out a bucket's access control list.""" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + for entry in bucket.acl: + print("{}: {}".format(entry["role"], entry["entity"])) + + +# [END storage_print_bucket_acl] + +if __name__ == "__main__": + print_bucket_acl(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_print_bucket_acl_for_user.py b/storage/samples/snippets/storage_print_bucket_acl_for_user.py new file mode 100644 index 00000000000..fa786d03af9 --- /dev/null +++ b/storage/samples/snippets/storage_print_bucket_acl_for_user.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_print_bucket_acl_for_user] +from google.cloud import storage + + +def print_bucket_acl_for_user(bucket_name, user_email): + """Prints out a bucket's access control list for a given user.""" + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + # Reload fetches the current ACL from Cloud Storage. + bucket.acl.reload() + + # You can also use `group`, `domain`, `all_authenticated` and `all` to + # get the roles for different types of entities. + roles = bucket.acl.user(user_email).get_roles() + + print(roles) + + +# [END storage_print_bucket_acl_for_user] + +if __name__ == "__main__": + print_bucket_acl_for_user(bucket_name=sys.argv[1], user_email=sys.argv[2]) diff --git a/storage/samples/snippets/storage_print_file_acl.py b/storage/samples/snippets/storage_print_file_acl.py new file mode 100644 index 00000000000..f34a5283b1d --- /dev/null +++ b/storage/samples/snippets/storage_print_file_acl.py @@ -0,0 +1,37 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_print_file_acl] +from google.cloud import storage + + +def print_blob_acl(bucket_name, blob_name): + """Prints out a blob's access control list.""" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + for entry in blob.acl: + print("{}: {}".format(entry["role"], entry["entity"])) + + +# [END storage_print_file_acl] + +if __name__ == "__main__": + print_blob_acl(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_print_file_acl_for_user.py b/storage/samples/snippets/storage_print_file_acl_for_user.py new file mode 100644 index 00000000000..e399b916013 --- /dev/null +++ b/storage/samples/snippets/storage_print_file_acl_for_user.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_print_file_acl_for_user] +from google.cloud import storage + + +def print_blob_acl_for_user(bucket_name, blob_name, user_email): + """Prints out a blob's access control list for a given user.""" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + # Reload fetches the current ACL from Cloud Storage. + blob.acl.reload() + + # You can also use `group`, `domain`, `all_authenticated` and `all` to + # get the roles for different types of entities. + roles = blob.acl.user(user_email).get_roles() + + print(roles) + + +# [END storage_print_file_acl_for_user] + +if __name__ == "__main__": + print_blob_acl_for_user( + bucket_name=sys.argv[1], blob_name=sys.argv[2], user_email=sys.argv[3], + ) diff --git a/storage/samples/snippets/storage_release_event_based_hold.py b/storage/samples/snippets/storage_release_event_based_hold.py new file mode 100644 index 00000000000..8c3c11b6ff0 --- /dev/null +++ b/storage/samples/snippets/storage_release_event_based_hold.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_release_event_based_hold] +from google.cloud import storage + + +def release_event_based_hold(bucket_name, blob_name): + """Releases the event based hold on a given blob""" + + # bucket_name = "my-bucket" + # blob_name = "my-blob" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + blob.event_based_hold = False + blob.patch() + + print("Event based hold was released for {}".format(blob_name)) + + +# [END storage_release_event_based_hold] + + +if __name__ == "__main__": + release_event_based_hold(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_release_temporary_hold.py b/storage/samples/snippets/storage_release_temporary_hold.py new file mode 100644 index 00000000000..02a6ca96c06 --- /dev/null +++ b/storage/samples/snippets/storage_release_temporary_hold.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_release_temporary_hold] +from google.cloud import storage + + +def release_temporary_hold(bucket_name, blob_name): + """Releases the temporary hold on a given blob""" + + # bucket_name = "my-bucket" + # blob_name = "my-blob" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + blob.temporary_hold = False + blob.patch() + + print("Temporary hold was release for #{blob_name}") + + +# [END storage_release_temporary_hold] + + +if __name__ == "__main__": + release_temporary_hold(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_remove_bucket_conditional_iam_binding.py b/storage/samples/snippets/storage_remove_bucket_conditional_iam_binding.py new file mode 100644 index 00000000000..242544d8ed2 --- /dev/null +++ b/storage/samples/snippets/storage_remove_bucket_conditional_iam_binding.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_remove_bucket_conditional_iam_binding] +from google.cloud import storage + + +def remove_bucket_conditional_iam_binding( + bucket_name, role, title, description, expression +): + """Remove a conditional IAM binding from a bucket's IAM policy.""" + # bucket_name = "your-bucket-name" + # role = "IAM role, e.g. roles/storage.objectViewer" + # title = "Condition title." + # description = "Condition description." + # expression = "Condition expression." + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + policy = bucket.get_iam_policy(requested_policy_version=3) + + # Set the policy's version to 3 to use condition in bindings. + policy.version = 3 + + condition = { + "title": title, + "description": description, + "expression": expression, + } + policy.bindings = [ + binding + for binding in policy.bindings + if not (binding["role"] == role and binding.get("condition") == condition) + ] + + bucket.set_iam_policy(policy) + + print("Conditional Binding was removed.") + + +# [END storage_remove_bucket_conditional_iam_binding] + + +if __name__ == "__main__": + remove_bucket_conditional_iam_binding( + bucket_name=sys.argv[1], + role=sys.argv[2], + title=sys.argv[3], + description=sys.argv[4], + expression=sys.argv[5], + ) diff --git a/storage/samples/snippets/storage_remove_bucket_default_owner.py b/storage/samples/snippets/storage_remove_bucket_default_owner.py new file mode 100644 index 00000000000..beaf6be84d4 --- /dev/null +++ b/storage/samples/snippets/storage_remove_bucket_default_owner.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_remove_bucket_default_owner] +from google.cloud import storage + + +def remove_bucket_default_owner(bucket_name, user_email): + """Removes a user from the access control list of the given bucket's + default object access control list.""" + # bucket_name = "your-bucket-name" + # user_email = "name@example.com" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + # Reload fetches the current ACL from Cloud Storage. + bucket.acl.reload() + + # You can also use `group`, `domain`, `all_authenticated` and `all` to + # remove access for different types of entities. + bucket.default_object_acl.user(user_email).revoke_read() + bucket.default_object_acl.user(user_email).revoke_write() + bucket.default_object_acl.user(user_email).revoke_owner() + bucket.default_object_acl.save() + + print( + "Removed user {} from the default acl of bucket {}.".format( + user_email, bucket_name + ) + ) + + +# [END storage_remove_bucket_default_owner] + +if __name__ == "__main__": + remove_bucket_default_owner( + bucket_name=sys.argv[1], user_email=sys.argv[2] + ) diff --git a/storage/samples/snippets/storage_remove_bucket_iam_member.py b/storage/samples/snippets/storage_remove_bucket_iam_member.py new file mode 100644 index 00000000000..ef75a1a15fc --- /dev/null +++ b/storage/samples/snippets/storage_remove_bucket_iam_member.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_remove_bucket_iam_member] +from google.cloud import storage + + +def remove_bucket_iam_member(bucket_name, role, member): + """Remove member from bucket IAM Policy""" + # bucket_name = "your-bucket-name" + # role = "IAM role, e.g. roles/storage.objectViewer" + # member = "IAM identity, e.g. user: name@example.com" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + policy = bucket.get_iam_policy(requested_policy_version=3) + + for binding in policy.bindings: + print(binding) + if binding["role"] == role and binding.get("condition") is None: + binding["members"].discard(member) + + bucket.set_iam_policy(policy) + + print("Removed {} with role {} from {}.".format(member, role, bucket_name)) + + +# [END storage_remove_bucket_iam_member] + +if __name__ == "__main__": + remove_bucket_iam_member( + bucket_name=sys.argv[1], role=sys.argv[2], member=sys.argv[3] + ) diff --git a/storage/samples/snippets/storage_remove_bucket_label.py b/storage/samples/snippets/storage_remove_bucket_label.py new file mode 100644 index 00000000000..58bbfef2d51 --- /dev/null +++ b/storage/samples/snippets/storage_remove_bucket_label.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# [START storage_remove_bucket_label] +import pprint +# [END storage_remove_bucket_label] +import sys +# [START storage_remove_bucket_label] + +from google.cloud import storage + + +def remove_bucket_label(bucket_name): + """Remove a label from a bucket.""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + labels = bucket.labels + + if "example" in labels: + del labels["example"] + + bucket.labels = labels + bucket.patch() + + print("Removed labels on {}.".format(bucket.name)) + pprint.pprint(bucket.labels) + + +# [END storage_remove_bucket_label] + +if __name__ == "__main__": + remove_bucket_label(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_remove_bucket_owner.py b/storage/samples/snippets/storage_remove_bucket_owner.py new file mode 100644 index 00000000000..f54e7a7cc56 --- /dev/null +++ b/storage/samples/snippets/storage_remove_bucket_owner.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_remove_bucket_owner] +from google.cloud import storage + + +def remove_bucket_owner(bucket_name, user_email): + """Removes a user from the access control list of the given bucket.""" + # bucket_name = "your-bucket-name" + # user_email = "name@example.com" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + # Reload fetches the current ACL from Cloud Storage. + bucket.acl.reload() + + # You can also use `group`, `domain`, `all_authenticated` and `all` to + # remove access for different types of entities. + bucket.acl.user(user_email).revoke_read() + bucket.acl.user(user_email).revoke_write() + bucket.acl.user(user_email).revoke_owner() + bucket.acl.save() + + print("Removed user {} from bucket {}.".format(user_email, bucket_name)) + + +# [END storage_remove_bucket_owner] + +if __name__ == "__main__": + remove_bucket_owner(bucket_name=sys.argv[1], user_email=sys.argv[2]) diff --git a/storage/samples/snippets/storage_remove_cors_configuration.py b/storage/samples/snippets/storage_remove_cors_configuration.py new file mode 100644 index 00000000000..48ee7433856 --- /dev/null +++ b/storage/samples/snippets/storage_remove_cors_configuration.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_remove_cors_configuration] +from google.cloud import storage + + +def remove_cors_configuration(bucket_name): + """Remove a bucket's CORS policies configuration.""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + bucket.cors = [] + bucket.patch() + + print("Remove CORS policies for bucket {}.".format(bucket.name)) + return bucket + + +# [END storage_remove_cors_configuration] + +if __name__ == "__main__": + remove_cors_configuration(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_remove_file_owner.py b/storage/samples/snippets/storage_remove_file_owner.py new file mode 100644 index 00000000000..9db83cce0cd --- /dev/null +++ b/storage/samples/snippets/storage_remove_file_owner.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_remove_file_owner] +from google.cloud import storage + + +def remove_blob_owner(bucket_name, blob_name, user_email): + """Removes a user from the access control list of the given blob in the + given bucket.""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + # user_email = "name@example.com" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + # You can also use `group`, `domain`, `all_authenticated` and `all` to + # remove access for different types of entities. + blob.acl.user(user_email).revoke_read() + blob.acl.user(user_email).revoke_write() + blob.acl.user(user_email).revoke_owner() + blob.acl.save() + + print( + "Removed user {} from blob {} in bucket {}.".format( + user_email, blob_name, bucket_name + ) + ) + + +# [END storage_remove_file_owner] + +if __name__ == "__main__": + remove_blob_owner( + bucket_name=sys.argv[1], blob_name=sys.argv[2], user_email=sys.argv[3], + ) diff --git a/storage/samples/snippets/storage_remove_retention_policy.py b/storage/samples/snippets/storage_remove_retention_policy.py new file mode 100644 index 00000000000..cb8ee548cf7 --- /dev/null +++ b/storage/samples/snippets/storage_remove_retention_policy.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_remove_retention_policy] +from google.cloud import storage + + +def remove_retention_policy(bucket_name): + """Removes the retention policy on a given bucket""" + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + bucket.reload() + + if bucket.retention_policy_locked: + print( + "Unable to remove retention period as retention policy is locked." + ) + return + + bucket.retention_period = None + bucket.patch() + + print("Removed bucket {} retention policy".format(bucket.name)) + + +# [END storage_remove_retention_policy] + + +if __name__ == "__main__": + remove_retention_policy(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_rename_file.py b/storage/samples/snippets/storage_rename_file.py new file mode 100644 index 00000000000..b47e186218f --- /dev/null +++ b/storage/samples/snippets/storage_rename_file.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_rename_file] +from google.cloud import storage + + +def rename_blob(bucket_name, blob_name, new_name): + """Renames a blob.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + # The ID of the GCS object to rename + # blob_name = "your-object-name" + # The new ID of the GCS object + # new_name = "new-object-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + new_blob = bucket.rename_blob(blob, new_name) + + print("Blob {} has been renamed to {}".format(blob.name, new_blob.name)) + + +# [END storage_rename_file] + +if __name__ == "__main__": + rename_blob(bucket_name=sys.argv[1], blob_name=sys.argv[2], new_name=sys.argv[3]) diff --git a/storage/samples/snippets/storage_rotate_encryption_key.py b/storage/samples/snippets/storage_rotate_encryption_key.py new file mode 100644 index 00000000000..663ee47964b --- /dev/null +++ b/storage/samples/snippets/storage_rotate_encryption_key.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# [START storage_rotate_encryption_key] +import base64 +# [END storage_rotate_encryption_key] +import sys +# [START storage_rotate_encryption_key] + +from google.cloud import storage + + +def rotate_encryption_key( + bucket_name, blob_name, base64_encryption_key, base64_new_encryption_key +): + """Performs a key rotation by re-writing an encrypted blob with a new + encryption key.""" + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + current_encryption_key = base64.b64decode(base64_encryption_key) + new_encryption_key = base64.b64decode(base64_new_encryption_key) + + # Both source_blob and destination_blob refer to the same storage object, + # but destination_blob has the new encryption key. + source_blob = bucket.blob( + blob_name, encryption_key=current_encryption_key + ) + destination_blob = bucket.blob( + blob_name, encryption_key=new_encryption_key + ) + + token = None + + while True: + token, bytes_rewritten, total_bytes = destination_blob.rewrite( + source_blob, token=token + ) + if token is None: + break + + print("Key rotation complete for Blob {}".format(blob_name)) + + +# [END storage_rotate_encryption_key] + +if __name__ == "__main__": + rotate_encryption_key( + bucket_name=sys.argv[1], + blob_name=sys.argv[2], + base64_encryption_key=sys.argv[3], + base64_new_encryption_key=sys.argv[4], + ) diff --git a/storage/samples/snippets/storage_set_bucket_default_kms_key.py b/storage/samples/snippets/storage_set_bucket_default_kms_key.py new file mode 100644 index 00000000000..7ba4718b2be --- /dev/null +++ b/storage/samples/snippets/storage_set_bucket_default_kms_key.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_set_bucket_default_kms_key] +from google.cloud import storage + + +def enable_default_kms_key(bucket_name, kms_key_name): + """Sets a bucket's default KMS key.""" + # bucket_name = "your-bucket-name" + # kms_key_name = "projects/PROJ/locations/LOC/keyRings/RING/cryptoKey/KEY" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + bucket.default_kms_key_name = kms_key_name + bucket.patch() + + print( + "Set default KMS key for bucket {} to {}.".format( + bucket.name, bucket.default_kms_key_name + ) + ) + + +# [END storage_set_bucket_default_kms_key] + +if __name__ == "__main__": + enable_default_kms_key(bucket_name=sys.argv[1], kms_key_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_set_bucket_public_iam.py b/storage/samples/snippets/storage_set_bucket_public_iam.py new file mode 100644 index 00000000000..c43b3eee5f3 --- /dev/null +++ b/storage/samples/snippets/storage_set_bucket_public_iam.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python + +# Copyright 2020 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_set_bucket_public_iam] +from google.cloud import storage + + +def set_bucket_public_iam(bucket_name): + """Set a public IAM Policy to bucket""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + policy = bucket.get_iam_policy(requested_policy_version=3) + policy.bindings.append( + {"role": "roles/storage.objectViewer", "members": {"allUsers"}} + ) + + bucket.set_iam_policy(policy) + + print("Bucket {} is now publicly readable".format(bucket.name)) + + +# [END storage_set_bucket_public_iam] + +if __name__ == "__main__": + set_bucket_public_iam( + bucket_name=sys.argv[1], + ) diff --git a/storage/samples/snippets/storage_set_event_based_hold.py b/storage/samples/snippets/storage_set_event_based_hold.py new file mode 100644 index 00000000000..52a89b88e41 --- /dev/null +++ b/storage/samples/snippets/storage_set_event_based_hold.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_set_event_based_hold] +from google.cloud import storage + + +def set_event_based_hold(bucket_name, blob_name): + """Sets a event based hold on a given blob""" + # bucket_name = "my-bucket" + # blob_name = "my-blob" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + blob.event_based_hold = True + blob.patch() + + print("Event based hold was set for {}".format(blob_name)) + + +# [END storage_set_event_based_hold] + + +if __name__ == "__main__": + set_event_based_hold(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_set_metadata.py b/storage/samples/snippets/storage_set_metadata.py new file mode 100644 index 00000000000..07529ac68d7 --- /dev/null +++ b/storage/samples/snippets/storage_set_metadata.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2020 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_set_metadata] +from google.cloud import storage + + +def set_blob_metadata(bucket_name, blob_name): + """Set a blob's metadata.""" + # bucket_name = 'your-bucket-name' + # blob_name = 'your-object-name' + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.get_blob(blob_name) + metadata = {'color': 'Red', 'name': 'Test'} + blob.metadata = metadata + blob.patch() + + print("The metadata for the blob {} is {}".format(blob.name, blob.metadata)) + + +# [END storage_set_metadata] + +if __name__ == "__main__": + set_blob_metadata(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_set_public_access_prevention_enforced.py b/storage/samples/snippets/storage_set_public_access_prevention_enforced.py new file mode 100644 index 00000000000..59ce5ce56ef --- /dev/null +++ b/storage/samples/snippets/storage_set_public_access_prevention_enforced.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_set_public_access_prevention_enforced] +from google.cloud import storage +from google.cloud.storage.constants import PUBLIC_ACCESS_PREVENTION_ENFORCED + + +def set_public_access_prevention_enforced(bucket_name): + """Enforce public access prevention for a bucket.""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + bucket.iam_configuration.public_access_prevention = ( + PUBLIC_ACCESS_PREVENTION_ENFORCED + ) + bucket.patch() + + print(f"Public access prevention is set to enforced for {bucket.name}.") + + +# [END storage_set_public_access_prevention_enforced] + +if __name__ == "__main__": + set_public_access_prevention_enforced(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_set_public_access_prevention_inherited.py b/storage/samples/snippets/storage_set_public_access_prevention_inherited.py new file mode 100644 index 00000000000..97e218f9d0a --- /dev/null +++ b/storage/samples/snippets/storage_set_public_access_prevention_inherited.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that sets public access prevention to inherited. +This sample is used on this page: + https://cloud.google.com/storage/docs/using-public-access-prevention +For more information, see README.md. +""" + +# [START storage_set_public_access_prevention_inherited] + +from google.cloud import storage +from google.cloud.storage.constants import PUBLIC_ACCESS_PREVENTION_INHERITED + + +def set_public_access_prevention_inherited(bucket_name): + """Sets the public access prevention status to inherited, so that the bucket inherits its setting from its parent project.""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + bucket.iam_configuration.public_access_prevention = ( + PUBLIC_ACCESS_PREVENTION_INHERITED + ) + bucket.patch() + + print(f"Public access prevention is 'inherited' for {bucket.name}.") + + +# [END storage_set_public_access_prevention_inherited] + +if __name__ == "__main__": + set_public_access_prevention_inherited(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_set_public_access_prevention_unspecified.py b/storage/samples/snippets/storage_set_public_access_prevention_unspecified.py new file mode 100644 index 00000000000..ae2c4701c57 --- /dev/null +++ b/storage/samples/snippets/storage_set_public_access_prevention_unspecified.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_set_public_access_prevention_unspecified] +from google.cloud import storage +from google.cloud.storage.constants import PUBLIC_ACCESS_PREVENTION_UNSPECIFIED + + +def set_public_access_prevention_unspecified(bucket_name): + """Sets the public access prevention status to unspecified, so that the bucket inherits its setting from its parent project.""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + bucket.iam_configuration.public_access_prevention = ( + PUBLIC_ACCESS_PREVENTION_UNSPECIFIED + ) + bucket.patch() + + print(f"Public access prevention is 'unspecified' for {bucket.name}.") + + +# [END storage_set_public_access_prevention_unspecified] + +if __name__ == "__main__": + set_public_access_prevention_unspecified(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_set_retention_policy.py b/storage/samples/snippets/storage_set_retention_policy.py new file mode 100644 index 00000000000..2b36024919a --- /dev/null +++ b/storage/samples/snippets/storage_set_retention_policy.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_set_retention_policy] +from google.cloud import storage + + +def set_retention_policy(bucket_name, retention_period): + """Defines a retention policy on a given bucket""" + # bucket_name = "my-bucket" + # retention_period = 10 + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + bucket.retention_period = retention_period + bucket.patch() + + print( + "Bucket {} retention period set for {} seconds".format( + bucket.name, bucket.retention_period + ) + ) + + +# [END storage_set_retention_policy] + + +if __name__ == "__main__": + set_retention_policy(bucket_name=sys.argv[1], retention_period=sys.argv[2]) diff --git a/storage/samples/snippets/storage_set_temporary_hold.py b/storage/samples/snippets/storage_set_temporary_hold.py new file mode 100644 index 00000000000..edeb3c57840 --- /dev/null +++ b/storage/samples/snippets/storage_set_temporary_hold.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_set_temporary_hold] +from google.cloud import storage + + +def set_temporary_hold(bucket_name, blob_name): + """Sets a temporary hold on a given blob""" + # bucket_name = "my-bucket" + # blob_name = "my-blob" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + blob.temporary_hold = True + blob.patch() + + print("Temporary hold was set for #{blob_name}") + + +# [END storage_set_temporary_hold] + + +if __name__ == "__main__": + set_temporary_hold(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_upload_encrypted_file.py b/storage/samples/snippets/storage_upload_encrypted_file.py new file mode 100644 index 00000000000..e7d02c67b7d --- /dev/null +++ b/storage/samples/snippets/storage_upload_encrypted_file.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python + +# Copyright 2019 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# [START storage_upload_encrypted_file] +import base64 +# [END storage_upload_encrypted_file] +import sys +# [START storage_upload_encrypted_file] + +from google.cloud import storage + + +def upload_encrypted_blob( + bucket_name, + source_file_name, + destination_blob_name, + base64_encryption_key, +): + """Uploads a file to a Google Cloud Storage bucket using a custom + encryption key. + + The file will be encrypted by Google Cloud Storage and only + retrievable using the provided encryption key. + """ + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + # Encryption key must be an AES256 key represented as a bytestring with + # 32 bytes. Since it's passed in as a base64 encoded string, it needs + # to be decoded. + encryption_key = base64.b64decode(base64_encryption_key) + blob = bucket.blob( + destination_blob_name, encryption_key=encryption_key + ) + + blob.upload_from_filename(source_file_name) + + print( + "File {} uploaded to {}.".format( + source_file_name, destination_blob_name + ) + ) + + +# [END storage_upload_encrypted_file] + +if __name__ == "__main__": + upload_encrypted_blob( + bucket_name=sys.argv[1], + source_file_name=sys.argv[2], + destination_blob_name=sys.argv[3], + base64_encryption_key=sys.argv[4], + ) diff --git a/storage/samples/snippets/storage_upload_file.py b/storage/samples/snippets/storage_upload_file.py new file mode 100644 index 00000000000..fb02c3632a9 --- /dev/null +++ b/storage/samples/snippets/storage_upload_file.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_upload_file] +from google.cloud import storage + + +def upload_blob(bucket_name, source_file_name, destination_blob_name): + """Uploads a file to the bucket.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + # The path to your file to upload + # source_file_name = "local/path/to/file" + # The ID of your GCS object + # destination_blob_name = "storage-object-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(destination_blob_name) + + blob.upload_from_filename(source_file_name) + + print( + "File {} uploaded to {}.".format( + source_file_name, destination_blob_name + ) + ) + + +# [END storage_upload_file] + +if __name__ == "__main__": + upload_blob( + bucket_name=sys.argv[1], + source_file_name=sys.argv[2], + destination_blob_name=sys.argv[3], + ) diff --git a/storage/samples/snippets/storage_upload_with_kms_key.py b/storage/samples/snippets/storage_upload_with_kms_key.py new file mode 100644 index 00000000000..e83c10aea19 --- /dev/null +++ b/storage/samples/snippets/storage_upload_with_kms_key.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_upload_with_kms_key] +from google.cloud import storage + + +def upload_blob_with_kms( + bucket_name, source_file_name, destination_blob_name, kms_key_name +): + """Uploads a file to the bucket, encrypting it with the given KMS key.""" + # bucket_name = "your-bucket-name" + # source_file_name = "local/path/to/file" + # destination_blob_name = "storage-object-name" + # kms_key_name = "projects/PROJ/locations/LOC/keyRings/RING/cryptoKey/KEY" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(destination_blob_name, kms_key_name=kms_key_name) + blob.upload_from_filename(source_file_name) + + print( + "File {} uploaded to {} with encryption key {}.".format( + source_file_name, destination_blob_name, kms_key_name + ) + ) + + +# [END storage_upload_with_kms_key] + +if __name__ == "__main__": + upload_blob_with_kms( + bucket_name=sys.argv[1], + source_file_name=sys.argv[2], + destination_blob_name=sys.argv[3], + kms_key_name=sys.argv[4], + ) diff --git a/storage/samples/snippets/storage_view_bucket_iam_members.py b/storage/samples/snippets/storage_view_bucket_iam_members.py new file mode 100644 index 00000000000..5272f0ddbdb --- /dev/null +++ b/storage/samples/snippets/storage_view_bucket_iam_members.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_view_bucket_iam_members] +from google.cloud import storage + + +def view_bucket_iam_members(bucket_name): + """View IAM Policy for a bucket""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + policy = bucket.get_iam_policy(requested_policy_version=3) + + for binding in policy.bindings: + print("Role: {}, Members: {}".format(binding["role"], binding["members"])) + + +# [END storage_view_bucket_iam_members] + + +if __name__ == "__main__": + view_bucket_iam_members(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/uniform_bucket_level_access_test.py b/storage/samples/snippets/uniform_bucket_level_access_test.py new file mode 100644 index 00000000000..b43fa016fe3 --- /dev/null +++ b/storage/samples/snippets/uniform_bucket_level_access_test.py @@ -0,0 +1,52 @@ +# Copyright 2019 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import storage_disable_uniform_bucket_level_access +import storage_enable_uniform_bucket_level_access +import storage_get_uniform_bucket_level_access + + +def test_get_uniform_bucket_level_access(bucket, capsys): + storage_get_uniform_bucket_level_access.get_uniform_bucket_level_access( + bucket.name + ) + out, _ = capsys.readouterr() + assert ( + "Uniform bucket-level access is disabled for {}.".format(bucket.name) + in out + ) + + +def test_enable_uniform_bucket_level_access(bucket, capsys): + short_name = storage_enable_uniform_bucket_level_access + short_name.enable_uniform_bucket_level_access( + bucket.name + ) + out, _ = capsys.readouterr() + assert ( + "Uniform bucket-level access was enabled for {}.".format(bucket.name) + in out + ) + + +def test_disable_uniform_bucket_level_access(bucket, capsys): + short_name = storage_disable_uniform_bucket_level_access + short_name.disable_uniform_bucket_level_access( + bucket.name + ) + out, _ = capsys.readouterr() + assert ( + "Uniform bucket-level access was disabled for {}.".format(bucket.name) + in out + ) From 5e50365089a552526effdc2774bf5fea0eb6e90b Mon Sep 17 00:00:00 2001 From: cojenco Date: Wed, 20 Oct 2021 11:12:10 -0700 Subject: [PATCH 002/172] test: update iam test public bucket visibility (#632) --- storage/samples/snippets/iam_test.py | 7 +++++-- .../samples/snippets/storage_set_bucket_public_iam.py | 9 +++++++-- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/storage/samples/snippets/iam_test.py b/storage/samples/snippets/iam_test.py index eb7638de5f4..edeb8427d38 100644 --- a/storage/samples/snippets/iam_test.py +++ b/storage/samples/snippets/iam_test.py @@ -137,10 +137,13 @@ def test_remove_bucket_conditional_iam_binding(bucket): def test_set_bucket_public_iam(public_bucket): - storage_set_bucket_public_iam.set_bucket_public_iam(public_bucket.name) + # The test project has org policy restricting identities by domain. + # Testing "domain:google.com" instead of "allUsers" + storage_set_bucket_public_iam.set_bucket_public_iam(public_bucket.name, ["domain:google.com"]) policy = public_bucket.get_iam_policy(requested_policy_version=3) + assert any( binding["role"] == "roles/storage.objectViewer" - and "allUsers" in binding["members"] + and "domain:google.com" in binding["members"] for binding in policy.bindings ) diff --git a/storage/samples/snippets/storage_set_bucket_public_iam.py b/storage/samples/snippets/storage_set_bucket_public_iam.py index c43b3eee5f3..4b7df89dfa1 100644 --- a/storage/samples/snippets/storage_set_bucket_public_iam.py +++ b/storage/samples/snippets/storage_set_bucket_public_iam.py @@ -17,10 +17,15 @@ import sys # [START storage_set_bucket_public_iam] +from typing import List + from google.cloud import storage -def set_bucket_public_iam(bucket_name): +def set_bucket_public_iam( + bucket_name: str = "your-bucket-name", + members: List[str] = ["allUsers"], +): """Set a public IAM Policy to bucket""" # bucket_name = "your-bucket-name" @@ -29,7 +34,7 @@ def set_bucket_public_iam(bucket_name): policy = bucket.get_iam_policy(requested_policy_version=3) policy.bindings.append( - {"role": "roles/storage.objectViewer", "members": {"allUsers"}} + {"role": "roles/storage.objectViewer", "members": members} ) bucket.set_iam_policy(policy) From a5ba2be082f32dc2c329736faf8455d28ec15506 Mon Sep 17 00:00:00 2001 From: cojenco Date: Thu, 21 Oct 2021 12:58:27 -0700 Subject: [PATCH 003/172] docs: add contributing and authoring guides under samples/ (#633) * add samples contributing and authoring guides * update CODEOWNERS for samples changes --- storage/samples/AUTHORING_GUIDE.md | 1 + storage/samples/CONTRIBUTING.md | 1 + 2 files changed, 2 insertions(+) create mode 100644 storage/samples/AUTHORING_GUIDE.md create mode 100644 storage/samples/CONTRIBUTING.md diff --git a/storage/samples/AUTHORING_GUIDE.md b/storage/samples/AUTHORING_GUIDE.md new file mode 100644 index 00000000000..55c97b32f4c --- /dev/null +++ b/storage/samples/AUTHORING_GUIDE.md @@ -0,0 +1 @@ +See https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md \ No newline at end of file diff --git a/storage/samples/CONTRIBUTING.md b/storage/samples/CONTRIBUTING.md new file mode 100644 index 00000000000..34c882b6f1a --- /dev/null +++ b/storage/samples/CONTRIBUTING.md @@ -0,0 +1 @@ +See https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/CONTRIBUTING.md \ No newline at end of file From c9c7664ce7dcd0018df14d77c4ce6024b74e7e5e Mon Sep 17 00:00:00 2001 From: Tres Seaver Date: Mon, 25 Oct 2021 13:20:37 -0400 Subject: [PATCH 004/172] tests: replace spurious googleapis exception w/ api_core (#636) Closes #630. See https://github.com/googleapis/python-storage/pull/626#discussion_r735722629 --- storage/samples/snippets/acl_test.py | 36 ++++++++++------------- storage/samples/snippets/requirements.txt | 1 - 2 files changed, 16 insertions(+), 21 deletions(-) diff --git a/storage/samples/snippets/acl_test.py b/storage/samples/snippets/acl_test.py index fd2088ad638..91856d81654 100644 --- a/storage/samples/snippets/acl_test.py +++ b/storage/samples/snippets/acl_test.py @@ -16,8 +16,8 @@ import uuid import backoff +from google.api_core.exceptions import GoogleAPIError from google.cloud import storage -from googleapiclient.errors import HttpError import pytest import storage_add_bucket_default_owner @@ -33,10 +33,7 @@ # Typically we'd use a @example.com address, but GCS requires a real Google # account. Retrieve a service account email with storage admin permissions. -TEST_EMAIL = ( - "py38-storage-test" - "@python-docs-samples-tests.iam.gserviceaccount.com" -) +TEST_EMAIL = "py38-storage-test" "@python-docs-samples-tests.iam.gserviceaccount.com" @pytest.fixture(scope="module") @@ -45,8 +42,8 @@ def test_bucket(): # The new projects have uniform bucket-level access and our tests don't # pass with those buckets. We need to use the old main project for now. - original_value = os.environ['GOOGLE_CLOUD_PROJECT'] - os.environ['GOOGLE_CLOUD_PROJECT'] = os.environ['MAIN_GOOGLE_CLOUD_PROJECT'] + original_value = os.environ["GOOGLE_CLOUD_PROJECT"] + os.environ["GOOGLE_CLOUD_PROJECT"] = os.environ["MAIN_GOOGLE_CLOUD_PROJECT"] bucket = None while bucket is None or bucket.exists(): bucket_name = "acl-test-{}".format(uuid.uuid4()) @@ -55,7 +52,7 @@ def test_bucket(): yield bucket bucket.delete(force=True) # Set the value back. - os.environ['GOOGLE_CLOUD_PROJECT'] = original_value + os.environ["GOOGLE_CLOUD_PROJECT"] = original_value @pytest.fixture @@ -85,7 +82,7 @@ def test_print_bucket_acl_for_user(test_bucket, capsys): assert "OWNER" in out -@backoff.on_exception(backoff.expo, HttpError, max_time=60) +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_add_bucket_owner(test_bucket): storage_add_bucket_owner.add_bucket_owner(test_bucket.name, TEST_EMAIL) @@ -93,19 +90,18 @@ def test_add_bucket_owner(test_bucket): assert "OWNER" in test_bucket.acl.user(TEST_EMAIL).get_roles() -@backoff.on_exception(backoff.expo, HttpError, max_time=60) +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_remove_bucket_owner(test_bucket): test_bucket.acl.user(TEST_EMAIL).grant_owner() test_bucket.acl.save() - storage_remove_bucket_owner.remove_bucket_owner( - test_bucket.name, TEST_EMAIL) + storage_remove_bucket_owner.remove_bucket_owner(test_bucket.name, TEST_EMAIL) test_bucket.acl.reload() assert "OWNER" not in test_bucket.acl.user(TEST_EMAIL).get_roles() -@backoff.on_exception(backoff.expo, HttpError, max_time=60) +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_add_bucket_default_owner(test_bucket): storage_add_bucket_default_owner.add_bucket_default_owner( test_bucket.name, TEST_EMAIL @@ -116,7 +112,7 @@ def test_add_bucket_default_owner(test_bucket): assert "OWNER" in roles -@backoff.on_exception(backoff.expo, HttpError, max_time=60) +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_remove_bucket_default_owner(test_bucket): test_bucket.acl.user(TEST_EMAIL).grant_owner() test_bucket.acl.save() @@ -131,13 +127,12 @@ def test_remove_bucket_default_owner(test_bucket): def test_print_blob_acl(test_blob, capsys): - storage_print_file_acl.print_blob_acl( - test_blob.bucket.name, test_blob.name) + storage_print_file_acl.print_blob_acl(test_blob.bucket.name, test_blob.name) out, _ = capsys.readouterr() assert out -@backoff.on_exception(backoff.expo, HttpError, max_time=60) +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_print_blob_acl_for_user(test_blob, capsys): test_blob.acl.user(TEST_EMAIL).grant_owner() test_blob.acl.save() @@ -150,16 +145,17 @@ def test_print_blob_acl_for_user(test_blob, capsys): assert "OWNER" in out -@backoff.on_exception(backoff.expo, HttpError, max_time=60) +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_add_blob_owner(test_blob): storage_add_file_owner.add_blob_owner( - test_blob.bucket.name, test_blob.name, TEST_EMAIL) + test_blob.bucket.name, test_blob.name, TEST_EMAIL + ) test_blob.acl.reload() assert "OWNER" in test_blob.acl.user(TEST_EMAIL).get_roles() -@backoff.on_exception(backoff.expo, HttpError, max_time=60) +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_remove_blob_owner(test_blob): test_blob.acl.user(TEST_EMAIL).grant_owner() test_blob.acl.save() diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 76ac6ee7c57..89a0718c645 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,3 +1,2 @@ google-cloud-pubsub==2.8.0 google-cloud-storage==1.42.3 -google-api-python-client==2.25.0 From 4535d58a898f1390934ae7d58b83e1e347201881 Mon Sep 17 00:00:00 2001 From: cojenco Date: Tue, 26 Oct 2021 15:59:20 -0700 Subject: [PATCH 005/172] docs: add README to samples subdirectory (#639) * streamline samples README outline * update comments in requester_pays_test * change list of samples to be collapsible * move readme to upper level for discoverability * add cloud shell to each sample * fix typo and links --- storage/samples/snippets/README.md | 10 ---------- storage/samples/snippets/requester_pays_test.py | 2 ++ 2 files changed, 2 insertions(+), 10 deletions(-) delete mode 100644 storage/samples/snippets/README.md diff --git a/storage/samples/snippets/README.md b/storage/samples/snippets/README.md deleted file mode 100644 index 3d7e3664f58..00000000000 --- a/storage/samples/snippets/README.md +++ /dev/null @@ -1,10 +0,0 @@ - -For requester_pays_test.py, we need to use a different Storage bucket. - -The test looks for an environment variable `REQUESTER_PAYS_TEST_BUCKET`. - -Also, the service account for the test needs to have `Billing Project -Manager` role in order to make changes on buckets with requester pays -enabled. - -We added that role to the test service account. diff --git a/storage/samples/snippets/requester_pays_test.py b/storage/samples/snippets/requester_pays_test.py index 9f85c6bdb20..9a178edb03e 100644 --- a/storage/samples/snippets/requester_pays_test.py +++ b/storage/samples/snippets/requester_pays_test.py @@ -25,6 +25,8 @@ # We use a different bucket from other tests. +# The service account for the test needs to have Billing Project Manager role +# in order to make changes on buckets with requester pays enabled. BUCKET = os.environ["REQUESTER_PAYS_TEST_BUCKET"] PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"] From 704e22c9713950abba704313dbeb68328796b712 Mon Sep 17 00:00:00 2001 From: Aaron Gabriel Neyer Date: Tue, 9 Nov 2021 11:33:37 -0700 Subject: [PATCH 006/172] samples: Add GCS fileio samples (#645) * Add GCS file-like io samples * now with pandas! * change pandas version * add to readme * requests from andrew * lint * canonical command line args * Update storage_fileio_pandas.py * Update storage_fileio_write_read.py --- storage/samples/snippets/fileio_test.py | 35 ++++++++ storage/samples/snippets/requirements.txt | 1 + .../samples/snippets/storage_fileio_pandas.py | 86 +++++++++++++++++++ .../snippets/storage_fileio_write_read.py | 53 ++++++++++++ 4 files changed, 175 insertions(+) create mode 100644 storage/samples/snippets/fileio_test.py create mode 100644 storage/samples/snippets/storage_fileio_pandas.py create mode 100644 storage/samples/snippets/storage_fileio_write_read.py diff --git a/storage/samples/snippets/fileio_test.py b/storage/samples/snippets/fileio_test.py new file mode 100644 index 00000000000..cf98ce1ab5e --- /dev/null +++ b/storage/samples/snippets/fileio_test.py @@ -0,0 +1,35 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import uuid + +import storage_fileio_pandas +import storage_fileio_write_read + + +def test_fileio_write_read(bucket, capsys): + blob_name = "test-fileio-{}".format(uuid.uuid4()) + storage_fileio_write_read.write_read(bucket.name, blob_name) + out, _ = capsys.readouterr() + assert "Hello world" in out + + +def test_fileio_pandas(bucket, capsys): + blob_name = "test-fileio-{}".format(uuid.uuid4()) + storage_fileio_pandas.pandas_write(bucket.name, blob_name) + out, _ = capsys.readouterr() + assert f"Wrote csv with pandas with name {blob_name} from bucket {bucket.name}." in out + storage_fileio_pandas.pandas_read(bucket.name, blob_name) + out, _ = capsys.readouterr() + assert f"Read csv with pandas with name {blob_name} from bucket {bucket.name}." in out diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 89a0718c645..e6d0833e5fe 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,2 +1,3 @@ google-cloud-pubsub==2.8.0 google-cloud-storage==1.42.3 +pandas==1.1.5 diff --git a/storage/samples/snippets/storage_fileio_pandas.py b/storage/samples/snippets/storage_fileio_pandas.py new file mode 100644 index 00000000000..d4d01edd784 --- /dev/null +++ b/storage/samples/snippets/storage_fileio_pandas.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python + +# Copyright 2021 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that creates and consumes a GCS blob using pandas with file-like IO +""" + +# [START storage_fileio_pandas_write] + + +def pandas_write(bucket_name, blob_name): + """Use pandas to interact with GCS using file-like IO""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The ID of your new GCS object + # blob_name = "storage-object-name" + + from google.cloud import storage + import pandas as pd + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + with blob.open("w") as f: + df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]}) + f.write(df.to_csv(index=False)) + + print(f"Wrote csv with pandas with name {blob_name} from bucket {bucket.name}.") + + +# [END storage_fileio_pandas_write] + + +# [START storage_fileio_pandas_read] + + +def pandas_read(bucket_name, blob_name): + """Use pandas to interact with GCS using file-like IO""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The ID of your new GCS object + # blob_name = "storage-object-name" + + from google.cloud import storage + import pandas as pd + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + with blob.open("r") as f: + pd.read_csv(f) + + print(f"Read csv with pandas with name {blob_name} from bucket {bucket.name}.") + + +# [END storage_fileio_pandas_read] + + +if __name__ == "__main__": + pandas_write( + bucket_name=sys.argv[1], + blob_name=sys.argv[2] + ) + + pandas_read( + bucket_name=sys.argv[1], + blob_name=sys.argv[2] + ) diff --git a/storage/samples/snippets/storage_fileio_write_read.py b/storage/samples/snippets/storage_fileio_write_read.py new file mode 100644 index 00000000000..5d35c84ab51 --- /dev/null +++ b/storage/samples/snippets/storage_fileio_write_read.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python + +# Copyright 2021 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that writes and read a blob in GCS using file-like IO +""" + +# [START storage_fileio_write_read] +from google.cloud import storage + + +def write_read(bucket_name, blob_name): + """Write and read a blob from GCS using file-like IO""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The ID of your new GCS object + # blob_name = "storage-object-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + # Mode can be specified as wb/rb for bytes mode. + # See: https://docs.python.org/3/library/io.html + with blob.open("w") as f: + f.write("Hello world") + + with blob.open("r") as f: + print(f.read()) + + +# [END storage_fileio_write_read] + +if __name__ == "__main__": + write_read( + bucket_name=sys.argv[1], + blob_name=sys.argv[2] + ) From 02831ad464fc1b2ae2d3750319b7b4c03962ba72 Mon Sep 17 00:00:00 2001 From: cojenco Date: Wed, 10 Nov 2021 10:20:33 -0800 Subject: [PATCH 007/172] samples: add pubsub notifications samples (#646) * samples: add sample list notifications * samples: add sample get notification * add tests for get list notifications samples * add sample and test for creating a pubsub notification * add sample for deleting a pubsub notification * revise notification samples tests * update samples readme * clean up pubsub topic in tests * revise readme per comment --- storage/samples/snippets/notification_test.py | 120 ++++++++++++++++++ .../storage_create_bucket_notifications.py | 47 +++++++ .../storage_delete_bucket_notification.py | 47 +++++++ .../storage_list_bucket_notifications.py | 45 +++++++ ...torage_print_pubsub_bucket_notification.py | 53 ++++++++ 5 files changed, 312 insertions(+) create mode 100644 storage/samples/snippets/notification_test.py create mode 100644 storage/samples/snippets/storage_create_bucket_notifications.py create mode 100644 storage/samples/snippets/storage_delete_bucket_notification.py create mode 100644 storage/samples/snippets/storage_list_bucket_notifications.py create mode 100644 storage/samples/snippets/storage_print_pubsub_bucket_notification.py diff --git a/storage/samples/snippets/notification_test.py b/storage/samples/snippets/notification_test.py new file mode 100644 index 00000000000..13553c84413 --- /dev/null +++ b/storage/samples/snippets/notification_test.py @@ -0,0 +1,120 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import uuid + +from google.api_core.exceptions import NotFound +from google.cloud import storage + +import pytest + +import storage_create_bucket_notifications +import storage_delete_bucket_notification +import storage_list_bucket_notifications +import storage_print_pubsub_bucket_notification + +_topic_name = f"notification-{uuid.uuid4()}" + + +@pytest.fixture(scope="module") +def storage_client(): + return storage.Client() + + +@pytest.fixture(scope="module") +def publisher_client(): + try: + from google.cloud.pubsub_v1 import PublisherClient + except ImportError: + pytest.skip("Cannot import pubsub") + + return PublisherClient() + + +@pytest.fixture(scope="module") +def _notification_topic(storage_client, publisher_client): + topic_path = publisher_client.topic_path(storage_client.project, _topic_name) + try: + topic = publisher_client.get_topic(request={"topic": topic_path}) + except NotFound: + topic = publisher_client.create_topic(request={"name": topic_path}) + + policy = publisher_client.get_iam_policy(request={"resource": topic_path}) + binding = policy.bindings.add() + binding.role = "roles/pubsub.publisher" + binding.members.append( + "serviceAccount:{}".format(storage_client.get_service_account_email()) + ) + publisher_client.set_iam_policy(request={"resource": topic_path, "policy": policy}) + + yield topic + + try: + publisher_client.delete_topic(request={"topic": topic.name}) + except NotFound: + pass + + +@pytest.fixture(scope="module") +def bucket_w_notification(storage_client, _notification_topic): + """Yields a bucket with notification that is deleted after the tests complete.""" + bucket = None + while bucket is None or bucket.exists(): + bucket_name = f"notification-test-{uuid.uuid4()}" + bucket = storage_client.bucket(bucket_name) + bucket.create() + + notification = bucket.notification(topic_name=_topic_name) + notification.create() + + yield bucket + + bucket.delete(force=True) + + +def test_list_bucket_notifications(bucket_w_notification, capsys): + storage_list_bucket_notifications.list_bucket_notifications(bucket_w_notification.name) + out, _ = capsys.readouterr() + assert "Notification ID" in out + + +def test_print_pubsub_bucket_notification(bucket_w_notification, capsys): + notification_id = 1 + storage_print_pubsub_bucket_notification.print_pubsub_bucket_notification(bucket_w_notification.name, notification_id) + out, _ = capsys.readouterr() + assert "Notification ID: 1" in out + + +def test_create_bucket_notifications(bucket_w_notification, capsys): + # test only bucket notification ID 1 was created in the fixture + assert bucket_w_notification.notification(notification_id=1).exists() is True + assert bucket_w_notification.notification(notification_id=2).exists() is False + + storage_create_bucket_notifications.create_bucket_notifications(bucket_w_notification.name, _topic_name) + out, _ = capsys.readouterr() + assert "Successfully created notification" in out + # test succesfully creates new bucket notification with ID 2 + assert bucket_w_notification.notification(notification_id=2).exists() is True + + +def test_delete_bucket_notification(bucket_w_notification, capsys): + # test bucket notification ID 1 was created in the fixture + notification_id = 1 + assert bucket_w_notification.notification(notification_id=notification_id).exists() is True + + storage_delete_bucket_notification.delete_bucket_notification(bucket_w_notification.name, notification_id) + out, _ = capsys.readouterr() + assert "Successfully deleted notification" in out + assert bucket_w_notification.notification(notification_id=notification_id).exists() is False diff --git a/storage/samples/snippets/storage_create_bucket_notifications.py b/storage/samples/snippets/storage_create_bucket_notifications.py new file mode 100644 index 00000000000..a6f218c36fa --- /dev/null +++ b/storage/samples/snippets/storage_create_bucket_notifications.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that creates a notification configuration for a bucket. +This sample is used on this page: + https://cloud.google.com/storage/docs/reporting-changes +For more information, see README.md. +""" + +# [START storage_create_bucket_notifications] +from google.cloud import storage + + +def create_bucket_notifications(bucket_name, topic_name): + """Creates a notification configuration for a bucket.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + # The name of a topic + # topic_name = "your-topic-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + notification = bucket.notification(topic_name=topic_name) + notification.create() + + print(f"Successfully created notification with ID {notification.notification_id} for bucket {bucket_name}") + +# [END storage_create_bucket_notifications] + + +if __name__ == "__main__": + create_bucket_notifications(bucket_name=sys.argv[1], topic_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_delete_bucket_notification.py b/storage/samples/snippets/storage_delete_bucket_notification.py new file mode 100644 index 00000000000..efd41771d60 --- /dev/null +++ b/storage/samples/snippets/storage_delete_bucket_notification.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that deletes a notification configuration for a bucket. +This sample is used on this page: + https://cloud.google.com/storage/docs/reporting-changes +For more information, see README.md. +""" + +# [START storage_delete_bucket_notification] +from google.cloud import storage + + +def delete_bucket_notification(bucket_name, notification_id): + """Deletes a notification configuration for a bucket.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + # The ID of the notification + # notification_id = "your-notification-id" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + notification = bucket.notification(notification_id=notification_id) + notification.delete() + + print(f"Successfully deleted notification with ID {notification_id} for bucket {bucket_name}") + +# [END storage_delete_bucket_notification] + + +if __name__ == "__main__": + delete_bucket_notification(bucket_name=sys.argv[1], notification_id=sys.argv[2]) diff --git a/storage/samples/snippets/storage_list_bucket_notifications.py b/storage/samples/snippets/storage_list_bucket_notifications.py new file mode 100644 index 00000000000..0d25138bc90 --- /dev/null +++ b/storage/samples/snippets/storage_list_bucket_notifications.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that lists notification configurations for a bucket. +This sample is used on this page: + https://cloud.google.com/storage/docs/reporting-changes +For more information, see README.md. +""" + +# [START storage_list_bucket_notifications] +from google.cloud import storage + + +def list_bucket_notifications(bucket_name): + """Lists notification configurations for a bucket.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + notifications = bucket.list_notifications() + + for notification in notifications: + print(f"Notification ID: {notification.notification_id}") + +# [END storage_list_bucket_notifications] + + +if __name__ == "__main__": + list_bucket_notifications(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_print_pubsub_bucket_notification.py b/storage/samples/snippets/storage_print_pubsub_bucket_notification.py new file mode 100644 index 00000000000..3df45dc1f57 --- /dev/null +++ b/storage/samples/snippets/storage_print_pubsub_bucket_notification.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that gets a notification configuration for a bucket. +This sample is used on this page: + https://cloud.google.com/storage/docs/reporting-changes +For more information, see README.md. +""" + +# [START storage_print_pubsub_bucket_notification] +from google.cloud import storage + + +def print_pubsub_bucket_notification(bucket_name, notification_id): + """Gets a notification configuration for a bucket.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + # The ID of the notification + # notification_id = "your-notification-id" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + notification = bucket.get_notification(notification_id) + + print(f"Notification ID: {notification.notification_id}") + print(f"Topic Name: {notification.topic_name}") + print(f"Event Types: {notification.event_types}") + print(f"Custom Attributes: {notification.custom_attributes}") + print(f"Payload Format: {notification.payload_format}") + print(f"Blob Name Prefix: {notification.blob_name_prefix}") + print(f"Etag: {notification.etag}") + print(f"Self Link: {notification.self_link}") + +# [END storage_print_pubsub_bucket_notification] + + +if __name__ == "__main__": + print_pubsub_bucket_notification(bucket_name=sys.argv[1], notification_id=sys.argv[2]) From f1ada718f54fe765c05f10a940bdd7c204f97be9 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 11 Nov 2021 04:06:03 +0100 Subject: [PATCH 008/172] chore(deps): update dependency google-cloud-pubsub to v2.9.0 (#649) Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index e6d0833e5fe..3650d395bcf 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,3 +1,3 @@ -google-cloud-pubsub==2.8.0 +google-cloud-pubsub==2.9.0 google-cloud-storage==1.42.3 pandas==1.1.5 From fce859ab87d89dca6f906ed5eac67d0536665852 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 12 Nov 2021 18:47:11 +0100 Subject: [PATCH 009/172] chore(deps): update dependency pandas to v1.3.4 (#648) * chore(deps): update dependency pandas to v1.3.4 * Update samples/snippets/requirements.txt Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> --- storage/samples/snippets/requirements.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 3650d395bcf..addda196042 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,3 +1,4 @@ google-cloud-pubsub==2.9.0 google-cloud-storage==1.42.3 -pandas==1.1.5 +pandas==1.3.4; python_version > '3.6' +pandas==1.1.5; python_version < '3.7' From 86ee74347bd6cd58c87d9fd7006866452b9c4c20 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 15 Nov 2021 23:04:15 +0100 Subject: [PATCH 010/172] chore(deps): update dependency pytest to v6.2.5 (#629) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [pytest](https://docs.pytest.org/en/latest/) ([source](https://togithub.com/pytest-dev/pytest), [changelog](https://docs.pytest.org/en/stable/changelog.html)) | `==6.2.4` -> `==6.2.5` | [![age](https://badges.renovateapi.com/packages/pypi/pytest/6.2.5/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/pytest/6.2.5/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/pytest/6.2.5/compatibility-slim/6.2.4)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/pytest/6.2.5/confidence-slim/6.2.4)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
pytest-dev/pytest ### [`v6.2.5`](https://togithub.com/pytest-dev/pytest/releases/6.2.5) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/6.2.4...6.2.5) # pytest 6.2.5 (2021-08-29) ## Trivial/Internal Changes - [#​8494](https://togithub.com/pytest-dev/pytest/issues/8494): Python 3.10 is now supported. - [#​9040](https://togithub.com/pytest-dev/pytest/issues/9040): Enable compatibility with `pluggy 1.0` or later.
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-storage). --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 2b550f467fa..0a75575806c 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==6.2.4 +pytest==6.2.5 mock==4.0.3 backoff==1.11.1 \ No newline at end of file From 91f25f70fb171e478db7cd2baff9115f993a3ef7 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 18 Nov 2021 18:51:49 +0100 Subject: [PATCH 011/172] chore(deps): update dependency google-cloud-storage to v1.43.0 (#654) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index addda196042..240aa5070c3 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.9.0 -google-cloud-storage==1.42.3 +google-cloud-storage==1.43.0 pandas==1.3.4; python_version > '3.6' pandas==1.1.5; python_version < '3.7' From dcca3d2836b27a6c63ffb3af44edb53a6d35c852 Mon Sep 17 00:00:00 2001 From: Bonnie Chan <52431539+cbonnie@users.noreply.github.com> Date: Fri, 10 Dec 2021 13:08:30 -0500 Subject: [PATCH 012/172] docs: Describe code sample more specifically (#660) docs: This is just a simple PR to better describe what the code is doing in the comments. --- .../samples/snippets/storage_create_bucket_class_location.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/storage/samples/snippets/storage_create_bucket_class_location.py b/storage/samples/snippets/storage_create_bucket_class_location.py index 64c2652d77a..51fa864405d 100644 --- a/storage/samples/snippets/storage_create_bucket_class_location.py +++ b/storage/samples/snippets/storage_create_bucket_class_location.py @@ -21,7 +21,10 @@ def create_bucket_class_location(bucket_name): - """Create a new bucket in specific location with storage class""" + """ + Create a new bucket in the US region with the coldline storage + class + """ # bucket_name = "your-new-bucket-name" storage_client = storage.Client() From 059a4f83bd22e5a18fc76628c34621bc19aaa3c0 Mon Sep 17 00:00:00 2001 From: Sameena Shaffeeullah Date: Fri, 10 Dec 2021 14:36:52 -0800 Subject: [PATCH 013/172] samples: added upload from/download into memory samples (#664) * samples: added upload from/download into memory samples * linted files: * responded to PR comments * renamed blob variable * responded to comments and updated readme * updated copyright * fixed test --- storage/samples/snippets/snippets_test.py | 20 +++++++ .../snippets/storage_download_into_memory.py | 55 +++++++++++++++++++ .../snippets/storage_upload_from_memory.py | 52 ++++++++++++++++++ 3 files changed, 127 insertions(+) create mode 100644 storage/samples/snippets/storage_download_into_memory.py create mode 100644 storage/samples/snippets/storage_upload_from_memory.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index dd8e6aeaf1e..7dc27ae1b49 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -38,6 +38,7 @@ import storage_disable_bucket_lifecycle_management import storage_disable_versioning import storage_download_file +import storage_download_into_memory import storage_download_public_file import storage_enable_bucket_lifecycle_management import storage_enable_versioning @@ -62,6 +63,7 @@ import storage_set_bucket_default_kms_key import storage_set_metadata import storage_upload_file +import storage_upload_from_memory import storage_upload_with_kms_key KMS_KEY = os.environ["CLOUD_KMS_KEY"] @@ -189,6 +191,15 @@ def test_upload_blob(test_bucket): ) +def test_upload_blob_from_memory(test_bucket, capsys): + storage_upload_from_memory.upload_blob_from_memory( + test_bucket.name, "Hello, is it me you're looking for?", "test_upload_blob" + ) + out, _ = capsys.readouterr() + + assert "Hello, is it me you're looking for?" in out + + def test_upload_blob_with_kms(test_bucket): with tempfile.NamedTemporaryFile() as source_file: source_file.write(b"test") @@ -209,6 +220,15 @@ def test_download_blob(test_blob): assert dest_file.read() +def test_download_blob_into_memory(test_blob, capsys): + storage_download_into_memory.download_blob_into_memory( + test_blob.bucket.name, test_blob.name + ) + out, _ = capsys.readouterr() + + assert "Hello, is it me you're looking for?" in out + + def test_blob_metadata(test_blob, capsys): storage_get_metadata.blob_metadata(test_blob.bucket.name, test_blob.name) out, _ = capsys.readouterr() diff --git a/storage/samples/snippets/storage_download_into_memory.py b/storage/samples/snippets/storage_download_into_memory.py new file mode 100644 index 00000000000..dd91302aa37 --- /dev/null +++ b/storage/samples/snippets/storage_download_into_memory.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_download_file] +from google.cloud import storage + + +def download_blob_into_memory(bucket_name, blob_name): + """Downloads a blob into memory.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The ID of your GCS object + # blob_name = "storage-object-name" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + + # Construct a client side representation of a blob. + # Note `Bucket.blob` differs from `Bucket.get_blob` as it doesn't retrieve + # any content from Google Cloud Storage. As we don't need additional data, + # using `Bucket.blob` is preferred here. + blob = bucket.blob(blob_name) + contents = blob.download_as_string() + + print( + "Downloaded storage object {} from bucket {} as the following string: {}.".format( + blob_name, bucket_name, contents + ) + ) + + +# [END storage_download_file] + +if __name__ == "__main__": + download_blob_into_memory( + bucket_name=sys.argv[1], + blob_name=sys.argv[2], + ) diff --git a/storage/samples/snippets/storage_upload_from_memory.py b/storage/samples/snippets/storage_upload_from_memory.py new file mode 100644 index 00000000000..e5f61ff932c --- /dev/null +++ b/storage/samples/snippets/storage_upload_from_memory.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_file_upload_from_memory] +from google.cloud import storage + + +def upload_blob_from_memory(bucket_name, contents, destination_blob_name): + """Uploads a file to the bucket.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + # The contents to upload to the file + # contents = "these are my contents" + # The ID of your GCS object + # destination_blob_name = "storage-object-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(destination_blob_name) + + blob.upload_from_string(contents) + + print( + "{} with contents {} uploaded to {}.".format( + destination_blob_name, contents, destination_blob_name + ) + ) + + +# [END storage_file_upload_from_memory] + +if __name__ == "__main__": + upload_blob_from_memory( + bucket_name=sys.argv[1], + contents=sys.argv[2], + destination_blob_name=sys.argv[3], + ) From 920601e9b3e49e222c5f9189e9602345bf27ba07 Mon Sep 17 00:00:00 2001 From: Sameena Shaffeeullah Date: Thu, 16 Dec 2021 10:40:37 -0800 Subject: [PATCH 014/172] samples: delete unspecified sample (#671) * samples: delete unspecified sample * fixed lint issue * updated readme --- .../snippets/public_access_prevention_test.py | 11 ----- ...et_public_access_prevention_unspecified.py | 43 ------------------- 2 files changed, 54 deletions(-) delete mode 100644 storage/samples/snippets/storage_set_public_access_prevention_unspecified.py diff --git a/storage/samples/snippets/public_access_prevention_test.py b/storage/samples/snippets/public_access_prevention_test.py index 40d3924b296..558a4ef1575 100644 --- a/storage/samples/snippets/public_access_prevention_test.py +++ b/storage/samples/snippets/public_access_prevention_test.py @@ -12,15 +12,12 @@ # See the License for the specific language governing permissions and # limitations under the License. -import pytest import storage_get_public_access_prevention import storage_set_public_access_prevention_enforced import storage_set_public_access_prevention_inherited -import storage_set_public_access_prevention_unspecified -@pytest.mark.skip(reason="Inconsistent due to unspecified->inherited change") def test_get_public_access_prevention(bucket, capsys): short_name = storage_get_public_access_prevention short_name.get_public_access_prevention(bucket.name) @@ -35,14 +32,6 @@ def test_set_public_access_prevention_enforced(bucket, capsys): assert f"Public access prevention is set to enforced for {bucket.name}." in out -@pytest.mark.skip(reason="Inconsistent due to unspecified->inherited change") -def test_set_public_access_prevention_unspecified(bucket, capsys): - short_name = storage_set_public_access_prevention_unspecified - short_name.set_public_access_prevention_unspecified(bucket.name) - out, _ = capsys.readouterr() - assert f"Public access prevention is 'unspecified' for {bucket.name}." in out - - def test_set_public_access_prevention_inherited(bucket, capsys): short_name = storage_set_public_access_prevention_inherited short_name.set_public_access_prevention_inherited(bucket.name) diff --git a/storage/samples/snippets/storage_set_public_access_prevention_unspecified.py b/storage/samples/snippets/storage_set_public_access_prevention_unspecified.py deleted file mode 100644 index ae2c4701c57..00000000000 --- a/storage/samples/snippets/storage_set_public_access_prevention_unspecified.py +++ /dev/null @@ -1,43 +0,0 @@ -#!/usr/bin/env python - -# Copyright 2021 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the 'License'); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import sys - -# [START storage_set_public_access_prevention_unspecified] -from google.cloud import storage -from google.cloud.storage.constants import PUBLIC_ACCESS_PREVENTION_UNSPECIFIED - - -def set_public_access_prevention_unspecified(bucket_name): - """Sets the public access prevention status to unspecified, so that the bucket inherits its setting from its parent project.""" - # The ID of your GCS bucket - # bucket_name = "my-bucket" - - storage_client = storage.Client() - bucket = storage_client.get_bucket(bucket_name) - - bucket.iam_configuration.public_access_prevention = ( - PUBLIC_ACCESS_PREVENTION_UNSPECIFIED - ) - bucket.patch() - - print(f"Public access prevention is 'unspecified' for {bucket.name}.") - - -# [END storage_set_public_access_prevention_unspecified] - -if __name__ == "__main__": - set_public_access_prevention_unspecified(bucket_name=sys.argv[1]) From d1863ba0ce764baa2ccef1cce66760d05f4e4753 Mon Sep 17 00:00:00 2001 From: Sameena Shaffeeullah Date: Mon, 20 Dec 2021 11:02:52 -0800 Subject: [PATCH 015/172] samples: storage_download_byte_range (#674) * samples: storage_download_byte_range * added spacing * updated readme * fixed test --- storage/samples/snippets/snippets_test.py | 9 +++ .../snippets/storage_download_byte_range.py | 69 +++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 storage/samples/snippets/storage_download_byte_range.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 7dc27ae1b49..9691388d513 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -37,6 +37,7 @@ import storage_delete_file_archived_generation import storage_disable_bucket_lifecycle_management import storage_disable_versioning +import storage_download_byte_range import storage_download_file import storage_download_into_memory import storage_download_public_file @@ -211,6 +212,14 @@ def test_upload_blob_with_kms(test_bucket): assert kms_blob.kms_key_name.startswith(KMS_KEY) +def test_download_byte_range(test_blob): + with tempfile.NamedTemporaryFile() as dest_file: + storage_download_byte_range.download_byte_range( + test_blob.bucket.name, test_blob.name, 0, 4, dest_file.name + ) + assert dest_file.read() == b'Hello' + + def test_download_blob(test_blob): with tempfile.NamedTemporaryFile() as dest_file: storage_download_file.download_blob( diff --git a/storage/samples/snippets/storage_download_byte_range.py b/storage/samples/snippets/storage_download_byte_range.py new file mode 100644 index 00000000000..e6143a04f46 --- /dev/null +++ b/storage/samples/snippets/storage_download_byte_range.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_download_byte_range] +from google.cloud import storage + + +def download_byte_range( + bucket_name, source_blob_name, start_byte, end_byte, destination_file_name +): + """Downloads a blob from the bucket.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The ID of your GCS object + # source_blob_name = "storage-object-name" + + # The starting byte at which to begin the download + # start_byte = 0 + + # The ending byte at which to end the download + # end_byte = 20 + + # The path to which the file should be downloaded + # destination_file_name = "local/path/to/file" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + + # Construct a client side representation of a blob. + # Note `Bucket.blob` differs from `Bucket.get_blob` as it doesn't retrieve + # any content from Google Cloud Storage. As we don't need additional data, + # using `Bucket.blob` is preferred here. + blob = bucket.blob(source_blob_name) + blob.download_to_filename(destination_file_name, start=start_byte, end=end_byte) + + print( + "Downloaded bytes {} to {} of object {} from bucket {} to local file {}.".format( + start_byte, end_byte, source_blob_name, bucket_name, destination_file_name + ) + ) + + +# [END storage_download_byte_range] + +if __name__ == "__main__": + download_byte_range( + bucket_name=sys.argv[1], + source_blob_name=sys.argv[2], + start_byte=sys.argv[3], + end_byte=sys.argv[4], + destination_file_name=sys.argv[5], + ) From 3129b4f251bf8a72578b0787c2479f0bb3c1ac60 Mon Sep 17 00:00:00 2001 From: Sameena Shaffeeullah Date: Wed, 22 Dec 2021 10:54:31 -0800 Subject: [PATCH 016/172] samples: update region tags (#675) --- storage/samples/snippets/storage_download_into_memory.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/storage_download_into_memory.py b/storage/samples/snippets/storage_download_into_memory.py index dd91302aa37..453a13e2112 100644 --- a/storage/samples/snippets/storage_download_into_memory.py +++ b/storage/samples/snippets/storage_download_into_memory.py @@ -16,7 +16,7 @@ import sys -# [START storage_download_file] +# [START storage_file_download_into_memory] from google.cloud import storage @@ -46,7 +46,7 @@ def download_blob_into_memory(bucket_name, blob_name): ) -# [END storage_download_file] +# [END storage_file_download_into_memory] if __name__ == "__main__": download_blob_into_memory( From 6ed493189b077efc88771f158fcbf0b8892fbfd3 Mon Sep 17 00:00:00 2001 From: cojenco Date: Wed, 5 Jan 2022 10:19:05 -0800 Subject: [PATCH 017/172] samples: add batch request sample and test (#656) * samples: add batch request sample and test * samples: update readme with new sample * add clarifying comment --- storage/samples/snippets/snippets_test.py | 15 +++++ .../samples/snippets/storage_batch_request.py | 60 +++++++++++++++++++ 2 files changed, 75 insertions(+) create mode 100644 storage/samples/snippets/storage_batch_request.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 9691388d513..7c0a5b91d66 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -23,6 +23,7 @@ import requests import storage_add_bucket_label +import storage_batch_request import storage_bucket_delete_default_kms_key import storage_change_default_storage_class import storage_change_file_storage_class @@ -538,3 +539,17 @@ def test_storage_configure_retries(test_blob, capsys): assert "The following library method is customized to be retried" in out assert "_should_retry" in out assert "initial=1.5, maximum=45.0, multiplier=1.2, deadline=500.0" in out + + +def test_batch_request(test_bucket): + blob1 = test_bucket.blob("b/1.txt") + blob2 = test_bucket.blob("b/2.txt") + blob1.upload_from_string("hello world") + blob2.upload_from_string("hello world") + + storage_batch_request.batch_request(test_bucket.name, "b/") + blob1.reload() + blob2.reload() + + assert blob1.metadata.get("your-metadata-key") == "your-metadata-value" + assert blob2.metadata.get("your-metadata-key") == "your-metadata-value" diff --git a/storage/samples/snippets/storage_batch_request.py b/storage/samples/snippets/storage_batch_request.py new file mode 100644 index 00000000000..863fc09cd11 --- /dev/null +++ b/storage/samples/snippets/storage_batch_request.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that uses a batch request. +This sample is used on this page: + https://cloud.google.com/storage/docs/batch +For more information, see README.md. +""" + +# [START storage_batch_request] + +from google.cloud import storage + + +def batch_request(bucket_name, prefix=None): + """Use a batch request to patch a list of objects with the given prefix in a bucket.""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + # The prefix of the object paths + # prefix = "directory-prefix/" + + client = storage.Client() + bucket = client.bucket(bucket_name) + + # Accumulate in a list the objects with a given prefix. + blobs_to_patch = [blob for blob in bucket.list_blobs(prefix=prefix)] + + # Use a batch context manager to edit metadata in the list of blobs. + # The batch request is sent out when the context manager closes. + # No more than 100 calls should be included in a single batch request. + with client.batch(): + for blob in blobs_to_patch: + metadata = {"your-metadata-key": "your-metadata-value"} + blob.metadata = metadata + blob.patch() + + print( + f"Batch request edited metadata for all objects with the given prefix in {bucket.name}." + ) + + +# [END storage_batch_request] + +if __name__ == "__main__": + batch_request(bucket_name=sys.argv[1], prefix=sys.argv[2]) From 6346889d4756d681b12a0705e2511592184f2741 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 8 Jan 2022 01:45:36 +0100 Subject: [PATCH 018/172] chore(deps): update dependency google-cloud-storage to v1.44.0 (#680) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 240aa5070c3..de124cad231 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.9.0 -google-cloud-storage==1.43.0 +google-cloud-storage==1.44.0 pandas==1.3.4; python_version > '3.6' pandas==1.1.5; python_version < '3.7' From a08e532a8fb64abb98ebda9795657e4c15cbae72 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 11 Jan 2022 00:42:52 +0100 Subject: [PATCH 019/172] chore(deps): update dependency pandas to v1.3.5 (#681) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [pandas](https://pandas.pydata.org) ([source](https://togithub.com/pandas-dev/pandas)) | `==1.1.5` -> `==1.3.5` | [![age](https://badges.renovateapi.com/packages/pypi/pandas/1.3.5/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/pandas/1.3.5/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/pandas/1.3.5/compatibility-slim/1.1.5)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/pandas/1.3.5/confidence-slim/1.1.5)](https://docs.renovatebot.com/merge-confidence/) | | [pandas](https://pandas.pydata.org) ([source](https://togithub.com/pandas-dev/pandas)) | `==1.3.4` -> `==1.3.5` | [![age](https://badges.renovateapi.com/packages/pypi/pandas/1.3.5/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/pandas/1.3.5/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/pandas/1.3.5/compatibility-slim/1.3.4)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/pandas/1.3.5/confidence-slim/1.3.4)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
pandas-dev/pandas ### [`v1.3.5`](https://togithub.com/pandas-dev/pandas/releases/v1.3.5) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.3.4...v1.3.5) This is a patch release in the 1.3.x series and includes some regression fixes. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.3.5/whatsnew/v1.3.5.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.3.4`](https://togithub.com/pandas-dev/pandas/releases/v1.3.4) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.3.3...v1.3.4) This is a patch release in the 1.3.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.3.4/whatsnew/v1.3.4.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.3.3`](https://togithub.com/pandas-dev/pandas/releases/v1.3.3) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.3.2...v1.3.3) This is a patch release in the 1.3.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.3.3/whatsnew/v1.3.3.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.3.2`](https://togithub.com/pandas-dev/pandas/releases/v1.3.2) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.3.1...v1.3.2) This is a patch release in the 1.3.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.3.2/whatsnew/v1.3.2.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.3.1`](https://togithub.com/pandas-dev/pandas/releases/v1.3.1) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.3.0...v1.3.1) This is the first patch release in the 1.3.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.3.1/whatsnew/v1.3.1.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.3.0`](https://togithub.com/pandas-dev/pandas/releases/v1.3.0) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.2.5...v1.3.0) This release includes some new features, bug fixes, and performance improvements. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.3.0/whatsnew/v1.3.0.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install -c conda-forge pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.2.5`](https://togithub.com/pandas-dev/pandas/releases/v1.2.5) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.2.4...v1.2.5) This is a patch release in the 1.2.x series and includes some regression fixes. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.2.5/whatsnew/v1.2.5.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.2.4`](https://togithub.com/pandas-dev/pandas/releases/v1.2.4) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.2.3...v1.2.4) This is a patch release in the 1.2.x series and includes some regression fixes. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.2.4/whatsnew/v1.2.4.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.2.3`](https://togithub.com/pandas-dev/pandas/releases/v1.2.3) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.2.2...v1.2.3) This is a patch release in the 1.2.x series and includes some regression fixes. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.2.3/whatsnew/v1.2.3.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.2.2`](https://togithub.com/pandas-dev/pandas/releases/v1.2.2) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.2.1...v1.2.2) This is a patch release in the 1.2.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.2.2/whatsnew/v1.2.2.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.2.1`](https://togithub.com/pandas-dev/pandas/releases/v1.2.1) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.2.0...v1.2.1) This is the first patch release in the 1.2.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.2.1/whatsnew/v1.2.1.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). ### [`v1.2.0`](https://togithub.com/pandas-dev/pandas/releases/v1.2.0) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v1.1.5...v1.2.0) This release includes some new features, bug fixes, and performance improvements. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/1.2.0/whatsnew/v1.2.0.html) for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install -c conda-forge pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues).
--- ### Configuration 📅 **Schedule**: At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-storage). --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index de124cad231..14c2e74b4ce 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.9.0 google-cloud-storage==1.44.0 -pandas==1.3.4; python_version > '3.6' +pandas==1.3.5; python_version > '3.6' pandas==1.1.5; python_version < '3.7' From d73b7ce7e9f87ec13e61516c75f1cd2740ae6e29 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Tue, 11 Jan 2022 10:14:49 -0500 Subject: [PATCH 020/172] chore(samples): Add check for tests in directory (#684) Source-Link: https://github.com/googleapis/synthtool/commit/52aef91f8d25223d9dbdb4aebd94ba8eea2101f3 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:36a95b8f494e4674dc9eee9af98961293b51b86b3649942aac800ae6c1f796d4 Co-authored-by: Owl Bot --- storage/samples/snippets/noxfile.py | 70 ++++++++++++++++------------- 1 file changed, 39 insertions(+), 31 deletions(-) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 93a9122cc45..3bbef5d54f4 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -14,6 +14,7 @@ from __future__ import print_function +import glob import os from pathlib import Path import sys @@ -184,37 +185,44 @@ def blacken(session: nox.sessions.Session) -> None: def _session_tests( session: nox.sessions.Session, post_install: Callable = None ) -> None: - if TEST_CONFIG["pip_version_override"]: - pip_version = TEST_CONFIG["pip_version_override"] - session.install(f"pip=={pip_version}") - """Runs py.test for a particular project.""" - if os.path.exists("requirements.txt"): - if os.path.exists("constraints.txt"): - session.install("-r", "requirements.txt", "-c", "constraints.txt") - else: - session.install("-r", "requirements.txt") - - if os.path.exists("requirements-test.txt"): - if os.path.exists("constraints-test.txt"): - session.install("-r", "requirements-test.txt", "-c", "constraints-test.txt") - else: - session.install("-r", "requirements-test.txt") - - if INSTALL_LIBRARY_FROM_SOURCE: - session.install("-e", _get_repo_root()) - - if post_install: - post_install(session) - - session.run( - "pytest", - *(PYTEST_COMMON_ARGS + session.posargs), - # Pytest will return 5 when no tests are collected. This can happen - # on travis where slow and flaky tests are excluded. - # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html - success_codes=[0, 5], - env=get_pytest_env_vars(), - ) + # check for presence of tests + test_list = glob.glob("*_test.py") + glob.glob("test_*.py") + if len(test_list) == 0: + print("No tests found, skipping directory.") + else: + if TEST_CONFIG["pip_version_override"]: + pip_version = TEST_CONFIG["pip_version_override"] + session.install(f"pip=={pip_version}") + """Runs py.test for a particular project.""" + if os.path.exists("requirements.txt"): + if os.path.exists("constraints.txt"): + session.install("-r", "requirements.txt", "-c", "constraints.txt") + else: + session.install("-r", "requirements.txt") + + if os.path.exists("requirements-test.txt"): + if os.path.exists("constraints-test.txt"): + session.install( + "-r", "requirements-test.txt", "-c", "constraints-test.txt" + ) + else: + session.install("-r", "requirements-test.txt") + + if INSTALL_LIBRARY_FROM_SOURCE: + session.install("-e", _get_repo_root()) + + if post_install: + post_install(session) + + session.run( + "pytest", + *(PYTEST_COMMON_ARGS + session.posargs), + # Pytest will return 5 when no tests are collected. This can happen + # on travis where slow and flaky tests are excluded. + # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html + success_codes=[0, 5], + env=get_pytest_env_vars(), + ) @nox.session(python=ALL_VERSIONS) From 0708feeb1e1db14c0852c6112fccf9d49832d994 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 13 Jan 2022 20:25:16 +0100 Subject: [PATCH 021/172] chore(deps): update dependency google-cloud-storage to v2 (#690) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 14c2e74b4ce..443e0a601f7 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.9.0 -google-cloud-storage==1.44.0 +google-cloud-storage==2.0.0 pandas==1.3.5; python_version > '3.6' pandas==1.1.5; python_version < '3.7' From 67d0da882c2b70b9b3bc35bf7c920c7dd20fcf18 Mon Sep 17 00:00:00 2001 From: Aaron Gabriel Neyer Date: Tue, 18 Jan 2022 11:47:49 -0700 Subject: [PATCH 022/172] feat: remove python 3.6 support (#689) * remove python 3.6 support * few more things * try deleting kokoro python3.6 samples --- storage/samples/snippets/noxfile_config.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/noxfile_config.py b/storage/samples/snippets/noxfile_config.py index 463da97de55..ecd7fdce7a3 100644 --- a/storage/samples/snippets/noxfile_config.py +++ b/storage/samples/snippets/noxfile_config.py @@ -72,7 +72,7 @@ def get_cloud_kms_key(): TEST_CONFIG_OVERRIDE = { # You can opt out from the test for specific Python versions. - 'ignored_versions': ["2.7"], + 'ignored_versions': ["2.7", "3.6"], # An envvar key for determining the project id to use. Change it # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a From b1148e31d4fcc266708ba5a462ee21a07c50d64a Mon Sep 17 00:00:00 2001 From: Aaron Gabriel Neyer Date: Tue, 18 Jan 2022 14:12:01 -0700 Subject: [PATCH 023/172] feat: add turbo replication support and samples (#622) * feat: add turbo replication support * lintfix * nother-lint-fix * hmm weird * i need to learn how to use lint better... * . * . * how about now? * take rpo out of constructor * add unit tests * add link to docs * ensure inclusion of "rpo" in bucket._changes * add rpo samples * lint it * address cathys nits * fix a little test thing * start to fix weirdness, creating issue to address more fully * change it back --- storage/samples/snippets/rpo_test.py | 61 +++++++++++++++++++ ...storage_create_bucket_turbo_replication.py | 48 +++++++++++++++ storage/samples/snippets/storage_get_rpo.py | 48 +++++++++++++++ .../snippets/storage_set_rpo_async_turbo.py | 48 +++++++++++++++ .../snippets/storage_set_rpo_default.py | 48 +++++++++++++++ 5 files changed, 253 insertions(+) create mode 100644 storage/samples/snippets/rpo_test.py create mode 100644 storage/samples/snippets/storage_create_bucket_turbo_replication.py create mode 100644 storage/samples/snippets/storage_get_rpo.py create mode 100644 storage/samples/snippets/storage_set_rpo_async_turbo.py create mode 100644 storage/samples/snippets/storage_set_rpo_default.py diff --git a/storage/samples/snippets/rpo_test.py b/storage/samples/snippets/rpo_test.py new file mode 100644 index 00000000000..d084710a994 --- /dev/null +++ b/storage/samples/snippets/rpo_test.py @@ -0,0 +1,61 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import uuid + +from google.cloud import storage +import pytest + +import storage_create_bucket_turbo_replication +import storage_get_rpo +import storage_set_rpo_async_turbo +import storage_set_rpo_default + + +@pytest.fixture +def dual_region_bucket(): + """Yields a dual region bucket that is deleted after the test completes.""" + bucket = None + while bucket is None or bucket.exists(): + bucket_name = "bucket-lock-{}".format(uuid.uuid4()) + bucket = storage.Client().bucket(bucket_name) + bucket.location = "NAM4" + bucket.create() + yield bucket + bucket.delete(force=True) + + +def test_get_rpo(dual_region_bucket, capsys): + storage_get_rpo.get_rpo(dual_region_bucket.name) + out, _ = capsys.readouterr() + assert f"RPO for {dual_region_bucket.name} is DEFAULT." in out + + +def test_set_rpo_async_turbo(dual_region_bucket, capsys): + storage_set_rpo_async_turbo.set_rpo_async_turbo(dual_region_bucket.name) + out, _ = capsys.readouterr() + assert f"RPO is ASYNC_TURBO for {dual_region_bucket.name}." in out + + +def test_set_rpo_default(dual_region_bucket, capsys): + storage_set_rpo_default.set_rpo_default(dual_region_bucket.name) + out, _ = capsys.readouterr() + assert f"RPO is DEFAULT for {dual_region_bucket.name}." in out + + +def test_create_bucket_turbo_replication(capsys): + bucket_name = "test-rpo-{}".format(uuid.uuid4()) + storage_create_bucket_turbo_replication.create_bucket_turbo_replication(bucket_name) + out, _ = capsys.readouterr() + assert f"{bucket_name} created with RPO ASYNC_TURBO in NAM4." in out diff --git a/storage/samples/snippets/storage_create_bucket_turbo_replication.py b/storage/samples/snippets/storage_create_bucket_turbo_replication.py new file mode 100644 index 00000000000..68f0ba48240 --- /dev/null +++ b/storage/samples/snippets/storage_create_bucket_turbo_replication.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that creates a new bucket with dual-region and turbo replication. +This sample is used on this page: + https://cloud.google.com/storage/docs/managing-turbo-replication +For more information, see README.md. +""" + +# [START storage_create_bucket_turbo_replication] + +from google.cloud import storage +from google.cloud.storage.constants import RPO_ASYNC_TURBO + + +def create_bucket_turbo_replication(bucket_name): + """Creates dual-region bucket with turbo replication enabled.""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + bucket.location = "NAM4" + bucket.rpo = RPO_ASYNC_TURBO + bucket.create() + + print(f"{bucket.name} created with RPO {bucket.rpo} in {bucket.location}.") + + +# [END storage_create_bucket_turbo_replication] + +if __name__ == "__main__": + create_bucket_turbo_replication(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_get_rpo.py b/storage/samples/snippets/storage_get_rpo.py new file mode 100644 index 00000000000..29ae186fa42 --- /dev/null +++ b/storage/samples/snippets/storage_get_rpo.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that gets RPO (Recovery Point Objective) of a bucket +This sample is used on this page: + https://cloud.google.com/storage/docs/managing-turbo-replication +For more information, see README.md. +""" + +# [START storage_get_rpo] + +from google.cloud import storage +from google.cloud.storage.constants import RPO_DEFAULT + + +def get_rpo(bucket_name): + """Gets the RPO of the bucket""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + bucket.rpo = RPO_DEFAULT + rpo = bucket.rpo + + print(f"RPO for {bucket.name} is {rpo}.") + + +# [END storage_get_rpo] + +if __name__ == "__main__": + get_rpo(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_set_rpo_async_turbo.py b/storage/samples/snippets/storage_set_rpo_async_turbo.py new file mode 100644 index 00000000000..10b4c67a334 --- /dev/null +++ b/storage/samples/snippets/storage_set_rpo_async_turbo.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that sets RPO (Recovery Point Objective) to ASYNC_TURBO +This sample is used on this page: + https://cloud.google.com/storage/docs/managing-turbo-replication +For more information, see README.md. +""" + +# [START storage_set_rpo_async_turbo] + +from google.cloud import storage +from google.cloud.storage.constants import RPO_ASYNC_TURBO + + +def set_rpo_async_turbo(bucket_name): + """Sets the RPO to ASYNC_TURBO, enabling the turbo replication feature""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + bucket.rpo = RPO_ASYNC_TURBO + bucket.patch() + + print(f"RPO is ASYNC_TURBO for {bucket.name}.") + + +# [END storage_set_rpo_async_turbo] + +if __name__ == "__main__": + set_rpo_async_turbo(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_set_rpo_default.py b/storage/samples/snippets/storage_set_rpo_default.py new file mode 100644 index 00000000000..8d41b1fe0fb --- /dev/null +++ b/storage/samples/snippets/storage_set_rpo_default.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that sets RPO (Recovery Point Objective) to default +This sample is used on this page: + https://cloud.google.com/storage/docs/managing-turbo-replication +For more information, see README.md. +""" + +# [START storage_set_rpo_default] + +from google.cloud import storage +from google.cloud.storage.constants import RPO_DEFAULT + + +def set_rpo_default(bucket_name): + """Sets the RPO to DEFAULT, disabling the turbo replication feature""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + bucket.rpo = RPO_DEFAULT + bucket.patch() + + print(f"RPO is DEFAULT for {bucket.name}.") + + +# [END storage_set_rpo_default] + +if __name__ == "__main__": + set_rpo_default(bucket_name=sys.argv[1]) From f2e5dba8eb0b891aa32d660b72d8fba7ce352731 Mon Sep 17 00:00:00 2001 From: Aaron Gabriel Neyer Date: Tue, 18 Jan 2022 14:53:05 -0700 Subject: [PATCH 024/172] samples: add async upload sample (#665) * samples: add async upload sample * make python3.6 compatible * woops, this one too * so annoying, can we deprecate 3.6 too? * be gone with you 3.6! * add comment clarifying how sample is run --- storage/samples/snippets/snippets_test.py | 8 +++ .../samples/snippets/storage_async_upload.py | 61 +++++++++++++++++++ 2 files changed, 69 insertions(+) create mode 100644 storage/samples/snippets/storage_async_upload.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 7c0a5b91d66..28b35340bbf 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -12,6 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. +import asyncio import os import tempfile import time @@ -23,6 +24,7 @@ import requests import storage_add_bucket_label +import storage_async_upload import storage_batch_request import storage_bucket_delete_default_kms_key import storage_change_default_storage_class @@ -213,6 +215,12 @@ def test_upload_blob_with_kms(test_bucket): assert kms_blob.kms_key_name.startswith(KMS_KEY) +def test_async_upload(bucket, capsys): + asyncio.run(storage_async_upload.async_upload_blob(bucket.name)) + out, _ = capsys.readouterr() + assert f"Uploaded 3 files to bucket {bucket.name}" in out + + def test_download_byte_range(test_blob): with tempfile.NamedTemporaryFile() as dest_file: storage_download_byte_range.download_byte_range( diff --git a/storage/samples/snippets/storage_async_upload.py b/storage/samples/snippets/storage_async_upload.py new file mode 100644 index 00000000000..25aabb63ee4 --- /dev/null +++ b/storage/samples/snippets/storage_async_upload.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python + +# Copyright 2021 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import sys + + +"""Sample that asynchronously uploads a file to GCS +""" + + +# [START storage_async_upload] +# This sample can be run by calling `async.run(async_upload_blob('bucket_name'))` +async def async_upload_blob(bucket_name): + """Uploads a number of files in parallel to the bucket.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + import asyncio + from functools import partial + from google.cloud import storage + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + loop = asyncio.get_running_loop() + + tasks = [] + count = 3 + for x in range(count): + blob_name = f"async_sample_blob_{x}" + content = f"Hello world #{x}" + blob = bucket.blob(blob_name) + # The first arg, None, tells it to use the default loops executor + tasks.append(loop.run_in_executor(None, partial(blob.upload_from_string, content))) + + # If the method returns a value (such as download_as_string), gather will return the values + await asyncio.gather(*tasks) + + print(f"Uploaded {count} files to bucket {bucket_name}") + + +# [END storage_async_upload] + + +if __name__ == "__main__": + asyncio.run(async_upload_blob( + bucket_name=sys.argv[1] + )) From 74bb41442e059296e27f5e1209212c6d791669bc Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Tue, 18 Jan 2022 20:59:28 -0500 Subject: [PATCH 025/172] chore(python): Noxfile recognizes that tests can live in a folder (#696) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(python): Noxfile recognizes that tests can live in a folder Source-Link: https://github.com/googleapis/synthtool/commit/4760d8dce1351d93658cb11d02a1b7ceb23ae5d7 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:f0e4b51deef56bed74d3e2359c583fc104a8d6367da3984fc5c66938db738828 * update owlbot.py to remove python 3.6 samples * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/noxfile.py | 1 + 1 file changed, 1 insertion(+) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 3bbef5d54f4..20cdfc62013 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -187,6 +187,7 @@ def _session_tests( ) -> None: # check for presence of tests test_list = glob.glob("*_test.py") + glob.glob("test_*.py") + test_list.extend(glob.glob("tests")) if len(test_list) == 0: print("No tests found, skipping directory.") else: From 04db8d2eb7b38607ff5f5dc92215193661a3da8c Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 20 Jan 2022 17:13:27 +0100 Subject: [PATCH 026/172] chore(deps): update dependency google-cloud-storage to v2.1.0 (#697) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 443e0a601f7..3f72e94b739 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.9.0 -google-cloud-storage==2.0.0 +google-cloud-storage==2.1.0 pandas==1.3.5; python_version > '3.6' pandas==1.1.5; python_version < '3.7' From b4e3ee7460695fe2aa085c57700a117d74364cb8 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sun, 23 Jan 2022 12:13:15 +0100 Subject: [PATCH 027/172] chore(deps): update dependency pandas to v1.4.0 (#700) * chore(deps): update dependency pandas to v1.4.0 * add pandas pin for python 3.7 * adjust pin Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 3f72e94b739..df515cde4e7 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.9.0 google-cloud-storage==2.1.0 -pandas==1.3.5; python_version > '3.6' -pandas==1.1.5; python_version < '3.7' +pandas==1.3.5; python_version == '3.7' +pandas==1.4.0; python_version >= '3.8' From ea57ab7d41b8b5b9eb2d176527d5f5b80aef9cfc Mon Sep 17 00:00:00 2001 From: Aaron Gabriel Neyer Date: Tue, 1 Feb 2022 14:55:09 -0700 Subject: [PATCH 028/172] cleanup: add turbo rep samples and fix spacing on fileio samples (#708) * cleanup: add turbo rep samples and fix spacing on fileio samples * fix links * woops, missed that --- .../storage_create_bucket_location.py | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 storage/samples/snippets/storage_create_bucket_location.py diff --git a/storage/samples/snippets/storage_create_bucket_location.py b/storage/samples/snippets/storage_create_bucket_location.py new file mode 100644 index 00000000000..c94894a5b77 --- /dev/null +++ b/storage/samples/snippets/storage_create_bucket_location.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python + +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +"""Sample that creates a new bucket in a specified region +""" + +# [START storage_create_bucket_location] + +from google.cloud import storage + + +def create_bucket_location(bucket_name, bucket_location): + """Creates bucket in specified region.""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + # bucket_location = 'us-west1' # region + # bucket_location = 'nam4' # dual-region + # bucket_location = 'us' #multi-region + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + bucket.create(location=bucket_location) + + print(f"{bucket.name} created in {bucket.location}.") + + +# [END storage_create_bucket_location] + +if __name__ == "__main__": + create_bucket_location(bucket_name=sys.argv[1], bucket_location=sys.argv[2]) From 77f568c4e94465d56810955d7998d64f8228046d Mon Sep 17 00:00:00 2001 From: Andrew Gorcester Date: Wed, 2 Feb 2022 16:30:57 -0800 Subject: [PATCH 029/172] samples: Add snippets for upload/download from file-like object/stream (#709) * samples: Add snippets for upload/download from file-like object/stream * change sample region tags to be uniform with other libraries * lint --- storage/samples/snippets/snippets_test.py | 28 ++++++++++ .../snippets/storage_download_to_stream.py | 50 ++++++++++++++++++ .../snippets/storage_upload_from_memory.py | 7 ++- .../snippets/storage_upload_from_stream.py | 52 +++++++++++++++++++ 4 files changed, 135 insertions(+), 2 deletions(-) create mode 100644 storage/samples/snippets/storage_download_to_stream.py create mode 100644 storage/samples/snippets/storage_upload_from_stream.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 28b35340bbf..9f1f444afd7 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -13,6 +13,7 @@ # limitations under the License. import asyncio +import io import os import tempfile import time @@ -44,6 +45,7 @@ import storage_download_file import storage_download_into_memory import storage_download_public_file +import storage_download_to_stream import storage_enable_bucket_lifecycle_management import storage_enable_versioning import storage_generate_signed_post_policy_v4 @@ -68,6 +70,7 @@ import storage_set_metadata import storage_upload_file import storage_upload_from_memory +import storage_upload_from_stream import storage_upload_with_kms_key KMS_KEY = os.environ["CLOUD_KMS_KEY"] @@ -204,6 +207,17 @@ def test_upload_blob_from_memory(test_bucket, capsys): assert "Hello, is it me you're looking for?" in out +def test_upload_blob_from_stream(test_bucket, capsys): + file_obj = io.StringIO() + file_obj.write("This is test data.") + storage_upload_from_stream.upload_blob_from_stream( + test_bucket.name, file_obj, "test_upload_blob" + ) + out, _ = capsys.readouterr() + + assert "Stream data uploaded to {}".format("test_upload_blob") in out + + def test_upload_blob_with_kms(test_bucket): with tempfile.NamedTemporaryFile() as source_file: source_file.write(b"test") @@ -247,6 +261,20 @@ def test_download_blob_into_memory(test_blob, capsys): assert "Hello, is it me you're looking for?" in out +def test_download_blob_to_stream(test_blob, capsys): + file_obj = io.BytesIO() + storage_download_to_stream.download_blob_to_stream( + test_blob.bucket.name, test_blob.name, file_obj + ) + out, _ = capsys.readouterr() + + file_obj.seek(0) + content = file_obj.read() + + assert "Downloaded blob" in out + assert b"Hello, is it me you're looking for?" in content + + def test_blob_metadata(test_blob, capsys): storage_get_metadata.blob_metadata(test_blob.bucket.name, test_blob.name) out, _ = capsys.readouterr() diff --git a/storage/samples/snippets/storage_download_to_stream.py b/storage/samples/snippets/storage_download_to_stream.py new file mode 100644 index 00000000000..1cb8dcc7b31 --- /dev/null +++ b/storage/samples/snippets/storage_download_to_stream.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python + +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_stream_file_download] +from google.cloud import storage + + +def download_blob_to_stream(bucket_name, source_blob_name, file_obj): + """Downloads a blob to a stream or other file-like object.""" + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The ID of your GCS object (blob) + # source_blob_name = "storage-object-name" + + # The stream or file (file-like object) to which the blob will be written + # import io + # file_obj = io.BytesIO() + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + + # Construct a client-side representation of a blob. + # Note `Bucket.blob` differs from `Bucket.get_blob` in that it doesn't + # retrieve metadata from Google Cloud Storage. As we don't use metadata in + # this example, using `Bucket.blob` is preferred here. + blob = bucket.blob(source_blob_name) + blob.download_to_file(file_obj) + + print("Downloaded blob {} to file-like object.".format(source_blob_name)) + + return file_obj + # Before reading from file_obj, remember to rewind with file_obj.seek(0). + +# [END storage_stream_file_download] diff --git a/storage/samples/snippets/storage_upload_from_memory.py b/storage/samples/snippets/storage_upload_from_memory.py index e5f61ff932c..ee8a9828cd4 100644 --- a/storage/samples/snippets/storage_upload_from_memory.py +++ b/storage/samples/snippets/storage_upload_from_memory.py @@ -22,10 +22,13 @@ def upload_blob_from_memory(bucket_name, contents, destination_blob_name): """Uploads a file to the bucket.""" + # The ID of your GCS bucket # bucket_name = "your-bucket-name" + # The contents to upload to the file # contents = "these are my contents" + # The ID of your GCS object # destination_blob_name = "storage-object-name" @@ -37,13 +40,13 @@ def upload_blob_from_memory(bucket_name, contents, destination_blob_name): print( "{} with contents {} uploaded to {}.".format( - destination_blob_name, contents, destination_blob_name + destination_blob_name, contents, bucket_name ) ) - # [END storage_file_upload_from_memory] + if __name__ == "__main__": upload_blob_from_memory( bucket_name=sys.argv[1], diff --git a/storage/samples/snippets/storage_upload_from_stream.py b/storage/samples/snippets/storage_upload_from_stream.py new file mode 100644 index 00000000000..d43365e08d1 --- /dev/null +++ b/storage/samples/snippets/storage_upload_from_stream.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python + +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_stream_file_upload] +from google.cloud import storage + + +def upload_blob_from_stream(bucket_name, file_obj, destination_blob_name): + """Uploads bytes from a stream or other file-like object to a blob.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The stream or file (file-like object) from which to read + # import io + # file_obj = io.StringIO() + # file_obj.write("This is test data.") + + # The desired name of the uploaded GCS object (blob) + # destination_blob_name = "storage-object-name" + + # Construct a client-side representation of the blob. + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(destination_blob_name) + + # Rewind the stream to the beginning. This step can be omitted if the input + # stream will always be at a correct position. + file_obj.seek(0) + + # Upload data from the stream to your bucket. + blob.upload_from_file(file_obj) + + print( + "Stream data uploaded to {} in bucket {}.".format( + destination_blob_name, bucket_name + ) + ) + +# [END storage_stream_file_upload] From e17952ba2359e70e37a0c5f60944a243841856a0 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 7 Feb 2022 17:15:02 +0100 Subject: [PATCH 030/172] chore(deps): update dependency pytest to v7 (#711) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 0a75575806c..15deafb7b15 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==6.2.5 +pytest==7.0.0 mock==4.0.3 backoff==1.11.1 \ No newline at end of file From a73aca8bf8d9391d30eb1be18c12200f59805539 Mon Sep 17 00:00:00 2001 From: Aaron Gabriel Neyer Date: Wed, 9 Feb 2022 17:16:23 -0700 Subject: [PATCH 031/172] samples: add endpoint sample (#710) * samples: add endpoint sample * lint * properly check endpoint from client --- storage/samples/snippets/snippets_test.py | 8 +++++++ ...tion.py => storage_set_client_endpoint.py} | 24 ++++++++----------- 2 files changed, 18 insertions(+), 14 deletions(-) rename storage/samples/snippets/{storage_create_bucket_location.py => storage_set_client_endpoint.py} (53%) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 9f1f444afd7..58113410204 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -67,6 +67,7 @@ import storage_remove_cors_configuration import storage_rename_file import storage_set_bucket_default_kms_key +import storage_set_client_endpoint import storage_set_metadata import storage_upload_file import storage_upload_from_memory @@ -589,3 +590,10 @@ def test_batch_request(test_bucket): assert blob1.metadata.get("your-metadata-key") == "your-metadata-value" assert blob2.metadata.get("your-metadata-key") == "your-metadata-value" + + +def test_storage_set_client_endpoint(capsys): + storage_set_client_endpoint.set_client_endpoint('https://storage.googleapis.com') + out, _ = capsys.readouterr() + + assert "client initiated with endpoint: https://storage.googleapis.com" in out diff --git a/storage/samples/snippets/storage_create_bucket_location.py b/storage/samples/snippets/storage_set_client_endpoint.py similarity index 53% rename from storage/samples/snippets/storage_create_bucket_location.py rename to storage/samples/snippets/storage_set_client_endpoint.py index c94894a5b77..99ca283a18b 100644 --- a/storage/samples/snippets/storage_create_bucket_location.py +++ b/storage/samples/snippets/storage_set_client_endpoint.py @@ -19,27 +19,23 @@ """Sample that creates a new bucket in a specified region """ -# [START storage_create_bucket_location] +# [START storage_set_client_endpoint] from google.cloud import storage -def create_bucket_location(bucket_name, bucket_location): - """Creates bucket in specified region.""" - # The ID of your GCS bucket - # bucket_name = "my-bucket" - # bucket_location = 'us-west1' # region - # bucket_location = 'nam4' # dual-region - # bucket_location = 'us' #multi-region +def set_client_endpoint(api_endpoint): + """Initiates client with specified endpoint.""" + # api_endpoint = 'https://storage.googleapis.com' - storage_client = storage.Client() - bucket = storage_client.bucket(bucket_name) - bucket.create(location=bucket_location) + storage_client = storage.Client(client_options={'api_endpoint': api_endpoint}) - print(f"{bucket.name} created in {bucket.location}.") + print(f"client initiated with endpoint: {storage_client._connection.API_BASE_URL}") + return storage_client -# [END storage_create_bucket_location] + +# [END storage_set_client_endpoint] if __name__ == "__main__": - create_bucket_location(bucket_name=sys.argv[1], bucket_location=sys.argv[2]) + set_client_endpoint(api_endpoint=sys.argv[1]) From 6fa39918b76da30dbcf8cdc6d9118f516ac67327 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 11 Feb 2022 23:59:35 +0100 Subject: [PATCH 032/172] chore(deps): update dependency pytest to v7.0.1 (#713) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 15deafb7b15..dbdc6bb8101 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.0.0 +pytest==7.0.1 mock==4.0.3 backoff==1.11.1 \ No newline at end of file From 8e20b770ad83176825b037916cabab48fa01187c Mon Sep 17 00:00:00 2001 From: Sydney Munro <97561403+sydney-munro@users.noreply.github.com> Date: Thu, 17 Feb 2022 16:57:51 -0800 Subject: [PATCH 033/172] samples: Adding MediaLink to Get Metadata sample (#717) --- storage/samples/snippets/storage_get_metadata.py | 1 + 1 file changed, 1 insertion(+) diff --git a/storage/samples/snippets/storage_get_metadata.py b/storage/samples/snippets/storage_get_metadata.py index c5ef0b4ccc5..3ce7ecea862 100644 --- a/storage/samples/snippets/storage_get_metadata.py +++ b/storage/samples/snippets/storage_get_metadata.py @@ -52,6 +52,7 @@ def blob_metadata(bucket_name, blob_name): print("Content-encoding: {}".format(blob.content_encoding)) print("Content-language: {}".format(blob.content_language)) print("Metadata: {}".format(blob.metadata)) + print("Medialink: {}".format(blob.media_link)) print("Custom Time: {}".format(blob.custom_time)) print("Temporary hold: ", "enabled" if blob.temporary_hold else "disabled") print( From 0666748b87874dab464a97f1dd1a607a0a699991 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 4 Mar 2022 19:16:46 +0100 Subject: [PATCH 034/172] chore(deps): update dependency google-cloud-pubsub to v2.10.0 (#724) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index df515cde4e7..e9487c310a4 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.9.0 +google-cloud-pubsub==2.10.0 google-cloud-storage==2.1.0 pandas==1.3.5; python_version == '3.7' pandas==1.4.0; python_version >= '3.8' From 7e198dbbd0a7ed148a1222f9b230c7a032b88ca5 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Fri, 4 Mar 2022 14:34:16 -0500 Subject: [PATCH 035/172] chore: Adding support for pytest-xdist and pytest-parallel (#723) Source-Link: https://github.com/googleapis/synthtool/commit/82f5cb283efffe96e1b6cd634738e0e7de2cd90a Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:5d8da01438ece4021d135433f2cf3227aa39ef0eaccc941d62aa35e6902832ae Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/noxfile.py | 80 +++++++++++++++++------------ 1 file changed, 46 insertions(+), 34 deletions(-) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 20cdfc62013..4c808af73ea 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -188,42 +188,54 @@ def _session_tests( # check for presence of tests test_list = glob.glob("*_test.py") + glob.glob("test_*.py") test_list.extend(glob.glob("tests")) + if len(test_list) == 0: print("No tests found, skipping directory.") - else: - if TEST_CONFIG["pip_version_override"]: - pip_version = TEST_CONFIG["pip_version_override"] - session.install(f"pip=={pip_version}") - """Runs py.test for a particular project.""" - if os.path.exists("requirements.txt"): - if os.path.exists("constraints.txt"): - session.install("-r", "requirements.txt", "-c", "constraints.txt") - else: - session.install("-r", "requirements.txt") - - if os.path.exists("requirements-test.txt"): - if os.path.exists("constraints-test.txt"): - session.install( - "-r", "requirements-test.txt", "-c", "constraints-test.txt" - ) - else: - session.install("-r", "requirements-test.txt") - - if INSTALL_LIBRARY_FROM_SOURCE: - session.install("-e", _get_repo_root()) - - if post_install: - post_install(session) - - session.run( - "pytest", - *(PYTEST_COMMON_ARGS + session.posargs), - # Pytest will return 5 when no tests are collected. This can happen - # on travis where slow and flaky tests are excluded. - # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html - success_codes=[0, 5], - env=get_pytest_env_vars(), - ) + return + + if TEST_CONFIG["pip_version_override"]: + pip_version = TEST_CONFIG["pip_version_override"] + session.install(f"pip=={pip_version}") + """Runs py.test for a particular project.""" + concurrent_args = [] + if os.path.exists("requirements.txt"): + if os.path.exists("constraints.txt"): + session.install("-r", "requirements.txt", "-c", "constraints.txt") + else: + session.install("-r", "requirements.txt") + with open("requirements.txt") as rfile: + packages = rfile.read() + + if os.path.exists("requirements-test.txt"): + if os.path.exists("constraints-test.txt"): + session.install( + "-r", "requirements-test.txt", "-c", "constraints-test.txt" + ) + else: + session.install("-r", "requirements-test.txt") + with open("requirements-test.txt") as rtfile: + packages += rtfile.read() + + if INSTALL_LIBRARY_FROM_SOURCE: + session.install("-e", _get_repo_root()) + + if post_install: + post_install(session) + + if "pytest-parallel" in packages: + concurrent_args.extend(['--workers', 'auto', '--tests-per-worker', 'auto']) + elif "pytest-xdist" in packages: + concurrent_args.extend(['-n', 'auto']) + + session.run( + "pytest", + *(PYTEST_COMMON_ARGS + session.posargs + concurrent_args), + # Pytest will return 5 when no tests are collected. This can happen + # on travis where slow and flaky tests are excluded. + # See http://doc.pytest.org/en/latest/_modules/_pytest/main.html + success_codes=[0, 5], + env=get_pytest_env_vars(), + ) @nox.session(python=ALL_VERSIONS) From feb0ca83f5073e09eb98f7490a17ab95f4bbfcfb Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 10 Mar 2022 11:41:30 +0100 Subject: [PATCH 036/172] chore(deps): update dependency google-cloud-pubsub to v2.11.0 (#729) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index e9487c310a4..8e7837b4356 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.10.0 +google-cloud-pubsub==2.11.0 google-cloud-storage==2.1.0 pandas==1.3.5; python_version == '3.7' pandas==1.4.0; python_version >= '3.8' From 73ec783bd92b43a6d463ad70f3334a7f13cf10d7 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sun, 13 Mar 2022 21:02:26 +0100 Subject: [PATCH 037/172] chore(deps): update dependency pytest to v7.1.0 (#732) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index dbdc6bb8101..2777a5d9f73 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.0.1 +pytest==7.1.0 mock==4.0.3 backoff==1.11.1 \ No newline at end of file From c9124e277f0a28f5a19767c178de0eb50f26a1f2 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 14 Mar 2022 21:07:04 +0100 Subject: [PATCH 038/172] chore(deps): update dependency google-cloud-storage to v2.2.0 (#733) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 8e7837b4356..b95a96f8823 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.11.0 -google-cloud-storage==2.1.0 +google-cloud-storage==2.2.0 pandas==1.3.5; python_version == '3.7' pandas==1.4.0; python_version >= '3.8' From 9279601e43928c536c62a5425d6d394639831e50 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 15 Mar 2022 21:40:16 +0100 Subject: [PATCH 039/172] chore(deps): update dependency google-cloud-storage to v2.2.1 (#737) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index b95a96f8823..82d925edae2 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.11.0 -google-cloud-storage==2.2.0 +google-cloud-storage==2.2.1 pandas==1.3.5; python_version == '3.7' pandas==1.4.0; python_version >= '3.8' From 65b15741788906f41a2bb887058b331b6156b1ef Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sat, 19 Mar 2022 11:10:35 +0100 Subject: [PATCH 040/172] chore(deps): update dependency pytest to v7.1.1 (#738) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 2777a5d9f73..6fa4b275361 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.1.0 +pytest==7.1.1 mock==4.0.3 backoff==1.11.1 \ No newline at end of file From 3ec74425f1418c1703637c746c363f17d1e6a472 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Wed, 30 Mar 2022 06:02:41 -0400 Subject: [PATCH 041/172] chore(python): use black==22.3.0 (#742) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(python): use black==22.3.0 Source-Link: https://github.com/googleapis/synthtool/commit/6fab84af09f2cf89a031fd8671d1def6b2931b11 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:7cffbc10910c3ab1b852c05114a08d374c195a81cdec1d4a67a1d129331d0bfe * chore(python): use black==22.3.0 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/noxfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 4c808af73ea..949e0fde9ae 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -29,7 +29,7 @@ # WARNING - WARNING - WARNING - WARNING - WARNING # WARNING - WARNING - WARNING - WARNING - WARNING -BLACK_VERSION = "black==19.10b0" +BLACK_VERSION = "black==22.3.0" # Copy `noxfile_config.py` to your directory and modify it instead. From aff11b6449c34261b279d11bc7fd14e729b6be97 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sun, 3 Apr 2022 22:02:20 +0200 Subject: [PATCH 042/172] chore(deps): update dependency pandas to v1.4.2 (#751) * chore(deps): update dependency pandas to v1.4.2 * revert pin change for py37; use === to prevent updates for environment specific pins Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 82d925edae2..f13df193d3d 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.11.0 google-cloud-storage==2.2.1 -pandas==1.3.5; python_version == '3.7' -pandas==1.4.0; python_version >= '3.8' +pandas===1.3.5; python_version == '3.7' +pandas==1.4.2; python_version >= '3.8' From 770dbdb298d204e26d17c0faf01b33f7057c60ca Mon Sep 17 00:00:00 2001 From: cojenco Date: Wed, 6 Apr 2022 14:18:35 -0700 Subject: [PATCH 043/172] feat: add dual region bucket support and sample (#748) * feat: add dual region bucket support and tests * add dual region bucket sample * fix lint * update docstrings and doc ref links Co-authored-by: Daniel Bankhead --- storage/samples/snippets/snippets_test.py | 11 ++++ .../storage_create_bucket_dual_region.py | 50 +++++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 storage/samples/snippets/storage_create_bucket_dual_region.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 58113410204..7a5a3a64f0b 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -36,6 +36,7 @@ import storage_copy_file_archived_generation import storage_cors_configuration import storage_create_bucket_class_location +import storage_create_bucket_dual_region import storage_define_bucket_website_configuration import storage_delete_file import storage_delete_file_archived_generation @@ -433,6 +434,16 @@ def test_create_bucket_class_location(test_bucket_create): assert bucket.storage_class == "COLDLINE" +def test_create_bucket_dual_region(test_bucket_create, capsys): + region_1 = "US-EAST1" + region_2 = "US-WEST1" + storage_create_bucket_dual_region.create_bucket_dual_region( + test_bucket_create.name, region_1, region_2 + ) + out, _ = capsys.readouterr() + assert f"Bucket {test_bucket_create.name} created in {region_1}+{region_2}" in out + + def test_bucket_delete_default_kms_key(test_bucket, capsys): test_bucket.default_kms_key_name = KMS_KEY test_bucket.patch() diff --git a/storage/samples/snippets/storage_create_bucket_dual_region.py b/storage/samples/snippets/storage_create_bucket_dual_region.py new file mode 100644 index 00000000000..e6f4ac01f3d --- /dev/null +++ b/storage/samples/snippets/storage_create_bucket_dual_region.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python + +# Copyright 2022 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +""" +Sample that creates a dual region bucket. +""" + +# [START storage_create_bucket_dual_region] +from google.cloud import storage + + +def create_bucket_dual_region(bucket_name, region_1, region_2): + """Creates a Dual-Region Bucket with provided locations.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The bucket's pair of regions. Case-insensitive. + # See this documentation for other valid locations: + # https://cloud.google.com/storage/docs/locations + # region_1 = "US-EAST1" + # region_2 = "US-WEST1" + + storage_client = storage.Client() + storage_client.create_bucket(bucket_name, location=f"{region_1}+{region_2}") + + print(f"Bucket {bucket_name} created in {region_1}+{region_2}.") + + +# [END storage_create_bucket_dual_region] + + +if __name__ == "__main__": + create_bucket_dual_region( + bucket_name=sys.argv[1], region_1=sys.argv[2], region_2=sys.argv[3] + ) From d09c7079570bf0bcd515edeb07bcfccabb62fbef Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 7 Apr 2022 13:15:05 +0200 Subject: [PATCH 044/172] chore(deps): update dependency google-cloud-pubsub to v2.12.0 (#758) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index f13df193d3d..9074c15734b 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.11.0 +google-cloud-pubsub==2.12.0 google-cloud-storage==2.2.1 pandas===1.3.5; python_version == '3.7' pandas==1.4.2; python_version >= '3.8' From 009b5cd41966d582f95cab78aafa7ed6861243fe Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 14 Apr 2022 01:54:21 +0200 Subject: [PATCH 045/172] chore(deps): update dependency google-cloud-storage to v2.3.0 (#766) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 9074c15734b..d87433b822a 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.12.0 -google-cloud-storage==2.2.1 +google-cloud-storage==2.3.0 pandas===1.3.5; python_version == '3.7' pandas==1.4.2; python_version >= '3.8' From c3733f1290c32e18bb1a6afd026fc5774dc1c10a Mon Sep 17 00:00:00 2001 From: Tianzi Cai Date: Thu, 14 Apr 2022 13:48:07 -0700 Subject: [PATCH 046/172] samples(docs): remove beta tag in gcloud command (#767) --- storage/samples/snippets/notification_polling.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/notification_polling.py b/storage/samples/snippets/notification_polling.py index 3182db6da34..34fd8cc3e19 100644 --- a/storage/samples/snippets/notification_polling.py +++ b/storage/samples/snippets/notification_polling.py @@ -38,7 +38,7 @@ $ gsutil notification create -f json -t testtopic gs://testbucket 5. Create a subscription for your new topic: - $ gcloud beta pubsub subscriptions create testsubscription --topic=testtopic + $ gcloud pubsub subscriptions create testsubscription --topic=testtopic 6. Run this program: $ python notification_polling.py my-project-id testsubscription From e4ea6b6ae6976407f576f09779668e3415b6e49c Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Wed, 20 Apr 2022 20:35:30 -0400 Subject: [PATCH 047/172] chore(python): add nox session to sort python imports (#774) Source-Link: https://github.com/googleapis/synthtool/commit/1b71c10e20de7ed3f97f692f99a0e3399b67049f Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:00c9d764fd1cd56265f12a5ef4b99a0c9e87cf261018099141e2ca5158890416 Co-authored-by: Owl Bot --- storage/samples/snippets/noxfile.py | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 949e0fde9ae..38bb0a572b8 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -30,6 +30,7 @@ # WARNING - WARNING - WARNING - WARNING - WARNING BLACK_VERSION = "black==22.3.0" +ISORT_VERSION = "isort==5.10.1" # Copy `noxfile_config.py` to your directory and modify it instead. @@ -168,12 +169,32 @@ def lint(session: nox.sessions.Session) -> None: @nox.session def blacken(session: nox.sessions.Session) -> None: + """Run black. Format code to uniform standard.""" session.install(BLACK_VERSION) python_files = [path for path in os.listdir(".") if path.endswith(".py")] session.run("black", *python_files) +# +# format = isort + black +# + +@nox.session +def format(session: nox.sessions.Session) -> None: + """ + Run isort to sort imports. Then run black + to format code to uniform standard. + """ + session.install(BLACK_VERSION, ISORT_VERSION) + python_files = [path for path in os.listdir(".") if path.endswith(".py")] + + # Use the --fss option to sort imports using strict alphabetical order. + # See https://pycqa.github.io/isort/docs/configuration/options.html#force-sort-within-sections + session.run("isort", "--fss", *python_files) + session.run("black", *python_files) + + # # Sample Tests # From 138ceccc03e6700a84b4f558633b8d2a0716e76b Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 25 Apr 2022 17:12:06 +0200 Subject: [PATCH 048/172] chore(deps): update dependency pytest to v7.1.2 (#777) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 6fa4b275361..cf4bca94203 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.1.1 +pytest==7.1.2 mock==4.0.3 backoff==1.11.1 \ No newline at end of file From 0775a98e2fc690ca071ca55d3297378a3f16ac38 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 26 Apr 2022 20:31:15 +0200 Subject: [PATCH 049/172] chore(deps): update dependency backoff to v2 (#778) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index cf4bca94203..c5483280930 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ pytest==7.1.2 mock==4.0.3 -backoff==1.11.1 \ No newline at end of file +backoff==2.0.0 \ No newline at end of file From 04ccbc4d140764a1adb34a1555837144cba855be Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 27 Apr 2022 18:10:35 +0200 Subject: [PATCH 050/172] chore(deps): update dependency backoff to v2.0.1 (#779) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index c5483280930..88beb7ba24a 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ pytest==7.1.2 mock==4.0.3 -backoff==2.0.0 \ No newline at end of file +backoff==2.0.1 \ No newline at end of file From 62fc51286fc26e4e4c42cd738997cecb4bf3631e Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 12 May 2022 21:48:18 +0200 Subject: [PATCH 051/172] chore(deps): update dependency google-cloud-pubsub to v2.12.1 (#788) Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index d87433b822a..843c608cd87 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.12.0 +google-cloud-pubsub==2.12.1 google-cloud-storage==2.3.0 pandas===1.3.5; python_version == '3.7' pandas==1.4.2; python_version >= '3.8' From 35da51e76a5cdbbcce5557d93f7615fe79de9ec7 Mon Sep 17 00:00:00 2001 From: Pal Szabo Date: Mon, 16 May 2022 17:39:39 +0200 Subject: [PATCH 052/172] cleanup: f-string formatting (#789) * cleanup: f-string formatting * cleanup: f-string formatting * remove unnecessary :d specifier Co-authored-by: Mariatta Wijaya * remove unnecessary :d specifier Co-authored-by: Mariatta Wijaya * cleanup: f-string formatting * cleanup: f-string formatting Co-authored-by: Mariatta Wijaya --- storage/samples/snippets/acl_test.py | 4 +- storage/samples/snippets/bucket_lock_test.py | 10 ++-- storage/samples/snippets/encryption_test.py | 4 +- storage/samples/snippets/fileio_test.py | 4 +- storage/samples/snippets/iam_test.py | 4 +- .../samples/snippets/notification_polling.py | 12 ++--- storage/samples/snippets/notification_test.py | 2 +- storage/samples/snippets/noxfile.py | 4 +- storage/samples/snippets/quickstart.py | 2 +- .../samples/snippets/requester_pays_test.py | 6 +-- storage/samples/snippets/rpo_test.py | 4 +- storage/samples/snippets/snippets_test.py | 20 ++++---- .../snippets/storage_activate_hmac_key.py | 16 +++---- ...rage_add_bucket_conditional_iam_binding.py | 10 ++-- .../snippets/storage_add_bucket_iam_member.py | 2 +- .../snippets/storage_add_bucket_label.py | 2 +- .../snippets/storage_add_bucket_owner.py | 4 +- .../storage_bucket_delete_default_kms_key.py | 2 +- .../storage_change_default_storage_class.py | 2 +- .../snippets/storage_configure_retries.py | 2 +- .../snippets/storage_cors_configuration.py | 2 +- .../samples/snippets/storage_create_bucket.py | 2 +- .../snippets/storage_create_hmac_key.py | 18 ++++---- .../snippets/storage_deactivate_hmac_key.py | 16 +++---- .../samples/snippets/storage_delete_bucket.py | 2 +- .../samples/snippets/storage_delete_file.py | 2 +- ...storage_delete_file_archived_generation.py | 4 +- ...age_disable_bucket_lifecycle_management.py | 2 +- ...torage_disable_default_event_based_hold.py | 2 +- .../storage_disable_requester_pays.py | 2 +- ...age_disable_uniform_bucket_level_access.py | 2 +- .../snippets/storage_disable_versioning.py | 2 +- .../storage_download_encrypted_file.py | 4 +- .../snippets/storage_download_to_stream.py | 2 +- ...rage_enable_bucket_lifecycle_management.py | 4 +- ...storage_enable_default_event_based_hold.py | 2 +- .../snippets/storage_enable_requester_pays.py | 2 +- ...rage_enable_uniform_bucket_level_access.py | 2 +- .../snippets/storage_enable_versioning.py | 2 +- .../storage_generate_encryption_key.py | 2 +- .../storage_generate_signed_post_policy_v4.py | 2 +- .../storage_generate_signed_url_v2.py | 2 +- .../storage_generate_signed_url_v4.py | 2 +- .../storage_get_default_event_based_hold.py | 6 +-- .../samples/snippets/storage_get_hmac_key.py | 16 +++---- .../samples/snippets/storage_get_metadata.py | 46 +++++++++---------- .../storage_get_requester_pays_status.py | 4 +- .../snippets/storage_get_retention_policy.py | 6 +-- .../snippets/storage_get_service_account.py | 4 +- ...storage_get_uniform_bucket_level_access.py | 8 +--- .../storage_list_file_archived_generations.py | 2 +- .../snippets/storage_list_hmac_keys.py | 4 +- .../snippets/storage_lock_retention_policy.py | 6 +-- .../samples/snippets/storage_make_public.py | 4 +- .../snippets/storage_object_get_kms_key.py | 2 +- .../snippets/storage_print_bucket_acl.py | 2 +- .../snippets/storage_print_file_acl.py | 2 +- .../storage_release_event_based_hold.py | 2 +- .../storage_remove_bucket_default_owner.py | 4 +- .../storage_remove_bucket_iam_member.py | 2 +- .../snippets/storage_remove_bucket_label.py | 2 +- .../snippets/storage_remove_bucket_owner.py | 2 +- .../storage_remove_cors_configuration.py | 2 +- .../snippets/storage_remove_file_owner.py | 4 +- .../storage_remove_retention_policy.py | 2 +- .../samples/snippets/storage_rename_file.py | 2 +- .../snippets/storage_rotate_encryption_key.py | 2 +- .../snippets/storage_set_bucket_public_iam.py | 2 +- .../snippets/storage_set_event_based_hold.py | 2 +- .../samples/snippets/storage_set_metadata.py | 2 +- .../snippets/storage_upload_encrypted_file.py | 4 +- .../samples/snippets/storage_upload_file.py | 4 +- .../snippets/storage_upload_from_memory.py | 4 +- .../snippets/storage_upload_from_stream.py | 4 +- .../storage_view_bucket_iam_members.py | 2 +- .../uniform_bucket_level_access_test.py | 6 +-- 76 files changed, 164 insertions(+), 200 deletions(-) diff --git a/storage/samples/snippets/acl_test.py b/storage/samples/snippets/acl_test.py index 91856d81654..eecee522b57 100644 --- a/storage/samples/snippets/acl_test.py +++ b/storage/samples/snippets/acl_test.py @@ -46,7 +46,7 @@ def test_bucket(): os.environ["GOOGLE_CLOUD_PROJECT"] = os.environ["MAIN_GOOGLE_CLOUD_PROJECT"] bucket = None while bucket is None or bucket.exists(): - bucket_name = "acl-test-{}".format(uuid.uuid4()) + bucket_name = f"acl-test-{uuid.uuid4()}" bucket = storage.Client().bucket(bucket_name) bucket.create() yield bucket @@ -59,7 +59,7 @@ def test_bucket(): def test_blob(test_bucket): """Yields a blob that is deleted after the test completes.""" bucket = test_bucket - blob = bucket.blob("storage_acl_test_sigil-{}".format(uuid.uuid4())) + blob = bucket.blob(f"storage_acl_test_sigil-{uuid.uuid4()}") blob.upload_from_string("Hello, is it me you're looking for?") yield blob diff --git a/storage/samples/snippets/bucket_lock_test.py b/storage/samples/snippets/bucket_lock_test.py index 67d4ec6853a..9b7b4fa2a8e 100644 --- a/storage/samples/snippets/bucket_lock_test.py +++ b/storage/samples/snippets/bucket_lock_test.py @@ -42,7 +42,7 @@ def bucket(): """Yields a bucket that is deleted after the test completes.""" bucket = None while bucket is None or bucket.exists(): - bucket_name = "bucket-lock-{}".format(uuid.uuid4()) + bucket_name = f"bucket-lock-{uuid.uuid4()}" bucket = storage.Client().bucket(bucket_name) bucket.create() yield bucket @@ -61,7 +61,7 @@ def test_retention_policy_no_lock(bucket, capsys): storage_get_retention_policy.get_retention_policy(bucket.name) out, _ = capsys.readouterr() - assert "Retention Policy for {}".format(bucket.name) in out + assert f"Retention Policy for {bucket.name}" in out assert "Retention Period: 5" in out assert "Effective Time: " in out assert "Retention Policy is locked" not in out @@ -100,11 +100,11 @@ def test_enable_disable_bucket_default_event_based_hold(bucket, capsys): ) out, _ = capsys.readouterr() assert ( - "Default event-based hold is not enabled for {}".format(bucket.name) + f"Default event-based hold is not enabled for {bucket.name}" in out ) assert ( - "Default event-based hold is enabled for {}".format(bucket.name) + f"Default event-based hold is enabled for {bucket.name}" not in out ) @@ -120,7 +120,7 @@ def test_enable_disable_bucket_default_event_based_hold(bucket, capsys): ) out, _ = capsys.readouterr() assert ( - "Default event-based hold is enabled for {}".format(bucket.name) in out + f"Default event-based hold is enabled for {bucket.name}" in out ) # Changes to the bucket will be readable immediately after writing, diff --git a/storage/samples/snippets/encryption_test.py b/storage/samples/snippets/encryption_test.py index 6c2377e0fe6..536c5d3343e 100644 --- a/storage/samples/snippets/encryption_test.py +++ b/storage/samples/snippets/encryption_test.py @@ -62,7 +62,7 @@ def test_upload_encrypted_blob(): def test_blob(): """Provides a pre-existing blob in the test bucket.""" bucket = storage.Client().bucket(BUCKET) - blob_name = "test_blob_{}".format(uuid.uuid4().hex) + blob_name = f"test_blob_{uuid.uuid4().hex}" blob = Blob( blob_name, bucket, @@ -81,7 +81,7 @@ def test_blob(): blob.delete() except NotFound as e: # For the case that the rotation succeeded. - print("Ignoring 404, detail: {}".format(e)) + print(f"Ignoring 404, detail: {e}") blob = Blob( blob_name, bucket, diff --git a/storage/samples/snippets/fileio_test.py b/storage/samples/snippets/fileio_test.py index cf98ce1ab5e..b8a4b8272f4 100644 --- a/storage/samples/snippets/fileio_test.py +++ b/storage/samples/snippets/fileio_test.py @@ -19,14 +19,14 @@ def test_fileio_write_read(bucket, capsys): - blob_name = "test-fileio-{}".format(uuid.uuid4()) + blob_name = f"test-fileio-{uuid.uuid4()}" storage_fileio_write_read.write_read(bucket.name, blob_name) out, _ = capsys.readouterr() assert "Hello world" in out def test_fileio_pandas(bucket, capsys): - blob_name = "test-fileio-{}".format(uuid.uuid4()) + blob_name = f"test-fileio-{uuid.uuid4()}" storage_fileio_pandas.pandas_write(bucket.name, blob_name) out, _ = capsys.readouterr() assert f"Wrote csv with pandas with name {blob_name} from bucket {bucket.name}." in out diff --git a/storage/samples/snippets/iam_test.py b/storage/samples/snippets/iam_test.py index edeb8427d38..7700b6c6a8a 100644 --- a/storage/samples/snippets/iam_test.py +++ b/storage/samples/snippets/iam_test.py @@ -42,7 +42,7 @@ def bucket(): bucket = None while bucket is None or bucket.exists(): storage_client = storage.Client() - bucket_name = "test-iam-{}".format(uuid.uuid4()) + bucket_name = f"test-iam-{uuid.uuid4()}" bucket = storage_client.bucket(bucket_name) bucket.iam_configuration.uniform_bucket_level_access_enabled = True storage_client.create_bucket(bucket) @@ -60,7 +60,7 @@ def public_bucket(): bucket = None while bucket is None or bucket.exists(): storage_client = storage.Client() - bucket_name = "test-iam-{}".format(uuid.uuid4()) + bucket_name = f"test-iam-{uuid.uuid4()}" bucket = storage_client.bucket(bucket_name) bucket.iam_configuration.uniform_bucket_level_access_enabled = True storage_client.create_bucket(bucket) diff --git a/storage/samples/snippets/notification_polling.py b/storage/samples/snippets/notification_polling.py index 34fd8cc3e19..2ee6789c32f 100644 --- a/storage/samples/snippets/notification_polling.py +++ b/storage/samples/snippets/notification_polling.py @@ -76,13 +76,9 @@ def summarize(message): ) if "overwroteGeneration" in attributes: - description += "\tOverwrote generation: %s\n" % ( - attributes["overwroteGeneration"] - ) + description += f"\tOverwrote generation: {attributes['overwroteGeneration']}\n" if "overwrittenByGeneration" in attributes: - description += "\tOverwritten by generation: %s\n" % ( - attributes["overwrittenByGeneration"] - ) + description += f"\tOverwritten by generation: {attributes['overwrittenByGeneration']}\n" payload_format = attributes["payloadFormat"] if payload_format == "JSON_API_V1": @@ -110,14 +106,14 @@ def poll_notifications(project, subscription_name): ) def callback(message): - print("Received message:\n{}".format(summarize(message))) + print(f"Received message:\n{summarize(message)}") message.ack() subscriber.subscribe(subscription_path, callback=callback) # The subscriber is non-blocking, so we must keep the main thread from # exiting to allow it to process messages in the background. - print("Listening for messages on {}".format(subscription_path)) + print(f"Listening for messages on {subscription_path}") while True: time.sleep(60) diff --git a/storage/samples/snippets/notification_test.py b/storage/samples/snippets/notification_test.py index 13553c84413..a2fdbe3ef39 100644 --- a/storage/samples/snippets/notification_test.py +++ b/storage/samples/snippets/notification_test.py @@ -55,7 +55,7 @@ def _notification_topic(storage_client, publisher_client): binding = policy.bindings.add() binding.role = "roles/pubsub.publisher" binding.members.append( - "serviceAccount:{}".format(storage_client.get_service_account_email()) + f"serviceAccount:{storage_client.get_service_account_email()}" ) publisher_client.set_iam_policy(request={"resource": topic_path, "policy": policy}) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 38bb0a572b8..9f1cc8fb1ac 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -66,7 +66,7 @@ sys.path.append(".") from noxfile_config import TEST_CONFIG_OVERRIDE except ImportError as e: - print("No user noxfile_config found: detail: {}".format(e)) + print(f"No user noxfile_config found: detail: {e}") TEST_CONFIG_OVERRIDE = {} # Update the TEST_CONFIG with the user supplied values. @@ -266,7 +266,7 @@ def py(session: nox.sessions.Session) -> None: _session_tests(session) else: session.skip( - "SKIPPED: {} tests are disabled for this sample.".format(session.python) + f"SKIPPED: {session.python} tests are disabled for this sample." ) diff --git a/storage/samples/snippets/quickstart.py b/storage/samples/snippets/quickstart.py index 578e50753f0..54148b1fb55 100644 --- a/storage/samples/snippets/quickstart.py +++ b/storage/samples/snippets/quickstart.py @@ -29,7 +29,7 @@ def run_quickstart(): # Creates the new bucket bucket = storage_client.create_bucket(bucket_name) - print("Bucket {} created.".format(bucket.name)) + print(f"Bucket {bucket.name} created.") # [END storage_quickstart] diff --git a/storage/samples/snippets/requester_pays_test.py b/storage/samples/snippets/requester_pays_test.py index 9a178edb03e..cf8c2d09799 100644 --- a/storage/samples/snippets/requester_pays_test.py +++ b/storage/samples/snippets/requester_pays_test.py @@ -34,19 +34,19 @@ def test_enable_requester_pays(capsys): storage_enable_requester_pays.enable_requester_pays(BUCKET) out, _ = capsys.readouterr() - assert "Requester Pays has been enabled for {}".format(BUCKET) in out + assert f"Requester Pays has been enabled for {BUCKET}" in out def test_disable_requester_pays(capsys): storage_disable_requester_pays.disable_requester_pays(BUCKET) out, _ = capsys.readouterr() - assert "Requester Pays has been disabled for {}".format(BUCKET) in out + assert f"Requester Pays has been disabled for {BUCKET}" in out def test_get_requester_pays_status(capsys): storage_get_requester_pays_status.get_requester_pays_status(BUCKET) out, _ = capsys.readouterr() - assert "Requester Pays is disabled for {}".format(BUCKET) in out + assert f"Requester Pays is disabled for {BUCKET}" in out @pytest.fixture diff --git a/storage/samples/snippets/rpo_test.py b/storage/samples/snippets/rpo_test.py index d084710a994..f1f16e7fbd8 100644 --- a/storage/samples/snippets/rpo_test.py +++ b/storage/samples/snippets/rpo_test.py @@ -28,7 +28,7 @@ def dual_region_bucket(): """Yields a dual region bucket that is deleted after the test completes.""" bucket = None while bucket is None or bucket.exists(): - bucket_name = "bucket-lock-{}".format(uuid.uuid4()) + bucket_name = f"bucket-lock-{uuid.uuid4()}" bucket = storage.Client().bucket(bucket_name) bucket.location = "NAM4" bucket.create() @@ -55,7 +55,7 @@ def test_set_rpo_default(dual_region_bucket, capsys): def test_create_bucket_turbo_replication(capsys): - bucket_name = "test-rpo-{}".format(uuid.uuid4()) + bucket_name = f"test-rpo-{uuid.uuid4()}" storage_create_bucket_turbo_replication.create_bucket_turbo_replication(bucket_name) out, _ = capsys.readouterr() assert f"{bucket_name} created with RPO ASYNC_TURBO in NAM4." in out diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 7a5a3a64f0b..bdd8c528ecd 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -111,7 +111,7 @@ def test_bucket(): """Yields a bucket that is deleted after the test completes.""" bucket = None while bucket is None or bucket.exists(): - bucket_name = "storage-snippets-test-{}".format(uuid.uuid4()) + bucket_name = f"storage-snippets-test-{uuid.uuid4()}" bucket = storage.Client().bucket(bucket_name) bucket.create() yield bucket @@ -127,7 +127,7 @@ def test_public_bucket(): bucket = None while bucket is None or bucket.exists(): storage_client = storage.Client() - bucket_name = "storage-snippets-test-{}".format(uuid.uuid4()) + bucket_name = f"storage-snippets-test-{uuid.uuid4()}" bucket = storage_client.bucket(bucket_name) storage_client.create_bucket(bucket) yield bucket @@ -140,7 +140,7 @@ def test_public_bucket(): def test_blob(test_bucket): """Yields a blob that is deleted after the test completes.""" bucket = test_bucket - blob = bucket.blob("storage_snippets_test_sigil-{}".format(uuid.uuid4())) + blob = bucket.blob(f"storage_snippets_test_sigil-{uuid.uuid4()}") blob.upload_from_string("Hello, is it me you're looking for?") yield blob @@ -149,7 +149,7 @@ def test_blob(test_bucket): def test_public_blob(test_public_bucket): """Yields a blob that is deleted after the test completes.""" bucket = test_public_bucket - blob = bucket.blob("storage_snippets_test_sigil-{}".format(uuid.uuid4())) + blob = bucket.blob(f"storage_snippets_test_sigil-{uuid.uuid4()}") blob.upload_from_string("Hello, is it me you're looking for?") yield blob @@ -159,7 +159,7 @@ def test_bucket_create(): """Yields a bucket object that is deleted after the test completes.""" bucket = None while bucket is None or bucket.exists(): - bucket_name = "storage-snippets-test-{}".format(uuid.uuid4()) + bucket_name = f"storage-snippets-test-{uuid.uuid4()}" bucket = storage.Client().bucket(bucket_name) yield bucket bucket.delete(force=True) @@ -217,7 +217,7 @@ def test_upload_blob_from_stream(test_bucket, capsys): ) out, _ = capsys.readouterr() - assert "Stream data uploaded to {}".format("test_upload_blob") in out + assert "Stream data uploaded to test_upload_blob" in out def test_upload_blob_with_kms(test_bucket): @@ -339,7 +339,7 @@ def test_generate_signed_policy_v4(test_bucket, capsys): blob_name = "storage_snippets_test_form" short_name = storage_generate_signed_post_policy_v4 form = short_name.generate_signed_post_policy_v4(test_bucket.name, blob_name) - assert "name='key' value='{}'".format(blob_name) in form + assert f"name='key' value='{blob_name}'" in form assert "name='x-goog-signature'" in form assert "name='x-goog-date'" in form assert "name='x-goog-credential'" in form @@ -355,7 +355,7 @@ def test_rename_blob(test_blob): try: bucket.delete_blob("test_rename_blob") except google.cloud.exceptions.exceptions.NotFound: - print("test_rename_blob not found in bucket {}".format(bucket.name)) + print(f"test_rename_blob not found in bucket {bucket.name}") storage_rename_file.rename_blob(bucket.name, test_blob.name, "test_rename_blob") @@ -370,7 +370,7 @@ def test_move_blob(test_bucket_create, test_blob): try: test_bucket_create.delete_blob("test_move_blob") except google.cloud.exceptions.NotFound: - print("test_move_blob not found in bucket {}".format(test_bucket_create.name)) + print(f"test_move_blob not found in bucket {test_bucket_create.name}") storage_move_file.move_blob( bucket.name, test_blob.name, test_bucket_create.name, "test_move_blob" @@ -551,7 +551,7 @@ def test_change_file_storage_class(test_blob, capsys): test_blob.bucket.name, test_blob.name ) out, _ = capsys.readouterr() - assert "Blob {} in bucket {}". format(blob.name, blob.bucket.name) in out + assert f"Blob {blob.name} in bucket {blob.bucket.name}" in out assert blob.storage_class == 'NEARLINE' diff --git a/storage/samples/snippets/storage_activate_hmac_key.py b/storage/samples/snippets/storage_activate_hmac_key.py index e77cd80665d..d3960eb622c 100644 --- a/storage/samples/snippets/storage_activate_hmac_key.py +++ b/storage/samples/snippets/storage_activate_hmac_key.py @@ -36,14 +36,14 @@ def activate_key(access_id, project_id): hmac_key.update() print("The HMAC key metadata is:") - print("Service Account Email: {}".format(hmac_key.service_account_email)) - print("Key ID: {}".format(hmac_key.id)) - print("Access ID: {}".format(hmac_key.access_id)) - print("Project ID: {}".format(hmac_key.project)) - print("State: {}".format(hmac_key.state)) - print("Created At: {}".format(hmac_key.time_created)) - print("Updated At: {}".format(hmac_key.updated)) - print("Etag: {}".format(hmac_key.etag)) + print(f"Service Account Email: {hmac_key.service_account_email}") + print(f"Key ID: {hmac_key.id}") + print(f"Access ID: {hmac_key.access_id}") + print(f"Project ID: {hmac_key.project}") + print(f"State: {hmac_key.state}") + print(f"Created At: {hmac_key.time_created}") + print(f"Updated At: {hmac_key.updated}") + print(f"Etag: {hmac_key.etag}") return hmac_key diff --git a/storage/samples/snippets/storage_add_bucket_conditional_iam_binding.py b/storage/samples/snippets/storage_add_bucket_conditional_iam_binding.py index ddc0fc0286d..d09f528cf72 100644 --- a/storage/samples/snippets/storage_add_bucket_conditional_iam_binding.py +++ b/storage/samples/snippets/storage_add_bucket_conditional_iam_binding.py @@ -53,15 +53,15 @@ def add_bucket_conditional_iam_binding( bucket.set_iam_policy(policy) - print("Added the following member(s) with role {} to {}:".format(role, bucket_name)) + print(f"Added the following member(s) with role {role} to {bucket_name}:") for member in members: - print(" {}".format(member)) + print(f" {member}") print("with condition:") - print(" Title: {}".format(title)) - print(" Description: {}".format(description)) - print(" Expression: {}".format(expression)) + print(f" Title: {title}") + print(f" Description: {description}") + print(f" Expression: {expression}") # [END storage_add_bucket_conditional_iam_binding] diff --git a/storage/samples/snippets/storage_add_bucket_iam_member.py b/storage/samples/snippets/storage_add_bucket_iam_member.py index 727f1848325..0d610eae7ce 100644 --- a/storage/samples/snippets/storage_add_bucket_iam_member.py +++ b/storage/samples/snippets/storage_add_bucket_iam_member.py @@ -35,7 +35,7 @@ def add_bucket_iam_member(bucket_name, role, member): bucket.set_iam_policy(policy) - print("Added {} with role {} to {}.".format(member, role, bucket_name)) + print(f"Added {member} with role {role} to {bucket_name}.") # [END storage_add_bucket_iam_member] diff --git a/storage/samples/snippets/storage_add_bucket_label.py b/storage/samples/snippets/storage_add_bucket_label.py index 8ae8fe1f407..9c6fcff7af3 100644 --- a/storage/samples/snippets/storage_add_bucket_label.py +++ b/storage/samples/snippets/storage_add_bucket_label.py @@ -36,7 +36,7 @@ def add_bucket_label(bucket_name): bucket.labels = labels bucket.patch() - print("Updated labels on {}.".format(bucket.name)) + print(f"Updated labels on {bucket.name}.") pprint.pprint(bucket.labels) diff --git a/storage/samples/snippets/storage_add_bucket_owner.py b/storage/samples/snippets/storage_add_bucket_owner.py index acdb60dc5a2..bac1f3f6440 100644 --- a/storage/samples/snippets/storage_add_bucket_owner.py +++ b/storage/samples/snippets/storage_add_bucket_owner.py @@ -40,9 +40,7 @@ def add_bucket_owner(bucket_name, user_email): bucket.acl.save() print( - "Added user {} as an owner on bucket {}.".format( - user_email, bucket_name - ) + f"Added user {user_email} as an owner on bucket {bucket_name}." ) diff --git a/storage/samples/snippets/storage_bucket_delete_default_kms_key.py b/storage/samples/snippets/storage_bucket_delete_default_kms_key.py index 3df23767df0..0db29375699 100644 --- a/storage/samples/snippets/storage_bucket_delete_default_kms_key.py +++ b/storage/samples/snippets/storage_bucket_delete_default_kms_key.py @@ -30,7 +30,7 @@ def bucket_delete_default_kms_key(bucket_name): bucket.default_kms_key_name = None bucket.patch() - print("Default KMS key was removed from {}".format(bucket.name)) + print(f"Default KMS key was removed from {bucket.name}") return bucket diff --git a/storage/samples/snippets/storage_change_default_storage_class.py b/storage/samples/snippets/storage_change_default_storage_class.py index 8a72719bad3..5d2f924ade7 100644 --- a/storage/samples/snippets/storage_change_default_storage_class.py +++ b/storage/samples/snippets/storage_change_default_storage_class.py @@ -31,7 +31,7 @@ def change_default_storage_class(bucket_name): bucket.storage_class = constants.COLDLINE_STORAGE_CLASS bucket.patch() - print("Default storage class for bucket {} has been set to {}".format(bucket_name, bucket.storage_class)) + print(f"Default storage class for bucket {bucket_name} has been set to {bucket.storage_class}") return bucket diff --git a/storage/samples/snippets/storage_configure_retries.py b/storage/samples/snippets/storage_configure_retries.py index 9543111b3a7..ef1e422b663 100644 --- a/storage/samples/snippets/storage_configure_retries.py +++ b/storage/samples/snippets/storage_configure_retries.py @@ -53,7 +53,7 @@ def configure_retries(bucket_name, blob_name): ) blob.delete(retry=modified_retry) - print("Blob {} deleted with a customized retry strategy.".format(blob_name)) + print(f"Blob {blob_name} deleted with a customized retry strategy.") # [END storage_configure_retries] diff --git a/storage/samples/snippets/storage_cors_configuration.py b/storage/samples/snippets/storage_cors_configuration.py index 3d2595a9ddb..2c5dd242870 100644 --- a/storage/samples/snippets/storage_cors_configuration.py +++ b/storage/samples/snippets/storage_cors_configuration.py @@ -38,7 +38,7 @@ def cors_configuration(bucket_name): ] bucket.patch() - print("Set CORS policies for bucket {} is {}".format(bucket.name, bucket.cors)) + print(f"Set CORS policies for bucket {bucket.name} is {bucket.cors}") return bucket diff --git a/storage/samples/snippets/storage_create_bucket.py b/storage/samples/snippets/storage_create_bucket.py index aaee9e23485..c95f32f569b 100644 --- a/storage/samples/snippets/storage_create_bucket.py +++ b/storage/samples/snippets/storage_create_bucket.py @@ -28,7 +28,7 @@ def create_bucket(bucket_name): bucket = storage_client.create_bucket(bucket_name) - print("Bucket {} created".format(bucket.name)) + print(f"Bucket {bucket.name} created") # [END storage_create_bucket] diff --git a/storage/samples/snippets/storage_create_hmac_key.py b/storage/samples/snippets/storage_create_hmac_key.py index 27a418c39bd..d845738b780 100644 --- a/storage/samples/snippets/storage_create_hmac_key.py +++ b/storage/samples/snippets/storage_create_hmac_key.py @@ -33,17 +33,17 @@ def create_key(project_id, service_account_email): service_account_email=service_account_email, project_id=project_id ) - print("The base64 encoded secret is {}".format(secret)) + print(f"The base64 encoded secret is {secret}") print("Do not miss that secret, there is no API to recover it.") print("The HMAC key metadata is:") - print("Service Account Email: {}".format(hmac_key.service_account_email)) - print("Key ID: {}".format(hmac_key.id)) - print("Access ID: {}".format(hmac_key.access_id)) - print("Project ID: {}".format(hmac_key.project)) - print("State: {}".format(hmac_key.state)) - print("Created At: {}".format(hmac_key.time_created)) - print("Updated At: {}".format(hmac_key.updated)) - print("Etag: {}".format(hmac_key.etag)) + print(f"Service Account Email: {hmac_key.service_account_email}") + print(f"Key ID: {hmac_key.id}") + print(f"Access ID: {hmac_key.access_id}") + print(f"Project ID: {hmac_key.project}") + print(f"State: {hmac_key.state}") + print(f"Created At: {hmac_key.time_created}") + print(f"Updated At: {hmac_key.updated}") + print(f"Etag: {hmac_key.etag}") return hmac_key diff --git a/storage/samples/snippets/storage_deactivate_hmac_key.py b/storage/samples/snippets/storage_deactivate_hmac_key.py index 389efb998d5..007f7b5a5f2 100644 --- a/storage/samples/snippets/storage_deactivate_hmac_key.py +++ b/storage/samples/snippets/storage_deactivate_hmac_key.py @@ -37,14 +37,14 @@ def deactivate_key(access_id, project_id): print("The HMAC key is now inactive.") print("The HMAC key metadata is:") - print("Service Account Email: {}".format(hmac_key.service_account_email)) - print("Key ID: {}".format(hmac_key.id)) - print("Access ID: {}".format(hmac_key.access_id)) - print("Project ID: {}".format(hmac_key.project)) - print("State: {}".format(hmac_key.state)) - print("Created At: {}".format(hmac_key.time_created)) - print("Updated At: {}".format(hmac_key.updated)) - print("Etag: {}".format(hmac_key.etag)) + print(f"Service Account Email: {hmac_key.service_account_email}") + print(f"Key ID: {hmac_key.id}") + print(f"Access ID: {hmac_key.access_id}") + print(f"Project ID: {hmac_key.project}") + print(f"State: {hmac_key.state}") + print(f"Created At: {hmac_key.time_created}") + print(f"Updated At: {hmac_key.updated}") + print(f"Etag: {hmac_key.etag}") return hmac_key diff --git a/storage/samples/snippets/storage_delete_bucket.py b/storage/samples/snippets/storage_delete_bucket.py index b3e264c74e7..b12c066361d 100644 --- a/storage/samples/snippets/storage_delete_bucket.py +++ b/storage/samples/snippets/storage_delete_bucket.py @@ -29,7 +29,7 @@ def delete_bucket(bucket_name): bucket = storage_client.get_bucket(bucket_name) bucket.delete() - print("Bucket {} deleted".format(bucket.name)) + print(f"Bucket {bucket.name} deleted") # [END storage_delete_bucket] diff --git a/storage/samples/snippets/storage_delete_file.py b/storage/samples/snippets/storage_delete_file.py index 1105f3725a1..b2997c86b40 100644 --- a/storage/samples/snippets/storage_delete_file.py +++ b/storage/samples/snippets/storage_delete_file.py @@ -31,7 +31,7 @@ def delete_blob(bucket_name, blob_name): blob = bucket.blob(blob_name) blob.delete() - print("Blob {} deleted.".format(blob_name)) + print(f"Blob {blob_name} deleted.") # [END storage_delete_file] diff --git a/storage/samples/snippets/storage_delete_file_archived_generation.py b/storage/samples/snippets/storage_delete_file_archived_generation.py index 4e490900100..ff02bca23dc 100644 --- a/storage/samples/snippets/storage_delete_file_archived_generation.py +++ b/storage/samples/snippets/storage_delete_file_archived_generation.py @@ -31,9 +31,7 @@ def delete_file_archived_generation(bucket_name, blob_name, generation): bucket = storage_client.get_bucket(bucket_name) bucket.delete_blob(blob_name, generation=generation) print( - "Generation {} of blob {} was deleted from {}".format( - generation, blob_name, bucket_name - ) + f"Generation {generation} of blob {blob_name} was deleted from {bucket_name}" ) diff --git a/storage/samples/snippets/storage_disable_bucket_lifecycle_management.py b/storage/samples/snippets/storage_disable_bucket_lifecycle_management.py index 9ef6971fb10..a5fa56fcf35 100644 --- a/storage/samples/snippets/storage_disable_bucket_lifecycle_management.py +++ b/storage/samples/snippets/storage_disable_bucket_lifecycle_management.py @@ -31,7 +31,7 @@ def disable_bucket_lifecycle_management(bucket_name): bucket.patch() rules = bucket.lifecycle_rules - print("Lifecycle management is disable for bucket {} and the rules are {}".format(bucket_name, list(rules))) + print(f"Lifecycle management is disable for bucket {bucket_name} and the rules are {list(rules)}") return bucket diff --git a/storage/samples/snippets/storage_disable_default_event_based_hold.py b/storage/samples/snippets/storage_disable_default_event_based_hold.py index dff3ed3c129..48becdac1c0 100644 --- a/storage/samples/snippets/storage_disable_default_event_based_hold.py +++ b/storage/samples/snippets/storage_disable_default_event_based_hold.py @@ -30,7 +30,7 @@ def disable_default_event_based_hold(bucket_name): bucket.default_event_based_hold = False bucket.patch() - print("Default event based hold was disabled for {}".format(bucket_name)) + print(f"Default event based hold was disabled for {bucket_name}") # [END storage_disable_default_event_based_hold] diff --git a/storage/samples/snippets/storage_disable_requester_pays.py b/storage/samples/snippets/storage_disable_requester_pays.py index c49cc28eaab..78e195d8a4a 100644 --- a/storage/samples/snippets/storage_disable_requester_pays.py +++ b/storage/samples/snippets/storage_disable_requester_pays.py @@ -30,7 +30,7 @@ def disable_requester_pays(bucket_name): bucket.requester_pays = False bucket.patch() - print("Requester Pays has been disabled for {}".format(bucket_name)) + print(f"Requester Pays has been disabled for {bucket_name}") # [END storage_disable_requester_pays] diff --git a/storage/samples/snippets/storage_disable_uniform_bucket_level_access.py b/storage/samples/snippets/storage_disable_uniform_bucket_level_access.py index 4f4691611b9..20a045686c3 100644 --- a/storage/samples/snippets/storage_disable_uniform_bucket_level_access.py +++ b/storage/samples/snippets/storage_disable_uniform_bucket_level_access.py @@ -31,7 +31,7 @@ def disable_uniform_bucket_level_access(bucket_name): bucket.patch() print( - "Uniform bucket-level access was disabled for {}.".format(bucket.name) + f"Uniform bucket-level access was disabled for {bucket.name}." ) diff --git a/storage/samples/snippets/storage_disable_versioning.py b/storage/samples/snippets/storage_disable_versioning.py index 98832ba6856..9dfd0ff909a 100644 --- a/storage/samples/snippets/storage_disable_versioning.py +++ b/storage/samples/snippets/storage_disable_versioning.py @@ -30,7 +30,7 @@ def disable_versioning(bucket_name): bucket.versioning_enabled = False bucket.patch() - print("Versioning was disabled for bucket {}".format(bucket)) + print(f"Versioning was disabled for bucket {bucket}") return bucket diff --git a/storage/samples/snippets/storage_download_encrypted_file.py b/storage/samples/snippets/storage_download_encrypted_file.py index ac7071fbefb..8a81b0de597 100644 --- a/storage/samples/snippets/storage_download_encrypted_file.py +++ b/storage/samples/snippets/storage_download_encrypted_file.py @@ -52,9 +52,7 @@ def download_encrypted_blob( blob.download_to_filename(destination_file_name) print( - "Blob {} downloaded to {}.".format( - source_blob_name, destination_file_name - ) + f"Blob {source_blob_name} downloaded to {destination_file_name}." ) diff --git a/storage/samples/snippets/storage_download_to_stream.py b/storage/samples/snippets/storage_download_to_stream.py index 1cb8dcc7b31..3834e34c917 100644 --- a/storage/samples/snippets/storage_download_to_stream.py +++ b/storage/samples/snippets/storage_download_to_stream.py @@ -42,7 +42,7 @@ def download_blob_to_stream(bucket_name, source_blob_name, file_obj): blob = bucket.blob(source_blob_name) blob.download_to_file(file_obj) - print("Downloaded blob {} to file-like object.".format(source_blob_name)) + print(f"Downloaded blob {source_blob_name} to file-like object.") return file_obj # Before reading from file_obj, remember to rewind with file_obj.seek(0). diff --git a/storage/samples/snippets/storage_enable_bucket_lifecycle_management.py b/storage/samples/snippets/storage_enable_bucket_lifecycle_management.py index 61c7d7b20d9..0bbff079c8a 100644 --- a/storage/samples/snippets/storage_enable_bucket_lifecycle_management.py +++ b/storage/samples/snippets/storage_enable_bucket_lifecycle_management.py @@ -29,12 +29,12 @@ def enable_bucket_lifecycle_management(bucket_name): bucket = storage_client.get_bucket(bucket_name) rules = bucket.lifecycle_rules - print("Lifecycle management rules for bucket {} are {}".format(bucket_name, list(rules))) + print(f"Lifecycle management rules for bucket {bucket_name} are {list(rules)}") bucket.add_lifecycle_delete_rule(age=2) bucket.patch() rules = bucket.lifecycle_rules - print("Lifecycle management is enable for bucket {} and the rules are {}".format(bucket_name, list(rules))) + print(f"Lifecycle management is enable for bucket {bucket_name} and the rules are {list(rules)}") return bucket diff --git a/storage/samples/snippets/storage_enable_default_event_based_hold.py b/storage/samples/snippets/storage_enable_default_event_based_hold.py index a535390c913..5dfdf94a983 100644 --- a/storage/samples/snippets/storage_enable_default_event_based_hold.py +++ b/storage/samples/snippets/storage_enable_default_event_based_hold.py @@ -30,7 +30,7 @@ def enable_default_event_based_hold(bucket_name): bucket.default_event_based_hold = True bucket.patch() - print("Default event based hold was enabled for {}".format(bucket_name)) + print(f"Default event based hold was enabled for {bucket_name}") # [END storage_enable_default_event_based_hold] diff --git a/storage/samples/snippets/storage_enable_requester_pays.py b/storage/samples/snippets/storage_enable_requester_pays.py index 9787008ddcf..fbecb04f47c 100644 --- a/storage/samples/snippets/storage_enable_requester_pays.py +++ b/storage/samples/snippets/storage_enable_requester_pays.py @@ -30,7 +30,7 @@ def enable_requester_pays(bucket_name): bucket.requester_pays = True bucket.patch() - print("Requester Pays has been enabled for {}".format(bucket_name)) + print(f"Requester Pays has been enabled for {bucket_name}") # [END storage_enable_requester_pays] diff --git a/storage/samples/snippets/storage_enable_uniform_bucket_level_access.py b/storage/samples/snippets/storage_enable_uniform_bucket_level_access.py index c689bb735c6..9ab71ae3730 100644 --- a/storage/samples/snippets/storage_enable_uniform_bucket_level_access.py +++ b/storage/samples/snippets/storage_enable_uniform_bucket_level_access.py @@ -31,7 +31,7 @@ def enable_uniform_bucket_level_access(bucket_name): bucket.patch() print( - "Uniform bucket-level access was enabled for {}.".format(bucket.name) + f"Uniform bucket-level access was enabled for {bucket.name}." ) diff --git a/storage/samples/snippets/storage_enable_versioning.py b/storage/samples/snippets/storage_enable_versioning.py index 89693e42656..9cdc980016e 100644 --- a/storage/samples/snippets/storage_enable_versioning.py +++ b/storage/samples/snippets/storage_enable_versioning.py @@ -30,7 +30,7 @@ def enable_versioning(bucket_name): bucket.versioning_enabled = True bucket.patch() - print("Versioning was enabled for bucket {}".format(bucket.name)) + print(f"Versioning was enabled for bucket {bucket.name}") return bucket diff --git a/storage/samples/snippets/storage_generate_encryption_key.py b/storage/samples/snippets/storage_generate_encryption_key.py index a973418a611..dbeb46b914b 100644 --- a/storage/samples/snippets/storage_generate_encryption_key.py +++ b/storage/samples/snippets/storage_generate_encryption_key.py @@ -30,7 +30,7 @@ def generate_encryption_key(): key = os.urandom(32) encoded_key = base64.b64encode(key).decode("utf-8") - print("Base 64 encoded encryption key: {}".format(encoded_key)) + print(f"Base 64 encoded encryption key: {encoded_key}") # [END storage_generate_encryption_key] diff --git a/storage/samples/snippets/storage_generate_signed_post_policy_v4.py b/storage/samples/snippets/storage_generate_signed_post_policy_v4.py index 8217714e2ed..0c06ddc2fd4 100644 --- a/storage/samples/snippets/storage_generate_signed_post_policy_v4.py +++ b/storage/samples/snippets/storage_generate_signed_post_policy_v4.py @@ -46,7 +46,7 @@ def generate_signed_post_policy_v4(bucket_name, blob_name): # Include all fields returned in the HTML form as they're required for key, value in policy["fields"].items(): - form += " \n".format(key, value) + form += f" \n" form += "
\n" form += "
\n" diff --git a/storage/samples/snippets/storage_generate_signed_url_v2.py b/storage/samples/snippets/storage_generate_signed_url_v2.py index abea3dd540b..f1317ea2fbf 100644 --- a/storage/samples/snippets/storage_generate_signed_url_v2.py +++ b/storage/samples/snippets/storage_generate_signed_url_v2.py @@ -44,7 +44,7 @@ def generate_signed_url(bucket_name, blob_name): method="GET", ) - print("The signed url for {} is {}".format(blob.name, url)) + print(f"The signed url for {blob.name} is {url}") return url diff --git a/storage/samples/snippets/storage_generate_signed_url_v4.py b/storage/samples/snippets/storage_generate_signed_url_v4.py index 2a45b23e9be..80625a7b34b 100644 --- a/storage/samples/snippets/storage_generate_signed_url_v4.py +++ b/storage/samples/snippets/storage_generate_signed_url_v4.py @@ -49,7 +49,7 @@ def generate_download_signed_url_v4(bucket_name, blob_name): print("Generated GET signed URL:") print(url) print("You can use this URL with any user agent, for example:") - print("curl '{}'".format(url)) + print(f"curl '{url}'") return url diff --git a/storage/samples/snippets/storage_get_default_event_based_hold.py b/storage/samples/snippets/storage_get_default_event_based_hold.py index 4cf13914d8e..08a05f8ef55 100644 --- a/storage/samples/snippets/storage_get_default_event_based_hold.py +++ b/storage/samples/snippets/storage_get_default_event_based_hold.py @@ -29,12 +29,10 @@ def get_default_event_based_hold(bucket_name): bucket = storage_client.get_bucket(bucket_name) if bucket.default_event_based_hold: - print("Default event-based hold is enabled for {}".format(bucket_name)) + print(f"Default event-based hold is enabled for {bucket_name}") else: print( - "Default event-based hold is not enabled for {}".format( - bucket_name - ) + f"Default event-based hold is not enabled for {bucket_name}" ) diff --git a/storage/samples/snippets/storage_get_hmac_key.py b/storage/samples/snippets/storage_get_hmac_key.py index 4dc52240d27..82b28ff99e4 100644 --- a/storage/samples/snippets/storage_get_hmac_key.py +++ b/storage/samples/snippets/storage_get_hmac_key.py @@ -34,14 +34,14 @@ def get_key(access_id, project_id): ) print("The HMAC key metadata is:") - print("Service Account Email: {}".format(hmac_key.service_account_email)) - print("Key ID: {}".format(hmac_key.id)) - print("Access ID: {}".format(hmac_key.access_id)) - print("Project ID: {}".format(hmac_key.project)) - print("State: {}".format(hmac_key.state)) - print("Created At: {}".format(hmac_key.time_created)) - print("Updated At: {}".format(hmac_key.updated)) - print("Etag: {}".format(hmac_key.etag)) + print(f"Service Account Email: {hmac_key.service_account_email}") + print(f"Key ID: {hmac_key.id}") + print(f"Access ID: {hmac_key.access_id}") + print(f"Project ID: {hmac_key.project}") + print(f"State: {hmac_key.state}") + print(f"Created At: {hmac_key.time_created}") + print(f"Updated At: {hmac_key.updated}") + print(f"Etag: {hmac_key.etag}") return hmac_key diff --git a/storage/samples/snippets/storage_get_metadata.py b/storage/samples/snippets/storage_get_metadata.py index 3ce7ecea862..eece8028a85 100644 --- a/storage/samples/snippets/storage_get_metadata.py +++ b/storage/samples/snippets/storage_get_metadata.py @@ -33,27 +33,27 @@ def blob_metadata(bucket_name, blob_name): # make an HTTP request. blob = bucket.get_blob(blob_name) - print("Blob: {}".format(blob.name)) - print("Bucket: {}".format(blob.bucket.name)) - print("Storage class: {}".format(blob.storage_class)) - print("ID: {}".format(blob.id)) - print("Size: {} bytes".format(blob.size)) - print("Updated: {}".format(blob.updated)) - print("Generation: {}".format(blob.generation)) - print("Metageneration: {}".format(blob.metageneration)) - print("Etag: {}".format(blob.etag)) - print("Owner: {}".format(blob.owner)) - print("Component count: {}".format(blob.component_count)) - print("Crc32c: {}".format(blob.crc32c)) - print("md5_hash: {}".format(blob.md5_hash)) - print("Cache-control: {}".format(blob.cache_control)) - print("Content-type: {}".format(blob.content_type)) - print("Content-disposition: {}".format(blob.content_disposition)) - print("Content-encoding: {}".format(blob.content_encoding)) - print("Content-language: {}".format(blob.content_language)) - print("Metadata: {}".format(blob.metadata)) - print("Medialink: {}".format(blob.media_link)) - print("Custom Time: {}".format(blob.custom_time)) + print(f"Blob: {blob.name}") + print(f"Bucket: {blob.bucket.name}") + print(f"Storage class: {blob.storage_class}") + print(f"ID: {blob.id}") + print(f"Size: {blob.size} bytes") + print(f"Updated: {blob.updated}") + print(f"Generation: {blob.generation}") + print(f"Metageneration: {blob.metageneration}") + print(f"Etag: {blob.etag}") + print(f"Owner: {blob.owner}") + print(f"Component count: {blob.component_count}") + print(f"Crc32c: {blob.crc32c}") + print(f"md5_hash: {blob.md5_hash}") + print(f"Cache-control: {blob.cache_control}") + print(f"Content-type: {blob.content_type}") + print(f"Content-disposition: {blob.content_disposition}") + print(f"Content-encoding: {blob.content_encoding}") + print(f"Content-language: {blob.content_language}") + print(f"Metadata: {blob.metadata}") + print(f"Medialink: {blob.media_link}") + print(f"Custom Time: {blob.custom_time}") print("Temporary hold: ", "enabled" if blob.temporary_hold else "disabled") print( "Event based hold: ", @@ -61,9 +61,7 @@ def blob_metadata(bucket_name, blob_name): ) if blob.retention_expiration_time: print( - "retentionExpirationTime: {}".format( - blob.retention_expiration_time - ) + f"retentionExpirationTime: {blob.retention_expiration_time}" ) diff --git a/storage/samples/snippets/storage_get_requester_pays_status.py b/storage/samples/snippets/storage_get_requester_pays_status.py index 2014d654c0b..a2eeb34d70f 100644 --- a/storage/samples/snippets/storage_get_requester_pays_status.py +++ b/storage/samples/snippets/storage_get_requester_pays_status.py @@ -29,9 +29,9 @@ def get_requester_pays_status(bucket_name): requester_pays_status = bucket.requester_pays if requester_pays_status: - print("Requester Pays is enabled for {}".format(bucket_name)) + print(f"Requester Pays is enabled for {bucket_name}") else: - print("Requester Pays is disabled for {}".format(bucket_name)) + print(f"Requester Pays is disabled for {bucket_name}") # [END storage_get_requester_pays_status] diff --git a/storage/samples/snippets/storage_get_retention_policy.py b/storage/samples/snippets/storage_get_retention_policy.py index f2ca26d2630..215f80d5a59 100644 --- a/storage/samples/snippets/storage_get_retention_policy.py +++ b/storage/samples/snippets/storage_get_retention_policy.py @@ -28,14 +28,14 @@ def get_retention_policy(bucket_name): bucket = storage_client.bucket(bucket_name) bucket.reload() - print("Retention Policy for {}".format(bucket_name)) - print("Retention Period: {}".format(bucket.retention_period)) + print(f"Retention Policy for {bucket_name}") + print(f"Retention Period: {bucket.retention_period}") if bucket.retention_policy_locked: print("Retention Policy is locked") if bucket.retention_policy_effective_time: print( - "Effective Time: {}".format(bucket.retention_policy_effective_time) + f"Effective Time: {bucket.retention_policy_effective_time}" ) diff --git a/storage/samples/snippets/storage_get_service_account.py b/storage/samples/snippets/storage_get_service_account.py index 58ababb91cb..5ac0e563835 100644 --- a/storage/samples/snippets/storage_get_service_account.py +++ b/storage/samples/snippets/storage_get_service_account.py @@ -25,9 +25,7 @@ def get_service_account(): email = storage_client.get_service_account_email() print( - "The GCS service account for project {} is: {} ".format( - storage_client.project, email - ) + f"The GCS service account for project {storage_client.project} is: {email} " ) diff --git a/storage/samples/snippets/storage_get_uniform_bucket_level_access.py b/storage/samples/snippets/storage_get_uniform_bucket_level_access.py index eddb8bc1ac8..206b9f1ff11 100644 --- a/storage/samples/snippets/storage_get_uniform_bucket_level_access.py +++ b/storage/samples/snippets/storage_get_uniform_bucket_level_access.py @@ -30,9 +30,7 @@ def get_uniform_bucket_level_access(bucket_name): if iam_configuration.uniform_bucket_level_access_enabled: print( - "Uniform bucket-level access is enabled for {}.".format( - bucket.name - ) + f"Uniform bucket-level access is enabled for {bucket.name}." ) print( "Bucket will be locked on {}.".format( @@ -41,9 +39,7 @@ def get_uniform_bucket_level_access(bucket_name): ) else: print( - "Uniform bucket-level access is disabled for {}.".format( - bucket.name - ) + f"Uniform bucket-level access is disabled for {bucket.name}." ) diff --git a/storage/samples/snippets/storage_list_file_archived_generations.py b/storage/samples/snippets/storage_list_file_archived_generations.py index dc2f5eaf5c2..419cc3da408 100644 --- a/storage/samples/snippets/storage_list_file_archived_generations.py +++ b/storage/samples/snippets/storage_list_file_archived_generations.py @@ -29,7 +29,7 @@ def list_file_archived_generations(bucket_name): blobs = storage_client.list_blobs(bucket_name, versions=True) for blob in blobs: - print("{},{}".format(blob.name, blob.generation)) + print(f"{blob.name},{blob.generation}") # [END storage_list_file_archived_generations] diff --git a/storage/samples/snippets/storage_list_hmac_keys.py b/storage/samples/snippets/storage_list_hmac_keys.py index 8e5c53b589d..a09616fa519 100644 --- a/storage/samples/snippets/storage_list_hmac_keys.py +++ b/storage/samples/snippets/storage_list_hmac_keys.py @@ -31,9 +31,9 @@ def list_keys(project_id): print("HMAC Keys:") for hmac_key in hmac_keys: print( - "Service Account Email: {}".format(hmac_key.service_account_email) + f"Service Account Email: {hmac_key.service_account_email}" ) - print("Access ID: {}".format(hmac_key.access_id)) + print(f"Access ID: {hmac_key.access_id}") return hmac_keys diff --git a/storage/samples/snippets/storage_lock_retention_policy.py b/storage/samples/snippets/storage_lock_retention_policy.py index d59572f5dad..adff364d749 100644 --- a/storage/samples/snippets/storage_lock_retention_policy.py +++ b/storage/samples/snippets/storage_lock_retention_policy.py @@ -33,11 +33,9 @@ def lock_retention_policy(bucket_name): # and retention period can only be increased. bucket.lock_retention_policy() - print("Retention policy for {} is now locked".format(bucket_name)) + print(f"Retention policy for {bucket_name} is now locked") print( - "Retention policy effective as of {}".format( - bucket.retention_policy_effective_time - ) + f"Retention policy effective as of {bucket.retention_policy_effective_time}" ) diff --git a/storage/samples/snippets/storage_make_public.py b/storage/samples/snippets/storage_make_public.py index 79ae40d123b..489508cf674 100644 --- a/storage/samples/snippets/storage_make_public.py +++ b/storage/samples/snippets/storage_make_public.py @@ -32,9 +32,7 @@ def make_blob_public(bucket_name, blob_name): blob.make_public() print( - "Blob {} is publicly accessible at {}".format( - blob.name, blob.public_url - ) + f"Blob {blob.name} is publicly accessible at {blob.public_url}" ) diff --git a/storage/samples/snippets/storage_object_get_kms_key.py b/storage/samples/snippets/storage_object_get_kms_key.py index dddfc9151b8..7604e6eba6e 100644 --- a/storage/samples/snippets/storage_object_get_kms_key.py +++ b/storage/samples/snippets/storage_object_get_kms_key.py @@ -32,7 +32,7 @@ def object_get_kms_key(bucket_name, blob_name): kms_key = blob.kms_key_name - print("The KMS key of a blob is {}".format(blob.kms_key_name)) + print(f"The KMS key of a blob is {blob.kms_key_name}") return kms_key diff --git a/storage/samples/snippets/storage_print_bucket_acl.py b/storage/samples/snippets/storage_print_bucket_acl.py index 0804f7a9a89..55417f1bc77 100644 --- a/storage/samples/snippets/storage_print_bucket_acl.py +++ b/storage/samples/snippets/storage_print_bucket_acl.py @@ -27,7 +27,7 @@ def print_bucket_acl(bucket_name): bucket = storage_client.bucket(bucket_name) for entry in bucket.acl: - print("{}: {}".format(entry["role"], entry["entity"])) + print(f"{entry['role']}: {entry['entity']}") # [END storage_print_bucket_acl] diff --git a/storage/samples/snippets/storage_print_file_acl.py b/storage/samples/snippets/storage_print_file_acl.py index f34a5283b1d..8dfc4e98464 100644 --- a/storage/samples/snippets/storage_print_file_acl.py +++ b/storage/samples/snippets/storage_print_file_acl.py @@ -28,7 +28,7 @@ def print_blob_acl(bucket_name, blob_name): blob = bucket.blob(blob_name) for entry in blob.acl: - print("{}: {}".format(entry["role"], entry["entity"])) + print(f"{entry['role']}: {entry['entity']}") # [END storage_print_file_acl] diff --git a/storage/samples/snippets/storage_release_event_based_hold.py b/storage/samples/snippets/storage_release_event_based_hold.py index 8c3c11b6ff0..1db637cd9e7 100644 --- a/storage/samples/snippets/storage_release_event_based_hold.py +++ b/storage/samples/snippets/storage_release_event_based_hold.py @@ -33,7 +33,7 @@ def release_event_based_hold(bucket_name, blob_name): blob.event_based_hold = False blob.patch() - print("Event based hold was released for {}".format(blob_name)) + print(f"Event based hold was released for {blob_name}") # [END storage_release_event_based_hold] diff --git a/storage/samples/snippets/storage_remove_bucket_default_owner.py b/storage/samples/snippets/storage_remove_bucket_default_owner.py index beaf6be84d4..e6f3c495e5f 100644 --- a/storage/samples/snippets/storage_remove_bucket_default_owner.py +++ b/storage/samples/snippets/storage_remove_bucket_default_owner.py @@ -40,9 +40,7 @@ def remove_bucket_default_owner(bucket_name, user_email): bucket.default_object_acl.save() print( - "Removed user {} from the default acl of bucket {}.".format( - user_email, bucket_name - ) + f"Removed user {user_email} from the default acl of bucket {bucket_name}." ) diff --git a/storage/samples/snippets/storage_remove_bucket_iam_member.py b/storage/samples/snippets/storage_remove_bucket_iam_member.py index ef75a1a15fc..2efc29e303c 100644 --- a/storage/samples/snippets/storage_remove_bucket_iam_member.py +++ b/storage/samples/snippets/storage_remove_bucket_iam_member.py @@ -38,7 +38,7 @@ def remove_bucket_iam_member(bucket_name, role, member): bucket.set_iam_policy(policy) - print("Removed {} with role {} from {}.".format(member, role, bucket_name)) + print(f"Removed {member} with role {role} from {bucket_name}.") # [END storage_remove_bucket_iam_member] diff --git a/storage/samples/snippets/storage_remove_bucket_label.py b/storage/samples/snippets/storage_remove_bucket_label.py index 58bbfef2d51..fc4a5b4e7b2 100644 --- a/storage/samples/snippets/storage_remove_bucket_label.py +++ b/storage/samples/snippets/storage_remove_bucket_label.py @@ -39,7 +39,7 @@ def remove_bucket_label(bucket_name): bucket.labels = labels bucket.patch() - print("Removed labels on {}.".format(bucket.name)) + print(f"Removed labels on {bucket.name}.") pprint.pprint(bucket.labels) diff --git a/storage/samples/snippets/storage_remove_bucket_owner.py b/storage/samples/snippets/storage_remove_bucket_owner.py index f54e7a7cc56..561ba9175a6 100644 --- a/storage/samples/snippets/storage_remove_bucket_owner.py +++ b/storage/samples/snippets/storage_remove_bucket_owner.py @@ -38,7 +38,7 @@ def remove_bucket_owner(bucket_name, user_email): bucket.acl.user(user_email).revoke_owner() bucket.acl.save() - print("Removed user {} from bucket {}.".format(user_email, bucket_name)) + print(f"Removed user {user_email} from bucket {bucket_name}.") # [END storage_remove_bucket_owner] diff --git a/storage/samples/snippets/storage_remove_cors_configuration.py b/storage/samples/snippets/storage_remove_cors_configuration.py index 48ee7433856..ad97371f494 100644 --- a/storage/samples/snippets/storage_remove_cors_configuration.py +++ b/storage/samples/snippets/storage_remove_cors_configuration.py @@ -29,7 +29,7 @@ def remove_cors_configuration(bucket_name): bucket.cors = [] bucket.patch() - print("Remove CORS policies for bucket {}.".format(bucket.name)) + print(f"Remove CORS policies for bucket {bucket.name}.") return bucket diff --git a/storage/samples/snippets/storage_remove_file_owner.py b/storage/samples/snippets/storage_remove_file_owner.py index 9db83cce0cd..315a747adbc 100644 --- a/storage/samples/snippets/storage_remove_file_owner.py +++ b/storage/samples/snippets/storage_remove_file_owner.py @@ -39,9 +39,7 @@ def remove_blob_owner(bucket_name, blob_name, user_email): blob.acl.save() print( - "Removed user {} from blob {} in bucket {}.".format( - user_email, blob_name, bucket_name - ) + f"Removed user {user_email} from blob {blob_name} in bucket {bucket_name}." ) diff --git a/storage/samples/snippets/storage_remove_retention_policy.py b/storage/samples/snippets/storage_remove_retention_policy.py index cb8ee548cf7..9ede8053afd 100644 --- a/storage/samples/snippets/storage_remove_retention_policy.py +++ b/storage/samples/snippets/storage_remove_retention_policy.py @@ -37,7 +37,7 @@ def remove_retention_policy(bucket_name): bucket.retention_period = None bucket.patch() - print("Removed bucket {} retention policy".format(bucket.name)) + print(f"Removed bucket {bucket.name} retention policy") # [END storage_remove_retention_policy] diff --git a/storage/samples/snippets/storage_rename_file.py b/storage/samples/snippets/storage_rename_file.py index b47e186218f..1125007c655 100644 --- a/storage/samples/snippets/storage_rename_file.py +++ b/storage/samples/snippets/storage_rename_file.py @@ -35,7 +35,7 @@ def rename_blob(bucket_name, blob_name, new_name): new_blob = bucket.rename_blob(blob, new_name) - print("Blob {} has been renamed to {}".format(blob.name, new_blob.name)) + print(f"Blob {blob.name} has been renamed to {new_blob.name}") # [END storage_rename_file] diff --git a/storage/samples/snippets/storage_rotate_encryption_key.py b/storage/samples/snippets/storage_rotate_encryption_key.py index 663ee47964b..828b7d5ef31 100644 --- a/storage/samples/snippets/storage_rotate_encryption_key.py +++ b/storage/samples/snippets/storage_rotate_encryption_key.py @@ -52,7 +52,7 @@ def rotate_encryption_key( if token is None: break - print("Key rotation complete for Blob {}".format(blob_name)) + print(f"Key rotation complete for Blob {blob_name}") # [END storage_rotate_encryption_key] diff --git a/storage/samples/snippets/storage_set_bucket_public_iam.py b/storage/samples/snippets/storage_set_bucket_public_iam.py index 4b7df89dfa1..0fb33f59c65 100644 --- a/storage/samples/snippets/storage_set_bucket_public_iam.py +++ b/storage/samples/snippets/storage_set_bucket_public_iam.py @@ -39,7 +39,7 @@ def set_bucket_public_iam( bucket.set_iam_policy(policy) - print("Bucket {} is now publicly readable".format(bucket.name)) + print(f"Bucket {bucket.name} is now publicly readable") # [END storage_set_bucket_public_iam] diff --git a/storage/samples/snippets/storage_set_event_based_hold.py b/storage/samples/snippets/storage_set_event_based_hold.py index 52a89b88e41..e04ed7552a9 100644 --- a/storage/samples/snippets/storage_set_event_based_hold.py +++ b/storage/samples/snippets/storage_set_event_based_hold.py @@ -32,7 +32,7 @@ def set_event_based_hold(bucket_name, blob_name): blob.event_based_hold = True blob.patch() - print("Event based hold was set for {}".format(blob_name)) + print(f"Event based hold was set for {blob_name}") # [END storage_set_event_based_hold] diff --git a/storage/samples/snippets/storage_set_metadata.py b/storage/samples/snippets/storage_set_metadata.py index 07529ac68d7..90b6838c026 100644 --- a/storage/samples/snippets/storage_set_metadata.py +++ b/storage/samples/snippets/storage_set_metadata.py @@ -32,7 +32,7 @@ def set_blob_metadata(bucket_name, blob_name): blob.metadata = metadata blob.patch() - print("The metadata for the blob {} is {}".format(blob.name, blob.metadata)) + print(f"The metadata for the blob {blob.name} is {blob.metadata}") # [END storage_set_metadata] diff --git a/storage/samples/snippets/storage_upload_encrypted_file.py b/storage/samples/snippets/storage_upload_encrypted_file.py index e7d02c67b7d..5f49872384e 100644 --- a/storage/samples/snippets/storage_upload_encrypted_file.py +++ b/storage/samples/snippets/storage_upload_encrypted_file.py @@ -51,9 +51,7 @@ def upload_encrypted_blob( blob.upload_from_filename(source_file_name) print( - "File {} uploaded to {}.".format( - source_file_name, destination_blob_name - ) + f"File {source_file_name} uploaded to {destination_blob_name}." ) diff --git a/storage/samples/snippets/storage_upload_file.py b/storage/samples/snippets/storage_upload_file.py index fb02c3632a9..8e7d98630e3 100644 --- a/storage/samples/snippets/storage_upload_file.py +++ b/storage/samples/snippets/storage_upload_file.py @@ -36,9 +36,7 @@ def upload_blob(bucket_name, source_file_name, destination_blob_name): blob.upload_from_filename(source_file_name) print( - "File {} uploaded to {}.".format( - source_file_name, destination_blob_name - ) + f"File {source_file_name} uploaded to {destination_blob_name}." ) diff --git a/storage/samples/snippets/storage_upload_from_memory.py b/storage/samples/snippets/storage_upload_from_memory.py index ee8a9828cd4..eff3d222afd 100644 --- a/storage/samples/snippets/storage_upload_from_memory.py +++ b/storage/samples/snippets/storage_upload_from_memory.py @@ -39,9 +39,7 @@ def upload_blob_from_memory(bucket_name, contents, destination_blob_name): blob.upload_from_string(contents) print( - "{} with contents {} uploaded to {}.".format( - destination_blob_name, contents, bucket_name - ) + f"{destination_blob_name} with contents {contents} uploaded to {bucket_name}." ) # [END storage_file_upload_from_memory] diff --git a/storage/samples/snippets/storage_upload_from_stream.py b/storage/samples/snippets/storage_upload_from_stream.py index d43365e08d1..e2d31a5e393 100644 --- a/storage/samples/snippets/storage_upload_from_stream.py +++ b/storage/samples/snippets/storage_upload_from_stream.py @@ -44,9 +44,7 @@ def upload_blob_from_stream(bucket_name, file_obj, destination_blob_name): blob.upload_from_file(file_obj) print( - "Stream data uploaded to {} in bucket {}.".format( - destination_blob_name, bucket_name - ) + f"Stream data uploaded to {destination_blob_name} in bucket {bucket_name}." ) # [END storage_stream_file_upload] diff --git a/storage/samples/snippets/storage_view_bucket_iam_members.py b/storage/samples/snippets/storage_view_bucket_iam_members.py index 5272f0ddbdb..184a1361f0f 100644 --- a/storage/samples/snippets/storage_view_bucket_iam_members.py +++ b/storage/samples/snippets/storage_view_bucket_iam_members.py @@ -30,7 +30,7 @@ def view_bucket_iam_members(bucket_name): policy = bucket.get_iam_policy(requested_policy_version=3) for binding in policy.bindings: - print("Role: {}, Members: {}".format(binding["role"], binding["members"])) + print(f"Role: {binding['role']}, Members: {binding['members']}") # [END storage_view_bucket_iam_members] diff --git a/storage/samples/snippets/uniform_bucket_level_access_test.py b/storage/samples/snippets/uniform_bucket_level_access_test.py index b43fa016fe3..8b7964038ac 100644 --- a/storage/samples/snippets/uniform_bucket_level_access_test.py +++ b/storage/samples/snippets/uniform_bucket_level_access_test.py @@ -23,7 +23,7 @@ def test_get_uniform_bucket_level_access(bucket, capsys): ) out, _ = capsys.readouterr() assert ( - "Uniform bucket-level access is disabled for {}.".format(bucket.name) + f"Uniform bucket-level access is disabled for {bucket.name}." in out ) @@ -35,7 +35,7 @@ def test_enable_uniform_bucket_level_access(bucket, capsys): ) out, _ = capsys.readouterr() assert ( - "Uniform bucket-level access was enabled for {}.".format(bucket.name) + f"Uniform bucket-level access was enabled for {bucket.name}." in out ) @@ -47,6 +47,6 @@ def test_disable_uniform_bucket_level_access(bucket, capsys): ) out, _ = capsys.readouterr() assert ( - "Uniform bucket-level access was disabled for {}.".format(bucket.name) + f"Uniform bucket-level access was disabled for {bucket.name}." in out ) From 3ff477f01574bde8d3539c5505a7a319f6b2de92 Mon Sep 17 00:00:00 2001 From: Rebecca Peterson <44721098+rebecca-pete@users.noreply.github.com> Date: Fri, 20 May 2022 10:54:14 -0700 Subject: [PATCH 053/172] docs(samples): Update the Recovery Point Objective (RPO) sample output (#725) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @ddelgrosso1 Related to b/217259317. Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly: - [ ] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/python-storage/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea - [ ] Ensure the tests and linter pass - [ ] Code coverage does not decrease (if any source code was changed) - [ ] Appropriate docs were updated (if necessary) Fixes # 🦕 --- storage/samples/snippets/rpo_test.py | 6 +++--- .../snippets/storage_create_bucket_turbo_replication.py | 2 +- storage/samples/snippets/storage_set_rpo_async_turbo.py | 2 +- storage/samples/snippets/storage_set_rpo_default.py | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/storage/samples/snippets/rpo_test.py b/storage/samples/snippets/rpo_test.py index f1f16e7fbd8..befc0334ace 100644 --- a/storage/samples/snippets/rpo_test.py +++ b/storage/samples/snippets/rpo_test.py @@ -45,17 +45,17 @@ def test_get_rpo(dual_region_bucket, capsys): def test_set_rpo_async_turbo(dual_region_bucket, capsys): storage_set_rpo_async_turbo.set_rpo_async_turbo(dual_region_bucket.name) out, _ = capsys.readouterr() - assert f"RPO is ASYNC_TURBO for {dual_region_bucket.name}." in out + assert f"RPO is set to ASYNC_TURBO for {dual_region_bucket.name}." in out def test_set_rpo_default(dual_region_bucket, capsys): storage_set_rpo_default.set_rpo_default(dual_region_bucket.name) out, _ = capsys.readouterr() - assert f"RPO is DEFAULT for {dual_region_bucket.name}." in out + assert f"RPO is set to DEFAULT for {dual_region_bucket.name}." in out def test_create_bucket_turbo_replication(capsys): bucket_name = f"test-rpo-{uuid.uuid4()}" storage_create_bucket_turbo_replication.create_bucket_turbo_replication(bucket_name) out, _ = capsys.readouterr() - assert f"{bucket_name} created with RPO ASYNC_TURBO in NAM4." in out + assert f"{bucket_name} created with the recovery point objective (RPO) set to ASYNC_TURBO in NAM4." in out diff --git a/storage/samples/snippets/storage_create_bucket_turbo_replication.py b/storage/samples/snippets/storage_create_bucket_turbo_replication.py index 68f0ba48240..3d26616ec61 100644 --- a/storage/samples/snippets/storage_create_bucket_turbo_replication.py +++ b/storage/samples/snippets/storage_create_bucket_turbo_replication.py @@ -39,7 +39,7 @@ def create_bucket_turbo_replication(bucket_name): bucket.rpo = RPO_ASYNC_TURBO bucket.create() - print(f"{bucket.name} created with RPO {bucket.rpo} in {bucket.location}.") + print(f"{bucket.name} created with the recovery point objective (RPO) set to {bucket.rpo} in {bucket.location}.") # [END storage_create_bucket_turbo_replication] diff --git a/storage/samples/snippets/storage_set_rpo_async_turbo.py b/storage/samples/snippets/storage_set_rpo_async_turbo.py index 10b4c67a334..a351cb8f82e 100644 --- a/storage/samples/snippets/storage_set_rpo_async_turbo.py +++ b/storage/samples/snippets/storage_set_rpo_async_turbo.py @@ -39,7 +39,7 @@ def set_rpo_async_turbo(bucket_name): bucket.rpo = RPO_ASYNC_TURBO bucket.patch() - print(f"RPO is ASYNC_TURBO for {bucket.name}.") + print(f"RPO is set to ASYNC_TURBO for {bucket.name}.") # [END storage_set_rpo_async_turbo] diff --git a/storage/samples/snippets/storage_set_rpo_default.py b/storage/samples/snippets/storage_set_rpo_default.py index 8d41b1fe0fb..883fee0c972 100644 --- a/storage/samples/snippets/storage_set_rpo_default.py +++ b/storage/samples/snippets/storage_set_rpo_default.py @@ -16,7 +16,7 @@ import sys -"""Sample that sets RPO (Recovery Point Objective) to default +"""Sample that sets the replication behavior or recovery point objective (RPO) to default. This sample is used on this page: https://cloud.google.com/storage/docs/managing-turbo-replication For more information, see README.md. @@ -39,7 +39,7 @@ def set_rpo_default(bucket_name): bucket.rpo = RPO_DEFAULT bucket.patch() - print(f"RPO is DEFAULT for {bucket.name}.") + print(f"RPO is set to DEFAULT for {bucket.name}.") # [END storage_set_rpo_default] From 66ed2845b95f692e31edaec920f51c69f0330c4d Mon Sep 17 00:00:00 2001 From: Dan Lee <71398022+dandhlee@users.noreply.github.com> Date: Wed, 1 Jun 2022 16:08:50 -0400 Subject: [PATCH 054/172] docs: fix changelog header to consistent size (#802) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs: fix changelog header to consistent size * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot Co-authored-by: cojenco --- storage/samples/snippets/noxfile.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 9f1cc8fb1ac..38bb0a572b8 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -66,7 +66,7 @@ sys.path.append(".") from noxfile_config import TEST_CONFIG_OVERRIDE except ImportError as e: - print(f"No user noxfile_config found: detail: {e}") + print("No user noxfile_config found: detail: {}".format(e)) TEST_CONFIG_OVERRIDE = {} # Update the TEST_CONFIG with the user supplied values. @@ -266,7 +266,7 @@ def py(session: nox.sessions.Session) -> None: _session_tests(session) else: session.skip( - f"SKIPPED: {session.python} tests are disabled for this sample." + "SKIPPED: {} tests are disabled for this sample.".format(session.python) ) From b063e8bffc49ab67d6783641d0c0a6a458dad5b6 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 7 Jun 2022 13:09:52 +0200 Subject: [PATCH 055/172] chore(deps): update dependency google-cloud-pubsub to v2.13.0 (#809) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 843c608cd87..44c1b701dc2 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.12.1 +google-cloud-pubsub==2.13.0 google-cloud-storage==2.3.0 pandas===1.3.5; python_version == '3.7' pandas==1.4.2; python_version >= '3.8' From 82ca951d5f59f5f98577d8c30adb697327d3cc39 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 7 Jun 2022 18:52:59 +0200 Subject: [PATCH 056/172] chore(deps): update dependency backoff to v2.1.0 (#810) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 88beb7ba24a..ed8c5dde8de 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ pytest==7.1.2 mock==4.0.3 -backoff==2.0.1 \ No newline at end of file +backoff==2.1.0 \ No newline at end of file From 668b161f103cbd2c09ffd091ac32b5189d1edb99 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 9 Jun 2022 02:21:55 +0200 Subject: [PATCH 057/172] chore(deps): update dependency google-cloud-storage to v2.4.0 (#813) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 44c1b701dc2..a2094a1fd6a 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.13.0 -google-cloud-storage==2.3.0 +google-cloud-storage==2.4.0 pandas===1.3.5; python_version == '3.7' pandas==1.4.2; python_version >= '3.8' From 5b8018dabab00ef8212a0b3173ff3373bcdd74f1 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Fri, 8 Jul 2022 11:57:33 -0400 Subject: [PATCH 058/172] chore: update templated files (#822) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(python): drop python 3.6 Source-Link: https://github.com/googleapis/synthtool/commit/4f89b13af10d086458f9b379e56a614f9d6dab7b Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:e7bb19d47c13839fe8c147e50e02e8b6cf5da8edd1af8b82208cd6f66cc2829c * add api_description to .repo-metadata.json * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/noxfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 38bb0a572b8..5fcb9d7461f 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -89,7 +89,7 @@ def get_pytest_env_vars() -> Dict[str, str]: # DO NOT EDIT - automatically generated. # All versions used to test samples. -ALL_VERSIONS = ["3.6", "3.7", "3.8", "3.9", "3.10"] +ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10"] # Any default versions that should be ignored. IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] From 22677d2f6311a0c5714f37c78c9b386e4b741619 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 8 Jul 2022 18:22:27 +0200 Subject: [PATCH 059/172] chore(deps): update dependency google-cloud-pubsub to v2.13.1 (#823) Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index a2094a1fd6a..7764b415391 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.13.0 +google-cloud-pubsub==2.13.1 google-cloud-storage==2.4.0 pandas===1.3.5; python_version == '3.7' pandas==1.4.2; python_version >= '3.8' From d2b4aad09fb220569ce47e4e1ad0b1d082201baf Mon Sep 17 00:00:00 2001 From: cojenco Date: Wed, 13 Jul 2022 09:48:14 -0700 Subject: [PATCH 060/172] feat: Custom Placement Config Dual Region Support (#819) * refactor: dual-region API update * test coverage --- storage/samples/snippets/snippets_test.py | 3 ++- .../snippets/storage_create_bucket_dual_region.py | 9 +++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index bdd8c528ecd..d0fefd488bc 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -435,10 +435,11 @@ def test_create_bucket_class_location(test_bucket_create): def test_create_bucket_dual_region(test_bucket_create, capsys): + location = "US" region_1 = "US-EAST1" region_2 = "US-WEST1" storage_create_bucket_dual_region.create_bucket_dual_region( - test_bucket_create.name, region_1, region_2 + test_bucket_create.name, location, region_1, region_2 ) out, _ = capsys.readouterr() assert f"Bucket {test_bucket_create.name} created in {region_1}+{region_2}" in out diff --git a/storage/samples/snippets/storage_create_bucket_dual_region.py b/storage/samples/snippets/storage_create_bucket_dual_region.py index e6f4ac01f3d..061f4c1db64 100644 --- a/storage/samples/snippets/storage_create_bucket_dual_region.py +++ b/storage/samples/snippets/storage_create_bucket_dual_region.py @@ -24,8 +24,8 @@ from google.cloud import storage -def create_bucket_dual_region(bucket_name, region_1, region_2): - """Creates a Dual-Region Bucket with provided locations.""" +def create_bucket_dual_region(bucket_name, location, region_1, region_2): + """Creates a Dual-Region Bucket with provided location and regions..""" # The ID of your GCS bucket # bucket_name = "your-bucket-name" @@ -34,9 +34,10 @@ def create_bucket_dual_region(bucket_name, region_1, region_2): # https://cloud.google.com/storage/docs/locations # region_1 = "US-EAST1" # region_2 = "US-WEST1" + # location = "US" storage_client = storage.Client() - storage_client.create_bucket(bucket_name, location=f"{region_1}+{region_2}") + storage_client.create_bucket(bucket_name, location=location, data_locations=[region_1, region_2]) print(f"Bucket {bucket_name} created in {region_1}+{region_2}.") @@ -46,5 +47,5 @@ def create_bucket_dual_region(bucket_name, region_1, region_2): if __name__ == "__main__": create_bucket_dual_region( - bucket_name=sys.argv[1], region_1=sys.argv[2], region_2=sys.argv[3] + bucket_name=sys.argv[1], location=sys.argv[2], region_1=sys.argv[3], region_2=sys.argv[4] ) From f114279ec6ab445906c3e39ab2ac17b538d773b8 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 19 Jul 2022 15:30:51 +0200 Subject: [PATCH 061/172] chore(deps): update dependency backoff to v2.1.2 (#811) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index ed8c5dde8de..077bdf92962 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ pytest==7.1.2 mock==4.0.3 -backoff==2.1.0 \ No newline at end of file +backoff==2.1.2 \ No newline at end of file From 78354ed0e923ac0f3cc72f80a795657252b359cf Mon Sep 17 00:00:00 2001 From: cojenco Date: Fri, 22 Jul 2022 05:28:12 -0700 Subject: [PATCH 062/172] docs: open file-like objects in byte mode for uploads (#824) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit File-like objects should be opened in binary mode for `blob.upload_from_file()` - cpython standard library accorded with [RFC 2616 Section 3.7.1](https://datatracker.ietf.org/doc/html/rfc2616#section-3.7.1) states the text default charset of iso-8859-1 - add clarifying notes in docstring - update code sample Fixes #818 🦕 --- storage/samples/snippets/storage_upload_from_stream.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/storage_upload_from_stream.py b/storage/samples/snippets/storage_upload_from_stream.py index e2d31a5e393..08eb2588907 100644 --- a/storage/samples/snippets/storage_upload_from_stream.py +++ b/storage/samples/snippets/storage_upload_from_stream.py @@ -25,8 +25,8 @@ def upload_blob_from_stream(bucket_name, file_obj, destination_blob_name): # The stream or file (file-like object) from which to read # import io - # file_obj = io.StringIO() - # file_obj.write("This is test data.") + # file_obj = io.BytesIO() + # file_obj.write(b"This is test data.") # The desired name of the uploaded GCS object (blob) # destination_blob_name = "storage-object-name" From 70556506ebf7c62e3b42c6f0b3444caec4e5a227 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Sun, 24 Jul 2022 14:46:29 +0200 Subject: [PATCH 063/172] chore(deps): update dependency pandas to v1.4.3 (#820) Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 7764b415391..fe1ba5907d4 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.13.1 google-cloud-storage==2.4.0 pandas===1.3.5; python_version == '3.7' -pandas==1.4.2; python_version >= '3.8' +pandas==1.4.3; python_version >= '3.8' From e382db2305c3ab2b76cc8f5d6879f33c96c70985 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 5 Aug 2022 19:06:47 +0200 Subject: [PATCH 064/172] chore(deps): update dependency google-cloud-storage to v2.5.0 (#830) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index fe1ba5907d4..29ee2d106b0 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.13.1 -google-cloud-storage==2.4.0 +google-cloud-storage==2.5.0 pandas===1.3.5; python_version == '3.7' pandas==1.4.3; python_version >= '3.8' From 9e5fd71fd619a17a1f8130f18a3452dc897adda4 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 10 Aug 2022 23:53:25 +0200 Subject: [PATCH 065/172] chore(deps): update dependency google-cloud-pubsub to v2.13.4 (#829) Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 29ee2d106b0..90c172cc5e1 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.13.1 +google-cloud-pubsub==2.13.4 google-cloud-storage==2.5.0 pandas===1.3.5; python_version == '3.7' pandas==1.4.3; python_version >= '3.8' From d1fb68bce5045cff9eeb8c15ed1880fa5500bcc5 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 11 Aug 2022 02:25:35 +0200 Subject: [PATCH 066/172] chore(deps): update dependency google-cloud-pubsub to v2.13.5 (#839) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 90c172cc5e1..969019c8ea7 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.13.4 +google-cloud-pubsub==2.13.5 google-cloud-storage==2.5.0 pandas===1.3.5; python_version == '3.7' pandas==1.4.3; python_version >= '3.8' From c4c7dbeb331e6ecaaa5f2f335a497174efec0d0b Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 12 Aug 2022 13:11:10 +0200 Subject: [PATCH 067/172] chore(deps): update dependency google-cloud-pubsub to v2.13.6 (#842) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 969019c8ea7..16383115e20 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.13.5 +google-cloud-pubsub==2.13.6 google-cloud-storage==2.5.0 pandas===1.3.5; python_version == '3.7' pandas==1.4.3; python_version >= '3.8' From 312a71a32136f12b3cc6e7ec2b920ba3eacf42fa Mon Sep 17 00:00:00 2001 From: cojenco Date: Fri, 12 Aug 2022 13:07:23 -0700 Subject: [PATCH 068/172] samples: update dual-region bucket creation sample output (#843) * samples: update dual-region bucket creation sample output * update snippet test --- storage/samples/snippets/snippets_test.py | 6 +++++- .../samples/snippets/storage_create_bucket_dual_region.py | 7 +++++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index d0fefd488bc..bc126010b95 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -442,7 +442,11 @@ def test_create_bucket_dual_region(test_bucket_create, capsys): test_bucket_create.name, location, region_1, region_2 ) out, _ = capsys.readouterr() - assert f"Bucket {test_bucket_create.name} created in {region_1}+{region_2}" in out + assert f"Created bucket {test_bucket_create.name}" in out + assert location in out + assert region_1 in out + assert region_2 in out + assert "dual-region" in out def test_bucket_delete_default_kms_key(test_bucket, capsys): diff --git a/storage/samples/snippets/storage_create_bucket_dual_region.py b/storage/samples/snippets/storage_create_bucket_dual_region.py index 061f4c1db64..c5a78fa0f9b 100644 --- a/storage/samples/snippets/storage_create_bucket_dual_region.py +++ b/storage/samples/snippets/storage_create_bucket_dual_region.py @@ -37,9 +37,12 @@ def create_bucket_dual_region(bucket_name, location, region_1, region_2): # location = "US" storage_client = storage.Client() - storage_client.create_bucket(bucket_name, location=location, data_locations=[region_1, region_2]) + bucket = storage_client.create_bucket(bucket_name, location=location, data_locations=[region_1, region_2]) - print(f"Bucket {bucket_name} created in {region_1}+{region_2}.") + print(f"Created bucket {bucket_name}") + print(f" - location: {bucket.location}") + print(f" - location_type: {bucket.location_type}") + print(f" - customPlacementConfig data_locations: {bucket.data_locations}") # [END storage_create_bucket_dual_region] From 3c3f120de037fbc64fa1ec1fd094cf0a5d473fb6 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 31 Aug 2022 16:39:14 +0200 Subject: [PATCH 069/172] chore(deps): update dependency pandas to v1.4.4 (#855) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 16383115e20..6ec6781218b 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.13.6 google-cloud-storage==2.5.0 pandas===1.3.5; python_version == '3.7' -pandas==1.4.3; python_version >= '3.8' +pandas==1.4.4; python_version >= '3.8' From 2170bda630d562499fdea9455cd15ce3479adb91 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Tue, 6 Sep 2022 19:36:05 +0200 Subject: [PATCH 070/172] chore(deps): update dependency pytest to v7.1.3 (#862) * chore(deps): update all dependencies * revert Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 077bdf92962..cbcfa2f4f4c 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.1.2 +pytest==7.1.3 mock==4.0.3 backoff==2.1.2 \ No newline at end of file From 664d111812c9bb728d6b8e2d55a62687c62760d6 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Tue, 13 Sep 2022 12:39:15 -0400 Subject: [PATCH 071/172] docs(nodejs_mono_repo): update broken links in README (#864) Source-Link: https://github.com/googleapis/synthtool/commit/50db768f450a50d7c1fd62513c113c9bb96fd434 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:e09366bdf0fd9c8976592988390b24d53583dd9f002d476934da43725adbb978 Co-authored-by: Owl Bot --- storage/samples/snippets/noxfile.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 5fcb9d7461f..0398d72ff69 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -207,8 +207,8 @@ def _session_tests( session: nox.sessions.Session, post_install: Callable = None ) -> None: # check for presence of tests - test_list = glob.glob("*_test.py") + glob.glob("test_*.py") - test_list.extend(glob.glob("tests")) + test_list = glob.glob("**/*_test.py", recursive=True) + glob.glob("**/test_*.py", recursive=True) + test_list.extend(glob.glob("**/tests", recursive=True)) if len(test_list) == 0: print("No tests found, skipping directory.") From 21b288bacccd89c2573754066a2af46d74c210cd Mon Sep 17 00:00:00 2001 From: cojenco Date: Thu, 15 Sep 2022 09:54:56 -0700 Subject: [PATCH 072/172] docs: clarify list_blobs usage (#866) --- storage/samples/snippets/storage_list_files.py | 1 + storage/samples/snippets/storage_list_files_with_prefix.py | 1 + 2 files changed, 2 insertions(+) diff --git a/storage/samples/snippets/storage_list_files.py b/storage/samples/snippets/storage_list_files.py index c6a80d9fadb..5e80c833afe 100644 --- a/storage/samples/snippets/storage_list_files.py +++ b/storage/samples/snippets/storage_list_files.py @@ -29,6 +29,7 @@ def list_blobs(bucket_name): # Note: Client.list_blobs requires at least package version 1.17.0. blobs = storage_client.list_blobs(bucket_name) + # Note: The call returns a response only when the iterator is consumed. for blob in blobs: print(blob.name) diff --git a/storage/samples/snippets/storage_list_files_with_prefix.py b/storage/samples/snippets/storage_list_files_with_prefix.py index f79413fb6f1..be7468cba3c 100644 --- a/storage/samples/snippets/storage_list_files_with_prefix.py +++ b/storage/samples/snippets/storage_list_files_with_prefix.py @@ -53,6 +53,7 @@ def list_blobs_with_prefix(bucket_name, prefix, delimiter=None): # Note: Client.list_blobs requires at least package version 1.17.0. blobs = storage_client.list_blobs(bucket_name, prefix=prefix, delimiter=delimiter) + # Note: The call returns a response only when the iterator is consumed. print("Blobs:") for blob in blobs: print(blob.name) From 048e255ad010586ede27b197248da60d7b09324f Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 19 Sep 2022 21:24:06 +0200 Subject: [PATCH 073/172] chore(deps): update dependency pandas to v1.5.0 (#867) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 6ec6781218b..56e80030378 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.13.6 google-cloud-storage==2.5.0 pandas===1.3.5; python_version == '3.7' -pandas==1.4.4; python_version >= '3.8' +pandas==1.5.0; python_version >= '3.8' From 2de2ae55b4fd124e91cd4d0418e777578ff08696 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 23 Sep 2022 23:14:03 +0200 Subject: [PATCH 074/172] chore(deps): update dependency google-cloud-pubsub to v2.13.7 (#869) Co-authored-by: cojenco --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 56e80030378..2910de3e1f9 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.13.6 +google-cloud-pubsub==2.13.7 google-cloud-storage==2.5.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.0; python_version >= '3.8' From 95e1da20883e8c71b2b7edf7a0c1ebd11c4b3a01 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 6 Oct 2022 15:40:08 +0200 Subject: [PATCH 075/172] chore(deps): update dependency backoff to v2.2.1 (#881) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index cbcfa2f4f4c..d999aa86c9a 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ pytest==7.1.3 mock==4.0.3 -backoff==2.1.2 \ No newline at end of file +backoff==2.2.1 \ No newline at end of file From 09129164b32dffb8a0d85a0475105803c983c838 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 12 Oct 2022 23:46:57 +0200 Subject: [PATCH 076/172] chore(deps): update dependency google-cloud-pubsub to v2.13.9 (#885) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 2910de3e1f9..0955d15fd2d 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.13.7 +google-cloud-pubsub==2.13.9 google-cloud-storage==2.5.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.0; python_version >= '3.8' From d2276ffc07bdc7014c5ddd3ad74a329fe9702f13 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Fri, 14 Oct 2022 20:21:36 +0200 Subject: [PATCH 077/172] chore(deps): update dependency google-cloud-pubsub to v2.13.10 (#888) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 0955d15fd2d..e17bf947490 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.13.9 +google-cloud-pubsub==2.13.10 google-cloud-storage==2.5.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.0; python_version >= '3.8' From 20c9dd115ea963a2eca606bb0d96e35be29eb09f Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 19 Oct 2022 16:04:13 +0200 Subject: [PATCH 078/172] chore(deps): update dependency pandas to v1.5.1 (#889) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index e17bf947490..8e71b2787e3 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.13.10 google-cloud-storage==2.5.0 pandas===1.3.5; python_version == '3.7' -pandas==1.5.0; python_version >= '3.8' +pandas==1.5.1; python_version >= '3.8' From f486baf5c05a3cc62f147eceee8a888b15453c93 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 26 Oct 2022 13:15:17 +0200 Subject: [PATCH 079/172] chore(deps): update dependency pytest to v7.2.0 (#893) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index d999aa86c9a..4e8a7389f16 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.1.3 +pytest==7.2.0 mock==4.0.3 backoff==2.2.1 \ No newline at end of file From 2b55d32d5de227f8ccfb38b98ae6ca75b3f3ef6a Mon Sep 17 00:00:00 2001 From: cojenco Date: Sun, 6 Nov 2022 22:56:13 -0800 Subject: [PATCH 080/172] feat: add Autoclass support and sample (#791) This adds support and samples for Autoclass For more info, see Internal: [go/gcs-dpe-autoclass](http://go/gcs-dpe-autoclass) Fixes #797 --- storage/samples/snippets/snippets_test.py | 38 +++++++++++++++ .../samples/snippets/storage_get_autoclass.py | 41 ++++++++++++++++ .../samples/snippets/storage_set_autoclass.py | 47 +++++++++++++++++++ 3 files changed, 126 insertions(+) create mode 100644 storage/samples/snippets/storage_get_autoclass.py create mode 100644 storage/samples/snippets/storage_set_autoclass.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index bc126010b95..9370ecbddee 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -53,6 +53,7 @@ import storage_generate_signed_url_v2 import storage_generate_signed_url_v4 import storage_generate_upload_signed_url_v4 +import storage_get_autoclass import storage_get_bucket_labels import storage_get_bucket_metadata import storage_get_metadata @@ -67,6 +68,7 @@ import storage_remove_bucket_label import storage_remove_cors_configuration import storage_rename_file +import storage_set_autoclass import storage_set_bucket_default_kms_key import storage_set_client_endpoint import storage_set_metadata @@ -136,6 +138,17 @@ def test_public_bucket(): os.environ['GOOGLE_CLOUD_PROJECT'] = original_value +@pytest.fixture(scope="module") +def new_bucket_obj(): + """Yields a new bucket object that is deleted after the test completes.""" + bucket = None + while bucket is None or bucket.exists(): + bucket_name = f"storage-snippets-test-{uuid.uuid4()}" + bucket = storage.Client().bucket(bucket_name) + yield bucket + bucket.delete(force=True) + + @pytest.fixture def test_blob(test_bucket): """Yields a blob that is deleted after the test completes.""" @@ -408,6 +421,31 @@ def test_versioning(test_bucket, capsys): assert bucket.versioning_enabled is False +def test_get_set_autoclass(new_bucket_obj, test_bucket, capsys): + # Test default values when Autoclass is unset + bucket = storage_get_autoclass.get_autoclass(test_bucket.name) + out, _ = capsys.readouterr() + assert "Autoclass enabled is set to False" in out + assert bucket.autoclass_toggle_time is None + + # Test enabling Autoclass at bucket creation + new_bucket_obj.autoclass_enabled = True + bucket = storage.Client().create_bucket(new_bucket_obj) + assert bucket.autoclass_enabled is True + + # Test disabling Autoclass + bucket = storage_set_autoclass.set_autoclass(bucket.name, False) + out, _ = capsys.readouterr() + assert "Autoclass enabled is set to False" in out + assert bucket.autoclass_enabled is False + + # Test get Autoclass + bucket = storage_get_autoclass.get_autoclass(bucket.name) + out, _ = capsys.readouterr() + assert "Autoclass enabled is set to False" in out + assert bucket.autoclass_toggle_time is not None + + def test_bucket_lifecycle_management(test_bucket, capsys): bucket = storage_enable_bucket_lifecycle_management.enable_bucket_lifecycle_management( test_bucket diff --git a/storage/samples/snippets/storage_get_autoclass.py b/storage/samples/snippets/storage_get_autoclass.py new file mode 100644 index 00000000000..d4bcbf3f4dd --- /dev/null +++ b/storage/samples/snippets/storage_get_autoclass.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_get_autoclass] +from google.cloud import storage + + +def get_autoclass(bucket_name): + """Get the Autoclass setting for a bucket.""" + # The ID of your GCS bucket + # bucket_name = "my-bucket" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + autoclass_enabled = bucket.autoclass_enabled + autoclass_toggle_time = bucket.autoclass_toggle_time + + print(f"Autoclass enabled is set to {autoclass_enabled} for {bucket.name} at {autoclass_toggle_time}.") + + return bucket + + +# [END storage_get_autoclass] + +if __name__ == "__main__": + get_autoclass(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_set_autoclass.py b/storage/samples/snippets/storage_set_autoclass.py new file mode 100644 index 00000000000..a25151f3bbc --- /dev/null +++ b/storage/samples/snippets/storage_set_autoclass.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python + +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_set_autoclass] +from google.cloud import storage + + +def set_autoclass(bucket_name, toggle): + """Disable Autoclass for a bucket. + + Note: Only patch requests that disable autoclass are currently supported. + To enable autoclass, you must set it at bucket creation time. + """ + # The ID of your GCS bucket + # bucket_name = "my-bucket" + # Boolean toggle - if true, enables Autoclass; if false, disables Autoclass + # toggle = False + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + bucket.autoclass_enabled = toggle + bucket.patch() + print(f"Autoclass enabled is set to {bucket.autoclass_enabled} for {bucket.name} at {bucket.autoclass_toggle_time}.") + + return bucket + + +# [END storage_set_autoclass] + +if __name__ == "__main__": + set_autoclass(bucket_name=sys.argv[1], toggle=sys.argv[2]) From d94b843688afc3aedb63d6be8725721acb0606ca Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Wed, 9 Nov 2022 01:02:25 +0100 Subject: [PATCH 081/172] chore(deps): update dependency google-cloud-storage to v2.6.0 (#899) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 8e71b2787e3..bcf31e6ba60 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.13.10 -google-cloud-storage==2.5.0 +google-cloud-storage==2.6.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.1; python_version >= '3.8' From 454aecc23262caf68df79d6a2603447f9c395208 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 15 Nov 2022 21:37:49 +0100 Subject: [PATCH 082/172] chore(deps): update dependency google-cloud-pubsub to v2.13.11 (#902) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index bcf31e6ba60..b676c0c6d54 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.13.10 +google-cloud-pubsub==2.13.11 google-cloud-storage==2.6.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.1; python_version >= '3.8' From 5f4a3298e44dbecb7bfa37257c06d827cc4309c6 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Sun, 27 Nov 2022 01:02:56 +0100 Subject: [PATCH 083/172] chore(deps): update dependency pandas to v1.5.2 (#915) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index b676c0c6d54..d5554b4d97e 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.13.11 google-cloud-storage==2.6.0 pandas===1.3.5; python_version == '3.7' -pandas==1.5.1; python_version >= '3.8' +pandas==1.5.2; python_version >= '3.8' From 3b4f66f3ecf6e0489bb1188ddbe6f0fe53637642 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Sun, 27 Nov 2022 07:54:02 -0500 Subject: [PATCH 084/172] chore(python): drop flake8-import-order in samples noxfile [autoapprove] (#916) * chore(python): drop flake8-import-order in samples noxfile Source-Link: https://github.com/googleapis/synthtool/commit/6ed3a831cb9ff69ef8a504c353e098ec0192ad93 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:3abfa0f1886adaf0b83f07cb117b24a639ea1cb9cffe56d43280b977033563eb * update python version for docs session Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/noxfile.py | 26 +++----------------------- 1 file changed, 3 insertions(+), 23 deletions(-) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 0398d72ff69..f5c32b22789 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -18,7 +18,7 @@ import os from pathlib import Path import sys -from typing import Callable, Dict, List, Optional +from typing import Callable, Dict, Optional import nox @@ -109,22 +109,6 @@ def get_pytest_env_vars() -> Dict[str, str]: # -def _determine_local_import_names(start_dir: str) -> List[str]: - """Determines all import names that should be considered "local". - - This is used when running the linter to insure that import order is - properly checked. - """ - file_ext_pairs = [os.path.splitext(path) for path in os.listdir(start_dir)] - return [ - basename - for basename, extension in file_ext_pairs - if extension == ".py" - or os.path.isdir(os.path.join(start_dir, basename)) - and basename not in ("__pycache__") - ] - - # Linting with flake8. # # We ignore the following rules: @@ -139,7 +123,6 @@ def _determine_local_import_names(start_dir: str) -> List[str]: "--show-source", "--builtin=gettext", "--max-complexity=20", - "--import-order-style=google", "--exclude=.nox,.cache,env,lib,generated_pb2,*_pb2.py,*_pb2_grpc.py", "--ignore=E121,E123,E126,E203,E226,E24,E266,E501,E704,W503,W504,I202", "--max-line-length=88", @@ -149,14 +132,11 @@ def _determine_local_import_names(start_dir: str) -> List[str]: @nox.session def lint(session: nox.sessions.Session) -> None: if not TEST_CONFIG["enforce_type_hints"]: - session.install("flake8", "flake8-import-order") + session.install("flake8") else: - session.install("flake8", "flake8-import-order", "flake8-annotations") + session.install("flake8", "flake8-annotations") - local_names = _determine_local_import_names(".") args = FLAKE8_COMMON_ARGS + [ - "--application-import-names", - ",".join(local_names), ".", ] session.run("flake8", *args) From e8d6c132cc1ee55cedf6bc54bb39e0d7ea81c6ca Mon Sep 17 00:00:00 2001 From: Andrew Gorcester Date: Tue, 6 Dec 2022 15:21:15 -0800 Subject: [PATCH 085/172] feat: Add "transfer_manager" module for concurrent uploads and downloads, as a preview feature (#943) * checkpoint before design doc impl * checkpoint * more tests * code and tests for transfer manager complete * proactively close temp files when finished reading * respond to comments; destroy tmp files as they are consumed * Add system tests, docstrings, address feedback * Respond to review comments * verify md5 hash of downloaded file in test * lint * default empty strings for root arguments * fix bug with blob constructor * add warning about files not being deleted if their downloads fail * docs: Add samples to multithread branch (#918) * add samples, tests pending * add snippet tests * snippet and snippets_test.py linting * snippets; recursive directory creation; rename some params * Add directory upload snippet * fix: remove chunked downloads; change max_workers to threads * update snippets to add thread info * fix snippets test issue due to change in dependency * snippet nomenclature * fix samples for real this time --- storage/samples/snippets/snippets_test.py | 118 +++++++++-- .../snippets/storage_transfer_manager.py | 184 ++++++++++++++++++ 2 files changed, 285 insertions(+), 17 deletions(-) create mode 100644 storage/samples/snippets/storage_transfer_manager.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 9370ecbddee..4ad0dc1a0ca 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -72,6 +72,7 @@ import storage_set_bucket_default_kms_key import storage_set_client_endpoint import storage_set_metadata +import storage_transfer_manager import storage_upload_file import storage_upload_from_memory import storage_upload_from_stream @@ -124,8 +125,8 @@ def test_bucket(): def test_public_bucket(): # The new projects don't allow to make a bucket available to public, so # for some tests we need to use the old main project for now. - original_value = os.environ['GOOGLE_CLOUD_PROJECT'] - os.environ['GOOGLE_CLOUD_PROJECT'] = os.environ['MAIN_GOOGLE_CLOUD_PROJECT'] + original_value = os.environ["GOOGLE_CLOUD_PROJECT"] + os.environ["GOOGLE_CLOUD_PROJECT"] = os.environ["MAIN_GOOGLE_CLOUD_PROJECT"] bucket = None while bucket is None or bucket.exists(): storage_client = storage.Client() @@ -135,7 +136,7 @@ def test_public_bucket(): yield bucket bucket.delete(force=True) # Set the value back. - os.environ['GOOGLE_CLOUD_PROJECT'] = original_value + os.environ["GOOGLE_CLOUD_PROJECT"] = original_value @pytest.fixture(scope="module") @@ -255,7 +256,7 @@ def test_download_byte_range(test_blob): storage_download_byte_range.download_byte_range( test_blob.bucket.name, test_blob.name, 0, 4, dest_file.name ) - assert dest_file.read() == b'Hello' + assert dest_file.read() == b"Hello" def test_download_blob(test_blob): @@ -308,7 +309,8 @@ def test_delete_blob(test_blob): def test_make_blob_public(test_public_blob): storage_make_public.make_blob_public( - test_public_blob.bucket.name, test_public_blob.name) + test_public_blob.bucket.name, test_public_blob.name + ) r = requests.get(test_public_blob.public_url) assert r.text == "Hello, is it me you're looking for?" @@ -340,7 +342,9 @@ def test_generate_upload_signed_url_v4(test_bucket, capsys): ) requests.put( - url, data=content, headers={"content-type": "application/octet-stream"}, + url, + data=content, + headers={"content-type": "application/octet-stream"}, ) bucket = storage.Client().bucket(test_bucket.name) @@ -447,16 +451,20 @@ def test_get_set_autoclass(new_bucket_obj, test_bucket, capsys): def test_bucket_lifecycle_management(test_bucket, capsys): - bucket = storage_enable_bucket_lifecycle_management.enable_bucket_lifecycle_management( - test_bucket + bucket = ( + storage_enable_bucket_lifecycle_management.enable_bucket_lifecycle_management( + test_bucket + ) ) out, _ = capsys.readouterr() assert "[]" in out assert "Lifecycle management is enable" in out assert len(list(bucket.lifecycle_rules)) > 0 - bucket = storage_disable_bucket_lifecycle_management.disable_bucket_lifecycle_management( - test_bucket + bucket = ( + storage_disable_bucket_lifecycle_management.disable_bucket_lifecycle_management( + test_bucket + ) ) out, _ = capsys.readouterr() assert "[]" in out @@ -512,7 +520,8 @@ def test_get_service_account(capsys): def test_download_public_file(test_public_blob): storage_make_public.make_blob_public( - test_public_blob.bucket.name, test_public_blob.name) + test_public_blob.bucket.name, test_public_blob.name + ) with tempfile.NamedTemporaryFile() as dest_file: storage_download_public_file.download_public_file( test_public_blob.bucket.name, test_public_blob.name, dest_file.name @@ -522,8 +531,10 @@ def test_download_public_file(test_public_blob): def test_define_bucket_website_configuration(test_bucket): - bucket = storage_define_bucket_website_configuration.define_bucket_website_configuration( - test_bucket.name, "index.html", "404.html" + bucket = ( + storage_define_bucket_website_configuration.define_bucket_website_configuration( + test_bucket.name, "index.html", "404.html" + ) ) website_val = {"mainPageSuffix": "index.html", "notFoundPage": "404.html"} @@ -586,7 +597,7 @@ def test_change_default_storage_class(test_bucket, capsys): ) out, _ = capsys.readouterr() assert "Default storage class for bucket" in out - assert bucket.storage_class == 'COLDLINE' + assert bucket.storage_class == "COLDLINE" def test_change_file_storage_class(test_blob, capsys): @@ -595,7 +606,7 @@ def test_change_file_storage_class(test_blob, capsys): ) out, _ = capsys.readouterr() assert f"Blob {blob.name} in bucket {blob.bucket.name}" in out - assert blob.storage_class == 'NEARLINE' + assert blob.storage_class == "NEARLINE" def test_copy_file_archived_generation(test_blob): @@ -629,7 +640,8 @@ def test_storage_configure_retries(test_blob, capsys): out, _ = capsys.readouterr() assert "The following library method is customized to be retried" in out assert "_should_retry" in out - assert "initial=1.5, maximum=45.0, multiplier=1.2, deadline=500.0" in out + assert "initial=1.5, maximum=45.0, multiplier=1.2" in out + assert "500" in out # "deadline" or "timeout" depending on dependency ver. def test_batch_request(test_bucket): @@ -647,7 +659,79 @@ def test_batch_request(test_bucket): def test_storage_set_client_endpoint(capsys): - storage_set_client_endpoint.set_client_endpoint('https://storage.googleapis.com') + storage_set_client_endpoint.set_client_endpoint("https://storage.googleapis.com") out, _ = capsys.readouterr() assert "client initiated with endpoint: https://storage.googleapis.com" in out + + +def test_transfer_manager_snippets(test_bucket, capsys): + BLOB_NAMES = [ + "test.txt", + "test2.txt", + "blobs/test.txt", + "blobs/nesteddir/test.txt", + ] + + with tempfile.TemporaryDirectory() as uploads: + # Create dirs and nested dirs + for name in BLOB_NAMES: + relpath = os.path.dirname(name) + os.makedirs(os.path.join(uploads, relpath), exist_ok=True) + + # Create files with nested dirs to exercise directory handling. + for name in BLOB_NAMES: + with open(os.path.join(uploads, name), "w") as f: + f.write(name) + + storage_transfer_manager.upload_many_blobs_with_transfer_manager( + test_bucket.name, + BLOB_NAMES, + source_directory="{}/".format(uploads), + threads=2, + ) + out, _ = capsys.readouterr() + + for name in BLOB_NAMES: + assert "Uploaded {}".format(name) in out + + with tempfile.TemporaryDirectory() as downloads: + # Download the files. + storage_transfer_manager.download_all_blobs_with_transfer_manager( + test_bucket.name, + destination_directory=os.path.join(downloads, ""), + threads=2, + ) + out, _ = capsys.readouterr() + + for name in BLOB_NAMES: + assert "Downloaded {}".format(name) in out + + +def test_transfer_manager_directory_upload(test_bucket, capsys): + BLOB_NAMES = [ + "dirtest/test.txt", + "dirtest/test2.txt", + "dirtest/blobs/test.txt", + "dirtest/blobs/nesteddir/test.txt", + ] + + with tempfile.TemporaryDirectory() as uploads: + # Create dirs and nested dirs + for name in BLOB_NAMES: + relpath = os.path.dirname(name) + os.makedirs(os.path.join(uploads, relpath), exist_ok=True) + + # Create files with nested dirs to exercise directory handling. + for name in BLOB_NAMES: + with open(os.path.join(uploads, name), "w") as f: + f.write(name) + + storage_transfer_manager.upload_directory_with_transfer_manager( + test_bucket.name, source_directory="{}/".format(uploads) + ) + out, _ = capsys.readouterr() + + assert "Found {}".format(len(BLOB_NAMES)) in out + for name in BLOB_NAMES: + assert "Uploaded {}".format(name) in out diff --git a/storage/samples/snippets/storage_transfer_manager.py b/storage/samples/snippets/storage_transfer_manager.py new file mode 100644 index 00000000000..0a02b96e338 --- /dev/null +++ b/storage/samples/snippets/storage_transfer_manager.py @@ -0,0 +1,184 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +def upload_many_blobs_with_transfer_manager( + bucket_name, filenames, source_directory="", threads=4 +): + """Upload every file in a list to a bucket, concurrently in a thread pool. + + Each blob name is derived from the filename, not including the + `source_directory` parameter. For complete control of the blob name for each + file (and other aspects of individual blob metadata), use + transfer_manager.upload_many() instead. + """ + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # A list (or other iterable) of filenames to upload. + # filenames = ["file_1.txt", "file_2.txt"] + + # The directory on your computer that is the root of all of the files in the + # list of filenames. This string is prepended (with os.path.join()) to each + # filename to get the full path to the file. Relative paths and absolute + # paths are both accepted. This string is not included in the name of the + # uploaded blob; it is only used to find the source files. An empty string + # means "the current working directory". Note that this parameter allows + # directory traversal (e.g. "/", "../") and is not intended for unsanitized + # end user input. + # source_directory="" + + # The number of threads to use for the operation. The performance impact of + # this value depends on the use case, but generally, smaller files benefit + # from more threads and larger files don't benefit from more threads. Too + # many threads can slow operations, especially with large files, due to + # contention over the Python GIL. + # threads=4 + + from google.cloud.storage import Client, transfer_manager + + storage_client = Client() + bucket = storage_client.bucket(bucket_name) + + results = transfer_manager.upload_many_from_filenames( + bucket, filenames, source_directory=source_directory, threads=threads + ) + + for name, result in zip(filenames, results): + # The results list is either `None` or an exception for each filename in + # the input list, in order. + + if isinstance(result, Exception): + print("Failed to upload {} due to exception: {}".format(name, result)) + else: + print("Uploaded {} to {}.".format(name, bucket.name)) + + +def upload_directory_with_transfer_manager(bucket_name, source_directory, threads=4): + """Upload every file in a directory, including all files in subdirectories. + + Each blob name is derived from the filename, not including the `directory` + parameter itself. For complete control of the blob name for each file (and + other aspects of individual blob metadata), use + transfer_manager.upload_many() instead. + """ + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The directory on your computer to upload. Files in the directory and its + # subdirectories will be uploaded. An empty string means "the current + # working directory". + # source_directory="" + + # The number of threads to use for the operation. The performance impact of + # this value depends on the use case, but generally, smaller files benefit + # from more threads and larger files don't benefit from more threads. Too + # many threads can slow operations, especially with large files, due to + # contention over the Python GIL. + # threads=4 + + from pathlib import Path + + from google.cloud.storage import Client, transfer_manager + + storage_client = Client() + bucket = storage_client.bucket(bucket_name) + + # Generate a list of paths (in string form) relative to the `directory`. + # This can be done in a single list comprehension, but is expanded into + # multiple lines here for clarity. + + # First, recursively get all files in `directory` as Path objects. + directory_as_path_obj = Path(source_directory) + paths = directory_as_path_obj.rglob("*") + + # Filter so the list only includes files, not directories themselves. + file_paths = [path for path in paths if path.is_file()] + + # These paths are relative to the current working directory. Next, make them + # relative to `directory` + relative_paths = [path.relative_to(source_directory) for path in file_paths] + + # Finally, convert them all to strings. + string_paths = [str(path) for path in relative_paths] + + print("Found {} files.".format(len(string_paths))) + + # Start the upload. + results = transfer_manager.upload_many_from_filenames( + bucket, string_paths, source_directory=source_directory, threads=threads + ) + + for name, result in zip(string_paths, results): + # The results list is either `None` or an exception for each filename in + # the input list, in order. + + if isinstance(result, Exception): + print("Failed to upload {} due to exception: {}".format(name, result)) + else: + print("Uploaded {} to {}.".format(name, bucket.name)) + + +def download_all_blobs_with_transfer_manager( + bucket_name, destination_directory="", threads=4 +): + """Download all of the blobs in a bucket, concurrently in a thread pool. + + The filename of each blob once downloaded is derived from the blob name and + the `destination_directory `parameter. For complete control of the filename + of each blob, use transfer_manager.download_many() instead. + + Directories will be created automatically as needed, for instance to + accommodate blob names that include slashes. + """ + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The directory on your computer to which to download all of the files. This + # string is prepended (with os.path.join()) to the name of each blob to form + # the full path. Relative paths and absolute paths are both accepted. An + # empty string means "the current working directory". Note that this + # parameter allows accepts directory traversal ("../" etc.) and is not + # intended for unsanitized end user input. + # destination_directory = "" + + # The number of threads to use for the operation. The performance impact of + # this value depends on the use case, but generally, smaller files benefit + # from more threads and larger files don't benefit from more threads. Too + # many threads can slow operations, especially with large files, due to + # contention over the Python GIL. + # threads=4 + + from google.cloud.storage import Client, transfer_manager + + storage_client = Client() + bucket = storage_client.bucket(bucket_name) + + blob_names = [blob.name for blob in bucket.list_blobs()] + + results = transfer_manager.download_many_to_path( + bucket, blob_names, destination_directory=destination_directory, threads=threads + ) + + for name, result in zip(blob_names, results): + # The results list is either `None` or an exception for each blob in + # the input list, in order. + + if isinstance(result, Exception): + print("Failed to download {} due to exception: {}".format(name, result)) + else: + print("Downloaded {} to {}.".format(name, destination_directory + name)) From a8aeca2298b6651631b3aa8fc8d7869a99edaf0e Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Wed, 7 Dec 2022 06:25:12 +0100 Subject: [PATCH 086/172] chore(deps): update dependency google-cloud-storage to v2.7.0 (#944) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index d5554b4d97e..1dee3070ee2 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.13.11 -google-cloud-storage==2.6.0 +google-cloud-storage==2.7.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.2; python_version >= '3.8' From aa084f2f20ba9265a1e2230a17d88270df33bd59 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Wed, 4 Jan 2023 21:03:03 +0100 Subject: [PATCH 087/172] chore(deps): update dependency mock to v5 (#946) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 4e8a7389f16..f3849b85913 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ pytest==7.2.0 -mock==4.0.3 +mock==5.0.0 backoff==2.2.1 \ No newline at end of file From f7507178d1cc87e5bb2ab99a7cdab635d590bcb6 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Fri, 6 Jan 2023 14:17:45 -0500 Subject: [PATCH 088/172] chore(python): add support for python 3.11 [autoapprove] (#952) * chore(python): add support for python 3.11 Source-Link: https://github.com/googleapis/synthtool/commit/7197a001ffb6d8ce7b0b9b11c280f0c536c1033a Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:c43f1d918bcf817d337aa29ff833439494a158a0831508fda4ec75dc4c0d0320 * add python 3.11 to noxfile * Add python 3.11 to noxfile and contributing doc Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/noxfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index f5c32b22789..7c8a63994cb 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -89,7 +89,7 @@ def get_pytest_env_vars() -> Dict[str, str]: # DO NOT EDIT - automatically generated. # All versions used to test samples. -ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10"] +ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"] # Any default versions that should be ignored. IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] From cc4c235c20886d9e662d6f0aeb93e979049427b2 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Mon, 9 Jan 2023 18:19:14 +0000 Subject: [PATCH 089/172] chore(deps): update dependency mock to v5.0.1 (#953) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index f3849b85913..3ab184d0cd6 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ pytest==7.2.0 -mock==5.0.0 +mock==5.0.1 backoff==2.2.1 \ No newline at end of file From a688913b925cd280a92e215b1f983044cb4dcca3 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 10 Jan 2023 18:42:59 +0000 Subject: [PATCH 090/172] chore(deps): update dependency google-cloud-pubsub to v2.13.12 (#972) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 1dee3070ee2..87fac49cf03 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.13.11 +google-cloud-pubsub==2.13.12 google-cloud-storage==2.7.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.2; python_version >= '3.8' From 954701a0bfd7b7ed370cae38a0033419ea6c0ab8 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Sat, 14 Jan 2023 18:08:39 +0000 Subject: [PATCH 091/172] chore(deps): update dependency pytest to v7.2.1 (#974) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 3ab184d0cd6..51c1be2e642 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.2.0 +pytest==7.2.1 mock==5.0.1 backoff==2.2.1 \ No newline at end of file From 0cf2cf7a96a6ceb96722a0505615436070979370 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Wed, 25 Jan 2023 18:43:47 +0000 Subject: [PATCH 092/172] chore(deps): update all dependencies (#976) --- storage/samples/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 87fac49cf03..b13cef2bfb0 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.13.12 +google-cloud-pubsub==2.14.0 google-cloud-storage==2.7.0 pandas===1.3.5; python_version == '3.7' -pandas==1.5.2; python_version >= '3.8' +pandas==1.5.3; python_version >= '3.8' From 80040f3e85a7176c62b4b7e1492ac9a4af317522 Mon Sep 17 00:00:00 2001 From: cojenco Date: Tue, 7 Feb 2023 12:00:08 -0800 Subject: [PATCH 093/172] samples: add generation-match preconditions to selected samples (#949) * samples: add preconditions to objects.delete * add preconditons to rewrite category samples * add compose and update previous changes * preconditions to rewrites and encrypted uploads * add preconditions to objects insert * refine optional block wording and flow * update test --- storage/samples/snippets/encryption_test.py | 5 ++++- storage/samples/snippets/snippets_test.py | 16 +++++++++------- .../storage_change_file_storage_class.py | 14 +++++++++++--- .../samples/snippets/storage_compose_file.py | 16 +++++++++++++--- storage/samples/snippets/storage_copy_file.py | 13 +++++++++++-- .../storage_copy_file_archived_generation.py | 13 +++++++++++-- .../samples/snippets/storage_delete_file.py | 10 +++++++++- storage/samples/snippets/storage_move_file.py | 13 +++++++++++-- .../snippets/storage_object_csek_to_cmek.py | 18 ++++++++++++++---- .../snippets/storage_rotate_encryption_key.py | 10 ++++++++-- .../snippets/storage_upload_encrypted_file.py | 14 +++++++++++++- .../samples/snippets/storage_upload_file.py | 10 +++++++++- .../snippets/storage_upload_with_kms_key.py | 13 +++++++++++-- 13 files changed, 134 insertions(+), 31 deletions(-) diff --git a/storage/samples/snippets/encryption_test.py b/storage/samples/snippets/encryption_test.py index 536c5d3343e..5a5eb7b2d53 100644 --- a/storage/samples/snippets/encryption_test.py +++ b/storage/samples/snippets/encryption_test.py @@ -47,15 +47,18 @@ def test_generate_encryption_key(capsys): def test_upload_encrypted_blob(): + blob_name = f"test_upload_encrypted_{uuid.uuid4().hex}" with tempfile.NamedTemporaryFile() as source_file: source_file.write(b"test") storage_upload_encrypted_file.upload_encrypted_blob( BUCKET, source_file.name, - "test_encrypted_upload_blob", + blob_name, TEST_ENCRYPTION_KEY, ) + bucket = storage.Client().bucket(BUCKET) + bucket.delete_blob(blob_name) @pytest.fixture(scope="module") diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 4ad0dc1a0ca..57751be6076 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -235,14 +235,16 @@ def test_upload_blob_from_stream(test_bucket, capsys): def test_upload_blob_with_kms(test_bucket): + blob_name = f"test_upload_with_kms_{uuid.uuid4().hex}" with tempfile.NamedTemporaryFile() as source_file: source_file.write(b"test") storage_upload_with_kms_key.upload_blob_with_kms( - test_bucket.name, source_file.name, "test_upload_blob_encrypted", KMS_KEY + test_bucket.name, source_file.name, blob_name, KMS_KEY, ) bucket = storage.Client().bucket(test_bucket.name) - kms_blob = bucket.get_blob("test_upload_blob_encrypted") + kms_blob = bucket.get_blob(blob_name) assert kms_blob.kms_key_name.startswith(KMS_KEY) + test_bucket.delete_blob(blob_name) def test_async_upload(bucket, capsys): @@ -390,7 +392,7 @@ def test_move_blob(test_bucket_create, test_blob): print(f"test_move_blob not found in bucket {test_bucket_create.name}") storage_move_file.move_blob( - bucket.name, test_blob.name, test_bucket_create.name, "test_move_blob" + bucket.name, test_blob.name, test_bucket_create.name, "test_move_blob", ) assert test_bucket_create.get_blob("test_move_blob") is not None @@ -406,7 +408,7 @@ def test_copy_blob(test_blob): pass storage_copy_file.copy_blob( - bucket.name, test_blob.name, bucket.name, "test_copy_blob" + bucket.name, test_blob.name, bucket.name, "test_copy_blob", ) assert bucket.get_blob("test_copy_blob") is not None @@ -545,7 +547,7 @@ def test_define_bucket_website_configuration(test_bucket): def test_object_get_kms_key(test_bucket): with tempfile.NamedTemporaryFile() as source_file: storage_upload_with_kms_key.upload_blob_with_kms( - test_bucket.name, source_file.name, "test_upload_blob_encrypted", KMS_KEY + test_bucket.name, source_file.name, "test_upload_blob_encrypted", KMS_KEY, ) kms_key = storage_object_get_kms_key.object_get_kms_key( test_bucket.name, "test_upload_blob_encrypted" @@ -562,7 +564,7 @@ def test_storage_compose_file(test_bucket): with tempfile.NamedTemporaryFile() as dest_file: destination = storage_compose_file.compose_file( - test_bucket.name, source_files[0], source_files[1], dest_file.name + test_bucket.name, source_files[0], source_files[1], dest_file.name, ) composed = destination.download_as_string() @@ -602,7 +604,7 @@ def test_change_default_storage_class(test_bucket, capsys): def test_change_file_storage_class(test_blob, capsys): blob = storage_change_file_storage_class.change_file_storage_class( - test_blob.bucket.name, test_blob.name + test_blob.bucket.name, test_blob.name, ) out, _ = capsys.readouterr() assert f"Blob {blob.name} in bucket {blob.bucket.name}" in out diff --git a/storage/samples/snippets/storage_change_file_storage_class.py b/storage/samples/snippets/storage_change_file_storage_class.py index d5dda56a709..a976ac8a4c8 100644 --- a/storage/samples/snippets/storage_change_file_storage_class.py +++ b/storage/samples/snippets/storage_change_file_storage_class.py @@ -27,9 +27,17 @@ def change_file_storage_class(bucket_name, blob_name): storage_client = storage.Client() - bucket = storage_client.get_bucket(bucket_name) - blob = bucket.get_blob(blob_name) - blob.update_storage_class("NEARLINE") + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + generation_match_precondition = None + + # Optional: set a generation-match precondition to avoid potential race + # conditions and data corruptions. The request is aborted if the + # object's generation number does not match your precondition. + blob.reload() # Fetch blob metadata to use in generation_match_precondition. + generation_match_precondition = blob.generation + + blob.update_storage_class("NEARLINE", if_generation_match=generation_match_precondition) print( "Blob {} in bucket {} had its storage class set to {}".format( diff --git a/storage/samples/snippets/storage_compose_file.py b/storage/samples/snippets/storage_compose_file.py index 2c1443f22f4..e673912725b 100644 --- a/storage/samples/snippets/storage_compose_file.py +++ b/storage/samples/snippets/storage_compose_file.py @@ -32,9 +32,19 @@ def compose_file(bucket_name, first_blob_name, second_blob_name, destination_blo destination = bucket.blob(destination_blob_name) destination.content_type = "text/plain" - # sources is a list of Blob instances, up to the max of 32 instances per request - sources = [bucket.get_blob(first_blob_name), bucket.get_blob(second_blob_name)] - destination.compose(sources) + # Note sources is a list of Blob instances, up to the max of 32 instances per request + sources = [bucket.blob(first_blob_name), bucket.blob(second_blob_name)] + + # Optional: set a generation-match precondition to avoid potential race conditions + # and data corruptions. The request to compose is aborted if the object's + # generation number does not match your precondition. For a destination + # object that does not yet exist, set the if_generation_match precondition to 0. + # If the destination object already exists in your bucket, set instead a + # generation-match precondition using its generation number. + # There is also an `if_source_generation_match` parameter, which is not used in this example. + destination_generation_match_precondition = 0 + + destination.compose(sources, if_generation_match=destination_generation_match_precondition) print( "New composite object {} in the bucket {} was created by combining {} and {}".format( diff --git a/storage/samples/snippets/storage_copy_file.py b/storage/samples/snippets/storage_copy_file.py index 5d36aa94b44..b802de28b1b 100644 --- a/storage/samples/snippets/storage_copy_file.py +++ b/storage/samples/snippets/storage_copy_file.py @@ -21,7 +21,7 @@ def copy_blob( - bucket_name, blob_name, destination_bucket_name, destination_blob_name + bucket_name, blob_name, destination_bucket_name, destination_blob_name, ): """Copies a blob from one bucket to another with a new name.""" # bucket_name = "your-bucket-name" @@ -35,8 +35,17 @@ def copy_blob( source_blob = source_bucket.blob(blob_name) destination_bucket = storage_client.bucket(destination_bucket_name) + # Optional: set a generation-match precondition to avoid potential race conditions + # and data corruptions. The request to copy is aborted if the object's + # generation number does not match your precondition. For a destination + # object that does not yet exist, set the if_generation_match precondition to 0. + # If the destination object already exists in your bucket, set instead a + # generation-match precondition using its generation number. + # There is also an `if_source_generation_match` parameter, which is not used in this example. + destination_generation_match_precondition = 0 + blob_copy = source_bucket.copy_blob( - source_blob, destination_bucket, destination_blob_name + source_blob, destination_bucket, destination_blob_name, if_generation_match=destination_generation_match_precondition, ) print( diff --git a/storage/samples/snippets/storage_copy_file_archived_generation.py b/storage/samples/snippets/storage_copy_file_archived_generation.py index 988ebcbebd8..419d8e5a369 100644 --- a/storage/samples/snippets/storage_copy_file_archived_generation.py +++ b/storage/samples/snippets/storage_copy_file_archived_generation.py @@ -36,13 +36,22 @@ def copy_file_archived_generation( source_blob = source_bucket.blob(blob_name) destination_bucket = storage_client.bucket(destination_bucket_name) + # Optional: set a generation-match precondition to avoid potential race conditions + # and data corruptions. The request to copy is aborted if the object's + # generation number does not match your precondition. For a destination + # object that does not yet exist, set the if_generation_match precondition to 0. + # If the destination object already exists in your bucket, set instead a + # generation-match precondition using its generation number. + destination_generation_match_precondition = 0 + + # source_generation selects a specific revision of the source object, as opposed to the latest version. blob_copy = source_bucket.copy_blob( - source_blob, destination_bucket, destination_blob_name, source_generation=generation + source_blob, destination_bucket, destination_blob_name, source_generation=generation, if_generation_match=destination_generation_match_precondition ) print( "Generation {} of the blob {} in bucket {} copied to blob {} in bucket {}.".format( - source_blob.generation, + generation, source_blob.name, source_bucket.name, blob_copy.name, diff --git a/storage/samples/snippets/storage_delete_file.py b/storage/samples/snippets/storage_delete_file.py index b2997c86b40..427604145dd 100644 --- a/storage/samples/snippets/storage_delete_file.py +++ b/storage/samples/snippets/storage_delete_file.py @@ -29,7 +29,15 @@ def delete_blob(bucket_name, blob_name): bucket = storage_client.bucket(bucket_name) blob = bucket.blob(blob_name) - blob.delete() + generation_match_precondition = None + + # Optional: set a generation-match precondition to avoid potential race conditions + # and data corruptions. The request to delete is aborted if the object's + # generation number does not match your precondition. + blob.reload() # Fetch blob metadata to use in generation_match_precondition. + generation_match_precondition = blob.generation + + blob.delete(if_generation_match=generation_match_precondition) print(f"Blob {blob_name} deleted.") diff --git a/storage/samples/snippets/storage_move_file.py b/storage/samples/snippets/storage_move_file.py index a881a38bade..b2e5144d0b2 100644 --- a/storage/samples/snippets/storage_move_file.py +++ b/storage/samples/snippets/storage_move_file.py @@ -20,7 +20,7 @@ from google.cloud import storage -def move_blob(bucket_name, blob_name, destination_bucket_name, destination_blob_name): +def move_blob(bucket_name, blob_name, destination_bucket_name, destination_blob_name,): """Moves a blob from one bucket to another with a new name.""" # The ID of your GCS bucket # bucket_name = "your-bucket-name" @@ -37,8 +37,17 @@ def move_blob(bucket_name, blob_name, destination_bucket_name, destination_blob_ source_blob = source_bucket.blob(blob_name) destination_bucket = storage_client.bucket(destination_bucket_name) + # Optional: set a generation-match precondition to avoid potential race conditions + # and data corruptions. The request is aborted if the object's + # generation number does not match your precondition. For a destination + # object that does not yet exist, set the if_generation_match precondition to 0. + # If the destination object already exists in your bucket, set instead a + # generation-match precondition using its generation number. + # There is also an `if_source_generation_match` parameter, which is not used in this example. + destination_generation_match_precondition = 0 + blob_copy = source_bucket.copy_blob( - source_blob, destination_bucket, destination_blob_name + source_blob, destination_bucket, destination_blob_name, if_generation_match=destination_generation_match_precondition, ) source_bucket.delete_blob(blob_name) diff --git a/storage/samples/snippets/storage_object_csek_to_cmek.py b/storage/samples/snippets/storage_object_csek_to_cmek.py index 9d4d710bf50..9a915f08d63 100644 --- a/storage/samples/snippets/storage_object_csek_to_cmek.py +++ b/storage/samples/snippets/storage_object_csek_to_cmek.py @@ -33,12 +33,22 @@ def object_csek_to_cmek(bucket_name, blob_name, encryption_key, kms_key_name): current_encryption_key = base64.b64decode(encryption_key) source_blob = bucket.blob(blob_name, encryption_key=current_encryption_key) - destination_blob = bucket.blob(blob_name, kms_key_name=kms_key_name) - token, rewritten, total = destination_blob.rewrite(source_blob) + generation_match_precondition = None + token = None + + # Optional: set a generation-match precondition to avoid potential race conditions + # and data corruptions. The request to rewrite is aborted if the object's + # generation number does not match your precondition. + source_blob.reload() # Fetch blob metadata to use in generation_match_precondition. + generation_match_precondition = source_blob.generation - while token is not None: - token, rewritten, total = destination_blob.rewrite(source_blob, token=token) + while True: + token, bytes_rewritten, total_bytes = destination_blob.rewrite( + source_blob, token=token, if_generation_match=generation_match_precondition + ) + if token is None: + break print( "Blob {} in bucket {} is now managed by the KMS key {} instead of a customer-supplied encryption key".format( diff --git a/storage/samples/snippets/storage_rotate_encryption_key.py b/storage/samples/snippets/storage_rotate_encryption_key.py index 828b7d5ef31..174947b843e 100644 --- a/storage/samples/snippets/storage_rotate_encryption_key.py +++ b/storage/samples/snippets/storage_rotate_encryption_key.py @@ -42,12 +42,18 @@ def rotate_encryption_key( destination_blob = bucket.blob( blob_name, encryption_key=new_encryption_key ) - + generation_match_precondition = None token = None + # Optional: set a generation-match precondition to avoid potential race conditions + # and data corruptions. The request to rewrite is aborted if the object's + # generation number does not match your precondition. + source_blob.reload() # Fetch blob metadata to use in generation_match_precondition. + generation_match_precondition = source_blob.generation + while True: token, bytes_rewritten, total_bytes = destination_blob.rewrite( - source_blob, token=token + source_blob, token=token, if_generation_match=generation_match_precondition ) if token is None: break diff --git a/storage/samples/snippets/storage_upload_encrypted_file.py b/storage/samples/snippets/storage_upload_encrypted_file.py index 5f49872384e..08f58154e07 100644 --- a/storage/samples/snippets/storage_upload_encrypted_file.py +++ b/storage/samples/snippets/storage_upload_encrypted_file.py @@ -36,6 +36,10 @@ def upload_encrypted_blob( The file will be encrypted by Google Cloud Storage and only retrievable using the provided encryption key. """ + # bucket_name = "your-bucket-name" + # source_file_name = "local/path/to/file" + # destination_blob_name = "storage-object-name" + # base64_encryption_key = "TIbv/fjexq+VmtXzAlc63J4z5kFmWJ6NdAPQulQBT7g=" storage_client = storage.Client() bucket = storage_client.bucket(bucket_name) @@ -48,7 +52,15 @@ def upload_encrypted_blob( destination_blob_name, encryption_key=encryption_key ) - blob.upload_from_filename(source_file_name) + # Optional: set a generation-match precondition to avoid potential race conditions + # and data corruptions. The request to upload is aborted if the object's + # generation number does not match your precondition. For a destination + # object that does not yet exist, set the if_generation_match precondition to 0. + # If the destination object already exists in your bucket, set instead a + # generation-match precondition using its generation number. + generation_match_precondition = 0 + + blob.upload_from_filename(source_file_name, if_generation_match=generation_match_precondition) print( f"File {source_file_name} uploaded to {destination_blob_name}." diff --git a/storage/samples/snippets/storage_upload_file.py b/storage/samples/snippets/storage_upload_file.py index 8e7d98630e3..1e7ceda5eb4 100644 --- a/storage/samples/snippets/storage_upload_file.py +++ b/storage/samples/snippets/storage_upload_file.py @@ -33,7 +33,15 @@ def upload_blob(bucket_name, source_file_name, destination_blob_name): bucket = storage_client.bucket(bucket_name) blob = bucket.blob(destination_blob_name) - blob.upload_from_filename(source_file_name) + # Optional: set a generation-match precondition to avoid potential race conditions + # and data corruptions. The request to upload is aborted if the object's + # generation number does not match your precondition. For a destination + # object that does not yet exist, set the if_generation_match precondition to 0. + # If the destination object already exists in your bucket, set instead a + # generation-match precondition using its generation number. + generation_match_precondition = 0 + + blob.upload_from_filename(source_file_name, if_generation_match=generation_match_precondition) print( f"File {source_file_name} uploaded to {destination_blob_name}." diff --git a/storage/samples/snippets/storage_upload_with_kms_key.py b/storage/samples/snippets/storage_upload_with_kms_key.py index e83c10aea19..6e8fe039404 100644 --- a/storage/samples/snippets/storage_upload_with_kms_key.py +++ b/storage/samples/snippets/storage_upload_with_kms_key.py @@ -21,7 +21,7 @@ def upload_blob_with_kms( - bucket_name, source_file_name, destination_blob_name, kms_key_name + bucket_name, source_file_name, destination_blob_name, kms_key_name, ): """Uploads a file to the bucket, encrypting it with the given KMS key.""" # bucket_name = "your-bucket-name" @@ -32,7 +32,16 @@ def upload_blob_with_kms( storage_client = storage.Client() bucket = storage_client.bucket(bucket_name) blob = bucket.blob(destination_blob_name, kms_key_name=kms_key_name) - blob.upload_from_filename(source_file_name) + + # Optional: set a generation-match precondition to avoid potential race conditions + # and data corruptions. The request to upload is aborted if the object's + # generation number does not match your precondition. For a destination + # object that does not yet exist, set the if_generation_match precondition to 0. + # If the destination object already exists in your bucket, set instead a + # generation-match precondition using its generation number. + generation_match_precondition = 0 + + blob.upload_from_filename(source_file_name, if_generation_match=generation_match_precondition) print( "File {} uploaded to {} with encryption key {}.".format( From a09765e72a13ace9e83e7ef3c6f6e229821f4fbd Mon Sep 17 00:00:00 2001 From: cojenco Date: Tue, 7 Feb 2023 14:52:27 -0800 Subject: [PATCH 094/172] samples: add metageneration-match preconditions to samples (#975) * samples: add metageneration-match preconditions to samples * update optional block wording and flow --- .../samples/snippets/storage_release_event_based_hold.py | 9 ++++++++- .../samples/snippets/storage_release_temporary_hold.py | 9 ++++++++- storage/samples/snippets/storage_set_event_based_hold.py | 9 ++++++++- storage/samples/snippets/storage_set_metadata.py | 9 ++++++++- storage/samples/snippets/storage_set_temporary_hold.py | 9 ++++++++- 5 files changed, 40 insertions(+), 5 deletions(-) diff --git a/storage/samples/snippets/storage_release_event_based_hold.py b/storage/samples/snippets/storage_release_event_based_hold.py index 1db637cd9e7..6b4a2ccb51c 100644 --- a/storage/samples/snippets/storage_release_event_based_hold.py +++ b/storage/samples/snippets/storage_release_event_based_hold.py @@ -29,9 +29,16 @@ def release_event_based_hold(bucket_name, blob_name): storage_client = storage.Client() bucket = storage_client.bucket(bucket_name) blob = bucket.blob(blob_name) + metageneration_match_precondition = None + + # Optional: set a metageneration-match precondition to avoid potential race + # conditions and data corruptions. The request to patch is aborted if the + # object's metageneration does not match your precondition. + blob.reload() # Fetch blob metadata to use in metageneration_match_precondition. + metageneration_match_precondition = blob.metageneration blob.event_based_hold = False - blob.patch() + blob.patch(if_metageneration_match=metageneration_match_precondition) print(f"Event based hold was released for {blob_name}") diff --git a/storage/samples/snippets/storage_release_temporary_hold.py b/storage/samples/snippets/storage_release_temporary_hold.py index 02a6ca96c06..64c7607c182 100644 --- a/storage/samples/snippets/storage_release_temporary_hold.py +++ b/storage/samples/snippets/storage_release_temporary_hold.py @@ -29,9 +29,16 @@ def release_temporary_hold(bucket_name, blob_name): storage_client = storage.Client() bucket = storage_client.bucket(bucket_name) blob = bucket.blob(blob_name) + metageneration_match_precondition = None + + # Optional: set a metageneration-match precondition to avoid potential race + # conditions and data corruptions. The request to patch is aborted if the + # object's metageneration does not match your precondition. + blob.reload() # Fetch blob metadata to use in metageneration_match_precondition. + metageneration_match_precondition = blob.metageneration blob.temporary_hold = False - blob.patch() + blob.patch(if_metageneration_match=metageneration_match_precondition) print("Temporary hold was release for #{blob_name}") diff --git a/storage/samples/snippets/storage_set_event_based_hold.py b/storage/samples/snippets/storage_set_event_based_hold.py index e04ed7552a9..76f7fd7eee4 100644 --- a/storage/samples/snippets/storage_set_event_based_hold.py +++ b/storage/samples/snippets/storage_set_event_based_hold.py @@ -28,9 +28,16 @@ def set_event_based_hold(bucket_name, blob_name): storage_client = storage.Client() bucket = storage_client.bucket(bucket_name) blob = bucket.blob(blob_name) + metageneration_match_precondition = None + + # Optional: set a metageneration-match precondition to avoid potential race + # conditions and data corruptions. The request to patch is aborted if the + # object's metageneration does not match your precondition. + blob.reload() # Fetch blob metadata to use in metageneration_match_precondition. + metageneration_match_precondition = blob.metageneration blob.event_based_hold = True - blob.patch() + blob.patch(if_metageneration_match=metageneration_match_precondition) print(f"Event based hold was set for {blob_name}") diff --git a/storage/samples/snippets/storage_set_metadata.py b/storage/samples/snippets/storage_set_metadata.py index 90b6838c026..6a4a9fb9e08 100644 --- a/storage/samples/snippets/storage_set_metadata.py +++ b/storage/samples/snippets/storage_set_metadata.py @@ -28,9 +28,16 @@ def set_blob_metadata(bucket_name, blob_name): storage_client = storage.Client() bucket = storage_client.bucket(bucket_name) blob = bucket.get_blob(blob_name) + metageneration_match_precondition = None + + # Optional: set a metageneration-match precondition to avoid potential race + # conditions and data corruptions. The request to patch is aborted if the + # object's metageneration does not match your precondition. + metageneration_match_precondition = blob.metageneration + metadata = {'color': 'Red', 'name': 'Test'} blob.metadata = metadata - blob.patch() + blob.patch(if_metageneration_match=metageneration_match_precondition) print(f"The metadata for the blob {blob.name} is {blob.metadata}") diff --git a/storage/samples/snippets/storage_set_temporary_hold.py b/storage/samples/snippets/storage_set_temporary_hold.py index edeb3c57840..a91521bcc11 100644 --- a/storage/samples/snippets/storage_set_temporary_hold.py +++ b/storage/samples/snippets/storage_set_temporary_hold.py @@ -28,9 +28,16 @@ def set_temporary_hold(bucket_name, blob_name): storage_client = storage.Client() bucket = storage_client.bucket(bucket_name) blob = bucket.blob(blob_name) + metageneration_match_precondition = None + + # Optional: set a metageneration-match precondition to avoid potential race + # conditions and data corruptions. The request to patch is aborted if the + # object's metageneration does not match your precondition. + blob.reload() # Fetch blob metadata to use in metageneration_match_precondition. + metageneration_match_precondition = blob.metageneration blob.temporary_hold = True - blob.patch() + blob.patch(if_metageneration_match=metageneration_match_precondition) print("Temporary hold was set for #{blob_name}") From 023372c151069275467da7e8e07e4d7b9acb24d3 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Wed, 15 Feb 2023 22:42:56 +0000 Subject: [PATCH 095/172] chore(deps): update dependency google-cloud-pubsub to v2.14.1 (#987) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index b13cef2bfb0..1ab012699f4 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.14.0 +google-cloud-pubsub==2.14.1 google-cloud-storage==2.7.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.3; python_version >= '3.8' From 8ab39e6ce5cdca928f61708eba156a50227f50cc Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Thu, 23 Feb 2023 21:49:53 +0000 Subject: [PATCH 096/172] chore(deps): update dependency google-cloud-pubsub to v2.15.0 (#995) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 1ab012699f4..19b0fdc99a8 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.14.1 +google-cloud-pubsub==2.15.0 google-cloud-storage==2.7.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.3; python_version >= '3.8' From 6f80a7fce818f6129e0147e56e0aac5b41f72ee3 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Sat, 4 Mar 2023 11:32:41 +0000 Subject: [PATCH 097/172] chore(deps): update dependency pytest to v7.2.2 (#999) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 51c1be2e642..2e805e1f8b2 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.2.1 +pytest==7.2.2 mock==5.0.1 backoff==2.2.1 \ No newline at end of file From ce659f96d9676a155cd21e5d3988b875f44d2190 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Thu, 23 Mar 2023 20:50:00 +0000 Subject: [PATCH 098/172] chore(deps): update dependency google-cloud-pubsub to v2.15.1 (#1007) Co-authored-by: cojenco --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 19b0fdc99a8..15ab153a28f 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.15.0 +google-cloud-pubsub==2.15.1 google-cloud-storage==2.7.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.3; python_version >= '3.8' From dc231e49fd6d7228ed839a53f042a9391fbb4923 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 28 Mar 2023 19:24:33 +0100 Subject: [PATCH 099/172] chore(deps): update dependency google-cloud-pubsub to v2.15.2 (#1009) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 15ab153a28f..1e06ff2d22f 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.15.1 +google-cloud-pubsub==2.15.2 google-cloud-storage==2.7.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.3; python_version >= '3.8' From aa76305d2a9b4beb6d11a7fbbe7d6d2c69f32af6 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Thu, 30 Mar 2023 18:41:51 +0100 Subject: [PATCH 100/172] chore(deps): update dependency google-cloud-storage to v2.8.0 (#1011) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 1e06ff2d22f..ae3bfecf510 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.15.2 -google-cloud-storage==2.7.0 +google-cloud-storage==2.8.0 pandas===1.3.5; python_version == '3.7' pandas==1.5.3; python_version >= '3.8' From 3309f31e4114f461824afc0a2b055818c33e3e6c Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Wed, 12 Apr 2023 22:21:03 +0100 Subject: [PATCH 101/172] chore(deps): update all dependencies (#1015) --- storage/samples/snippets/requirements-test.txt | 2 +- storage/samples/snippets/requirements.txt | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 2e805e1f8b2..be69289c139 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.2.2 +pytest==7.3.0 mock==5.0.1 backoff==2.2.1 \ No newline at end of file diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index ae3bfecf510..3c3f29db9e0 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.15.2 +google-cloud-pubsub==2.16.0 google-cloud-storage==2.8.0 pandas===1.3.5; python_version == '3.7' -pandas==1.5.3; python_version >= '3.8' +pandas==2.0.0; python_version >= '3.8' From 4c8bbd09e72a2621b85ae806f6920756de034391 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 18 Apr 2023 19:00:14 +0200 Subject: [PATCH 102/172] chore(deps): update dependency pytest to v7.3.1 (#1018) Co-authored-by: cojenco --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index be69289c139..4358edb87ee 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.3.0 +pytest==7.3.1 mock==5.0.1 backoff==2.2.1 \ No newline at end of file From 558d91ad1abb86e87fb27fbf3fd2682d15c531b5 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Wed, 19 Apr 2023 15:44:28 +0200 Subject: [PATCH 103/172] chore(deps): update dependency mock to v5.0.2 (#1019) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 4358edb87ee..0068826c5d3 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ pytest==7.3.1 -mock==5.0.1 +mock==5.0.2 backoff==2.2.1 \ No newline at end of file From a50d6cb5dcd1a1ea3dda5063fa6b68d9fd56addc Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Wed, 26 Apr 2023 19:40:39 +0200 Subject: [PATCH 104/172] chore(deps): update dependency pandas to v2.0.1 (#1021) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 3c3f29db9e0..c55ebd518fc 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.16.0 google-cloud-storage==2.8.0 pandas===1.3.5; python_version == '3.7' -pandas==2.0.0; python_version >= '3.8' +pandas==2.0.1; python_version >= '3.8' From 5317112c359d229c96c50c880a5929487f3f027a Mon Sep 17 00:00:00 2001 From: cojenco Date: Tue, 2 May 2023 13:13:41 -0700 Subject: [PATCH 105/172] chore: update samples testing and readme (#1022) * chore: update samples testing and readme * update version --------- Co-authored-by: Andrew Gorcester --- storage/samples/snippets/requester_pays_test.py | 6 ++++++ storage/samples/snippets/snippets_test.py | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/storage/samples/snippets/requester_pays_test.py b/storage/samples/snippets/requester_pays_test.py index cf8c2d09799..4bef0cb8968 100644 --- a/storage/samples/snippets/requester_pays_test.py +++ b/storage/samples/snippets/requester_pays_test.py @@ -12,9 +12,11 @@ # See the License for the specific language governing permissions and # limitations under the License. +import backoff import os import tempfile +from google.api_core.exceptions import GoogleAPIError from google.cloud import storage import pytest @@ -31,18 +33,21 @@ PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"] +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_enable_requester_pays(capsys): storage_enable_requester_pays.enable_requester_pays(BUCKET) out, _ = capsys.readouterr() assert f"Requester Pays has been enabled for {BUCKET}" in out +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_disable_requester_pays(capsys): storage_disable_requester_pays.disable_requester_pays(BUCKET) out, _ = capsys.readouterr() assert f"Requester Pays has been disabled for {BUCKET}" in out +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_get_requester_pays_status(capsys): storage_get_requester_pays_status.get_requester_pays_status(BUCKET) out, _ = capsys.readouterr() @@ -58,6 +63,7 @@ def test_blob(): return blob +@backoff.on_exception(backoff.expo, GoogleAPIError, max_time=60) def test_download_file_requester_pays(test_blob, capsys): with tempfile.NamedTemporaryFile() as dest_file: storage_download_file_requester_pays.download_file_requester_pays( diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 57751be6076..ee6f790f29c 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -78,7 +78,7 @@ import storage_upload_from_stream import storage_upload_with_kms_key -KMS_KEY = os.environ["CLOUD_KMS_KEY"] +KMS_KEY = os.environ.get("CLOUD_KMS_KEY") def test_enable_default_kms_key(test_bucket): From 3e0db08b8d9aab47bff5218b24d76e264b168c68 Mon Sep 17 00:00:00 2001 From: MiaCY <97990237+MiaCY@users.noreply.github.com> Date: Tue, 2 May 2023 14:41:01 -0700 Subject: [PATCH 106/172] docs: add sample and sample test for transfer manager (#1027) * add sample and sample test for transfer manager download blob as chunks concurrently method * chore: modify format for int * chore: refactor transfer manager sample names and tests --------- Co-authored-by: Andrew Gorcester --- storage/samples/snippets/snippets_test.py | 34 +++- .../snippets/storage_transfer_manager.py | 184 ------------------ ...age_transfer_manager_download_all_blobs.py | 65 +++++++ ...er_manager_download_chunks_concurrently.py | 44 +++++ ...orage_transfer_manager_upload_directory.py | 79 ++++++++ ...rage_transfer_manager_upload_many_blobs.py | 66 +++++++ 6 files changed, 284 insertions(+), 188 deletions(-) delete mode 100644 storage/samples/snippets/storage_transfer_manager.py create mode 100644 storage/samples/snippets/storage_transfer_manager_download_all_blobs.py create mode 100644 storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py create mode 100644 storage/samples/snippets/storage_transfer_manager_upload_directory.py create mode 100644 storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index ee6f790f29c..6be8e176765 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -72,7 +72,10 @@ import storage_set_bucket_default_kms_key import storage_set_client_endpoint import storage_set_metadata -import storage_transfer_manager +import storage_transfer_manager_download_all_blobs +import storage_transfer_manager_download_chunks_concurrently +import storage_transfer_manager_upload_directory +import storage_transfer_manager_upload_many_blobs import storage_upload_file import storage_upload_from_memory import storage_upload_from_stream @@ -686,7 +689,7 @@ def test_transfer_manager_snippets(test_bucket, capsys): with open(os.path.join(uploads, name), "w") as f: f.write(name) - storage_transfer_manager.upload_many_blobs_with_transfer_manager( + storage_transfer_manager_upload_many_blobs.upload_many_blobs_with_transfer_manager( test_bucket.name, BLOB_NAMES, source_directory="{}/".format(uploads), @@ -699,7 +702,7 @@ def test_transfer_manager_snippets(test_bucket, capsys): with tempfile.TemporaryDirectory() as downloads: # Download the files. - storage_transfer_manager.download_all_blobs_with_transfer_manager( + storage_transfer_manager_download_all_blobs.download_all_blobs_with_transfer_manager( test_bucket.name, destination_directory=os.path.join(downloads, ""), threads=2, @@ -729,7 +732,7 @@ def test_transfer_manager_directory_upload(test_bucket, capsys): with open(os.path.join(uploads, name), "w") as f: f.write(name) - storage_transfer_manager.upload_directory_with_transfer_manager( + storage_transfer_manager_upload_directory.upload_directory_with_transfer_manager( test_bucket.name, source_directory="{}/".format(uploads) ) out, _ = capsys.readouterr() @@ -737,3 +740,26 @@ def test_transfer_manager_directory_upload(test_bucket, capsys): assert "Found {}".format(len(BLOB_NAMES)) in out for name in BLOB_NAMES: assert "Uploaded {}".format(name) in out + + +def test_transfer_manager_download_chunks_concurrently(test_bucket, capsys): + BLOB_NAME = "test_file.txt" + + with tempfile.NamedTemporaryFile() as file: + file.write(b"test") + + storage_upload_file.upload_blob( + test_bucket.name, file.name, BLOB_NAME + ) + + with tempfile.TemporaryDirectory() as downloads: + # Download the file. + storage_transfer_manager_download_chunks_concurrently.download_chunks_concurrently( + test_bucket.name, + BLOB_NAME, + os.path.join(downloads, BLOB_NAME), + processes=8, + ) + out, _ = capsys.readouterr() + + assert "Downloaded {} to {}".format(BLOB_NAME, os.path.join(downloads, BLOB_NAME)) in out diff --git a/storage/samples/snippets/storage_transfer_manager.py b/storage/samples/snippets/storage_transfer_manager.py deleted file mode 100644 index 0a02b96e338..00000000000 --- a/storage/samples/snippets/storage_transfer_manager.py +++ /dev/null @@ -1,184 +0,0 @@ -# Copyright 2022 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the 'License'); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - - -def upload_many_blobs_with_transfer_manager( - bucket_name, filenames, source_directory="", threads=4 -): - """Upload every file in a list to a bucket, concurrently in a thread pool. - - Each blob name is derived from the filename, not including the - `source_directory` parameter. For complete control of the blob name for each - file (and other aspects of individual blob metadata), use - transfer_manager.upload_many() instead. - """ - - # The ID of your GCS bucket - # bucket_name = "your-bucket-name" - - # A list (or other iterable) of filenames to upload. - # filenames = ["file_1.txt", "file_2.txt"] - - # The directory on your computer that is the root of all of the files in the - # list of filenames. This string is prepended (with os.path.join()) to each - # filename to get the full path to the file. Relative paths and absolute - # paths are both accepted. This string is not included in the name of the - # uploaded blob; it is only used to find the source files. An empty string - # means "the current working directory". Note that this parameter allows - # directory traversal (e.g. "/", "../") and is not intended for unsanitized - # end user input. - # source_directory="" - - # The number of threads to use for the operation. The performance impact of - # this value depends on the use case, but generally, smaller files benefit - # from more threads and larger files don't benefit from more threads. Too - # many threads can slow operations, especially with large files, due to - # contention over the Python GIL. - # threads=4 - - from google.cloud.storage import Client, transfer_manager - - storage_client = Client() - bucket = storage_client.bucket(bucket_name) - - results = transfer_manager.upload_many_from_filenames( - bucket, filenames, source_directory=source_directory, threads=threads - ) - - for name, result in zip(filenames, results): - # The results list is either `None` or an exception for each filename in - # the input list, in order. - - if isinstance(result, Exception): - print("Failed to upload {} due to exception: {}".format(name, result)) - else: - print("Uploaded {} to {}.".format(name, bucket.name)) - - -def upload_directory_with_transfer_manager(bucket_name, source_directory, threads=4): - """Upload every file in a directory, including all files in subdirectories. - - Each blob name is derived from the filename, not including the `directory` - parameter itself. For complete control of the blob name for each file (and - other aspects of individual blob metadata), use - transfer_manager.upload_many() instead. - """ - - # The ID of your GCS bucket - # bucket_name = "your-bucket-name" - - # The directory on your computer to upload. Files in the directory and its - # subdirectories will be uploaded. An empty string means "the current - # working directory". - # source_directory="" - - # The number of threads to use for the operation. The performance impact of - # this value depends on the use case, but generally, smaller files benefit - # from more threads and larger files don't benefit from more threads. Too - # many threads can slow operations, especially with large files, due to - # contention over the Python GIL. - # threads=4 - - from pathlib import Path - - from google.cloud.storage import Client, transfer_manager - - storage_client = Client() - bucket = storage_client.bucket(bucket_name) - - # Generate a list of paths (in string form) relative to the `directory`. - # This can be done in a single list comprehension, but is expanded into - # multiple lines here for clarity. - - # First, recursively get all files in `directory` as Path objects. - directory_as_path_obj = Path(source_directory) - paths = directory_as_path_obj.rglob("*") - - # Filter so the list only includes files, not directories themselves. - file_paths = [path for path in paths if path.is_file()] - - # These paths are relative to the current working directory. Next, make them - # relative to `directory` - relative_paths = [path.relative_to(source_directory) for path in file_paths] - - # Finally, convert them all to strings. - string_paths = [str(path) for path in relative_paths] - - print("Found {} files.".format(len(string_paths))) - - # Start the upload. - results = transfer_manager.upload_many_from_filenames( - bucket, string_paths, source_directory=source_directory, threads=threads - ) - - for name, result in zip(string_paths, results): - # The results list is either `None` or an exception for each filename in - # the input list, in order. - - if isinstance(result, Exception): - print("Failed to upload {} due to exception: {}".format(name, result)) - else: - print("Uploaded {} to {}.".format(name, bucket.name)) - - -def download_all_blobs_with_transfer_manager( - bucket_name, destination_directory="", threads=4 -): - """Download all of the blobs in a bucket, concurrently in a thread pool. - - The filename of each blob once downloaded is derived from the blob name and - the `destination_directory `parameter. For complete control of the filename - of each blob, use transfer_manager.download_many() instead. - - Directories will be created automatically as needed, for instance to - accommodate blob names that include slashes. - """ - - # The ID of your GCS bucket - # bucket_name = "your-bucket-name" - - # The directory on your computer to which to download all of the files. This - # string is prepended (with os.path.join()) to the name of each blob to form - # the full path. Relative paths and absolute paths are both accepted. An - # empty string means "the current working directory". Note that this - # parameter allows accepts directory traversal ("../" etc.) and is not - # intended for unsanitized end user input. - # destination_directory = "" - - # The number of threads to use for the operation. The performance impact of - # this value depends on the use case, but generally, smaller files benefit - # from more threads and larger files don't benefit from more threads. Too - # many threads can slow operations, especially with large files, due to - # contention over the Python GIL. - # threads=4 - - from google.cloud.storage import Client, transfer_manager - - storage_client = Client() - bucket = storage_client.bucket(bucket_name) - - blob_names = [blob.name for blob in bucket.list_blobs()] - - results = transfer_manager.download_many_to_path( - bucket, blob_names, destination_directory=destination_directory, threads=threads - ) - - for name, result in zip(blob_names, results): - # The results list is either `None` or an exception for each blob in - # the input list, in order. - - if isinstance(result, Exception): - print("Failed to download {} due to exception: {}".format(name, result)) - else: - print("Downloaded {} to {}.".format(name, destination_directory + name)) diff --git a/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py b/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py new file mode 100644 index 00000000000..b07739d2077 --- /dev/null +++ b/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py @@ -0,0 +1,65 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +def download_all_blobs_with_transfer_manager( + bucket_name, destination_directory="", threads=4 +): + """Download all of the blobs in a bucket, concurrently in a thread pool. + + The filename of each blob once downloaded is derived from the blob name and + the `destination_directory `parameter. For complete control of the filename + of each blob, use transfer_manager.download_many() instead. + + Directories will be created automatically as needed, for instance to + accommodate blob names that include slashes. + """ + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The directory on your computer to which to download all of the files. This + # string is prepended (with os.path.join()) to the name of each blob to form + # the full path. Relative paths and absolute paths are both accepted. An + # empty string means "the current working directory". Note that this + # parameter allows accepts directory traversal ("../" etc.) and is not + # intended for unsanitized end user input. + # destination_directory = "" + + # The number of threads to use for the operation. The performance impact of + # this value depends on the use case, but generally, smaller files benefit + # from more threads and larger files don't benefit from more threads. Too + # many threads can slow operations, especially with large files, due to + # contention over the Python GIL. + # threads=4 + + from google.cloud.storage import Client, transfer_manager + + storage_client = Client() + bucket = storage_client.bucket(bucket_name) + + blob_names = [blob.name for blob in bucket.list_blobs()] + + results = transfer_manager.download_many_to_path( + bucket, blob_names, destination_directory=destination_directory, threads=threads + ) + + for name, result in zip(blob_names, results): + # The results list is either `None` or an exception for each blob in + # the input list, in order. + + if isinstance(result, Exception): + print("Failed to download {} due to exception: {}".format(name, result)) + else: + print("Downloaded {} to {}.".format(name, destination_directory + name)) diff --git a/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py b/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py new file mode 100644 index 00000000000..633c5ae651a --- /dev/null +++ b/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py @@ -0,0 +1,44 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +def download_chunks_concurrently(bucket_name, blob_name, filename, processes=8): + """Download a single file in chunks, concurrently.""" + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The file to be downloaded + # blob_name = "target-file" + + # The destination filename or path + # filename = "" + + # The maximum number of worker processes that should be used to handle the + # workload of downloading the blob concurrently. PROCESS worker type uses more + # system resources (both memory and CPU) and can result in faster operations + # when working with large files. The optimal number of workers depends heavily + # on the specific use case. Refer to the docstring of the underlining method + # for more details. + # processes=8 + + from google.cloud.storage import Client, transfer_manager + + storage_client = Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + + transfer_manager.download_chunks_concurrently(blob, filename, max_workers=processes) + + print("Downloaded {} to {}.".format(blob_name, filename)) diff --git a/storage/samples/snippets/storage_transfer_manager_upload_directory.py b/storage/samples/snippets/storage_transfer_manager_upload_directory.py new file mode 100644 index 00000000000..6f5171c5468 --- /dev/null +++ b/storage/samples/snippets/storage_transfer_manager_upload_directory.py @@ -0,0 +1,79 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +def upload_directory_with_transfer_manager(bucket_name, source_directory, threads=4): + """Upload every file in a directory, including all files in subdirectories. + + Each blob name is derived from the filename, not including the `directory` + parameter itself. For complete control of the blob name for each file (and + other aspects of individual blob metadata), use + transfer_manager.upload_many() instead. + """ + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The directory on your computer to upload. Files in the directory and its + # subdirectories will be uploaded. An empty string means "the current + # working directory". + # source_directory="" + + # The number of threads to use for the operation. The performance impact of + # this value depends on the use case, but generally, smaller files benefit + # from more threads and larger files don't benefit from more threads. Too + # many threads can slow operations, especially with large files, due to + # contention over the Python GIL. + # threads=4 + + from pathlib import Path + + from google.cloud.storage import Client, transfer_manager + + storage_client = Client() + bucket = storage_client.bucket(bucket_name) + + # Generate a list of paths (in string form) relative to the `directory`. + # This can be done in a single list comprehension, but is expanded into + # multiple lines here for clarity. + + # First, recursively get all files in `directory` as Path objects. + directory_as_path_obj = Path(source_directory) + paths = directory_as_path_obj.rglob("*") + + # Filter so the list only includes files, not directories themselves. + file_paths = [path for path in paths if path.is_file()] + + # These paths are relative to the current working directory. Next, make them + # relative to `directory` + relative_paths = [path.relative_to(source_directory) for path in file_paths] + + # Finally, convert them all to strings. + string_paths = [str(path) for path in relative_paths] + + print("Found {} files.".format(len(string_paths))) + + # Start the upload. + results = transfer_manager.upload_many_from_filenames( + bucket, string_paths, source_directory=source_directory, threads=threads + ) + + for name, result in zip(string_paths, results): + # The results list is either `None` or an exception for each filename in + # the input list, in order. + + if isinstance(result, Exception): + print("Failed to upload {} due to exception: {}".format(name, result)) + else: + print("Uploaded {} to {}.".format(name, bucket.name)) diff --git a/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py b/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py new file mode 100644 index 00000000000..995571b22dd --- /dev/null +++ b/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py @@ -0,0 +1,66 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +def upload_many_blobs_with_transfer_manager( + bucket_name, filenames, source_directory="", threads=4 +): + """Upload every file in a list to a bucket, concurrently in a thread pool. + + Each blob name is derived from the filename, not including the + `source_directory` parameter. For complete control of the blob name for each + file (and other aspects of individual blob metadata), use + transfer_manager.upload_many() instead. + """ + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # A list (or other iterable) of filenames to upload. + # filenames = ["file_1.txt", "file_2.txt"] + + # The directory on your computer that is the root of all of the files in the + # list of filenames. This string is prepended (with os.path.join()) to each + # filename to get the full path to the file. Relative paths and absolute + # paths are both accepted. This string is not included in the name of the + # uploaded blob; it is only used to find the source files. An empty string + # means "the current working directory". Note that this parameter allows + # directory traversal (e.g. "/", "../") and is not intended for unsanitized + # end user input. + # source_directory="" + + # The number of threads to use for the operation. The performance impact of + # this value depends on the use case, but generally, smaller files benefit + # from more threads and larger files don't benefit from more threads. Too + # many threads can slow operations, especially with large files, due to + # contention over the Python GIL. + # threads=4 + + from google.cloud.storage import Client, transfer_manager + + storage_client = Client() + bucket = storage_client.bucket(bucket_name) + + results = transfer_manager.upload_many_from_filenames( + bucket, filenames, source_directory=source_directory, threads=threads + ) + + for name, result in zip(filenames, results): + # The results list is either `None` or an exception for each filename in + # the input list, in order. + + if isinstance(result, Exception): + print("Failed to upload {} due to exception: {}".format(name, result)) + else: + print("Uploaded {} to {}.".format(name, bucket.name)) From 266ed8d2c15c93dce10b1d4ff28ea95f100676f3 Mon Sep 17 00:00:00 2001 From: MiaCY <97990237+MiaCY@users.noreply.github.com> Date: Thu, 4 May 2023 10:14:15 -0700 Subject: [PATCH 107/172] docs: remove threads in transfer manager samples (#1029) * docs: remove threads in transfer manager samples * omit worker type in transfer manager sample processes comments --------- Co-authored-by: Andrew Gorcester --- storage/samples/snippets/snippets_test.py | 4 ++-- ...storage_transfer_manager_download_all_blobs.py | 15 +++++++-------- ...ansfer_manager_download_chunks_concurrently.py | 10 ++++------ .../storage_transfer_manager_upload_directory.py | 15 +++++++-------- .../storage_transfer_manager_upload_many_blobs.py | 15 +++++++-------- 5 files changed, 27 insertions(+), 32 deletions(-) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 6be8e176765..6e5879eeb78 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -693,7 +693,7 @@ def test_transfer_manager_snippets(test_bucket, capsys): test_bucket.name, BLOB_NAMES, source_directory="{}/".format(uploads), - threads=2, + processes=8, ) out, _ = capsys.readouterr() @@ -705,7 +705,7 @@ def test_transfer_manager_snippets(test_bucket, capsys): storage_transfer_manager_download_all_blobs.download_all_blobs_with_transfer_manager( test_bucket.name, destination_directory=os.path.join(downloads, ""), - threads=2, + processes=8, ) out, _ = capsys.readouterr() diff --git a/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py b/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py index b07739d2077..2285f673f1d 100644 --- a/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py +++ b/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py @@ -14,7 +14,7 @@ def download_all_blobs_with_transfer_manager( - bucket_name, destination_directory="", threads=4 + bucket_name, destination_directory="", processes=8 ): """Download all of the blobs in a bucket, concurrently in a thread pool. @@ -37,12 +37,11 @@ def download_all_blobs_with_transfer_manager( # intended for unsanitized end user input. # destination_directory = "" - # The number of threads to use for the operation. The performance impact of - # this value depends on the use case, but generally, smaller files benefit - # from more threads and larger files don't benefit from more threads. Too - # many threads can slow operations, especially with large files, due to - # contention over the Python GIL. - # threads=4 + # The maximum number of processes to use for the operation. The performance + # impact of this value depends on the use case, but smaller files usually + # benefit from a higher number of processes. Each additional process occupies + # some CPU and memory resources until finished. + # processes=8 from google.cloud.storage import Client, transfer_manager @@ -52,7 +51,7 @@ def download_all_blobs_with_transfer_manager( blob_names = [blob.name for blob in bucket.list_blobs()] results = transfer_manager.download_many_to_path( - bucket, blob_names, destination_directory=destination_directory, threads=threads + bucket, blob_names, destination_directory=destination_directory, max_workers=processes ) for name, result in zip(blob_names, results): diff --git a/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py b/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py index 633c5ae651a..50541fb9353 100644 --- a/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py +++ b/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py @@ -25,12 +25,10 @@ def download_chunks_concurrently(bucket_name, blob_name, filename, processes=8): # The destination filename or path # filename = "" - # The maximum number of worker processes that should be used to handle the - # workload of downloading the blob concurrently. PROCESS worker type uses more - # system resources (both memory and CPU) and can result in faster operations - # when working with large files. The optimal number of workers depends heavily - # on the specific use case. Refer to the docstring of the underlining method - # for more details. + # The maximum number of processes to use for the operation. The performance + # impact of this value depends on the use case, but smaller files usually + # benefit from a higher number of processes. Each additional process occupies + # some CPU and memory resources until finished. # processes=8 from google.cloud.storage import Client, transfer_manager diff --git a/storage/samples/snippets/storage_transfer_manager_upload_directory.py b/storage/samples/snippets/storage_transfer_manager_upload_directory.py index 6f5171c5468..e4a369969bf 100644 --- a/storage/samples/snippets/storage_transfer_manager_upload_directory.py +++ b/storage/samples/snippets/storage_transfer_manager_upload_directory.py @@ -13,7 +13,7 @@ # limitations under the License. -def upload_directory_with_transfer_manager(bucket_name, source_directory, threads=4): +def upload_directory_with_transfer_manager(bucket_name, source_directory, processes=8): """Upload every file in a directory, including all files in subdirectories. Each blob name is derived from the filename, not including the `directory` @@ -30,12 +30,11 @@ def upload_directory_with_transfer_manager(bucket_name, source_directory, thread # working directory". # source_directory="" - # The number of threads to use for the operation. The performance impact of - # this value depends on the use case, but generally, smaller files benefit - # from more threads and larger files don't benefit from more threads. Too - # many threads can slow operations, especially with large files, due to - # contention over the Python GIL. - # threads=4 + # The maximum number of processes to use for the operation. The performance + # impact of this value depends on the use case, but smaller files usually + # benefit from a higher number of processes. Each additional process occupies + # some CPU and memory resources until finished. + # processes=8 from pathlib import Path @@ -66,7 +65,7 @@ def upload_directory_with_transfer_manager(bucket_name, source_directory, thread # Start the upload. results = transfer_manager.upload_many_from_filenames( - bucket, string_paths, source_directory=source_directory, threads=threads + bucket, string_paths, source_directory=source_directory, max_workers=processes ) for name, result in zip(string_paths, results): diff --git a/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py b/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py index 995571b22dd..600134bd6b8 100644 --- a/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py +++ b/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py @@ -14,7 +14,7 @@ def upload_many_blobs_with_transfer_manager( - bucket_name, filenames, source_directory="", threads=4 + bucket_name, filenames, source_directory="", processes=8 ): """Upload every file in a list to a bucket, concurrently in a thread pool. @@ -40,12 +40,11 @@ def upload_many_blobs_with_transfer_manager( # end user input. # source_directory="" - # The number of threads to use for the operation. The performance impact of - # this value depends on the use case, but generally, smaller files benefit - # from more threads and larger files don't benefit from more threads. Too - # many threads can slow operations, especially with large files, due to - # contention over the Python GIL. - # threads=4 + # The maximum number of processes to use for the operation. The performance + # impact of this value depends on the use case, but smaller files usually + # benefit from a higher number of processes. Each additional process occupies + # some CPU and memory resources until finished. + # processes=8 from google.cloud.storage import Client, transfer_manager @@ -53,7 +52,7 @@ def upload_many_blobs_with_transfer_manager( bucket = storage_client.bucket(bucket_name) results = transfer_manager.upload_many_from_filenames( - bucket, filenames, source_directory=source_directory, threads=threads + bucket, filenames, source_directory=source_directory, max_workers=processes ) for name, result in zip(filenames, results): From 6e3d60036a3391fa125019b1c70053c59c2d6648 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Fri, 5 May 2023 01:05:24 +0200 Subject: [PATCH 108/172] chore(deps): update dependency google-cloud-storage to v2.9.0 (#1032) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index c55ebd518fc..c0e68145021 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.16.0 -google-cloud-storage==2.8.0 +google-cloud-storage==2.9.0 pandas===1.3.5; python_version == '3.7' pandas==2.0.1; python_version >= '3.8' From 14ba42203df24e603ceaa5971c4497b35e01427d Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 9 May 2023 00:53:36 +0200 Subject: [PATCH 109/172] chore(deps): update dependency google-cloud-pubsub to v2.16.1 (#1034) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index c0e68145021..4e5bce1a9b8 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.16.0 +google-cloud-pubsub==2.16.1 google-cloud-storage==2.9.0 pandas===1.3.5; python_version == '3.7' pandas==2.0.1; python_version >= '3.8' From 106c5fd3b4621e58f09cbf330e213d4d59fd01fd Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Thu, 1 Jun 2023 13:31:37 +0200 Subject: [PATCH 110/172] chore(deps): update all dependencies (#1042) Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 4e5bce1a9b8..16b45191016 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.16.1 +google-cloud-pubsub==2.17.1 google-cloud-storage==2.9.0 pandas===1.3.5; python_version == '3.7' -pandas==2.0.1; python_version >= '3.8' +pandas==2.0.2; python_version >= '3.8' From 3f3fde75cba2cc2a4e73b478b9d3de6c43bcb983 Mon Sep 17 00:00:00 2001 From: cojenco Date: Tue, 6 Jun 2023 15:57:00 -0700 Subject: [PATCH 111/172] docs: add clarification to batch module (#1045) * docs: add clarification to batch module * clarify constraints with batch * update docs --- storage/samples/snippets/storage_batch_request.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/storage/samples/snippets/storage_batch_request.py b/storage/samples/snippets/storage_batch_request.py index 863fc09cd11..7fe11fb1cf7 100644 --- a/storage/samples/snippets/storage_batch_request.py +++ b/storage/samples/snippets/storage_batch_request.py @@ -28,7 +28,14 @@ def batch_request(bucket_name, prefix=None): - """Use a batch request to patch a list of objects with the given prefix in a bucket.""" + """ + Use a batch request to patch a list of objects with the given prefix in a bucket. + + Note that Cloud Storage does not support batch operations for uploading or downloading. + Additionally, the current batch design does not support library methods whose return values + depend on the response payload. + See https://cloud.google.com/python/docs/reference/storage/latest/google.cloud.storage.batch + """ # The ID of your GCS bucket # bucket_name = "my-bucket" # The prefix of the object paths From d4244c75d324207858ea6955394aa4b8a4e65347 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 13 Jun 2023 16:48:30 +0200 Subject: [PATCH 112/172] chore(deps): update dependency pytest to v7.3.2 (#1061) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 0068826c5d3..e389934acf1 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.3.1 +pytest==7.3.2 mock==5.0.2 backoff==2.2.1 \ No newline at end of file From 46ea463ad86991c629617d9890605169230108fe Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Thu, 29 Jun 2023 00:08:28 +0200 Subject: [PATCH 113/172] chore(deps): update all dependencies (#1064) Co-authored-by: cojenco --- storage/samples/snippets/requirements-test.txt | 2 +- storage/samples/snippets/requirements.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index e389934acf1..c426be49324 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.3.2 +pytest==7.4.0 mock==5.0.2 backoff==2.2.1 \ No newline at end of file diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 16b45191016..f748902ea7f 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.17.1 -google-cloud-storage==2.9.0 +google-cloud-storage==2.10.0 pandas===1.3.5; python_version == '3.7' pandas==2.0.2; python_version >= '3.8' From 41470697eefe552eebd63e5f938fb86c1b343ab7 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Wed, 5 Jul 2023 17:40:32 +0200 Subject: [PATCH 114/172] chore(deps): update dependency pandas to v2.0.3 (#1069) Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index f748902ea7f..d8bd1cf6a47 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.17.1 google-cloud-storage==2.10.0 pandas===1.3.5; python_version == '3.7' -pandas==2.0.2; python_version >= '3.8' +pandas==2.0.3; python_version >= '3.8' From 2cf8358047cc299107e58ff978c0d841cf902216 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Wed, 12 Jul 2023 17:34:01 +0200 Subject: [PATCH 115/172] chore(deps): update dependency mock to v5.1.0 (#1075) --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index c426be49324..2883c5abca9 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ pytest==7.4.0 -mock==5.0.2 +mock==5.1.0 backoff==2.2.1 \ No newline at end of file From 6e351ca26d11bc81cd6925c187bb2bc94f0a36db Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Thu, 13 Jul 2023 03:47:22 +0200 Subject: [PATCH 116/172] chore(deps): update dependency google-cloud-pubsub to v2.18.0 (#1078) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index d8bd1cf6a47..cdd2d923961 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.17.1 +google-cloud-pubsub==2.18.0 google-cloud-storage==2.10.0 pandas===1.3.5; python_version == '3.7' pandas==2.0.3; python_version >= '3.8' From 66eb6010e4920c652b8f9176ec1d19e0b6da4dc9 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Mon, 31 Jul 2023 15:59:31 +0200 Subject: [PATCH 117/172] chore(deps): update dependency google-cloud-pubsub to v2.18.1 (#1097) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index cdd2d923961..2388d18edeb 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.18.0 +google-cloud-pubsub==2.18.1 google-cloud-storage==2.10.0 pandas===1.3.5; python_version == '3.7' pandas==2.0.3; python_version >= '3.8' From 641c2ef5a1c877a1b02f76d3e616e65aabb39d80 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 8 Aug 2023 17:08:29 +0200 Subject: [PATCH 118/172] chore(deps): update dependency google-cloud-pubsub to v2.18.2 (#1104) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 2388d18edeb..d8ee4094cd0 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.18.1 +google-cloud-pubsub==2.18.2 google-cloud-storage==2.10.0 pandas===1.3.5; python_version == '3.7' pandas==2.0.3; python_version >= '3.8' From ebfe5f5adf27ac63d18c12650ca0539aea2e49de Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Sat, 19 Aug 2023 00:01:53 +0200 Subject: [PATCH 119/172] chore(deps): update dependency google-cloud-pubsub to v2.18.3 (#1111) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index d8ee4094cd0..f9b37be52aa 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.18.2 +google-cloud-pubsub==2.18.3 google-cloud-storage==2.10.0 pandas===1.3.5; python_version == '3.7' pandas==2.0.3; python_version >= '3.8' From 548266f27da0ff423376105902785a4d5d6c39a0 Mon Sep 17 00:00:00 2001 From: Andrew Gorcester Date: Tue, 22 Aug 2023 17:52:13 -0700 Subject: [PATCH 120/172] chore: Add region tags for Transfer Manager samples (#1110) --- .../snippets/storage_transfer_manager_download_all_blobs.py | 3 ++- .../storage_transfer_manager_download_chunks_concurrently.py | 3 ++- .../snippets/storage_transfer_manager_upload_directory.py | 3 ++- .../snippets/storage_transfer_manager_upload_many_blobs.py | 3 ++- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py b/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py index 2285f673f1d..a99ba6b7a1b 100644 --- a/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py +++ b/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. - +# [START storage_transfer_manager_download_all_blobs] def download_all_blobs_with_transfer_manager( bucket_name, destination_directory="", processes=8 ): @@ -62,3 +62,4 @@ def download_all_blobs_with_transfer_manager( print("Failed to download {} due to exception: {}".format(name, result)) else: print("Downloaded {} to {}.".format(name, destination_directory + name)) +# [END storage_transfer_manager_download_all_blobs] diff --git a/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py b/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py index 50541fb9353..33080b52dd9 100644 --- a/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py +++ b/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. - +# [START storage_transfer_manager_download_chunks_concurrently] def download_chunks_concurrently(bucket_name, blob_name, filename, processes=8): """Download a single file in chunks, concurrently.""" @@ -40,3 +40,4 @@ def download_chunks_concurrently(bucket_name, blob_name, filename, processes=8): transfer_manager.download_chunks_concurrently(blob, filename, max_workers=processes) print("Downloaded {} to {}.".format(blob_name, filename)) +# [END storage_transfer_manager_download_chunks_concurrently] diff --git a/storage/samples/snippets/storage_transfer_manager_upload_directory.py b/storage/samples/snippets/storage_transfer_manager_upload_directory.py index e4a369969bf..c0dbb9c9cac 100644 --- a/storage/samples/snippets/storage_transfer_manager_upload_directory.py +++ b/storage/samples/snippets/storage_transfer_manager_upload_directory.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. - +# [START storage_transfer_manager_upload_directory] def upload_directory_with_transfer_manager(bucket_name, source_directory, processes=8): """Upload every file in a directory, including all files in subdirectories. @@ -76,3 +76,4 @@ def upload_directory_with_transfer_manager(bucket_name, source_directory, proces print("Failed to upload {} due to exception: {}".format(name, result)) else: print("Uploaded {} to {}.".format(name, bucket.name)) +# [END storage_transfer_manager_upload_directory] diff --git a/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py b/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py index 600134bd6b8..a085cfc2b3e 100644 --- a/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py +++ b/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. - +# [START storage_transfer_manager_upload_many_blobs] def upload_many_blobs_with_transfer_manager( bucket_name, filenames, source_directory="", processes=8 ): @@ -63,3 +63,4 @@ def upload_many_blobs_with_transfer_manager( print("Failed to upload {} due to exception: {}".format(name, result)) else: print("Uploaded {} to {}.".format(name, bucket.name)) +# [END storage_transfer_manager_upload_many_blobs] From d916d9cc3b503f5ec9f2126f6f2627e3e21ddd97 Mon Sep 17 00:00:00 2001 From: Andrew Gorcester Date: Thu, 31 Aug 2023 13:09:41 -0700 Subject: [PATCH 121/172] chore: Amend Transfer Manager samples (#1113) * chore: Amend Transfer Manager samples * tests * tests again * respond to feedback --- storage/samples/snippets/snippets_test.py | 23 +++++- ...torage_transfer_manager_download_bucket.py | 74 +++++++++++++++++++ ...er_manager_download_chunks_concurrently.py | 2 +- ...storage_transfer_manager_download_many.py} | 24 +++--- ...> storage_transfer_manager_upload_many.py} | 6 +- 5 files changed, 111 insertions(+), 18 deletions(-) create mode 100644 storage/samples/snippets/storage_transfer_manager_download_bucket.py rename storage/samples/snippets/{storage_transfer_manager_download_all_blobs.py => storage_transfer_manager_download_many.py} (75%) rename storage/samples/snippets/{storage_transfer_manager_upload_many_blobs.py => storage_transfer_manager_upload_many.py} (95%) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 6e5879eeb78..2da7bb94c59 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -72,10 +72,11 @@ import storage_set_bucket_default_kms_key import storage_set_client_endpoint import storage_set_metadata -import storage_transfer_manager_download_all_blobs +import storage_transfer_manager_download_bucket import storage_transfer_manager_download_chunks_concurrently +import storage_transfer_manager_download_many import storage_transfer_manager_upload_directory -import storage_transfer_manager_upload_many_blobs +import storage_transfer_manager_upload_many import storage_upload_file import storage_upload_from_memory import storage_upload_from_stream @@ -689,7 +690,7 @@ def test_transfer_manager_snippets(test_bucket, capsys): with open(os.path.join(uploads, name), "w") as f: f.write(name) - storage_transfer_manager_upload_many_blobs.upload_many_blobs_with_transfer_manager( + storage_transfer_manager_upload_many.upload_many_blobs_with_transfer_manager( test_bucket.name, BLOB_NAMES, source_directory="{}/".format(uploads), @@ -702,10 +703,24 @@ def test_transfer_manager_snippets(test_bucket, capsys): with tempfile.TemporaryDirectory() as downloads: # Download the files. - storage_transfer_manager_download_all_blobs.download_all_blobs_with_transfer_manager( + storage_transfer_manager_download_bucket.download_bucket_with_transfer_manager( test_bucket.name, destination_directory=os.path.join(downloads, ""), processes=8, + max_results=10000, + ) + out, _ = capsys.readouterr() + + for name in BLOB_NAMES: + assert "Downloaded {}".format(name) in out + + with tempfile.TemporaryDirectory() as downloads: + # Download the files. + storage_transfer_manager_download_many.download_many_blobs_with_transfer_manager( + test_bucket.name, + blob_names=BLOB_NAMES, + destination_directory=os.path.join(downloads, ""), + processes=8, ) out, _ = capsys.readouterr() diff --git a/storage/samples/snippets/storage_transfer_manager_download_bucket.py b/storage/samples/snippets/storage_transfer_manager_download_bucket.py new file mode 100644 index 00000000000..4f21ee6e9fe --- /dev/null +++ b/storage/samples/snippets/storage_transfer_manager_download_bucket.py @@ -0,0 +1,74 @@ +# Copyright 2022 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_transfer_manager_download_bucket] +def download_bucket_with_transfer_manager( + bucket_name, destination_directory="", processes=8, max_results=1000 +): + """Download all of the blobs in a bucket, concurrently in a process pool. + + The filename of each blob once downloaded is derived from the blob name and + the `destination_directory `parameter. For complete control of the filename + of each blob, use transfer_manager.download_many() instead. + + Directories will be created automatically as needed, for instance to + accommodate blob names that include slashes. + """ + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The directory on your computer to which to download all of the files. This + # string is prepended (with os.path.join()) to the name of each blob to form + # the full path. Relative paths and absolute paths are both accepted. An + # empty string means "the current working directory". Note that this + # parameter allows accepts directory traversal ("../" etc.) and is not + # intended for unsanitized end user input. + # destination_directory = "" + + # The maximum number of processes to use for the operation. The performance + # impact of this value depends on the use case, but smaller files usually + # benefit from a higher number of processes. Each additional process occupies + # some CPU and memory resources until finished. + # processes=8 + + # The maximum number of results to fetch from bucket.list_blobs(). This + # sample code fetches all of the blobs up to max_results and queues them all + # for download at once. Though they will still be executed in batches up to + # the processes limit, queueing them all at once can be taxing on system + # memory if buckets are very large. Adjust max_results as needed for your + # system environment, or set it to None if you are sure the bucket is not + # too large to hold in memory easily. + # max_results=1000 + + from google.cloud.storage import Client, transfer_manager + + storage_client = Client() + bucket = storage_client.bucket(bucket_name) + + blob_names = [blob.name for blob in bucket.list_blobs(max_results=max_results)] + + results = transfer_manager.download_many_to_path( + bucket, blob_names, destination_directory=destination_directory, max_workers=processes + ) + + for name, result in zip(blob_names, results): + # The results list is either `None` or an exception for each blob in + # the input list, in order. + + if isinstance(result, Exception): + print("Failed to download {} due to exception: {}".format(name, result)) + else: + print("Downloaded {} to {}.".format(name, destination_directory + name)) +# [END storage_transfer_manager_download_bucket] diff --git a/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py b/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py index 33080b52dd9..9ddec094e7d 100644 --- a/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py +++ b/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py @@ -14,7 +14,7 @@ # [START storage_transfer_manager_download_chunks_concurrently] def download_chunks_concurrently(bucket_name, blob_name, filename, processes=8): - """Download a single file in chunks, concurrently.""" + """Download a single file in chunks, concurrently in a process pool.""" # The ID of your GCS bucket # bucket_name = "your-bucket-name" diff --git a/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py b/storage/samples/snippets/storage_transfer_manager_download_many.py similarity index 75% rename from storage/samples/snippets/storage_transfer_manager_download_all_blobs.py rename to storage/samples/snippets/storage_transfer_manager_download_many.py index a99ba6b7a1b..500eea1ce7a 100644 --- a/storage/samples/snippets/storage_transfer_manager_download_all_blobs.py +++ b/storage/samples/snippets/storage_transfer_manager_download_many.py @@ -1,4 +1,4 @@ -# Copyright 2022 Google LLC +# Copyright 2023 Google LLC # # Licensed under the Apache License, Version 2.0 (the 'License'); # you may not use this file except in compliance with the License. @@ -12,23 +12,29 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START storage_transfer_manager_download_all_blobs] -def download_all_blobs_with_transfer_manager( - bucket_name, destination_directory="", processes=8 +# [START storage_transfer_manager_download_many] +def download_many_blobs_with_transfer_manager( + bucket_name, blob_names, destination_directory="", processes=8 ): - """Download all of the blobs in a bucket, concurrently in a thread pool. + """Download blobs in a list by name, concurrently in a process pool. The filename of each blob once downloaded is derived from the blob name and the `destination_directory `parameter. For complete control of the filename of each blob, use transfer_manager.download_many() instead. - Directories will be created automatically as needed, for instance to - accommodate blob names that include slashes. + Directories will be created automatically as needed to accommodate blob + names that include slashes. """ # The ID of your GCS bucket # bucket_name = "your-bucket-name" + # The list of blob names to download. The names of each blobs will also + # be the name of each destination file (use transfer_manager.download_many() + # instead to control each destination file name). If there is a "/" in the + # blob name, then corresponding directories will be created on download. + # blob_names = ["myblob", "myblob2"] + # The directory on your computer to which to download all of the files. This # string is prepended (with os.path.join()) to the name of each blob to form # the full path. Relative paths and absolute paths are both accepted. An @@ -48,8 +54,6 @@ def download_all_blobs_with_transfer_manager( storage_client = Client() bucket = storage_client.bucket(bucket_name) - blob_names = [blob.name for blob in bucket.list_blobs()] - results = transfer_manager.download_many_to_path( bucket, blob_names, destination_directory=destination_directory, max_workers=processes ) @@ -62,4 +66,4 @@ def download_all_blobs_with_transfer_manager( print("Failed to download {} due to exception: {}".format(name, result)) else: print("Downloaded {} to {}.".format(name, destination_directory + name)) -# [END storage_transfer_manager_download_all_blobs] +# [END storage_transfer_manager_download_many] diff --git a/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py b/storage/samples/snippets/storage_transfer_manager_upload_many.py similarity index 95% rename from storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py rename to storage/samples/snippets/storage_transfer_manager_upload_many.py index a085cfc2b3e..2ed64765046 100644 --- a/storage/samples/snippets/storage_transfer_manager_upload_many_blobs.py +++ b/storage/samples/snippets/storage_transfer_manager_upload_many.py @@ -12,11 +12,11 @@ # See the License for the specific language governing permissions and # limitations under the License. -# [START storage_transfer_manager_upload_many_blobs] +# [START storage_transfer_manager_upload_many] def upload_many_blobs_with_transfer_manager( bucket_name, filenames, source_directory="", processes=8 ): - """Upload every file in a list to a bucket, concurrently in a thread pool. + """Upload every file in a list to a bucket, concurrently in a process pool. Each blob name is derived from the filename, not including the `source_directory` parameter. For complete control of the blob name for each @@ -63,4 +63,4 @@ def upload_many_blobs_with_transfer_manager( print("Failed to upload {} due to exception: {}".format(name, result)) else: print("Uploaded {} to {}.".format(name, bucket.name)) -# [END storage_transfer_manager_upload_many_blobs] +# [END storage_transfer_manager_upload_many] From f91f03420d773636423a19a691d45be7a21339c3 Mon Sep 17 00:00:00 2001 From: Andrew Gorcester Date: Mon, 9 Oct 2023 15:18:48 -0700 Subject: [PATCH 122/172] docs: Add snippets for upload_chunks_concurrently and add chunk_size (#1135) * docs: Add snippets for upload_chunks_concurrently and add chunk_size * switch from 'processes' to 'workers' in sample nomenclature * copyright * tests --- storage/samples/snippets/snippets_test.py | 61 ++++++++++++++----- ...torage_transfer_manager_download_bucket.py | 9 +-- ...er_manager_download_chunks_concurrently.py | 20 ++++-- .../storage_transfer_manager_download_many.py | 9 +-- ...sfer_manager_upload_chunks_concurrently.py | 57 +++++++++++++++++ ...orage_transfer_manager_upload_directory.py | 9 +-- .../storage_transfer_manager_upload_many.py | 9 +-- 7 files changed, 140 insertions(+), 34 deletions(-) create mode 100644 storage/samples/snippets/storage_transfer_manager_upload_chunks_concurrently.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 2da7bb94c59..8014411e833 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -75,6 +75,7 @@ import storage_transfer_manager_download_bucket import storage_transfer_manager_download_chunks_concurrently import storage_transfer_manager_download_many +import storage_transfer_manager_upload_chunks_concurrently import storage_transfer_manager_upload_directory import storage_transfer_manager_upload_many import storage_upload_file @@ -243,7 +244,10 @@ def test_upload_blob_with_kms(test_bucket): with tempfile.NamedTemporaryFile() as source_file: source_file.write(b"test") storage_upload_with_kms_key.upload_blob_with_kms( - test_bucket.name, source_file.name, blob_name, KMS_KEY, + test_bucket.name, + source_file.name, + blob_name, + KMS_KEY, ) bucket = storage.Client().bucket(test_bucket.name) kms_blob = bucket.get_blob(blob_name) @@ -396,7 +400,10 @@ def test_move_blob(test_bucket_create, test_blob): print(f"test_move_blob not found in bucket {test_bucket_create.name}") storage_move_file.move_blob( - bucket.name, test_blob.name, test_bucket_create.name, "test_move_blob", + bucket.name, + test_blob.name, + test_bucket_create.name, + "test_move_blob", ) assert test_bucket_create.get_blob("test_move_blob") is not None @@ -412,7 +419,10 @@ def test_copy_blob(test_blob): pass storage_copy_file.copy_blob( - bucket.name, test_blob.name, bucket.name, "test_copy_blob", + bucket.name, + test_blob.name, + bucket.name, + "test_copy_blob", ) assert bucket.get_blob("test_copy_blob") is not None @@ -551,7 +561,10 @@ def test_define_bucket_website_configuration(test_bucket): def test_object_get_kms_key(test_bucket): with tempfile.NamedTemporaryFile() as source_file: storage_upload_with_kms_key.upload_blob_with_kms( - test_bucket.name, source_file.name, "test_upload_blob_encrypted", KMS_KEY, + test_bucket.name, + source_file.name, + "test_upload_blob_encrypted", + KMS_KEY, ) kms_key = storage_object_get_kms_key.object_get_kms_key( test_bucket.name, "test_upload_blob_encrypted" @@ -568,7 +581,10 @@ def test_storage_compose_file(test_bucket): with tempfile.NamedTemporaryFile() as dest_file: destination = storage_compose_file.compose_file( - test_bucket.name, source_files[0], source_files[1], dest_file.name, + test_bucket.name, + source_files[0], + source_files[1], + dest_file.name, ) composed = destination.download_as_string() @@ -608,7 +624,8 @@ def test_change_default_storage_class(test_bucket, capsys): def test_change_file_storage_class(test_blob, capsys): blob = storage_change_file_storage_class.change_file_storage_class( - test_blob.bucket.name, test_blob.name, + test_blob.bucket.name, + test_blob.name, ) out, _ = capsys.readouterr() assert f"Blob {blob.name} in bucket {blob.bucket.name}" in out @@ -694,7 +711,7 @@ def test_transfer_manager_snippets(test_bucket, capsys): test_bucket.name, BLOB_NAMES, source_directory="{}/".format(uploads), - processes=8, + workers=8, ) out, _ = capsys.readouterr() @@ -706,7 +723,7 @@ def test_transfer_manager_snippets(test_bucket, capsys): storage_transfer_manager_download_bucket.download_bucket_with_transfer_manager( test_bucket.name, destination_directory=os.path.join(downloads, ""), - processes=8, + workers=8, max_results=10000, ) out, _ = capsys.readouterr() @@ -720,7 +737,7 @@ def test_transfer_manager_snippets(test_bucket, capsys): test_bucket.name, blob_names=BLOB_NAMES, destination_directory=os.path.join(downloads, ""), - processes=8, + workers=8, ) out, _ = capsys.readouterr() @@ -763,9 +780,7 @@ def test_transfer_manager_download_chunks_concurrently(test_bucket, capsys): with tempfile.NamedTemporaryFile() as file: file.write(b"test") - storage_upload_file.upload_blob( - test_bucket.name, file.name, BLOB_NAME - ) + storage_upload_file.upload_blob(test_bucket.name, file.name, BLOB_NAME) with tempfile.TemporaryDirectory() as downloads: # Download the file. @@ -773,8 +788,26 @@ def test_transfer_manager_download_chunks_concurrently(test_bucket, capsys): test_bucket.name, BLOB_NAME, os.path.join(downloads, BLOB_NAME), - processes=8, + workers=8, ) out, _ = capsys.readouterr() - assert "Downloaded {} to {}".format(BLOB_NAME, os.path.join(downloads, BLOB_NAME)) in out + assert ( + "Downloaded {} to {}".format(BLOB_NAME, os.path.join(downloads, BLOB_NAME)) + in out + ) + + +def test_transfer_manager_upload_chunks_concurrently(test_bucket, capsys): + BLOB_NAME = "test_file.txt" + + with tempfile.NamedTemporaryFile() as file: + file.write(b"test") + file.flush() + + storage_transfer_manager_upload_chunks_concurrently.upload_chunks_concurrently( + test_bucket.name, file.name, BLOB_NAME + ) + + out, _ = capsys.readouterr() + assert "File {} uploaded to {}".format(file.name, BLOB_NAME) in out diff --git a/storage/samples/snippets/storage_transfer_manager_download_bucket.py b/storage/samples/snippets/storage_transfer_manager_download_bucket.py index 4f21ee6e9fe..5d94a67aeea 100644 --- a/storage/samples/snippets/storage_transfer_manager_download_bucket.py +++ b/storage/samples/snippets/storage_transfer_manager_download_bucket.py @@ -14,7 +14,7 @@ # [START storage_transfer_manager_download_bucket] def download_bucket_with_transfer_manager( - bucket_name, destination_directory="", processes=8, max_results=1000 + bucket_name, destination_directory="", workers=8, max_results=1000 ): """Download all of the blobs in a bucket, concurrently in a process pool. @@ -40,8 +40,9 @@ def download_bucket_with_transfer_manager( # The maximum number of processes to use for the operation. The performance # impact of this value depends on the use case, but smaller files usually # benefit from a higher number of processes. Each additional process occupies - # some CPU and memory resources until finished. - # processes=8 + # some CPU and memory resources until finished. Threads can be used instead + # of processes by passing `worker_type=transfer_manager.THREAD`. + # workers=8 # The maximum number of results to fetch from bucket.list_blobs(). This # sample code fetches all of the blobs up to max_results and queues them all @@ -60,7 +61,7 @@ def download_bucket_with_transfer_manager( blob_names = [blob.name for blob in bucket.list_blobs(max_results=max_results)] results = transfer_manager.download_many_to_path( - bucket, blob_names, destination_directory=destination_directory, max_workers=processes + bucket, blob_names, destination_directory=destination_directory, max_workers=workers ) for name, result in zip(blob_names, results): diff --git a/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py b/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py index 9ddec094e7d..b6ac9982d61 100644 --- a/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py +++ b/storage/samples/snippets/storage_transfer_manager_download_chunks_concurrently.py @@ -13,7 +13,9 @@ # limitations under the License. # [START storage_transfer_manager_download_chunks_concurrently] -def download_chunks_concurrently(bucket_name, blob_name, filename, processes=8): +def download_chunks_concurrently( + bucket_name, blob_name, filename, chunk_size=32 * 1024 * 1024, workers=8 +): """Download a single file in chunks, concurrently in a process pool.""" # The ID of your GCS bucket @@ -25,11 +27,17 @@ def download_chunks_concurrently(bucket_name, blob_name, filename, processes=8): # The destination filename or path # filename = "" + # The size of each chunk. The performance impact of this value depends on + # the use case. The remote service has a minimum of 5 MiB and a maximum of + # 5 GiB. + # chunk_size = 32 * 1024 * 1024 (32 MiB) + # The maximum number of processes to use for the operation. The performance # impact of this value depends on the use case, but smaller files usually # benefit from a higher number of processes. Each additional process occupies - # some CPU and memory resources until finished. - # processes=8 + # some CPU and memory resources until finished. Threads can be used instead + # of processes by passing `worker_type=transfer_manager.THREAD`. + # workers=8 from google.cloud.storage import Client, transfer_manager @@ -37,7 +45,11 @@ def download_chunks_concurrently(bucket_name, blob_name, filename, processes=8): bucket = storage_client.bucket(bucket_name) blob = bucket.blob(blob_name) - transfer_manager.download_chunks_concurrently(blob, filename, max_workers=processes) + transfer_manager.download_chunks_concurrently( + blob, filename, chunk_size=chunk_size, max_workers=workers + ) print("Downloaded {} to {}.".format(blob_name, filename)) + + # [END storage_transfer_manager_download_chunks_concurrently] diff --git a/storage/samples/snippets/storage_transfer_manager_download_many.py b/storage/samples/snippets/storage_transfer_manager_download_many.py index 500eea1ce7a..02cb9b8877c 100644 --- a/storage/samples/snippets/storage_transfer_manager_download_many.py +++ b/storage/samples/snippets/storage_transfer_manager_download_many.py @@ -14,7 +14,7 @@ # [START storage_transfer_manager_download_many] def download_many_blobs_with_transfer_manager( - bucket_name, blob_names, destination_directory="", processes=8 + bucket_name, blob_names, destination_directory="", workers=8 ): """Download blobs in a list by name, concurrently in a process pool. @@ -46,8 +46,9 @@ def download_many_blobs_with_transfer_manager( # The maximum number of processes to use for the operation. The performance # impact of this value depends on the use case, but smaller files usually # benefit from a higher number of processes. Each additional process occupies - # some CPU and memory resources until finished. - # processes=8 + # some CPU and memory resources until finished. Threads can be used instead + # of processes by passing `worker_type=transfer_manager.THREAD`. + # workers=8 from google.cloud.storage import Client, transfer_manager @@ -55,7 +56,7 @@ def download_many_blobs_with_transfer_manager( bucket = storage_client.bucket(bucket_name) results = transfer_manager.download_many_to_path( - bucket, blob_names, destination_directory=destination_directory, max_workers=processes + bucket, blob_names, destination_directory=destination_directory, max_workers=workers ) for name, result in zip(blob_names, results): diff --git a/storage/samples/snippets/storage_transfer_manager_upload_chunks_concurrently.py b/storage/samples/snippets/storage_transfer_manager_upload_chunks_concurrently.py new file mode 100644 index 00000000000..009f0964870 --- /dev/null +++ b/storage/samples/snippets/storage_transfer_manager_upload_chunks_concurrently.py @@ -0,0 +1,57 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_transfer_manager_upload_chunks_concurrently] +def upload_chunks_concurrently( + bucket_name, + source_filename, + destination_blob_name, + chunk_size=32 * 1024 * 1024, + workers=8, +): + """Upload a single file, in chunks, concurrently in a process pool.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The path to your file to upload + # source_filename = "local/path/to/file" + + # The ID of your GCS object + # destination_blob_name = "storage-object-name" + + # The size of each chunk. The performance impact of this value depends on + # the use case. The remote service has a minimum of 5 MiB and a maximum of + # 5 GiB. + # chunk_size = 32 * 1024 * 1024 (32 MiB) + + # The maximum number of processes to use for the operation. The performance + # impact of this value depends on the use case. Each additional process + # occupies some CPU and memory resources until finished. Threads can be used + # instead of processes by passing `worker_type=transfer_manager.THREAD`. + # workers=8 + + from google.cloud.storage import Client, transfer_manager + + storage_client = Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(destination_blob_name) + + transfer_manager.upload_chunks_concurrently( + source_filename, blob, chunk_size=chunk_size, max_workers=workers + ) + + print(f"File {source_filename} uploaded to {destination_blob_name}.") + + +# [END storage_transfer_manager_upload_chunks_concurrently] diff --git a/storage/samples/snippets/storage_transfer_manager_upload_directory.py b/storage/samples/snippets/storage_transfer_manager_upload_directory.py index c0dbb9c9cac..329ca108133 100644 --- a/storage/samples/snippets/storage_transfer_manager_upload_directory.py +++ b/storage/samples/snippets/storage_transfer_manager_upload_directory.py @@ -13,7 +13,7 @@ # limitations under the License. # [START storage_transfer_manager_upload_directory] -def upload_directory_with_transfer_manager(bucket_name, source_directory, processes=8): +def upload_directory_with_transfer_manager(bucket_name, source_directory, workers=8): """Upload every file in a directory, including all files in subdirectories. Each blob name is derived from the filename, not including the `directory` @@ -33,8 +33,9 @@ def upload_directory_with_transfer_manager(bucket_name, source_directory, proces # The maximum number of processes to use for the operation. The performance # impact of this value depends on the use case, but smaller files usually # benefit from a higher number of processes. Each additional process occupies - # some CPU and memory resources until finished. - # processes=8 + # some CPU and memory resources until finished. Threads can be used instead + # of processes by passing `worker_type=transfer_manager.THREAD`. + # workers=8 from pathlib import Path @@ -65,7 +66,7 @@ def upload_directory_with_transfer_manager(bucket_name, source_directory, proces # Start the upload. results = transfer_manager.upload_many_from_filenames( - bucket, string_paths, source_directory=source_directory, max_workers=processes + bucket, string_paths, source_directory=source_directory, max_workers=workers ) for name, result in zip(string_paths, results): diff --git a/storage/samples/snippets/storage_transfer_manager_upload_many.py b/storage/samples/snippets/storage_transfer_manager_upload_many.py index 2ed64765046..1b9b9fc8983 100644 --- a/storage/samples/snippets/storage_transfer_manager_upload_many.py +++ b/storage/samples/snippets/storage_transfer_manager_upload_many.py @@ -14,7 +14,7 @@ # [START storage_transfer_manager_upload_many] def upload_many_blobs_with_transfer_manager( - bucket_name, filenames, source_directory="", processes=8 + bucket_name, filenames, source_directory="", workers=8 ): """Upload every file in a list to a bucket, concurrently in a process pool. @@ -43,8 +43,9 @@ def upload_many_blobs_with_transfer_manager( # The maximum number of processes to use for the operation. The performance # impact of this value depends on the use case, but smaller files usually # benefit from a higher number of processes. Each additional process occupies - # some CPU and memory resources until finished. - # processes=8 + # some CPU and memory resources until finished. Threads can be used instead + # of processes by passing `worker_type=transfer_manager.THREAD`. + # workers=8 from google.cloud.storage import Client, transfer_manager @@ -52,7 +53,7 @@ def upload_many_blobs_with_transfer_manager( bucket = storage_client.bucket(bucket_name) results = transfer_manager.upload_many_from_filenames( - bucket, filenames, source_directory=source_directory, max_workers=processes + bucket, filenames, source_directory=source_directory, max_workers=workers ) for name, result in zip(filenames, results): From def3efa093983ee8e5c534623d7963e9337b6a26 Mon Sep 17 00:00:00 2001 From: Andrew Gorcester Date: Wed, 11 Oct 2023 13:46:14 -0700 Subject: [PATCH 123/172] feat: add crc32c_checksum argument to download_chunks_concurrently (#1138) --- storage/samples/snippets/snippets_test.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 8014411e833..7a5f8c96002 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -213,6 +213,7 @@ def test_list_blobs_with_prefix(test_blob, capsys): def test_upload_blob(test_bucket): with tempfile.NamedTemporaryFile() as source_file: source_file.write(b"test") + source_file.flush() storage_upload_file.upload_blob( test_bucket.name, source_file.name, "test_upload_blob" @@ -243,6 +244,7 @@ def test_upload_blob_with_kms(test_bucket): blob_name = f"test_upload_with_kms_{uuid.uuid4().hex}" with tempfile.NamedTemporaryFile() as source_file: source_file.write(b"test") + source_file.flush() storage_upload_with_kms_key.upload_blob_with_kms( test_bucket.name, source_file.name, @@ -779,6 +781,7 @@ def test_transfer_manager_download_chunks_concurrently(test_bucket, capsys): with tempfile.NamedTemporaryFile() as file: file.write(b"test") + file.flush() storage_upload_file.upload_blob(test_bucket.name, file.name, BLOB_NAME) From fbab2cf9025fe63f60bd4879d21f09c135cad991 Mon Sep 17 00:00:00 2001 From: cojenco Date: Mon, 30 Oct 2023 11:02:20 -0700 Subject: [PATCH 124/172] feat: add Autoclass v2.1 support (#1117) * feat: add Autoclass v2.1 support * update tests and coverage * update samples with v2.1 additions * fix lint * update samples --- storage/samples/snippets/snippets_test.py | 14 ++++++++----- .../samples/snippets/storage_get_autoclass.py | 3 +++ .../samples/snippets/storage_set_autoclass.py | 20 +++++++++++-------- 3 files changed, 24 insertions(+), 13 deletions(-) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 7a5f8c96002..7add151845e 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -449,23 +449,27 @@ def test_get_set_autoclass(new_bucket_obj, test_bucket, capsys): out, _ = capsys.readouterr() assert "Autoclass enabled is set to False" in out assert bucket.autoclass_toggle_time is None + assert bucket.autoclass_terminal_storage_class_update_time is None # Test enabling Autoclass at bucket creation new_bucket_obj.autoclass_enabled = True bucket = storage.Client().create_bucket(new_bucket_obj) assert bucket.autoclass_enabled is True + assert bucket.autoclass_terminal_storage_class == "NEARLINE" - # Test disabling Autoclass - bucket = storage_set_autoclass.set_autoclass(bucket.name, False) + # Test set terminal_storage_class to ARCHIVE + bucket = storage_set_autoclass.set_autoclass(bucket.name) out, _ = capsys.readouterr() - assert "Autoclass enabled is set to False" in out - assert bucket.autoclass_enabled is False + assert "Autoclass enabled is set to True" in out + assert bucket.autoclass_enabled is True + assert bucket.autoclass_terminal_storage_class == "ARCHIVE" # Test get Autoclass bucket = storage_get_autoclass.get_autoclass(bucket.name) out, _ = capsys.readouterr() - assert "Autoclass enabled is set to False" in out + assert "Autoclass enabled is set to True" in out assert bucket.autoclass_toggle_time is not None + assert bucket.autoclass_terminal_storage_class_update_time is not None def test_bucket_lifecycle_management(test_bucket, capsys): diff --git a/storage/samples/snippets/storage_get_autoclass.py b/storage/samples/snippets/storage_get_autoclass.py index d4bcbf3f4dd..30fa0c4f6b3 100644 --- a/storage/samples/snippets/storage_get_autoclass.py +++ b/storage/samples/snippets/storage_get_autoclass.py @@ -29,8 +29,11 @@ def get_autoclass(bucket_name): bucket = storage_client.get_bucket(bucket_name) autoclass_enabled = bucket.autoclass_enabled autoclass_toggle_time = bucket.autoclass_toggle_time + terminal_storage_class = bucket.autoclass_terminal_storage_class + tsc_update_time = bucket.autoclass_terminal_storage_class_update_time print(f"Autoclass enabled is set to {autoclass_enabled} for {bucket.name} at {autoclass_toggle_time}.") + print(f"Autoclass terminal storage class is set to {terminal_storage_class} for {bucket.name} at {tsc_update_time}.") return bucket diff --git a/storage/samples/snippets/storage_set_autoclass.py b/storage/samples/snippets/storage_set_autoclass.py index a25151f3bbc..eec5a550f8c 100644 --- a/storage/samples/snippets/storage_set_autoclass.py +++ b/storage/samples/snippets/storage_set_autoclass.py @@ -20,23 +20,27 @@ from google.cloud import storage -def set_autoclass(bucket_name, toggle): - """Disable Autoclass for a bucket. +def set_autoclass(bucket_name): + """Configure the Autoclass setting for a bucket. - Note: Only patch requests that disable autoclass are currently supported. - To enable autoclass, you must set it at bucket creation time. + terminal_storage_class field is optional and defaults to NEARLINE if not otherwise specified. + Valid terminal_storage_class values are NEARLINE and ARCHIVE. """ # The ID of your GCS bucket # bucket_name = "my-bucket" - # Boolean toggle - if true, enables Autoclass; if false, disables Autoclass - # toggle = False + # Enable Autoclass for a bucket. Set enabled to false to disable Autoclass. + # Set Autoclass.TerminalStorageClass, valid values are NEARLINE and ARCHIVE. + enabled = True + terminal_storage_class = "ARCHIVE" storage_client = storage.Client() bucket = storage_client.bucket(bucket_name) - bucket.autoclass_enabled = toggle + bucket.autoclass_enabled = enabled + bucket.autoclass_terminal_storage_class = terminal_storage_class bucket.patch() print(f"Autoclass enabled is set to {bucket.autoclass_enabled} for {bucket.name} at {bucket.autoclass_toggle_time}.") + print(f"Autoclass terminal storage class is {bucket.autoclass_terminal_storage_class}.") return bucket @@ -44,4 +48,4 @@ def set_autoclass(bucket_name, toggle): # [END storage_set_autoclass] if __name__ == "__main__": - set_autoclass(bucket_name=sys.argv[1], toggle=sys.argv[2]) + set_autoclass(bucket_name=sys.argv[1]) From 746e68ef3272d03239ad28217f20c0cd9c51df09 Mon Sep 17 00:00:00 2001 From: cojenco Date: Tue, 7 Nov 2023 20:50:19 -0800 Subject: [PATCH 125/172] chore(samples): bump storage to latest (#1177) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index f9b37be52aa..617ae8a3394 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ google-cloud-pubsub==2.18.3 -google-cloud-storage==2.10.0 +google-cloud-storage==2.13.0 pandas===1.3.5; python_version == '3.7' pandas==2.0.3; python_version >= '3.8' From 4bf3e904bef6941d5d3491d37a3885e7715ff8b9 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Fri, 1 Dec 2023 13:46:24 -0800 Subject: [PATCH 126/172] feat: Add support for Python 3.12 (#1187) * chore(python): Add Python 3.12 Source-Link: https://github.com/googleapis/synthtool/commit/af16e6d4672cc7b400f144de2fc3068b54ff47d2 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:bacc3af03bff793a03add584537b36b5644342931ad989e3ba1171d3bd5399f5 * Add trove classifier for python 3.12 * Update contributing.rst, noxfile and constraints to include python 3.12 * remove usage of deprecated assertDictContainsSubset https://github.com/python/cpython/pull/28268 * update tests KMS key settings --------- Co-authored-by: Owl Bot Co-authored-by: Anthonios Partheniou Co-authored-by: Cathy Ouyang --- storage/samples/snippets/encryption_test.py | 2 +- storage/samples/snippets/noxfile.py | 2 +- storage/samples/snippets/noxfile_config.py | 10 +++++++++- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/storage/samples/snippets/encryption_test.py b/storage/samples/snippets/encryption_test.py index 5a5eb7b2d53..ff7a568e00a 100644 --- a/storage/samples/snippets/encryption_test.py +++ b/storage/samples/snippets/encryption_test.py @@ -29,7 +29,7 @@ import storage_upload_encrypted_file BUCKET = os.environ["CLOUD_STORAGE_BUCKET"] -KMS_KEY = os.environ["CLOUD_KMS_KEY"] +KMS_KEY = os.environ["MAIN_CLOUD_KMS_KEY"] TEST_ENCRYPTION_KEY = "brtJUWneL92g5q0N2gyDSnlPSYAiIVZ/cWgjyZNeMy0=" TEST_ENCRYPTION_KEY_DECODED = base64.b64decode(TEST_ENCRYPTION_KEY) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 7c8a63994cb..483b5590179 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -89,7 +89,7 @@ def get_pytest_env_vars() -> Dict[str, str]: # DO NOT EDIT - automatically generated. # All versions used to test samples. -ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"] +ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12"] # Any default versions that should be ignored. IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] diff --git a/storage/samples/snippets/noxfile_config.py b/storage/samples/snippets/noxfile_config.py index ecd7fdce7a3..4f184ede095 100644 --- a/storage/samples/snippets/noxfile_config.py +++ b/storage/samples/snippets/noxfile_config.py @@ -67,6 +67,12 @@ def get_cloud_kms_key(): if session == 'py-3.10': return ('projects/python-docs-samples-tests-310/locations/us/' 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') + if session == 'py-3.11': + return ('projects/python-docs-samples-tests-311/locations/us/' + 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') + if session == 'py-3.12': + return ('projects/python-docs-samples-tests-312/locations/us/' + 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') return os.environ['CLOUD_KMS_KEY'] @@ -91,6 +97,8 @@ def get_cloud_kms_key(): # 'constraints/iam.disableServiceAccountKeyCreation' policy. # 2. The new projects buckets need to have universal permission model. # For those tests, we'll use the original project. - 'MAIN_GOOGLE_CLOUD_PROJECT': 'python-docs-samples-tests' + 'MAIN_GOOGLE_CLOUD_PROJECT': 'python-docs-samples-tests', + 'MAIN_CLOUD_KMS_KEY': ('projects/python-docs-samples-tests/locations/us/' + 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') }, } From 2341e67cd7afe2ab5a28181ab4fa00e13c3dfa2a Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Sat, 2 Dec 2023 17:30:22 +0100 Subject: [PATCH 127/172] chore(deps): update all dependencies (#1114) * chore(deps): update all dependencies * See https://github.com/pandas-dev/pandas/releases/tag/v2.1.0 --------- Co-authored-by: Anthonios Partheniou --- storage/samples/snippets/requirements-test.txt | 2 +- storage/samples/snippets/requirements.txt | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 2883c5abca9..52e47f6e303 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.4.0 +pytest==7.4.3 mock==5.1.0 backoff==2.2.1 \ No newline at end of file diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 617ae8a3394..f5205187240 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,5 @@ -google-cloud-pubsub==2.18.3 +google-cloud-pubsub==2.18.4 google-cloud-storage==2.13.0 pandas===1.3.5; python_version == '3.7' -pandas==2.0.3; python_version >= '3.8' +pandas===2.0.3; python_version == '3.8' +pandas==2.1.3; python_version >= '3.9' From 444ba80581d3adc72449f79bc6e450a9bab9f123 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Sun, 10 Dec 2023 15:23:53 +0100 Subject: [PATCH 128/172] chore(deps): update dependency pandas to v2.1.4 (#1198) --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index f5205187240..5f6d540030c 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -2,4 +2,4 @@ google-cloud-pubsub==2.18.4 google-cloud-storage==2.13.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' -pandas==2.1.3; python_version >= '3.9' +pandas==2.1.4; python_version >= '3.9' From 61d3a7500b774248073a00701eed54179bcb57e9 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Thu, 14 Dec 2023 13:26:33 +0100 Subject: [PATCH 129/172] chore(deps): update all dependencies (#1199) --- storage/samples/snippets/requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 5f6d540030c..15a68497302 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,5 +1,5 @@ -google-cloud-pubsub==2.18.4 -google-cloud-storage==2.13.0 +google-cloud-pubsub==2.19.0 +google-cloud-storage==2.14.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' pandas==2.1.4; python_version >= '3.9' From e4d6ec33060b1b848c6acd36781146e0cdc4d0fe Mon Sep 17 00:00:00 2001 From: Chris Cotter Date: Wed, 3 Jan 2024 19:08:28 -0500 Subject: [PATCH 130/172] chore: fix get RPO sample (#1207) * chore: fix get RPO sample There was an extra line here that doesn't make sense. Fixing based on external user feedback. * Update samples/snippets/storage_get_rpo.py Co-authored-by: cojenco * remove constant import --------- Co-authored-by: cojenco --- storage/samples/snippets/storage_get_rpo.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/storage/samples/snippets/storage_get_rpo.py b/storage/samples/snippets/storage_get_rpo.py index 29ae186fa42..ab40ca3a5f4 100644 --- a/storage/samples/snippets/storage_get_rpo.py +++ b/storage/samples/snippets/storage_get_rpo.py @@ -25,7 +25,6 @@ # [START storage_get_rpo] from google.cloud import storage -from google.cloud.storage.constants import RPO_DEFAULT def get_rpo(bucket_name): @@ -34,9 +33,7 @@ def get_rpo(bucket_name): # bucket_name = "my-bucket" storage_client = storage.Client() - bucket = storage_client.bucket(bucket_name) - - bucket.rpo = RPO_DEFAULT + bucket = storage_client.get_bucket(bucket_name) rpo = bucket.rpo print(f"RPO for {bucket.name} is {rpo}.") From 68dd182174300cd7d7f723ba5e6776b401c9c5da Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Fri, 5 Jan 2024 00:06:56 +0100 Subject: [PATCH 131/172] chore(deps): update dependency pytest to v7.4.4 (#1204) Co-authored-by: cojenco --- storage/samples/snippets/requirements-test.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 52e47f6e303..9035a0f9176 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,3 @@ -pytest==7.4.3 +pytest==7.4.4 mock==5.1.0 backoff==2.2.1 \ No newline at end of file From 8fefcdc959b1912f281e2b4a69802469f825adf9 Mon Sep 17 00:00:00 2001 From: cojenco Date: Wed, 28 Feb 2024 13:25:37 -0800 Subject: [PATCH 132/172] samples: replace deprecated method (#1211) * samples: replace deprecated method * update print statement, decode --- storage/samples/snippets/snippets_test.py | 4 ++-- storage/samples/snippets/storage_download_into_memory.py | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 7add151845e..ff1d230057f 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -361,7 +361,7 @@ def test_generate_upload_signed_url_v4(test_bucket, capsys): bucket = storage.Client().bucket(test_bucket.name) blob = bucket.blob(blob_name) - assert blob.download_as_string() == content + assert blob.download_as_bytes() == content def test_generate_signed_policy_v4(test_bucket, capsys): @@ -592,7 +592,7 @@ def test_storage_compose_file(test_bucket): source_files[1], dest_file.name, ) - composed = destination.download_as_string() + composed = destination.download_as_bytes() assert composed.decode("utf-8") == source_files[0] + source_files[1] diff --git a/storage/samples/snippets/storage_download_into_memory.py b/storage/samples/snippets/storage_download_into_memory.py index 453a13e2112..97f677054d5 100644 --- a/storage/samples/snippets/storage_download_into_memory.py +++ b/storage/samples/snippets/storage_download_into_memory.py @@ -37,11 +37,11 @@ def download_blob_into_memory(bucket_name, blob_name): # any content from Google Cloud Storage. As we don't need additional data, # using `Bucket.blob` is preferred here. blob = bucket.blob(blob_name) - contents = blob.download_as_string() + contents = blob.download_as_bytes() print( - "Downloaded storage object {} from bucket {} as the following string: {}.".format( - blob_name, bucket_name, contents + "Downloaded storage object {} from bucket {} as the following bytes object: {}.".format( + blob_name, bucket_name, contents.decode("utf-8") ) ) From 603f76e58a5206ebeb1b16a452a5cb6a969e311f Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Thu, 28 Mar 2024 17:46:17 +0100 Subject: [PATCH 133/172] chore(deps): update all dependencies (#1213) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [filelock](https://togithub.com/tox-dev/py-filelock) | `==3.13.1` -> `==3.13.3` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/filelock/3.13.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/filelock/3.13.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/filelock/3.13.1/3.13.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/filelock/3.13.1/3.13.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [google-cloud-pubsub](https://togithub.com/googleapis/python-pubsub) | `==2.19.0` -> `==2.21.0` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/google-cloud-pubsub/2.21.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/google-cloud-pubsub/2.21.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/google-cloud-pubsub/2.19.0/2.21.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/google-cloud-pubsub/2.19.0/2.21.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | `==2.14.0` -> `==2.16.0` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/google-cloud-storage/2.16.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/google-cloud-storage/2.16.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/google-cloud-storage/2.14.0/2.16.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/google-cloud-storage/2.14.0/2.16.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [pandas](https://pandas.pydata.org) ([source](https://togithub.com/pandas-dev/pandas)) | `==2.1.4` -> `==2.2.1` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/pandas/2.2.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/pandas/2.2.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/pandas/2.1.4/2.2.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/pandas/2.1.4/2.2.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [pytest](https://togithub.com/pytest-dev/pytest) ([changelog](https://docs.pytest.org/en/stable/changelog.html)) | `==7.4.4` -> `==8.1.1` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/pytest/8.1.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/pytest/8.1.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/pytest/7.4.4/8.1.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/pytest/7.4.4/8.1.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes
tox-dev/py-filelock (filelock) ### [`v3.13.3`](https://togithub.com/tox-dev/filelock/releases/tag/3.13.3) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.13.2...3.13.3) #### What's Changed - Make singleton class instance dict unique per subclass by [@​nefrob](https://togithub.com/nefrob) in [https://togithub.com/tox-dev/filelock/pull/318](https://togithub.com/tox-dev/filelock/pull/318) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.13.2...3.13.3 ### [`v3.13.2`](https://togithub.com/tox-dev/filelock/releases/tag/3.13.2) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.13.1...3.13.2) ##### What's Changed - Fixed small typo in \_unix.py by [@​snemes](https://togithub.com/snemes) in [https://togithub.com/tox-dev/filelock/pull/302](https://togithub.com/tox-dev/filelock/pull/302) - Update SECURITY.md to reflect Python 3.7 support dropoff by [@​kemzeb](https://togithub.com/kemzeb) in [https://togithub.com/tox-dev/filelock/pull/304](https://togithub.com/tox-dev/filelock/pull/304) - Update index.rst to improve the demo usage by [@​youkaichao](https://togithub.com/youkaichao) in [https://togithub.com/tox-dev/filelock/pull/314](https://togithub.com/tox-dev/filelock/pull/314) - \[BugFix] fix permission denied error when lock file is placed in `/tmp` by [@​kota-iizuka](https://togithub.com/kota-iizuka) in [https://togithub.com/tox-dev/filelock/pull/317](https://togithub.com/tox-dev/filelock/pull/317) ##### New Contributors - [@​snemes](https://togithub.com/snemes) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/302](https://togithub.com/tox-dev/filelock/pull/302) - [@​kemzeb](https://togithub.com/kemzeb) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/304](https://togithub.com/tox-dev/filelock/pull/304) - [@​youkaichao](https://togithub.com/youkaichao) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/314](https://togithub.com/tox-dev/filelock/pull/314) - [@​kota-iizuka](https://togithub.com/kota-iizuka) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/317](https://togithub.com/tox-dev/filelock/pull/317) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.13.1...3.13.2
googleapis/python-pubsub (google-cloud-pubsub) ### [`v2.21.0`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2210-2024-03-26) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.20.3...v2.21.0) ##### Features - Add custom datetime format for Cloud Storage subscriptions ([#​1131](https://togithub.com/googleapis/python-pubsub/issues/1131)) ([4da6744](https://togithub.com/googleapis/python-pubsub/commit/4da67441ddab01a173620d8c03bc640271c785c6)) ### [`v2.20.3`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2203-2024-03-21) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.20.2...v2.20.3) ##### Documentation - **samples:** Update Region Tags ([#​1128](https://togithub.com/googleapis/python-pubsub/issues/1128)) ([e3bc89e](https://togithub.com/googleapis/python-pubsub/commit/e3bc89eaa51337c93144d6c3100486353d494ad9)) ### [`v2.20.2`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2202-2024-03-15) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.20.1...v2.20.2) ##### Documentation - **samples:** Add Create Topic with Kinesis IngestionDataSourceSettings Sample ([#​1120](https://togithub.com/googleapis/python-pubsub/issues/1120)) ([83dc9ff](https://togithub.com/googleapis/python-pubsub/commit/83dc9fff13aa35518fb9b6a73472816da852d975)) - **samples:** Update Topic with Kinesis Ingestion Settings ([#​1123](https://togithub.com/googleapis/python-pubsub/issues/1123)) ([e0e2d83](https://togithub.com/googleapis/python-pubsub/commit/e0e2d831da8d17288c3ae8900bea2388ce8758af)) ### [`v2.20.1`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2201-2024-03-06) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.20.0...v2.20.1) ##### Bug Fixes - Catch and surface BaseException() ([#​1108](https://togithub.com/googleapis/python-pubsub/issues/1108)) ([07e427f](https://togithub.com/googleapis/python-pubsub/commit/07e427f675464b9aa79c68dede67082529054980)) ### [`v2.20.0`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2200-2024-03-05) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.19.8...v2.20.0) ##### Features - Add include_recaptcha_script for as a new action in firewall policies ([#​1109](https://togithub.com/googleapis/python-pubsub/issues/1109)) ([54041a5](https://togithub.com/googleapis/python-pubsub/commit/54041a527398eb0ec5daa97a346ba3202ce349f3)) ##### Documentation - **samples:** Correct type and description of `timeout` parameter in subscriber quickstart ([#​1051](https://togithub.com/googleapis/python-pubsub/issues/1051)) ([141a473](https://togithub.com/googleapis/python-pubsub/commit/141a473561bd0e45d3137a02cbefddb454ab3af4)) ### [`v2.19.8`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2198-2024-03-05) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.19.7...v2.19.8) ##### Bug Fixes - **deps:** Exclude google-auth 2.24.0 and 2.25.0 ([#​1102](https://togithub.com/googleapis/python-pubsub/issues/1102)) ([165c983](https://togithub.com/googleapis/python-pubsub/commit/165c983803c48a17141765395cf9ec2e6a7056fa)) ### [`v2.19.7`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2197-2024-02-24) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.19.6...v2.19.7) ##### Bug Fixes - **deps:** Require `google-api-core>=1.34.1` ([#​1080](https://togithub.com/googleapis/python-pubsub/issues/1080)) ([1a5a134](https://togithub.com/googleapis/python-pubsub/commit/1a5a1342de8736c6a2b1ac63476667f8a02b5bb8)) ### [`v2.19.6`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2196-2024-02-23) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.19.5...v2.19.6) ##### Bug Fixes - Remove LOGGER.exception() line ([#​1087](https://togithub.com/googleapis/python-pubsub/issues/1087)) ([a395d26](https://togithub.com/googleapis/python-pubsub/commit/a395d26ed0fffaee8662f988da97dd35c480af4f)) ### [`v2.19.5`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2195-2024-02-22) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.19.4...v2.19.5) ##### Bug Fixes - Update system_test_python_versions ([#​1096](https://togithub.com/googleapis/python-pubsub/issues/1096)) ([c659ac7](https://togithub.com/googleapis/python-pubsub/commit/c659ac777f177e54d7272a8de93fa9f554b15d46)) ### [`v2.19.4`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2194-2024-02-09) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.19.3...v2.19.4) ##### Bug Fixes - **diregapic:** S/bazel/bazelisk/ in DIREGAPIC build GitHub action ([#​1064](https://togithub.com/googleapis/python-pubsub/issues/1064)) ([d56ad12](https://togithub.com/googleapis/python-pubsub/commit/d56ad12f197e9e379d2a4a0a38be108808985c23)) ### [`v2.19.3`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2193-2024-02-08) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.19.2...v2.19.3) ##### Bug Fixes - Add google-auth as a direct dependency ([#​1076](https://togithub.com/googleapis/python-pubsub/issues/1076)) ([5ce7301](https://togithub.com/googleapis/python-pubsub/commit/5ce7301b3056191203bc89bbcf1f33083de72a2d)) ### [`v2.19.2`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2192-2024-02-08) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.19.1...v2.19.2) ##### Bug Fixes - Unit test failures in https://togithub.com/googleapis/python-pubsu… ([#​1074](https://togithub.com/googleapis/python-pubsub/issues/1074)) ([3c6d128](https://togithub.com/googleapis/python-pubsub/commit/3c6d128a53d83439036aaec1f1fd48331152935b)) ### [`v2.19.1`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2191-2024-02-02) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.19.0...v2.19.1) ##### Documentation - **samples:** Swap writer and reader schema to correct places ([265f410](https://togithub.com/googleapis/python-pubsub/commit/265f4106f499ec5d2d01a127ba192404c1836a28))
googleapis/python-storage (google-cloud-storage) ### [`v2.16.0`](https://togithub.com/googleapis/python-storage/blob/HEAD/CHANGELOG.md#2160-2024-03-18) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v2.15.0...v2.16.0) ##### Features - Add support for soft delete ([#​1229](https://togithub.com/googleapis/python-storage/issues/1229)) ([3928aa0](https://togithub.com/googleapis/python-storage/commit/3928aa0680ec03addae1f792c73abb5c9dc8586f)) - Support includeFoldersAsPrefixes ([#​1223](https://togithub.com/googleapis/python-storage/issues/1223)) ([7bb8065](https://togithub.com/googleapis/python-storage/commit/7bb806538cf3d7a5e16390db1983620933d5e51a)) ### [`v2.15.0`](https://togithub.com/googleapis/python-storage/blob/HEAD/CHANGELOG.md#2150-2024-02-28) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v2.14.0...v2.15.0) ##### Features - Support custom universe domains/TPC ([#​1212](https://togithub.com/googleapis/python-storage/issues/1212)) ([f4cf041](https://togithub.com/googleapis/python-storage/commit/f4cf041a5f2075cecf5f4993f8b7afda0476a52b)) ##### Bug Fixes - Add "updated" as property for Bucket ([#​1220](https://togithub.com/googleapis/python-storage/issues/1220)) ([ae9a53b](https://togithub.com/googleapis/python-storage/commit/ae9a53b464e7d82c79a019a4111c49a4cdcc3ae0)) - Remove utcnow usage ([#​1215](https://togithub.com/googleapis/python-storage/issues/1215)) ([8d8a53a](https://togithub.com/googleapis/python-storage/commit/8d8a53a1368392ad7a1c4352f559c12932c5a9c9))
pandas-dev/pandas (pandas) ### [`v2.2.1`](https://togithub.com/pandas-dev/pandas/releases/tag/v2.2.1): Pandas 2.2.1 [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v2.2.0...v2.2.1) We are pleased to announce the release of pandas 2.2.1. This release includes some new features, bug fixes, and performance improvements. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/2.2.1/whatsnew/v2.2.1.html) for a list of all the changes. Pandas 2.2.1 supports Python 3.9 and higher. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker](https://togithub.com/pandas-dev/pandas/issues). Thanks to all the contributors who made this release possible. ### [`v2.2.0`](https://togithub.com/pandas-dev/pandas/compare/v2.1.4...v2.2.0) [Compare Source](https://togithub.com/pandas-dev/pandas/compare/v2.1.4...v2.2.0)
pytest-dev/pytest (pytest) ### [`v8.1.1`](https://togithub.com/pytest-dev/pytest/releases/tag/8.1.1) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/8.1.0...8.1.1) # pytest 8.1.1 (2024-03-08) ::: {.note} ::: {.title} Note ::: This release is not a usual bug fix release -- it contains features and improvements, being a follow up to `8.1.0`, which has been yanked from PyPI. ::: ## Features - [#​11475](https://togithub.com/pytest-dev/pytest/issues/11475): Added the new `consider_namespace_packages`{.interpreted-text role="confval"} configuration option, defaulting to `False`. If set to `True`, pytest will attempt to identify modules that are part of [namespace packages](https://packaging.python.org/en/latest/guides/packaging-namespace-packages) when importing modules. - [#​11653](https://togithub.com/pytest-dev/pytest/issues/11653): Added the new `verbosity_test_cases`{.interpreted-text role="confval"} configuration option for fine-grained control of test execution verbosity. See `Fine-grained verbosity `{.interpreted-text role="ref"} for more details. ## Improvements - [#​10865](https://togithub.com/pytest-dev/pytest/issues/10865): `pytest.warns`{.interpreted-text role="func"} now validates that `warnings.warn`{.interpreted-text role="func"} was called with a \[str]{.title-ref} or a \[Warning]{.title-ref}. Currently in Python it is possible to use other types, however this causes an exception when `warnings.filterwarnings`{.interpreted-text role="func"} is used to filter those warnings (see [CPython #​103577](https://togithub.com/python/cpython/issues/103577) for a discussion). While this can be considered a bug in CPython, we decided to put guards in pytest as the error message produced without this check in place is confusing. - [#​11311](https://togithub.com/pytest-dev/pytest/issues/11311): When using `--override-ini` for paths in invocations without a configuration file defined, the current working directory is used as the relative directory. Previoulsy this would raise an `AssertionError`{.interpreted-text role="class"}. - [#​11475](https://togithub.com/pytest-dev/pytest/issues/11475): `--import-mode=importlib `{.interpreted-text role="ref"} now tries to import modules using the standard import mechanism (but still without changing :py`sys.path`{.interpreted-text role="data"}), falling back to importing modules directly only if that fails. This means that installed packages will be imported under their canonical name if possible first, for example `app.core.models`, instead of having the module name always be derived from their path (for example `.env310.lib.site_packages.app.core.models`). - [#​11801](https://togithub.com/pytest-dev/pytest/issues/11801): Added the `iter_parents() <_pytest.nodes.Node.iter_parents>`{.interpreted-text role="func"} helper method on nodes. It is similar to `listchain <_pytest.nodes.Node.listchain>`{.interpreted-text role="func"}, but goes from bottom to top, and returns an iterator, not a list. - [#​11850](https://togithub.com/pytest-dev/pytest/issues/11850): Added support for `sys.last_exc`{.interpreted-text role="data"} for post-mortem debugging on Python>=3.12. - [#​11962](https://togithub.com/pytest-dev/pytest/issues/11962): In case no other suitable candidates for configuration file are found, a `pyproject.toml` (even without a `[tool.pytest.ini_options]` table) will be considered as the configuration file and define the `rootdir`. - [#​11978](https://togithub.com/pytest-dev/pytest/issues/11978): Add `--log-file-mode` option to the logging plugin, enabling appending to log-files. This option accepts either `"w"` or `"a"` and defaults to `"w"`. Previously, the mode was hard-coded to be `"w"` which truncates the file before logging. - [#​12047](https://togithub.com/pytest-dev/pytest/issues/12047): When multiple finalizers of a fixture raise an exception, now all exceptions are reported as an exception group. Previously, only the first exception was reported. ## Bug Fixes - [#​11475](https://togithub.com/pytest-dev/pytest/issues/11475): Fixed regression where `--importmode=importlib` would import non-test modules more than once. - [#​11904](https://togithub.com/pytest-dev/pytest/issues/11904): Fixed a regression in pytest 8.0.0 that would cause test collection to fail due to permission errors when using `--pyargs`. This change improves the collection tree for tests specified using `--pyargs`, see `12043`{.interpreted-text role="pull"} for a comparison with pytest 8.0 and <8. - [#​12011](https://togithub.com/pytest-dev/pytest/issues/12011): Fixed a regression in 8.0.1 whereby `setup_module` xunit-style fixtures are not executed when `--doctest-modules` is passed. - [#​12014](https://togithub.com/pytest-dev/pytest/issues/12014): Fix the `stacklevel` used when warning about marks used on fixtures. - [#​12039](https://togithub.com/pytest-dev/pytest/issues/12039): Fixed a regression in `8.0.2` where tests created using `tmp_path`{.interpreted-text role="fixture"} have been collected multiple times in CI under Windows. ## Improved Documentation - [#​11790](https://togithub.com/pytest-dev/pytest/issues/11790): Documented the retention of temporary directories created using the `tmp_path` fixture in more detail. ## Trivial/Internal Changes - [#​11785](https://togithub.com/pytest-dev/pytest/issues/11785): Some changes were made to private functions which may affect plugins which access them: - `FixtureManager._getautousenames()` now takes a `Node` itself instead of the nodeid. - `FixtureManager.getfixturedefs()` now takes the `Node` itself instead of the nodeid. - The `_pytest.nodes.iterparentnodeids()` function is removed without replacement. Prefer to traverse the node hierarchy itself instead. If you really need to, copy the function from the previous pytest release. - [#​12069](https://togithub.com/pytest-dev/pytest/issues/12069): Delayed the deprecation of the following features to `9.0.0`: - `node-ctor-fspath-deprecation`{.interpreted-text role="ref"}. - `legacy-path-hooks-deprecated`{.interpreted-text role="ref"}. It was discovered after `8.1.0` was released that the warnings about the impeding removal were not being displayed, so the team decided to revert the removal. This is the reason for `8.1.0` being yanked. # pytest 8.1.0 (YANKED) ::: {.note} ::: {.title} Note ::: This release has been **yanked**: it broke some plugins without the proper warning period, due to some warnings not showing up as expected. See [#​12069](https://togithub.com/pytest-dev/pytest/issues/12069). ::: ### [`v8.1.0`](https://togithub.com/pytest-dev/pytest/releases/tag/8.1.0) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/8.0.2...8.1.0) # pytest 8.1.0 (YANKED) > \[!IMPORTANT]\ > This release has been **yanked**: it broke some plugins without the proper warning period, due to some warnings not showing up as expected. See [#​12069](https://togithub.com/pytest-dev/pytest/issues/12069). ## Features - [#​11475](https://togithub.com/pytest-dev/pytest/issues/11475): Added the new `consider_namespace_packages`{.interpreted-text role="confval"} configuration option, defaulting to `False`. If set to `True`, pytest will attempt to identify modules that are part of [namespace packages](https://packaging.python.org/en/latest/guides/packaging-namespace-packages) when importing modules. - [#​11653](https://togithub.com/pytest-dev/pytest/issues/11653): Added the new `verbosity_test_cases`{.interpreted-text role="confval"} configuration option for fine-grained control of test execution verbosity. See `Fine-grained verbosity `{.interpreted-text role="ref"} for more details. ## Improvements - [#​10865](https://togithub.com/pytest-dev/pytest/issues/10865): `pytest.warns`{.interpreted-text role="func"} now validates that `warnings.warn`{.interpreted-text role="func"} was called with a \[str]{.title-ref} or a \[Warning]{.title-ref}. Currently in Python it is possible to use other types, however this causes an exception when `warnings.filterwarnings`{.interpreted-text role="func"} is used to filter those warnings (see [CPython #​103577](https://togithub.com/python/cpython/issues/103577) for a discussion). While this can be considered a bug in CPython, we decided to put guards in pytest as the error message produced without this check in place is confusing. - [#​11311](https://togithub.com/pytest-dev/pytest/issues/11311): When using `--override-ini` for paths in invocations without a configuration file defined, the current working directory is used as the relative directory. Previoulsy this would raise an `AssertionError`{.interpreted-text role="class"}. - [#​11475](https://togithub.com/pytest-dev/pytest/issues/11475): `--import-mode=importlib `{.interpreted-text role="ref"} now tries to import modules using the standard import mechanism (but still without changing :py`sys.path`{.interpreted-text role="data"}), falling back to importing modules directly only if that fails. This means that installed packages will be imported under their canonical name if possible first, for example `app.core.models`, instead of having the module name always be derived from their path (for example `.env310.lib.site_packages.app.core.models`). - [#​11801](https://togithub.com/pytest-dev/pytest/issues/11801): Added the `iter_parents() <_pytest.nodes.Node.iter_parents>`{.interpreted-text role="func"} helper method on nodes. It is similar to `listchain <_pytest.nodes.Node.listchain>`{.interpreted-text role="func"}, but goes from bottom to top, and returns an iterator, not a list. - [#​11850](https://togithub.com/pytest-dev/pytest/issues/11850): Added support for `sys.last_exc`{.interpreted-text role="data"} for post-mortem debugging on Python>=3.12. - [#​11962](https://togithub.com/pytest-dev/pytest/issues/11962): In case no other suitable candidates for configuration file are found, a `pyproject.toml` (even without a `[tool.pytest.ini_options]` table) will be considered as the configuration file and define the `rootdir`. - [#​11978](https://togithub.com/pytest-dev/pytest/issues/11978): Add `--log-file-mode` option to the logging plugin, enabling appending to log-files. This option accepts either `"w"` or `"a"` and defaults to `"w"`. Previously, the mode was hard-coded to be `"w"` which truncates the file before logging. - [#​12047](https://togithub.com/pytest-dev/pytest/issues/12047): When multiple finalizers of a fixture raise an exception, now all exceptions are reported as an exception group. Previously, only the first exception was reported. ## Bug Fixes - [#​11904](https://togithub.com/pytest-dev/pytest/issues/11904): Fixed a regression in pytest 8.0.0 that would cause test collection to fail due to permission errors when using `--pyargs`. This change improves the collection tree for tests specified using `--pyargs`, see `12043`{.interpreted-text role="pull"} for a comparison with pytest 8.0 and <8. - [#​12011](https://togithub.com/pytest-dev/pytest/issues/12011): Fixed a regression in 8.0.1 whereby `setup_module` xunit-style fixtures are not executed when `--doctest-modules` is passed. - [#​12014](https://togithub.com/pytest-dev/pytest/issues/12014): Fix the `stacklevel` used when warning about marks used on fixtures. - [#​12039](https://togithub.com/pytest-dev/pytest/issues/12039): Fixed a regression in `8.0.2` where tests created using `tmp_path`{.interpreted-text role="fixture"} have been collected multiple times in CI under Windows. ## Improved Documentation - [#​11790](https://togithub.com/pytest-dev/pytest/issues/11790): Documented the retention of temporary directories created using the `tmp_path` fixture in more detail. ## Trivial/Internal Changes - [#​11785](https://togithub.com/pytest-dev/pytest/issues/11785): Some changes were made to private functions which may affect plugins which access them: - `FixtureManager._getautousenames()` now takes a `Node` itself instead of the nodeid. - `FixtureManager.getfixturedefs()` now takes the `Node` itself instead of the nodeid. - The `_pytest.nodes.iterparentnodeids()` function is removed without replacement. Prefer to traverse the node hierarchy itself instead. If you really need to, copy the function from the previous pytest release. ### [`v8.0.2`](https://togithub.com/pytest-dev/pytest/releases/tag/8.0.2) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/8.0.1...8.0.2) # pytest 8.0.2 (2024-02-24) ## Bug Fixes - [#​11895](https://togithub.com/pytest-dev/pytest/issues/11895): Fix collection on Windows where initial paths contain the short version of a path (for example `c:\PROGRA~1\tests`). - [#​11953](https://togithub.com/pytest-dev/pytest/issues/11953): Fix an `IndexError` crash raising from `getstatementrange_ast`. - [#​12021](https://togithub.com/pytest-dev/pytest/issues/12021): Reverted a fix to \[--maxfail]{.title-ref} handling in pytest 8.0.0 because it caused a regression in pytest-xdist whereby session fixture teardowns may get executed multiple times when the max-fails is reached. ### [`v8.0.1`](https://togithub.com/pytest-dev/pytest/releases/tag/8.0.1) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/8.0.0...8.0.1) # pytest 8.0.1 (2024-02-16) ## Bug Fixes - [#​11875](https://togithub.com/pytest-dev/pytest/issues/11875): Correctly handle errors from `getpass.getuser`{.interpreted-text role="func"} in Python 3.13. - [#​11879](https://togithub.com/pytest-dev/pytest/issues/11879): Fix an edge case where `ExceptionInfo._stringify_exception` could crash `pytest.raises`{.interpreted-text role="func"}. - [#​11906](https://togithub.com/pytest-dev/pytest/issues/11906): Fix regression with `pytest.warns`{.interpreted-text role="func"} using custom warning subclasses which have more than one parameter in their \[\__init\_\_]{.title-ref}. - [#​11907](https://togithub.com/pytest-dev/pytest/issues/11907): Fix a regression in pytest 8.0.0 whereby calling `pytest.skip`{.interpreted-text role="func"} and similar control-flow exceptions within a `pytest.warns()`{.interpreted-text role="func"} block would get suppressed instead of propagating. - [#​11929](https://togithub.com/pytest-dev/pytest/issues/11929): Fix a regression in pytest 8.0.0 whereby autouse fixtures defined in a module get ignored by the doctests in the module. - [#​11937](https://togithub.com/pytest-dev/pytest/issues/11937): Fix a regression in pytest 8.0.0 whereby items would be collected in reverse order in some circumstances. ### [`v8.0.0`](https://togithub.com/pytest-dev/pytest/releases/tag/8.0.0): pytest 8.0.0 (2024-01-27) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/7.4.4...8.0.0) See [8.0.0rc1](https://togithub.com/pytest-dev/pytest/releases/tag/8.0.0rc1) and [8.0.0rc2](https://togithub.com/pytest-dev/pytest/releases/tag/8.0.0rc2) for the full changes since pytest 7.4! #### Bug Fixes - [#​11842](https://togithub.com/pytest-dev/pytest/issues/11842): Properly escape the `reason` of a `skip `{.interpreted-text role="ref"} mark when writing JUnit XML files. - [#​11861](https://togithub.com/pytest-dev/pytest/issues/11861): Avoid microsecond exceeds `1_000_000` when using `log-date-format` with `%f` specifier, which might cause the test suite to crash.
--- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/googleapis/python-storage). --- storage/samples/snippets/requirements-test.txt | 5 +++-- storage/samples/snippets/requirements.txt | 6 +++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 9035a0f9176..86a3ade300a 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,3 +1,4 @@ -pytest==7.4.4 +pytest===7.4.4; python_version == '3.7' +pytest==8.1.1; python_version >= '3.8' mock==5.1.0 -backoff==2.2.1 \ No newline at end of file +backoff==2.2.1 diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 15a68497302..782275255c1 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,5 +1,5 @@ -google-cloud-pubsub==2.19.0 -google-cloud-storage==2.14.0 +google-cloud-pubsub==2.21.0 +google-cloud-storage==2.16.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' -pandas==2.1.4; python_version >= '3.9' +pandas==2.2.1; python_version >= '3.9' From a1d1c8dc9ea53eb06dca5dc6e7b7129554e1ea63 Mon Sep 17 00:00:00 2001 From: cojenco Date: Thu, 11 Apr 2024 12:09:43 -0700 Subject: [PATCH 134/172] samples: add samples for object retention (#1247) --- storage/samples/snippets/snippets_test.py | 23 +++++++ .../storage_create_bucket_object_retention.py | 38 +++++++++++ .../snippets/storage_get_bucket_metadata.py | 1 + .../samples/snippets/storage_get_metadata.py | 2 + .../storage_set_object_retention_policy.py | 67 +++++++++++++++++++ 5 files changed, 131 insertions(+) create mode 100644 storage/samples/snippets/storage_create_bucket_object_retention.py create mode 100644 storage/samples/snippets/storage_set_object_retention_policy.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index ff1d230057f..f9851b0ec8b 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -37,6 +37,7 @@ import storage_cors_configuration import storage_create_bucket_class_location import storage_create_bucket_dual_region +import storage_create_bucket_object_retention import storage_define_bucket_website_configuration import storage_delete_file import storage_delete_file_archived_generation @@ -71,6 +72,7 @@ import storage_set_autoclass import storage_set_bucket_default_kms_key import storage_set_client_endpoint +import storage_set_object_retention_policy import storage_set_metadata import storage_transfer_manager_download_bucket import storage_transfer_manager_download_chunks_concurrently @@ -818,3 +820,24 @@ def test_transfer_manager_upload_chunks_concurrently(test_bucket, capsys): out, _ = capsys.readouterr() assert "File {} uploaded to {}".format(file.name, BLOB_NAME) in out + + +def test_object_retention_policy(test_bucket_create, capsys): + storage_create_bucket_object_retention.create_bucket_object_retention( + test_bucket_create.name + ) + out, _ = capsys.readouterr() + assert f"Created bucket {test_bucket_create.name} with object retention enabled setting" in out + + blob_name = "test_object_retention" + storage_set_object_retention_policy.set_object_retention_policy( + test_bucket_create.name, "hello world", blob_name + ) + out, _ = capsys.readouterr() + assert f"Retention policy for file {blob_name}" in out + + # Remove retention policy for test cleanup + blob = test_bucket_create.blob(blob_name) + blob.retention.mode = None + blob.retention.retain_until_time = None + blob.patch(override_unlocked_retention=True) diff --git a/storage/samples/snippets/storage_create_bucket_object_retention.py b/storage/samples/snippets/storage_create_bucket_object_retention.py new file mode 100644 index 00000000000..4ebc32c0a25 --- /dev/null +++ b/storage/samples/snippets/storage_create_bucket_object_retention.py @@ -0,0 +1,38 @@ +#!/usr/bin/env python + +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_create_bucket_with_object_retention] +from google.cloud import storage + + +def create_bucket_object_retention(bucket_name): + """Creates a bucket with object retention enabled.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.create_bucket(bucket_name, enable_object_retention=True) + + print(f"Created bucket {bucket_name} with object retention enabled setting: {bucket.object_retention_mode}") + + +# [END storage_create_bucket_with_object_retention] + + +if __name__ == "__main__": + create_bucket_object_retention(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_get_bucket_metadata.py b/storage/samples/snippets/storage_get_bucket_metadata.py index 87cd5eddc4e..c86e154de10 100644 --- a/storage/samples/snippets/storage_get_bucket_metadata.py +++ b/storage/samples/snippets/storage_get_bucket_metadata.py @@ -44,6 +44,7 @@ def bucket_metadata(bucket_name): print(f"Retention Effective Time: {bucket.retention_policy_effective_time}") print(f"Retention Period: {bucket.retention_period}") print(f"Retention Policy Locked: {bucket.retention_policy_locked}") + print(f"Object Retention Mode: {bucket.object_retention_mode}") print(f"Requester Pays: {bucket.requester_pays}") print(f"Self Link: {bucket.self_link}") print(f"Time Created: {bucket.time_created}") diff --git a/storage/samples/snippets/storage_get_metadata.py b/storage/samples/snippets/storage_get_metadata.py index eece8028a85..7216efdb4de 100644 --- a/storage/samples/snippets/storage_get_metadata.py +++ b/storage/samples/snippets/storage_get_metadata.py @@ -59,6 +59,8 @@ def blob_metadata(bucket_name, blob_name): "Event based hold: ", "enabled" if blob.event_based_hold else "disabled", ) + print(f"Retention mode: {blob.retention.mode}") + print(f"Retention retain until time: {blob.retention.retain_until_time}") if blob.retention_expiration_time: print( f"retentionExpirationTime: {blob.retention_expiration_time}" diff --git a/storage/samples/snippets/storage_set_object_retention_policy.py b/storage/samples/snippets/storage_set_object_retention_policy.py new file mode 100644 index 00000000000..d0d3a54ec50 --- /dev/null +++ b/storage/samples/snippets/storage_set_object_retention_policy.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python + +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import datetime +import sys + +# [START storage_set_object_retention_policy] +from google.cloud import storage + + +def set_object_retention_policy(bucket_name, contents, destination_blob_name): + """Set the object retention policy of a file.""" + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + # The contents to upload to the file + # contents = "these are my contents" + + # The ID of your GCS object + # destination_blob_name = "storage-object-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(destination_blob_name) + blob.upload_from_string(contents) + + # Set the retention policy for the file. + blob.retention.mode = "Unlocked" + retention_date = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=10) + blob.retention.retain_until_time = retention_date + blob.patch() + print( + f"Retention policy for file {destination_blob_name} was set to: {blob.retention.mode}." + ) + + # To modify an existing policy on an unlocked file object, pass in the override parameter. + new_retention_date = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=9) + blob.retention.retain_until_time = new_retention_date + blob.patch(override_unlocked_retention=True) + print( + f"Retention policy for file {destination_blob_name} was updated to: {blob.retention.retain_until_time}." + ) + + +# [END storage_set_object_retention_policy] + + +if __name__ == "__main__": + set_object_retention_policy( + bucket_name=sys.argv[1], + contents=sys.argv[2], + destination_blob_name=sys.argv[3], + ) From 62fad403c7921b75032622ace67edf74cebe04db Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Fri, 12 Apr 2024 17:35:59 +0200 Subject: [PATCH 135/172] chore(deps): update all dependencies (#1250) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 782275255c1..f081966857a 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.21.0 +google-cloud-pubsub==2.21.1 google-cloud-storage==2.16.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' From 517e52b59948d7fe833e6e4ba52a83559b9b6a59 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Fri, 12 Apr 2024 18:02:21 +0200 Subject: [PATCH 136/172] chore(deps): update all dependencies (#1260) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index f081966857a..b0e41fa8445 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -2,4 +2,4 @@ google-cloud-pubsub==2.21.1 google-cloud-storage==2.16.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' -pandas==2.2.1; python_version >= '3.9' +pandas==2.2.2; python_version >= '3.9' From 1033c70533b86ad18a497a7d00b3b97374e7f775 Mon Sep 17 00:00:00 2001 From: cojenco Date: Thu, 16 May 2024 14:42:23 -0700 Subject: [PATCH 137/172] fix: remove deprecated methods in samples and tests (#1274) * chore: remove deprecated methods in samples and tests * update method --- storage/samples/snippets/encryption_test.py | 2 +- storage/samples/snippets/rpo_test.py | 4 ++-- .../snippets/storage_create_bucket_turbo_replication.py | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/storage/samples/snippets/encryption_test.py b/storage/samples/snippets/encryption_test.py index ff7a568e00a..9039b1fad8d 100644 --- a/storage/samples/snippets/encryption_test.py +++ b/storage/samples/snippets/encryption_test.py @@ -125,4 +125,4 @@ def test_object_csek_to_cmek(test_blob): BUCKET, test_blob_name, TEST_ENCRYPTION_KEY_2, KMS_KEY ) - assert cmek_blob.download_as_string(), test_blob_content + assert cmek_blob.download_as_bytes(), test_blob_content diff --git a/storage/samples/snippets/rpo_test.py b/storage/samples/snippets/rpo_test.py index befc0334ace..0dcf1574646 100644 --- a/storage/samples/snippets/rpo_test.py +++ b/storage/samples/snippets/rpo_test.py @@ -27,11 +27,11 @@ def dual_region_bucket(): """Yields a dual region bucket that is deleted after the test completes.""" bucket = None + location = "NAM4" while bucket is None or bucket.exists(): bucket_name = f"bucket-lock-{uuid.uuid4()}" bucket = storage.Client().bucket(bucket_name) - bucket.location = "NAM4" - bucket.create() + bucket.create(location=location) yield bucket bucket.delete(force=True) diff --git a/storage/samples/snippets/storage_create_bucket_turbo_replication.py b/storage/samples/snippets/storage_create_bucket_turbo_replication.py index 3d26616ec61..bc05597958f 100644 --- a/storage/samples/snippets/storage_create_bucket_turbo_replication.py +++ b/storage/samples/snippets/storage_create_bucket_turbo_replication.py @@ -35,9 +35,9 @@ def create_bucket_turbo_replication(bucket_name): storage_client = storage.Client() bucket = storage_client.bucket(bucket_name) - bucket.location = "NAM4" + bucket_location = "NAM4" bucket.rpo = RPO_ASYNC_TURBO - bucket.create() + bucket.create(location=bucket_location) print(f"{bucket.name} created with the recovery point objective (RPO) set to {bucket.rpo} in {bucket.location}.") From 9bba2aa3b86bb5f5b130b9708e18229a13d16450 Mon Sep 17 00:00:00 2001 From: cojenco Date: Tue, 11 Jun 2024 09:55:14 -0700 Subject: [PATCH 138/172] samples: create bucket with HNS enabled (#1285) * samples: create bucket with HNS enabled * allow sample tests to run in specific runtimes --- storage/samples/snippets/noxfile_config.py | 2 +- storage/samples/snippets/snippets_test.py | 9 ++++ ...ge_create_bucket_hierarchical_namespace.py | 41 +++++++++++++++++++ 3 files changed, 51 insertions(+), 1 deletion(-) create mode 100644 storage/samples/snippets/storage_create_bucket_hierarchical_namespace.py diff --git a/storage/samples/snippets/noxfile_config.py b/storage/samples/snippets/noxfile_config.py index 4f184ede095..17a05b9f27d 100644 --- a/storage/samples/snippets/noxfile_config.py +++ b/storage/samples/snippets/noxfile_config.py @@ -78,7 +78,7 @@ def get_cloud_kms_key(): TEST_CONFIG_OVERRIDE = { # You can opt out from the test for specific Python versions. - 'ignored_versions': ["2.7", "3.6"], + 'ignored_versions': ["2.7", "3.6", "3.7", "3.11", "3.12"], # An envvar key for determining the project id to use. Change it # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index f9851b0ec8b..8c021f87066 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -37,6 +37,7 @@ import storage_cors_configuration import storage_create_bucket_class_location import storage_create_bucket_dual_region +import storage_create_bucket_hierarchical_namespace import storage_create_bucket_object_retention import storage_define_bucket_website_configuration import storage_delete_file @@ -841,3 +842,11 @@ def test_object_retention_policy(test_bucket_create, capsys): blob.retention.mode = None blob.retention.retain_until_time = None blob.patch(override_unlocked_retention=True) + + +def test_create_bucket_hierarchical_namespace(test_bucket_create, capsys): + storage_create_bucket_hierarchical_namespace.create_bucket_hierarchical_namespace( + test_bucket_create.name + ) + out, _ = capsys.readouterr() + assert f"Created bucket {test_bucket_create.name} with hierarchical namespace enabled" in out diff --git a/storage/samples/snippets/storage_create_bucket_hierarchical_namespace.py b/storage/samples/snippets/storage_create_bucket_hierarchical_namespace.py new file mode 100644 index 00000000000..d9d31077251 --- /dev/null +++ b/storage/samples/snippets/storage_create_bucket_hierarchical_namespace.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_create_bucket_hierarchical_namespace] +from google.cloud import storage + + +def create_bucket_hierarchical_namespace(bucket_name): + """Creates a bucket with hierarchical namespace enabled.""" + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + bucket.iam_configuration.uniform_bucket_level_access_enabled = True + bucket.hierarchical_namespace_enabled = True + bucket.create() + + print(f"Created bucket {bucket_name} with hierarchical namespace enabled.") + + +# [END storage_create_bucket_hierarchical_namespace] + + +if __name__ == "__main__": + create_bucket_hierarchical_namespace(bucket_name=sys.argv[1]) From 159232edb1510dca668f6bd1accbe925ea42c90c Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 9 Jul 2024 19:30:18 +0200 Subject: [PATCH 139/172] chore(deps): update all dependencies (#1308) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | Type | Update | |---|---|---|---|---|---|---|---| | [argcomplete](https://togithub.com/kislyuk/argcomplete) ([changelog](https://togithub.com/kislyuk/argcomplete/blob/master/Changes.rst)) | `==3.2.3` -> `==3.4.0` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/argcomplete/3.4.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/argcomplete/3.4.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/argcomplete/3.2.3/3.4.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/argcomplete/3.2.3/3.4.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | minor | | [filelock](https://togithub.com/tox-dev/py-filelock) | `==3.13.1` -> `==3.15.4` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/filelock/3.15.4?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/filelock/3.15.4?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/filelock/3.13.1/3.15.4?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/filelock/3.13.1/3.15.4?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | minor | | [google-cloud-pubsub](https://togithub.com/googleapis/python-pubsub) | `==2.21.1` -> `==2.21.5` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/google-cloud-pubsub/2.21.5?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/google-cloud-pubsub/2.21.5?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/google-cloud-pubsub/2.21.1/2.21.5?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/google-cloud-pubsub/2.21.1/2.21.5?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | patch | | [google-cloud-storage](https://togithub.com/googleapis/python-storage) | `==2.16.0` -> `==2.17.0` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/google-cloud-storage/2.17.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/google-cloud-storage/2.17.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/google-cloud-storage/2.16.0/2.17.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/google-cloud-storage/2.16.0/2.17.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | minor | | [nox](https://togithub.com/wntrblm/nox) | `==2024.3.2` -> `==2024.4.15` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/nox/2024.4.15?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/nox/2024.4.15?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/nox/2024.3.2/2024.4.15?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/nox/2024.3.2/2024.4.15?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | minor | | [packaging](https://togithub.com/pypa/packaging) | `==24.0` -> `==24.1` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/packaging/24.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/packaging/24.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/packaging/24.0/24.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/packaging/24.0/24.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | minor | | [platformdirs](https://togithub.com/platformdirs/platformdirs) | `==4.2.0` -> `==4.2.2` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/platformdirs/4.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/platformdirs/4.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/platformdirs/4.2.0/4.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/platformdirs/4.2.0/4.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | patch | | [pytest](https://togithub.com/pytest-dev/pytest) ([changelog](https://docs.pytest.org/en/stable/changelog.html)) | `==8.1.1` -> `==8.2.2` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/pytest/8.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/pytest/8.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/pytest/8.1.1/8.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/pytest/8.1.1/8.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | minor | | ubuntu | `22.04` -> `24.04` | [![age](https://developer.mend.io/api/mc/badges/age/docker/ubuntu/noble?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/docker/ubuntu/noble?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/docker/ubuntu/22.04/noble?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/docker/ubuntu/22.04/noble?slim=true)](https://docs.renovatebot.com/merge-confidence/) | final | major | | [virtualenv](https://togithub.com/pypa/virtualenv) | `==20.25.1` -> `==20.26.3` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/virtualenv/20.26.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/virtualenv/20.26.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/virtualenv/20.25.1/20.26.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/virtualenv/20.25.1/20.26.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | minor | --- ### Release Notes
kislyuk/argcomplete (argcomplete) ### [`v3.4.0`](https://togithub.com/kislyuk/argcomplete/blob/HEAD/Changes.rst#Changes-for-v340-2024-06-16) [Compare Source](https://togithub.com/kislyuk/argcomplete/compare/v3.3.0...v3.4.0) \=============================== - No stdin for python calls from bash completion functions ([#​488](https://togithub.com/kislyuk/argcomplete/issues/488)) Prevents usage of stdin by (python) executables that are called during completion generation. This prevents the completion locking up the entire shell when the python script is broken i.e. it enters an interactive mode (REPL) instead of generating the completions, as expected. - Localize shell variable REPLY to avoid overwriting users’ value ([#​489](https://togithub.com/kislyuk/argcomplete/issues/489)) The variable REPLY is used by default by the `read` shell builtin to store the return value, and like all bash/zsh variables, is scoped globally. This change allows this variable to be used for other needs by appropriately scoping its internal use by an argcomplete utility function that uses `read`. ### [`v3.3.0`](https://togithub.com/kislyuk/argcomplete/blob/HEAD/Changes.rst#Changes-for-v330-2024-04-14) [Compare Source](https://togithub.com/kislyuk/argcomplete/compare/v3.2.3...v3.3.0) \=============================== - Preserve compatibility with argparse option tuples of length 4. This update is required to use argcomplete on Python 3.11.9+ or 3.12.3+.
tox-dev/py-filelock (filelock) ### [`v3.15.4`](https://togithub.com/tox-dev/filelock/releases/tag/3.15.4) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.15.3...3.15.4) #### What's Changed - Pass `file_lock` as positional argument by [@​kwist-sgr](https://togithub.com/kwist-sgr) in [https://togithub.com/tox-dev/filelock/pull/347](https://togithub.com/tox-dev/filelock/pull/347) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.15.3...3.15.4 ### [`v3.15.3`](https://togithub.com/tox-dev/filelock/releases/tag/3.15.3) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.15.2...3.15.3) #### What's Changed - Add test for virtualenv stability by [@​gaborbernat](https://togithub.com/gaborbernat) in [https://togithub.com/tox-dev/filelock/pull/344](https://togithub.com/tox-dev/filelock/pull/344) - Fix `TypeError: _CountedFileLock.__init__() got an unexpected keyword argument 'timeout'` by [@​kwist-sgr](https://togithub.com/kwist-sgr) in [https://togithub.com/tox-dev/filelock/pull/345](https://togithub.com/tox-dev/filelock/pull/345) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.15.2...3.15.3 ### [`v3.15.2`](https://togithub.com/tox-dev/filelock/releases/tag/3.15.2) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.15.1...3.15.2) #### What's Changed - Use a metaclass to implement the singleton pattern by [@​kwist-sgr](https://togithub.com/kwist-sgr) in [https://togithub.com/tox-dev/filelock/pull/340](https://togithub.com/tox-dev/filelock/pull/340) #### New Contributors - [@​kwist-sgr](https://togithub.com/kwist-sgr) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/340](https://togithub.com/tox-dev/filelock/pull/340) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.15.1...3.15.2 ### [`v3.15.1`](https://togithub.com/tox-dev/filelock/releases/tag/3.15.1) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.15.0...3.15.1) #### What's Changed - Hotfix: Restore **init** method; more robust initialization for singleton locks by [@​ethanbb](https://togithub.com/ethanbb) in [https://togithub.com/tox-dev/filelock/pull/338](https://togithub.com/tox-dev/filelock/pull/338) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.15.0...3.15.1 ### [`v3.15.0`](https://togithub.com/tox-dev/filelock/releases/tag/3.15.0) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.14.0...3.15.0) #### What's Changed - asyncio support by [@​Ovizro](https://togithub.com/Ovizro) in [https://togithub.com/tox-dev/filelock/pull/332](https://togithub.com/tox-dev/filelock/pull/332) - Don't initialize BaseFileLock when just returning existing instance by [@​ethanbb](https://togithub.com/ethanbb) in [https://togithub.com/tox-dev/filelock/pull/334](https://togithub.com/tox-dev/filelock/pull/334) #### New Contributors - [@​Ovizro](https://togithub.com/Ovizro) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/332](https://togithub.com/tox-dev/filelock/pull/332) - [@​ethanbb](https://togithub.com/ethanbb) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/334](https://togithub.com/tox-dev/filelock/pull/334) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.14.0...3.15.0 ### [`v3.14.0`](https://togithub.com/tox-dev/filelock/releases/tag/3.14.0) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.13.4...3.14.0) #### What's Changed - feat: `blocking` parameter on lock constructor with tests and docs by [@​iamkhav](https://togithub.com/iamkhav) in [https://togithub.com/tox-dev/filelock/pull/325](https://togithub.com/tox-dev/filelock/pull/325) #### New Contributors - [@​iamkhav](https://togithub.com/iamkhav) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/325](https://togithub.com/tox-dev/filelock/pull/325) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.13.4...3.14.0 ### [`v3.13.4`](https://togithub.com/tox-dev/filelock/releases/tag/3.13.4) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.13.3...3.13.4) #### What's Changed - Raise error on incompatible singleton timeout and mode args by [@​nefrob](https://togithub.com/nefrob) in [https://togithub.com/tox-dev/filelock/pull/320](https://togithub.com/tox-dev/filelock/pull/320) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.13.3...3.13.4 ### [`v3.13.3`](https://togithub.com/tox-dev/filelock/releases/tag/3.13.3) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.13.2...3.13.3) #### What's Changed - Make singleton class instance dict unique per subclass by [@​nefrob](https://togithub.com/nefrob) in [https://togithub.com/tox-dev/filelock/pull/318](https://togithub.com/tox-dev/filelock/pull/318) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.13.2...3.13.3 ### [`v3.13.2`](https://togithub.com/tox-dev/filelock/releases/tag/3.13.2) [Compare Source](https://togithub.com/tox-dev/py-filelock/compare/3.13.1...3.13.2) #### What's Changed - Fixed small typo in \_unix.py by [@​snemes](https://togithub.com/snemes) in [https://togithub.com/tox-dev/filelock/pull/302](https://togithub.com/tox-dev/filelock/pull/302) - Update SECURITY.md to reflect Python 3.7 support dropoff by [@​kemzeb](https://togithub.com/kemzeb) in [https://togithub.com/tox-dev/filelock/pull/304](https://togithub.com/tox-dev/filelock/pull/304) - Update index.rst to improve the demo usage by [@​youkaichao](https://togithub.com/youkaichao) in [https://togithub.com/tox-dev/filelock/pull/314](https://togithub.com/tox-dev/filelock/pull/314) - \[BugFix] fix permission denied error when lock file is placed in `/tmp` by [@​kota-iizuka](https://togithub.com/kota-iizuka) in [https://togithub.com/tox-dev/filelock/pull/317](https://togithub.com/tox-dev/filelock/pull/317) #### New Contributors - [@​snemes](https://togithub.com/snemes) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/302](https://togithub.com/tox-dev/filelock/pull/302) - [@​kemzeb](https://togithub.com/kemzeb) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/304](https://togithub.com/tox-dev/filelock/pull/304) - [@​youkaichao](https://togithub.com/youkaichao) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/314](https://togithub.com/tox-dev/filelock/pull/314) - [@​kota-iizuka](https://togithub.com/kota-iizuka) made their first contribution in [https://togithub.com/tox-dev/filelock/pull/317](https://togithub.com/tox-dev/filelock/pull/317) **Full Changelog**: https://togithub.com/tox-dev/filelock/compare/3.13.1...3.13.2
googleapis/python-pubsub (google-cloud-pubsub) ### [`v2.21.5`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2215-2024-06-20) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.21.4...v2.21.5) ##### Bug Fixes - Allow Protobuf 5.x ([a369f04](https://togithub.com/googleapis/python-pubsub/commit/a369f04c46e4b3db34dcf8cc2ef7cda4ea491e26)) ### [`v2.21.4`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2214-2024-06-18) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.21.3...v2.21.4) ##### Documentation - **samples:** Add code sample for optimistic subscribe ([#​1182](https://togithub.com/googleapis/python-pubsub/issues/1182)) ([d8e8aa5](https://togithub.com/googleapis/python-pubsub/commit/d8e8aa59ab0288fdaf5a1cc5e476581e73d0f82c)) ### [`v2.21.3`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2213-2024-06-10) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.21.2...v2.21.3) ##### Bug Fixes - Race condition where future callbacks invoked before client is in paused state ([#​1145](https://togithub.com/googleapis/python-pubsub/issues/1145)) ([d12bac6](https://togithub.com/googleapis/python-pubsub/commit/d12bac6d94b337aa8978006600fb00e5b13d741d)) - Suppress warnings caused during pytest runs ([#​1189](https://togithub.com/googleapis/python-pubsub/issues/1189)) ([cd51149](https://togithub.com/googleapis/python-pubsub/commit/cd51149c9e0d3c59d1c75395c05308e860908bf9)) - Typecheck errors in samples/snippets/subscriber.py ([#​1186](https://togithub.com/googleapis/python-pubsub/issues/1186)) ([3698450](https://togithub.com/googleapis/python-pubsub/commit/3698450041cb4db0e2957832c24450f674b89c11)) ### [`v2.21.2`](https://togithub.com/googleapis/python-pubsub/blob/HEAD/CHANGELOG.md#2212-2024-05-30) [Compare Source](https://togithub.com/googleapis/python-pubsub/compare/v2.21.1...v2.21.2) ##### Bug Fixes - Test failures due to grpcio changes ([#​1178](https://togithub.com/googleapis/python-pubsub/issues/1178)) ([086dd46](https://togithub.com/googleapis/python-pubsub/commit/086dd4660ec56d9ff2d41a32ec0b8e8dc44acc55))
googleapis/python-storage (google-cloud-storage) ### [`v2.17.0`](https://togithub.com/googleapis/python-storage/blob/HEAD/CHANGELOG.md#2170-2024-05-22) [Compare Source](https://togithub.com/googleapis/python-storage/compare/v2.16.0...v2.17.0) ##### Features - Support HNS enablement in bucket metadata ([#​1278](https://togithub.com/googleapis/python-storage/issues/1278)) ([add3c01](https://togithub.com/googleapis/python-storage/commit/add3c01f0974e22df7f0b50504d5e83e4235fd81)) - Support page_size in bucket.list_blobs ([#​1275](https://togithub.com/googleapis/python-storage/issues/1275)) ([c52e882](https://togithub.com/googleapis/python-storage/commit/c52e882f65583a7739392926308cc34984561165)) ##### Bug Fixes - Remove deprecated methods in samples and tests ([#​1274](https://togithub.com/googleapis/python-storage/issues/1274)) ([4db96c9](https://togithub.com/googleapis/python-storage/commit/4db96c960b07e503c1031c9fa879cf2af195f513)) ##### Documentation - Reference Storage Control in readme ([#​1254](https://togithub.com/googleapis/python-storage/issues/1254)) ([3d6d369](https://togithub.com/googleapis/python-storage/commit/3d6d3693d5c1b24cd3d2bbdeabfd78b8bfd4161a)) - Update DEFAULT_RETRY_IF_GENERATION_SPECIFIED docstrings ([#​1234](https://togithub.com/googleapis/python-storage/issues/1234)) ([bdd426a](https://togithub.com/googleapis/python-storage/commit/bdd426adf5901faa36115885af868ef50e356a36))
wntrblm/nox (nox) ### [`v2024.4.15`](https://togithub.com/wntrblm/nox/compare/2024.03.02...2024.04.15) [Compare Source](https://togithub.com/wntrblm/nox/compare/2024.03.02...2024.04.15)
pypa/packaging (packaging) ### [`v24.1`](https://togithub.com/pypa/packaging/releases/tag/24.1) [Compare Source](https://togithub.com/pypa/packaging/compare/24.0...24.1) #### What's Changed - pyupgrade/black/isort/flake8 → ruff by [@​DimitriPapadopoulos](https://togithub.com/DimitriPapadopoulos) in [https://togithub.com/pypa/packaging/pull/769](https://togithub.com/pypa/packaging/pull/769) - Add support for Python 3.13 and drop EOL 3.7 by [@​hugovk](https://togithub.com/hugovk) in [https://togithub.com/pypa/packaging/pull/783](https://togithub.com/pypa/packaging/pull/783) - Bump the github-actions group with 4 updates by [@​dependabot](https://togithub.com/dependabot) in [https://togithub.com/pypa/packaging/pull/782](https://togithub.com/pypa/packaging/pull/782) - Fix typo in `_parser` docstring by [@​pradyunsg](https://togithub.com/pradyunsg) in [https://togithub.com/pypa/packaging/pull/784](https://togithub.com/pypa/packaging/pull/784) - Modernise type annotations using FA rules from ruff by [@​pradyunsg](https://togithub.com/pradyunsg) in [https://togithub.com/pypa/packaging/pull/785](https://togithub.com/pypa/packaging/pull/785) - Document `markers.default_environment()` by [@​edgarrmondragon](https://togithub.com/edgarrmondragon) in [https://togithub.com/pypa/packaging/pull/753](https://togithub.com/pypa/packaging/pull/753) - Bump the github-actions group with 3 updates by [@​dependabot](https://togithub.com/dependabot) in [https://togithub.com/pypa/packaging/pull/789](https://togithub.com/pypa/packaging/pull/789) - Work around platform.python_version() returning non PEP 440 compliant version for non-tagged CPython builds by [@​sbidoul](https://togithub.com/sbidoul) in [https://togithub.com/pypa/packaging/pull/802](https://togithub.com/pypa/packaging/pull/802) #### New Contributors - [@​dependabot](https://togithub.com/dependabot) made their first contribution in [https://togithub.com/pypa/packaging/pull/782](https://togithub.com/pypa/packaging/pull/782) - [@​edgarrmondragon](https://togithub.com/edgarrmondragon) made their first contribution in [https://togithub.com/pypa/packaging/pull/753](https://togithub.com/pypa/packaging/pull/753) **Full Changelog**: https://togithub.com/pypa/packaging/compare/24.0...24.1
platformdirs/platformdirs (platformdirs) ### [`v4.2.2`](https://togithub.com/platformdirs/platformdirs/releases/tag/4.2.2) [Compare Source](https://togithub.com/platformdirs/platformdirs/compare/4.2.1...4.2.2) #### What's Changed - Fix android detection when python4android is present by [@​tmolitor-stud-tu](https://togithub.com/tmolitor-stud-tu) in [https://togithub.com/platformdirs/platformdirs/pull/277](https://togithub.com/platformdirs/platformdirs/pull/277) #### New Contributors - [@​tmolitor-stud-tu](https://togithub.com/tmolitor-stud-tu) made their first contribution in [https://togithub.com/platformdirs/platformdirs/pull/277](https://togithub.com/platformdirs/platformdirs/pull/277) **Full Changelog**: https://togithub.com/platformdirs/platformdirs/compare/4.2.1...4.2.2 ### [`v4.2.1`](https://togithub.com/platformdirs/platformdirs/releases/tag/4.2.1) [Compare Source](https://togithub.com/platformdirs/platformdirs/compare/4.2.0...4.2.1) #### What's Changed - Switch to ruff for formatting and use codespell and docformatter by [@​gaborbernat](https://togithub.com/gaborbernat) in [https://togithub.com/platformdirs/platformdirs/pull/261](https://togithub.com/platformdirs/platformdirs/pull/261) - Use hatch over tox by [@​gaborbernat](https://togithub.com/gaborbernat) in [https://togithub.com/platformdirs/platformdirs/pull/262](https://togithub.com/platformdirs/platformdirs/pull/262) - chore: various minor fixes by [@​deronnax](https://togithub.com/deronnax) in [https://togithub.com/platformdirs/platformdirs/pull/263](https://togithub.com/platformdirs/platformdirs/pull/263) - chore: update dead Microsoft's known folders documentation link by [@​deronnax](https://togithub.com/deronnax) in [https://togithub.com/platformdirs/platformdirs/pull/267](https://togithub.com/platformdirs/platformdirs/pull/267) - Allow working without ctypes by [@​youknowone](https://togithub.com/youknowone) in [https://togithub.com/platformdirs/platformdirs/pull/275](https://togithub.com/platformdirs/platformdirs/pull/275) #### New Contributors - [@​deronnax](https://togithub.com/deronnax) made their first contribution in [https://togithub.com/platformdirs/platformdirs/pull/263](https://togithub.com/platformdirs/platformdirs/pull/263) - [@​youknowone](https://togithub.com/youknowone) made their first contribution in [https://togithub.com/platformdirs/platformdirs/pull/275](https://togithub.com/platformdirs/platformdirs/pull/275) **Full Changelog**: https://togithub.com/platformdirs/platformdirs/compare/4.2.0...4.2.1
pytest-dev/pytest (pytest) ### [`v8.2.2`](https://togithub.com/pytest-dev/pytest/releases/tag/8.2.2) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/8.2.1...8.2.2) # pytest 8.2.2 (2024-06-04) ## Bug Fixes - [#​12355](https://togithub.com/pytest-dev/pytest/issues/12355): Fix possible catastrophic performance slowdown on a certain parametrization pattern involving many higher-scoped parameters. - [#​12367](https://togithub.com/pytest-dev/pytest/issues/12367): Fix a regression in pytest 8.2.0 where unittest class instances (a fresh one is created for each test) were not released promptly on test teardown but only on session teardown. - [#​12381](https://togithub.com/pytest-dev/pytest/issues/12381): Fix possible "Directory not empty" crashes arising from concurent cache dir (`.pytest_cache`) creation. Regressed in pytest 8.2.0. ## Improved Documentation - [#​12290](https://togithub.com/pytest-dev/pytest/issues/12290): Updated Sphinx theme to use Furo instead of Flask, enabling Dark mode theme. - [#​12356](https://togithub.com/pytest-dev/pytest/issues/12356): Added a subsection to the documentation for debugging flaky tests to mention lack of thread safety in pytest as a possible source of flakyness. - [#​12363](https://togithub.com/pytest-dev/pytest/issues/12363): The documentation webpages now links to a canonical version to reduce outdated documentation in search engine results. ### [`v8.2.1`](https://togithub.com/pytest-dev/pytest/releases/tag/8.2.1) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/8.2.0...8.2.1) # pytest 8.2.1 (2024-05-19) ## Improvements - [#​12334](https://togithub.com/pytest-dev/pytest/issues/12334): Support for Python 3.13 (beta1 at the time of writing). ## Bug Fixes - [#​12120](https://togithub.com/pytest-dev/pytest/issues/12120): Fix \[PermissionError]{.title-ref} crashes arising from directories which are not selected on the command-line. - [#​12191](https://togithub.com/pytest-dev/pytest/issues/12191): Keyboard interrupts and system exits are now properly handled during the test collection. - [#​12300](https://togithub.com/pytest-dev/pytest/issues/12300): Fixed handling of 'Function not implemented' error under squashfuse_ll, which is a different way to say that the mountpoint is read-only. - [#​12308](https://togithub.com/pytest-dev/pytest/issues/12308): Fix a regression in pytest 8.2.0 where the permissions of automatically-created `.pytest_cache` directories became `rwx------` instead of the expected `rwxr-xr-x`. ## Trivial/Internal Changes - [#​12333](https://togithub.com/pytest-dev/pytest/issues/12333): pytest releases are now attested using the recent [Artifact Attestation](https://github.blog/2024-05-02-introducing-artifact-attestations-now-in-public-beta/) support from GitHub, allowing users to verify the provenance of pytest's sdist and wheel artifacts. ### [`v8.2.0`](https://togithub.com/pytest-dev/pytest/releases/tag/8.2.0) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/8.1.2...8.2.0) # pytest 8.2.0 (2024-04-27) ## Deprecations - [#​12069](https://togithub.com/pytest-dev/pytest/issues/12069): A deprecation warning is now raised when implementations of one of the following hooks request a deprecated `py.path.local` parameter instead of the `pathlib.Path` parameter which replaced it: - `pytest_ignore_collect`{.interpreted-text role="hook"} - the `path` parameter - use `collection_path` instead. - `pytest_collect_file`{.interpreted-text role="hook"} - the `path` parameter - use `file_path` instead. - `pytest_pycollect_makemodule`{.interpreted-text role="hook"} - the `path` parameter - use `module_path` instead. - `pytest_report_header`{.interpreted-text role="hook"} - the `startdir` parameter - use `start_path` instead. - `pytest_report_collectionfinish`{.interpreted-text role="hook"} - the `startdir` parameter - use `start_path` instead. The replacement parameters are available since pytest 7.0.0. The old parameters will be removed in pytest 9.0.0. See `legacy-path-hooks-deprecated`{.interpreted-text role="ref"} for more details. ## Features - [#​11871](https://togithub.com/pytest-dev/pytest/issues/11871): Added support for reading command line arguments from a file using the prefix character `@`, like e.g.: `pytest @​tests.txt`. The file must have one argument per line. See `Read arguments from file `{.interpreted-text role="ref"} for details. ## Improvements - [#​11523](https://togithub.com/pytest-dev/pytest/issues/11523): `pytest.importorskip`{.interpreted-text role="func"} will now issue a warning if the module could be found, but raised `ImportError`{.interpreted-text role="class"} instead of `ModuleNotFoundError`{.interpreted-text role="class"}. The warning can be suppressed by passing `exc_type=ImportError` to `pytest.importorskip`{.interpreted-text role="func"}. See `import-or-skip-import-error`{.interpreted-text role="ref"} for details. - [#​11728](https://togithub.com/pytest-dev/pytest/issues/11728): For `unittest`-based tests, exceptions during class cleanup (as raised by functions registered with `TestCase.addClassCleanup `{.interpreted-text role="meth"}) are now reported instead of silently failing. - [#​11777](https://togithub.com/pytest-dev/pytest/issues/11777): Text is no longer truncated in the `short test summary info` section when `-vv` is given. - [#​12112](https://togithub.com/pytest-dev/pytest/issues/12112): Improved namespace packages detection when `consider_namespace_packages`{.interpreted-text role="confval"} is enabled, covering more situations (like editable installs). - [#​9502](https://togithub.com/pytest-dev/pytest/issues/9502): Added `PYTEST_VERSION`{.interpreted-text role="envvar"} environment variable which is defined at the start of the pytest session and undefined afterwards. It contains the value of `pytest.__version__`, and among other things can be used to easily check if code is running from within a pytest run. ## Bug Fixes - [#​12065](https://togithub.com/pytest-dev/pytest/issues/12065): Fixed a regression in pytest 8.0.0 where test classes containing `setup_method` and tests using `@staticmethod` or `@classmethod` would crash with `AttributeError: 'NoneType' object has no attribute 'setup_method'`. Now the `request.instance `{.interpreted-text role="attr"} attribute of tests using `@staticmethod` and `@classmethod` is no longer `None`, but a fresh instance of the class, like in non-static methods. Previously it was `None`, and all fixtures of such tests would share a single `self`. - [#​12135](https://togithub.com/pytest-dev/pytest/issues/12135): Fixed issue where fixtures adding their finalizer multiple times to fixtures they request would cause unreliable and non-intuitive teardown ordering in some instances. - [#​12194](https://togithub.com/pytest-dev/pytest/issues/12194): Fixed a bug with `--importmode=importlib` and `--doctest-modules` where child modules did not appear as attributes in parent modules. - [#​1489](https://togithub.com/pytest-dev/pytest/issues/1489): Fixed some instances where teardown of higher-scoped fixtures was not happening in the reverse order they were initialized in. ## Trivial/Internal Changes - [#​12069](https://togithub.com/pytest-dev/pytest/issues/12069): `pluggy>=1.5.0` is now required. - [#​12167](https://togithub.com/pytest-dev/pytest/issues/12167): `cache `{.interpreted-text role="ref"}: create supporting files (`CACHEDIR.TAG`, `.gitignore`, etc.) in a temporary directory to provide atomic semantics. ### [`v8.1.2`](https://togithub.com/pytest-dev/pytest/releases/tag/8.1.2) [Compare Source](https://togithub.com/pytest-dev/pytest/compare/8.1.1...8.1.2) # pytest 8.1.2 (2024-04-26) ## Bug Fixes - [#​12114](https://togithub.com/pytest-dev/pytest/issues/12114): Fixed error in `pytest.approx`{.interpreted-text role="func"} when used with \[numpy]{.title-ref} arrays and comparing with other types.
pypa/virtualenv (virtualenv) ### [`v20.26.3`](https://togithub.com/pypa/virtualenv/releases/tag/20.26.3) [Compare Source](https://togithub.com/pypa/virtualenv/compare/20.26.2...20.26.3) #### What's Changed - release 20.26.2 by [@​gaborbernat](https://togithub.com/gaborbernat) in [https://togithub.com/pypa/virtualenv/pull/2724](https://togithub.com/pypa/virtualenv/pull/2724) - Bump embeded wheels by [@​gaborbernat](https://togithub.com/gaborbernat) in [https://togithub.com/pypa/virtualenv/pull/2741](https://togithub.com/pypa/virtualenv/pull/2741) **Full Changelog**: https://togithub.com/pypa/virtualenv/compare/20.26.2...20.26.3 ### [`v20.26.2`](https://togithub.com/pypa/virtualenv/compare/20.26.1...20.26.2) [Compare Source](https://togithub.com/pypa/virtualenv/compare/20.26.1...20.26.2) ### [`v20.26.1`](https://togithub.com/pypa/virtualenv/compare/20.26.0...20.26.1) [Compare Source](https://togithub.com/pypa/virtualenv/compare/20.26.0...20.26.1) ### [`v20.26.0`](https://togithub.com/pypa/virtualenv/releases/tag/20.26.0) [Compare Source](https://togithub.com/pypa/virtualenv/compare/20.25.3...20.26.0) ##### What's Changed - release 20.25.3 by [@​gaborbernat](https://togithub.com/gaborbernat) in [https://togithub.com/pypa/virtualenv/pull/2704](https://togithub.com/pypa/virtualenv/pull/2704) - Fixed a case when template variable is WindowsPath by [@​NtWriteCode](https://togithub.com/NtWriteCode) in [https://togithub.com/pypa/virtualenv/pull/2707](https://togithub.com/pypa/virtualenv/pull/2707) - Allow builtin interpreter discovery to find specific Python versions given a general spec by [@​flying-sheep](https://togithub.com/flying-sheep) in [https://togithub.com/pypa/virtualenv/pull/2709](https://togithub.com/pypa/virtualenv/pull/2709) ##### New Contributors - [@​NtWriteCode](https://togithub.com/NtWriteCode) made their first contribution in [https://togithub.com/pypa/virtualenv/pull/2707](https://togithub.com/pypa/virtualenv/pull/2707) - [@​flying-sheep](https://togithub.com/flying-sheep) made their first contribution in [https://togithub.com/pypa/virtualenv/pull/2709](https://togithub.com/pypa/virtualenv/pull/2709) **Full Changelog**: https://togithub.com/pypa/virtualenv/compare/20.25.3...20.26.0 ### [`v20.25.3`](https://togithub.com/pypa/virtualenv/releases/tag/20.25.3) [Compare Source](https://togithub.com/pypa/virtualenv/compare/20.25.2...20.25.3) #### What's Changed - release 20.25.2 by [@​gaborbernat](https://togithub.com/gaborbernat) in [https://togithub.com/pypa/virtualenv/pull/2703](https://togithub.com/pypa/virtualenv/pull/2703) - Fix for tests: Python 3.13.0a6 renamed pathmod to parser by [@​befeleme](https://togithub.com/befeleme) in [https://togithub.com/pypa/virtualenv/pull/2702](https://togithub.com/pypa/virtualenv/pull/2702) #### New Contributors - [@​befeleme](https://togithub.com/befeleme) made their first contribution in [https://togithub.com/pypa/virtualenv/pull/2702](https://togithub.com/pypa/virtualenv/pull/2702) **Full Changelog**: https://togithub.com/pypa/virtualenv/compare/20.25.2...20.25.3 ### [`v20.25.2`](https://togithub.com/pypa/virtualenv/releases/tag/20.25.2) [Compare Source](https://togithub.com/pypa/virtualenv/compare/20.25.1...20.25.2) #### What's Changed - release 20.25.1 by [@​gaborbernat](https://togithub.com/gaborbernat) in [https://togithub.com/pypa/virtualenv/pull/2692](https://togithub.com/pypa/virtualenv/pull/2692) - Fix windows utf8 encoding issue by [@​PzaThief](https://togithub.com/PzaThief) in [https://togithub.com/pypa/virtualenv/pull/2687](https://togithub.com/pypa/virtualenv/pull/2687) - Update changelog.rst by [@​Callek](https://togithub.com/Callek) in [https://togithub.com/pypa/virtualenv/pull/2701](https://togithub.com/pypa/virtualenv/pull/2701) - Fix indentation in activate.fish by [@​junzh0u](https://togithub.com/junzh0u) in [https://togithub.com/pypa/virtualenv/pull/2700](https://togithub.com/pypa/virtualenv/pull/2700) #### New Contributors - [@​PzaThief](https://togithub.com/PzaThief) made their first contribution in [https://togithub.com/pypa/virtualenv/pull/2687](https://togithub.com/pypa/virtualenv/pull/2687) - [@​Callek](https://togithub.com/Callek) made their first contribution in [https://togithub.com/pypa/virtualenv/pull/2701](https://togithub.com/pypa/virtualenv/pull/2701) - [@​junzh0u](https://togithub.com/junzh0u) made their first contribution in [https://togithub.com/pypa/virtualenv/pull/2700](https://togithub.com/pypa/virtualenv/pull/2700) **Full Changelog**: https://togithub.com/pypa/virtualenv/compare/20.25.1...20.25.2
--- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/googleapis/python-storage). --- storage/samples/snippets/requirements-test.txt | 2 +- storage/samples/snippets/requirements.txt | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 86a3ade300a..054670d8b6d 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,4 +1,4 @@ pytest===7.4.4; python_version == '3.7' -pytest==8.1.1; python_version >= '3.8' +pytest==8.2.2; python_version >= '3.8' mock==5.1.0 backoff==2.2.1 diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index b0e41fa8445..5e3e93d9323 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,5 +1,5 @@ -google-cloud-pubsub==2.21.1 -google-cloud-storage==2.16.0 +google-cloud-pubsub==2.21.5 +google-cloud-storage==2.17.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' pandas==2.2.2; python_version >= '3.9' From 6fb446804a327606c41ced27049a2222f2dc719f Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 9 Jul 2024 19:49:48 +0200 Subject: [PATCH 140/172] chore(deps): update all dependencies (#1324) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 5e3e93d9323..c5b45a4a248 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.21.5 +google-cloud-pubsub==2.22.0 google-cloud-storage==2.17.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' From 31e6e886f34ea77807ba5819822965d2a7c27bcd Mon Sep 17 00:00:00 2001 From: cojenco Date: Fri, 27 Sep 2024 16:40:50 -0700 Subject: [PATCH 141/172] tests: unflake ud system test to only run in prod and hmac sample test (#1353) * test: test universe domain client only in prod * unflake hmac snippet test --- storage/samples/snippets/hmac_samples_test.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/storage/samples/snippets/hmac_samples_test.py b/storage/samples/snippets/hmac_samples_test.py index 60eba240184..988b4030502 100644 --- a/storage/samples/snippets/hmac_samples_test.py +++ b/storage/samples/snippets/hmac_samples_test.py @@ -64,7 +64,10 @@ def new_hmac_key(): if not hmac_key.state == "INACTIVE": hmac_key.state = "INACTIVE" hmac_key.update() - hmac_key.delete() + try: + hmac_key.delete() + except google.api_core.exceptions.BadRequest: + pass def test_list_keys(capsys, new_hmac_key): From 0539e356f805c23efb6d22dd8f5d99896eb4611b Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Mon, 30 Sep 2024 18:13:30 +0200 Subject: [PATCH 142/172] chore(deps): update all dependencies (#1329) Co-authored-by: Holt Skinner <13262395+holtskinner@users.noreply.github.com> Co-authored-by: cojenco --- storage/samples/snippets/requirements-test.txt | 2 +- storage/samples/snippets/requirements.txt | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 054670d8b6d..68fb21c1cad 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,4 +1,4 @@ pytest===7.4.4; python_version == '3.7' -pytest==8.2.2; python_version >= '3.8' +pytest==8.3.2; python_version >= '3.8' mock==5.1.0 backoff==2.2.1 diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index c5b45a4a248..54f6f78069d 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,5 +1,5 @@ -google-cloud-pubsub==2.22.0 -google-cloud-storage==2.17.0 +google-cloud-pubsub==2.23.0 +google-cloud-storage==2.18.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' pandas==2.2.2; python_version >= '3.9' From 133e631f1301774a94a3fbf9f35ef3fc50cd0289 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 1 Oct 2024 01:59:47 +0200 Subject: [PATCH 143/172] chore(deps): update all dependencies (#1354) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot --- storage/samples/snippets/requirements-test.txt | 2 +- storage/samples/snippets/requirements.txt | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 68fb21c1cad..a1dda582fb9 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,4 +1,4 @@ pytest===7.4.4; python_version == '3.7' -pytest==8.3.2; python_version >= '3.8' +pytest==8.3.3; python_version >= '3.8' mock==5.1.0 backoff==2.2.1 diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 54f6f78069d..4eb727236a1 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,5 +1,5 @@ -google-cloud-pubsub==2.23.0 -google-cloud-storage==2.18.0 +google-cloud-pubsub==2.25.1 +google-cloud-storage==2.18.2 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' -pandas==2.2.2; python_version >= '3.9' +pandas==2.2.3; python_version >= '3.9' From bc3fad69aa57535cd2a519db9b5039f6f730c374 Mon Sep 17 00:00:00 2001 From: Andrew Gorcester Date: Tue, 10 Dec 2024 15:59:06 -0800 Subject: [PATCH 144/172] chore: update sample tests and README to reflect new checksum defaults --- storage/samples/snippets/snippets_test.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 8c021f87066..71435785391 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -233,8 +233,8 @@ def test_upload_blob_from_memory(test_bucket, capsys): def test_upload_blob_from_stream(test_bucket, capsys): - file_obj = io.StringIO() - file_obj.write("This is test data.") + file_obj = io.BytesIO() + file_obj.write(b"This is test data.") storage_upload_from_stream.upload_blob_from_stream( test_bucket.name, file_obj, "test_upload_blob" ) From 3d469d65469cfbdf9673d3d0e7c862e22055c1cb Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Fri, 3 Jan 2025 09:32:32 -0500 Subject: [PATCH 145/172] chore(python): Update the python version in docs presubmit to use 3.10 (#1403) Source-Link: https://github.com/googleapis/synthtool/commit/de3def663b75d8b9ae1e5d548364c960ff13af8f Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:a1c5112b81d645f5bbc4d4bbc99d7dcb5089a52216c0e3fb1203a0eeabadd7d5 Co-authored-by: Owl Bot --- storage/samples/snippets/noxfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index 483b5590179..a169b5b5b46 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -89,7 +89,7 @@ def get_pytest_env_vars() -> Dict[str, str]: # DO NOT EDIT - automatically generated. # All versions used to test samples. -ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12"] +ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "3.13"] # Any default versions that should be ignored. IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] From 433bc99f1e69ba128ea69abb001bd026c5682d56 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Tue, 14 Jan 2025 17:08:12 +0100 Subject: [PATCH 146/172] chore(deps): update all dependencies (#1405) --- storage/samples/snippets/requirements-test.txt | 2 +- storage/samples/snippets/requirements.txt | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index a1dda582fb9..7f13e54c93b 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,4 +1,4 @@ pytest===7.4.4; python_version == '3.7' -pytest==8.3.3; python_version >= '3.8' +pytest==8.3.4; python_version >= '3.8' mock==5.1.0 backoff==2.2.1 diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index 4eb727236a1..da7850ebd9c 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,5 +1,5 @@ -google-cloud-pubsub==2.25.1 -google-cloud-storage==2.18.2 +google-cloud-pubsub==2.27.2 +google-cloud-storage==2.19.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' pandas==2.2.3; python_version >= '3.9' From 81188d90cc79591b994d546ee0092190013b45d1 Mon Sep 17 00:00:00 2001 From: cojenco Date: Tue, 28 Jan 2025 09:41:56 -0800 Subject: [PATCH 147/172] samples: add OTel Tracing quickstart (#1371) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * samples: add otel tracing quickstart * update test * fix lint * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * review comments * align samples to use ALWAYS_ON * address comments --------- Co-authored-by: Owl Bot Co-authored-by: Frank Natividad --- storage/samples/snippets/requirements.txt | 3 + storage/samples/snippets/snippets_test.py | 13 +++ .../snippets/storage_trace_quickstart.py | 83 +++++++++++++++++++ 3 files changed, 99 insertions(+) create mode 100644 storage/samples/snippets/storage_trace_quickstart.py diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index da7850ebd9c..a5a006ab20f 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -3,3 +3,6 @@ google-cloud-storage==2.19.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' pandas==2.2.3; python_version >= '3.9' +opentelemetry-exporter-gcp-trace +opentelemetry-propagator-gcp +opentelemetry-instrumentation-requests diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 71435785391..339693dd89b 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -75,6 +75,7 @@ import storage_set_client_endpoint import storage_set_object_retention_policy import storage_set_metadata +import storage_trace_quickstart import storage_transfer_manager_download_bucket import storage_transfer_manager_download_chunks_concurrently import storage_transfer_manager_download_many @@ -850,3 +851,15 @@ def test_create_bucket_hierarchical_namespace(test_bucket_create, capsys): ) out, _ = capsys.readouterr() assert f"Created bucket {test_bucket_create.name} with hierarchical namespace enabled" in out + + +def test_storage_trace_quickstart(test_bucket, capsys): + blob_name = f"trace_quickstart_{uuid.uuid4().hex}" + contents = "The quick brown fox jumps over the lazy dog." + storage_trace_quickstart.run_quickstart(test_bucket.name, blob_name, contents) + out, _ = capsys.readouterr() + + assert f"{blob_name} uploaded to {test_bucket.name}" in out + assert ( + f"Downloaded storage object {blob_name} from bucket {test_bucket.name}" in out + ) diff --git a/storage/samples/snippets/storage_trace_quickstart.py b/storage/samples/snippets/storage_trace_quickstart.py new file mode 100644 index 00000000000..322edc24051 --- /dev/null +++ b/storage/samples/snippets/storage_trace_quickstart.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python + +# Copyright 2024 Google LLC. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +""" +Sample that exports OpenTelemetry Traces collected from the Storage client to Cloud Trace. +""" + + +def run_quickstart(bucket_name, blob_name, data): + # [START storage_enable_otel_tracing] + + from opentelemetry import trace + from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter + from opentelemetry.resourcedetector.gcp_resource_detector import ( + GoogleCloudResourceDetector, + ) + from opentelemetry.sdk.trace import TracerProvider + from opentelemetry.sdk.trace.export import BatchSpanProcessor + from opentelemetry.sdk.trace.sampling import ALWAYS_ON + # Optional: Enable traces emitted from the requests HTTP library. + from opentelemetry.instrumentation.requests import RequestsInstrumentor + + from google.cloud import storage + + # The ID of your GCS bucket + # bucket_name = "your-bucket-name" + # The ID of your GCS object + # blob_name = "your-object-name" + # The contents to upload to the file + # data = "The quick brown fox jumps over the lazy dog." + + # In this sample, we use Google Cloud Trace to export the OpenTelemetry + # traces: https://cloud.google.com/trace/docs/setup/python-ot + # Choose and configure the exporter for your environment. + + tracer_provider = TracerProvider( + # Sampling is set to ALWAYS_ON. + # It is recommended to sample based on a ratio to control trace ingestion volume, + # for instance, sampler=TraceIdRatioBased(0.2) + sampler=ALWAYS_ON, + resource=GoogleCloudResourceDetector().detect(), + ) + + # Export to Google Cloud Trace. + tracer_provider.add_span_processor(BatchSpanProcessor(CloudTraceSpanExporter())) + trace.set_tracer_provider(tracer_provider) + + # Optional: Enable traces emitted from the requests HTTP library. + RequestsInstrumentor().instrument(tracer_provider=tracer_provider) + + # Get the tracer and create a new root span. + tracer = tracer_provider.get_tracer("My App") + with tracer.start_as_current_span("trace-quickstart"): + # Instantiate a storage client and perform a write and read workload. + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + blob = bucket.blob(blob_name) + blob.upload_from_string(data) + print(f"{blob_name} uploaded to {bucket_name}.") + + blob.download_as_bytes() + print("Downloaded storage object {} from bucket {}.".format(blob_name, bucket_name)) + + # [END storage_enable_otel_tracing] + + +if __name__ == "__main__": + run_quickstart(bucket_name=sys.argv[1], blob_name=sys.argv[2], data=sys.argv[3]) From d0f3966983c22e86448533780a8f43b44aea892b Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Mon, 10 Mar 2025 16:01:04 +0100 Subject: [PATCH 148/172] chore(deps): update all dependencies (#1413) --- storage/samples/snippets/requirements-test.txt | 4 ++-- storage/samples/snippets/requirements.txt | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/storage/samples/snippets/requirements-test.txt b/storage/samples/snippets/requirements-test.txt index 7f13e54c93b..5644295d03e 100644 --- a/storage/samples/snippets/requirements-test.txt +++ b/storage/samples/snippets/requirements-test.txt @@ -1,4 +1,4 @@ pytest===7.4.4; python_version == '3.7' -pytest==8.3.4; python_version >= '3.8' -mock==5.1.0 +pytest==8.3.5; python_version >= '3.8' +mock==5.2.0 backoff==2.2.1 diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index a5a006ab20f..b5201ffa98b 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,5 +1,5 @@ -google-cloud-pubsub==2.27.2 -google-cloud-storage==2.19.0 +google-cloud-pubsub==2.28.0 +google-cloud-storage==3.1.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' pandas==2.2.3; python_version >= '3.9' From 662748c0c9fc428bff32e2e7232b59f4083c835d Mon Sep 17 00:00:00 2001 From: Anthonios Partheniou Date: Tue, 18 Mar 2025 19:02:23 -0400 Subject: [PATCH 149/172] fix: allow Protobuf 6.x (#1445) * fix: allow Protobuf 6.x * add prerelease nox session * add python 3.13 * lint * add test dependencies * add test dependencies * fix(deps): require google-crc32c >= 1.1.3 * fix(deps): require requests >= 2.22.0 * add dependencies for system tests * clean up * clean up * clean up * clean up * clean up * fix cover * clean up * Install dependencies needed for system tests * add dependencies for system test * update noxfile config * add credentials * exclude .kokoro/presubmit/prerelease-deps.cfg template * remove obsolete excludes * clean up * clean up * exclude .kokoro/continuous/prerelease-deps.cfg from templates; remove obsolete replacement * migrate prerelease test from presubmit to continuous build --- storage/samples/snippets/noxfile_config.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/storage/samples/snippets/noxfile_config.py b/storage/samples/snippets/noxfile_config.py index 17a05b9f27d..7eba203a4b4 100644 --- a/storage/samples/snippets/noxfile_config.py +++ b/storage/samples/snippets/noxfile_config.py @@ -73,12 +73,15 @@ def get_cloud_kms_key(): if session == 'py-3.12': return ('projects/python-docs-samples-tests-312/locations/us/' 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') + if session == 'py-3.13': + return ('projects/python-docs-samples-tests-313/locations/us/' + 'keyRings/gcs-kms-key-ring/cryptoKeys/gcs-kms-key') return os.environ['CLOUD_KMS_KEY'] TEST_CONFIG_OVERRIDE = { # You can opt out from the test for specific Python versions. - 'ignored_versions': ["2.7", "3.6", "3.7", "3.11", "3.12"], + 'ignored_versions': ["2.7", "3.6", "3.7", "3.11", "3.12", "3.13"], # An envvar key for determining the project id to use. Change it # to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a From 35a7dacab97d7214a0c98db51cebf83bf16094e5 Mon Sep 17 00:00:00 2001 From: Mend Renovate Date: Mon, 7 Apr 2025 19:56:54 +0200 Subject: [PATCH 150/172] chore(deps): update dependency google-cloud-pubsub to v2.29.0 (#1453) Co-authored-by: cojenco --- storage/samples/snippets/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/requirements.txt b/storage/samples/snippets/requirements.txt index b5201ffa98b..751f8cfbe53 100644 --- a/storage/samples/snippets/requirements.txt +++ b/storage/samples/snippets/requirements.txt @@ -1,4 +1,4 @@ -google-cloud-pubsub==2.28.0 +google-cloud-pubsub==2.29.0 google-cloud-storage==3.1.0 pandas===1.3.5; python_version == '3.7' pandas===2.0.3; python_version == '3.8' From 133543a4e933887b403f0a21742c18b8ce8f2ab5 Mon Sep 17 00:00:00 2001 From: Chandrashekhar Sirimala Date: Mon, 5 May 2025 21:39:37 +0530 Subject: [PATCH 151/172] samples: Add samples for async download files #1470 (#1471) * samples: Add samples for async download files #1470 * Add argument description for `async_download_blobs` function. Co-authored-by: cojenco * Addressed comments from cojenco@ * change download_as_string to bytes * Don't print blob contents after downloading * remove Google Inc , add Google LLC * pass list of file_names as one of the params. * fix lint issues * fix whitespace lints * remove unused variable i --------- Co-authored-by: cojenco --- storage/samples/snippets/snippets_test.py | 14 ++++ .../snippets/storage_async_download.py | 70 +++++++++++++++++++ 2 files changed, 84 insertions(+) create mode 100755 storage/samples/snippets/storage_async_download.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 339693dd89b..6d9cfc3177c 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -26,6 +26,7 @@ import storage_add_bucket_label import storage_async_upload +import storage_async_download import storage_batch_request import storage_bucket_delete_default_kms_key import storage_change_default_storage_class @@ -267,6 +268,19 @@ def test_async_upload(bucket, capsys): assert f"Uploaded 3 files to bucket {bucket.name}" in out +def test_async_download(test_bucket, capsys): + object_count = 3 + source_files = [f"async_sample_blob_{x}" for x in range(object_count)] + for source in source_files: + blob = test_bucket.blob(source) + blob.upload_from_string(source) + + asyncio.run(storage_async_download.async_download_blobs(test_bucket.name, *source_files)) + out, _ = capsys.readouterr() + for x in range(object_count): + assert f"Downloaded storage object async_sample_blob_{x}" in out + + def test_download_byte_range(test_blob): with tempfile.NamedTemporaryFile() as dest_file: storage_download_byte_range.download_byte_range( diff --git a/storage/samples/snippets/storage_async_download.py b/storage/samples/snippets/storage_async_download.py new file mode 100755 index 00000000000..ed8f3f304f9 --- /dev/null +++ b/storage/samples/snippets/storage_async_download.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import argparse + +"""Sample that asynchronously downloads multiple files from GCS to application's memory. +""" + + +# [START storage_async_download] +# This sample can be run by calling `async.run(async_download_blobs('bucket_name', ['file1', 'file2']))` +async def async_download_blobs(bucket_name, *file_names): + """Downloads a number of files in parallel from the bucket. + """ + # The ID of your GCS bucket. + # bucket_name = "your-bucket-name" + + # The list of files names to download, these files should be present in bucket. + # file_names = ["myfile1", "myfile2"] + + import asyncio + from google.cloud import storage + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + loop = asyncio.get_running_loop() + + tasks = [] + for file_name in file_names: + blob = bucket.blob(file_name) + # The first arg, None, tells it to use the default loops executor + tasks.append(loop.run_in_executor(None, blob.download_as_bytes)) + + # If the method returns a value (such as download_as_bytes), gather will return the values + _ = await asyncio.gather(*tasks) + for file_name in file_names: + print(f"Downloaded storage object {file_name}") + + +# [END storage_async_download] + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument('-b', '--bucket_name', type=str, dest='bucket_name', help='provide the name of the GCS bucket') + parser.add_argument( + '-f', '--file_name', + action='append', + type=str, + dest='file_names', + help='Example: -f file1.txt or --file_name my_fav.mp4 . It can be used multiple times.' + ) + args = parser.parse_args() + + asyncio.run(async_download_blobs(args.bucket_name, *args.file_names)) From 2857c990922adad022f808e643d4bb500fcfeb61 Mon Sep 17 00:00:00 2001 From: Chandrashekhar Sirimala Date: Tue, 6 May 2025 21:20:30 +0530 Subject: [PATCH 152/172] samples: Add samples for soft_deleted_buckets (#1463) * samples: Add samples for soft_deleted_buckets * fix: fix linting errors * fix: fix linting errors on #1455 - attempt2 * fix: fix linting errors on #1455 - attempt3 * fix: test_list_buckets errors * fix: address comments by @JesseLovelace * samples: Add storage_list_soft_deleted_buckets.py sample and test cases for all * fix: lint errors space b/w methods. * fix: lint issues. * fix: undo changes in storage_list_buckets.py * fix: lint errors * Change copyright statement. * fix minor typos in doc strings as per code comment --- storage/samples/snippets/snippets_test.py | 34 +++++++++++++ .../storage_get_soft_deleted_bucket.py | 48 +++++++++++++++++++ .../storage_list_soft_deleted_buckets.py | 36 ++++++++++++++ .../storage_restore_soft_deleted_bucket.py | 38 +++++++++++++++ 4 files changed, 156 insertions(+) create mode 100644 storage/samples/snippets/storage_get_soft_deleted_bucket.py create mode 100644 storage/samples/snippets/storage_list_soft_deleted_buckets.py create mode 100644 storage/samples/snippets/storage_restore_soft_deleted_bucket.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 6d9cfc3177c..4f98884b538 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -59,9 +59,12 @@ import storage_get_autoclass import storage_get_bucket_labels import storage_get_bucket_metadata +import storage_get_soft_deleted_bucket import storage_get_metadata import storage_get_service_account import storage_list_buckets +import storage_list_soft_deleted_buckets +import storage_restore_soft_deleted_bucket import storage_list_file_archived_generations import storage_list_files import storage_list_files_with_prefix @@ -131,6 +134,19 @@ def test_bucket(): bucket.delete(force=True) +@pytest.fixture(scope="module") +def test_soft_deleted_bucket(): + """Yields a soft-deleted bucket.""" + bucket = None + while bucket is None or bucket.exists(): + bucket_name = f"storage-snippets-test-{uuid.uuid4()}" + bucket = storage.Client().bucket(bucket_name) + bucket.create() + # [Assumption] Bucket is created with default policy , ie soft delete on. + bucket.delete() + yield bucket + + @pytest.fixture(scope="function") def test_public_bucket(): # The new projects don't allow to make a bucket available to public, so @@ -195,6 +211,12 @@ def test_list_buckets(test_bucket, capsys): assert test_bucket.name in out +def test_list_soft_deleted_buckets(test_soft_deleted_bucket, capsys): + storage_list_soft_deleted_buckets.list_soft_deleted_buckets() + out, _ = capsys.readouterr() + assert test_soft_deleted_bucket.name in out + + def test_list_blobs(test_blob, capsys): storage_list_files.list_blobs(test_blob.bucket.name) out, _ = capsys.readouterr() @@ -207,6 +229,18 @@ def test_bucket_metadata(test_bucket, capsys): assert test_bucket.name in out +def test_get_soft_deleted_bucket(test_soft_deleted_bucket, capsys): + storage_get_soft_deleted_bucket.get_soft_deleted_bucket(test_soft_deleted_bucket.name, test_soft_deleted_bucket.generation) + out, _ = capsys.readouterr() + assert test_soft_deleted_bucket.name in out + + +def test_restore_soft_deleted_bucket(test_soft_deleted_bucket, capsys): + storage_restore_soft_deleted_bucket.restore_bucket(test_soft_deleted_bucket.name, test_soft_deleted_bucket.generation) + out, _ = capsys.readouterr() + assert test_soft_deleted_bucket.name in out + + def test_list_blobs_with_prefix(test_blob, capsys): storage_list_files_with_prefix.list_blobs_with_prefix( test_blob.bucket.name, prefix="storage_snippets" diff --git a/storage/samples/snippets/storage_get_soft_deleted_bucket.py b/storage/samples/snippets/storage_get_soft_deleted_bucket.py new file mode 100644 index 00000000000..2b795504657 --- /dev/null +++ b/storage/samples/snippets/storage_get_soft_deleted_bucket.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import sys + +# [START storage_get_soft_deleted_bucket] + +from google.cloud import storage + + +def get_soft_deleted_bucket(bucket_name, generation): + """Prints out a soft-deleted bucket's metadata. + + Args: + bucket_name: str + The name of the bucket to get. + + generation: + The generation of the bucket. + + """ + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name, soft_deleted=True, generation=generation) + + print(f"ID: {bucket.id}") + print(f"Name: {bucket.name}") + print(f"Soft Delete time: {bucket.soft_delete_time}") + print(f"Hard Delete Time : {bucket.hard_delete_time}") + + +# [END storage_get_soft_deleted_bucket] + +if __name__ == "__main__": + get_soft_deleted_bucket(bucket_name=sys.argv[1], generation=sys.argv[2]) diff --git a/storage/samples/snippets/storage_list_soft_deleted_buckets.py b/storage/samples/snippets/storage_list_soft_deleted_buckets.py new file mode 100644 index 00000000000..16abd90f02a --- /dev/null +++ b/storage/samples/snippets/storage_list_soft_deleted_buckets.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_list_soft_deleted_buckets] + +from google.cloud import storage + + +def list_soft_deleted_buckets(): + """Lists all soft-deleted buckets.""" + + storage_client = storage.Client() + buckets = storage_client.list_buckets(soft_deleted=True) + + for bucket in buckets: + print(bucket.name) + + +# [END storage_list_soft_deleted_buckets] + + +if __name__ == "__main__": + list_soft_deleted_buckets() diff --git a/storage/samples/snippets/storage_restore_soft_deleted_bucket.py b/storage/samples/snippets/storage_restore_soft_deleted_bucket.py new file mode 100644 index 00000000000..fb62919978e --- /dev/null +++ b/storage/samples/snippets/storage_restore_soft_deleted_bucket.py @@ -0,0 +1,38 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import sys + +# [START storage_restore_soft_deleted_bucket] + +from google.cloud import storage + + +def restore_bucket(bucket_name, bucket_generation): + storage_client = storage.Client() + bucket = storage_client.restore_bucket(bucket_name=bucket_name, generation=bucket_generation) + print(f"Soft-deleted bucket {bucket.name} with ID: {bucket.id} was restored.") + print(f"Bucket Generation: {bucket.generation}") + + +# [END storage_restore_soft_deleted_bucket] + +if __name__ == "__main__": + if len(sys.argv) != 3: + print("Wrong inputs!! Usage of script - \"python storage_restore_soft_deleted_bucket.py \" ") + sys.exit(1) + restore_bucket(bucket_name=sys.argv[1], bucket_generation=sys.argv[2]) From 478bd1ed3430dd463e9960f8b0edae1b28f3e230 Mon Sep 17 00:00:00 2001 From: cojenco Date: Mon, 19 May 2025 13:08:32 -0700 Subject: [PATCH 153/172] samples: update retry sample and comments (#1485) --- storage/samples/snippets/storage_configure_retries.py | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/storage/samples/snippets/storage_configure_retries.py b/storage/samples/snippets/storage_configure_retries.py index ef1e422b663..25c2529a42e 100644 --- a/storage/samples/snippets/storage_configure_retries.py +++ b/storage/samples/snippets/storage_configure_retries.py @@ -38,16 +38,15 @@ def configure_retries(bucket_name, blob_name): bucket = storage_client.bucket(bucket_name) blob = bucket.blob(blob_name) - # Customize retry with a deadline of 500 seconds (default=120 seconds). - modified_retry = DEFAULT_RETRY.with_deadline(500.0) + # Customize retry with a timeout of 500 seconds (default=120 seconds). + modified_retry = DEFAULT_RETRY.with_timeout(500.0) # Customize retry with an initial wait time of 1.5 (default=1.0). # Customize retry with a wait time multiplier per iteration of 1.2 (default=2.0). # Customize retry with a maximum wait time of 45.0 (default=60.0). modified_retry = modified_retry.with_delay(initial=1.5, multiplier=1.2, maximum=45.0) - # blob.delete() uses DEFAULT_RETRY_IF_GENERATION_SPECIFIED by default. - # Override with modified_retry so the function retries even if the generation - # number is not specified. + # blob.delete() uses DEFAULT_RETRY by default. + # Pass in modified_retry to override the default retry behavior. print( f"The following library method is customized to be retried according to the following configurations: {modified_retry}" ) From 2bc8bc1afb0198928b55616e7a2a6b897a00ee37 Mon Sep 17 00:00:00 2001 From: shubham-up-47 Date: Sat, 7 Jun 2025 08:51:59 +0000 Subject: [PATCH 154/172] samples(storage): add samples for soft delete objects (#1486) --- storage/samples/snippets/snippets_test.py | 146 ++++++++++++++++-- .../snippets/storage_disable_soft_delete.py | 40 +++++ .../storage_get_soft_delete_policy.py | 47 ++++++ ...orage_list_soft_deleted_object_versions.py | 41 +++++ .../storage_list_soft_deleted_objects.py | 40 +++++ .../snippets/storage_restore_object.py | 47 ++++++ .../storage_set_soft_delete_policy.py | 42 +++++ 7 files changed, 393 insertions(+), 10 deletions(-) create mode 100644 storage/samples/snippets/storage_disable_soft_delete.py create mode 100644 storage/samples/snippets/storage_get_soft_delete_policy.py create mode 100644 storage/samples/snippets/storage_list_soft_deleted_object_versions.py create mode 100644 storage/samples/snippets/storage_list_soft_deleted_objects.py create mode 100644 storage/samples/snippets/storage_restore_object.py create mode 100644 storage/samples/snippets/storage_set_soft_delete_policy.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 4f98884b538..3fe377b6b5b 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -25,8 +25,8 @@ import requests import storage_add_bucket_label -import storage_async_upload import storage_async_download +import storage_async_upload import storage_batch_request import storage_bucket_delete_default_kms_key import storage_change_default_storage_class @@ -44,6 +44,7 @@ import storage_delete_file import storage_delete_file_archived_generation import storage_disable_bucket_lifecycle_management +import storage_disable_soft_delete import storage_disable_versioning import storage_download_byte_range import storage_download_file @@ -59,26 +60,31 @@ import storage_get_autoclass import storage_get_bucket_labels import storage_get_bucket_metadata -import storage_get_soft_deleted_bucket import storage_get_metadata import storage_get_service_account +import storage_get_soft_delete_policy +import storage_get_soft_deleted_bucket import storage_list_buckets -import storage_list_soft_deleted_buckets -import storage_restore_soft_deleted_bucket import storage_list_file_archived_generations import storage_list_files import storage_list_files_with_prefix +import storage_list_soft_deleted_buckets +import storage_list_soft_deleted_object_versions +import storage_list_soft_deleted_objects import storage_make_public import storage_move_file import storage_object_get_kms_key import storage_remove_bucket_label import storage_remove_cors_configuration import storage_rename_file +import storage_restore_object +import storage_restore_soft_deleted_bucket import storage_set_autoclass import storage_set_bucket_default_kms_key import storage_set_client_endpoint -import storage_set_object_retention_policy import storage_set_metadata +import storage_set_object_retention_policy +import storage_set_soft_delete_policy import storage_trace_quickstart import storage_transfer_manager_download_bucket import storage_transfer_manager_download_chunks_concurrently @@ -147,6 +153,21 @@ def test_soft_deleted_bucket(): yield bucket +@pytest.fixture(scope="function") +def test_soft_delete_enabled_bucket(): + """Yields a bucket with soft-delete enabled that is deleted after the test completes.""" + bucket = None + while bucket is None or bucket.exists(): + bucket_name = f"storage-snippets-test-{uuid.uuid4()}" + bucket = storage.Client().bucket(bucket_name) + # Soft-delete retention for 7 days (minimum allowed by API) + bucket.soft_delete_policy.retention_duration_seconds = 7 * 24 * 60 * 60 + # Soft-delete requires a region + bucket.create(location="US-CENTRAL1") + yield bucket + bucket.delete(force=True) + + @pytest.fixture(scope="function") def test_public_bucket(): # The new projects don't allow to make a bucket available to public, so @@ -230,13 +251,17 @@ def test_bucket_metadata(test_bucket, capsys): def test_get_soft_deleted_bucket(test_soft_deleted_bucket, capsys): - storage_get_soft_deleted_bucket.get_soft_deleted_bucket(test_soft_deleted_bucket.name, test_soft_deleted_bucket.generation) + storage_get_soft_deleted_bucket.get_soft_deleted_bucket( + test_soft_deleted_bucket.name, test_soft_deleted_bucket.generation + ) out, _ = capsys.readouterr() assert test_soft_deleted_bucket.name in out def test_restore_soft_deleted_bucket(test_soft_deleted_bucket, capsys): - storage_restore_soft_deleted_bucket.restore_bucket(test_soft_deleted_bucket.name, test_soft_deleted_bucket.generation) + storage_restore_soft_deleted_bucket.restore_bucket( + test_soft_deleted_bucket.name, test_soft_deleted_bucket.generation + ) out, _ = capsys.readouterr() assert test_soft_deleted_bucket.name in out @@ -309,7 +334,9 @@ def test_async_download(test_bucket, capsys): blob = test_bucket.blob(source) blob.upload_from_string(source) - asyncio.run(storage_async_download.async_download_blobs(test_bucket.name, *source_files)) + asyncio.run( + storage_async_download.async_download_blobs(test_bucket.name, *source_files) + ) out, _ = capsys.readouterr() for x in range(object_count): assert f"Downloaded storage object async_sample_blob_{x}" in out @@ -877,7 +904,10 @@ def test_object_retention_policy(test_bucket_create, capsys): test_bucket_create.name ) out, _ = capsys.readouterr() - assert f"Created bucket {test_bucket_create.name} with object retention enabled setting" in out + assert ( + f"Created bucket {test_bucket_create.name} with object retention enabled setting" + in out + ) blob_name = "test_object_retention" storage_set_object_retention_policy.set_object_retention_policy( @@ -898,7 +928,10 @@ def test_create_bucket_hierarchical_namespace(test_bucket_create, capsys): test_bucket_create.name ) out, _ = capsys.readouterr() - assert f"Created bucket {test_bucket_create.name} with hierarchical namespace enabled" in out + assert ( + f"Created bucket {test_bucket_create.name} with hierarchical namespace enabled" + in out + ) def test_storage_trace_quickstart(test_bucket, capsys): @@ -911,3 +944,96 @@ def test_storage_trace_quickstart(test_bucket, capsys): assert ( f"Downloaded storage object {blob_name} from bucket {test_bucket.name}" in out ) + + +def test_storage_disable_soft_delete(test_soft_delete_enabled_bucket, capsys): + bucket_name = test_soft_delete_enabled_bucket.name + storage_disable_soft_delete.disable_soft_delete(bucket_name) + out, _ = capsys.readouterr() + assert f"Soft-delete policy is disabled for bucket {bucket_name}" in out + + +def test_storage_get_soft_delete_policy(test_soft_delete_enabled_bucket, capsys): + bucket_name = test_soft_delete_enabled_bucket.name + storage_get_soft_delete_policy.get_soft_delete_policy(bucket_name) + out, _ = capsys.readouterr() + assert f"Soft-delete policy for {bucket_name}" in out + assert "Object soft-delete policy is enabled" in out + assert "Object retention duration: " in out + assert "Policy effective time: " in out + + # Disable the soft-delete policy + test_soft_delete_enabled_bucket.soft_delete_policy.retention_duration_seconds = 0 + test_soft_delete_enabled_bucket.patch() + storage_get_soft_delete_policy.get_soft_delete_policy(bucket_name) + out, _ = capsys.readouterr() + assert f"Soft-delete policy for {bucket_name}" in out + assert "Object soft-delete policy is disabled" in out + + +def test_storage_set_soft_delete_policy(test_soft_delete_enabled_bucket, capsys): + bucket_name = test_soft_delete_enabled_bucket.name + retention_duration_seconds = 10 * 24 * 60 * 60 # 10 days + storage_set_soft_delete_policy.set_soft_delete_policy( + bucket_name, retention_duration_seconds + ) + out, _ = capsys.readouterr() + assert ( + f"Soft delete policy for bucket {bucket_name} was set to {retention_duration_seconds} seconds retention period" + in out + ) + + +def test_storage_list_soft_deleted_objects(test_soft_delete_enabled_bucket, capsys): + bucket_name = test_soft_delete_enabled_bucket.name + blob_name = f"test_object_{uuid.uuid4().hex}.txt" + blob_content = "This object will be soft-deleted for listing." + blob = test_soft_delete_enabled_bucket.blob(blob_name) + blob.upload_from_string(blob_content) + blob_generation = blob.generation + + blob.delete() # Soft-delete the object + storage_list_soft_deleted_objects.list_soft_deleted_objects(bucket_name) + out, _ = capsys.readouterr() + assert f"Name: {blob_name}, Generation: {blob_generation}" in out + + +def test_storage_list_soft_deleted_object_versions( + test_soft_delete_enabled_bucket, capsys +): + bucket_name = test_soft_delete_enabled_bucket.name + blob_name = f"test_object_{uuid.uuid4().hex}.txt" + blob_content = "This object will be soft-deleted for version listing." + blob = test_soft_delete_enabled_bucket.blob(blob_name) + blob.upload_from_string(blob_content) + blob_generation = blob.generation + + blob.delete() # Soft-delete the object + storage_list_soft_deleted_object_versions.list_soft_deleted_object_versions( + bucket_name, blob_name + ) + out, _ = capsys.readouterr() + assert f"Version ID: {blob_generation}" in out + + +def test_storage_restore_soft_deleted_object(test_soft_delete_enabled_bucket, capsys): + bucket_name = test_soft_delete_enabled_bucket.name + blob_name = f"test-restore-sd-obj-{uuid.uuid4().hex}.txt" + blob_content = "This object will be soft-deleted and restored." + blob = test_soft_delete_enabled_bucket.blob(blob_name) + blob.upload_from_string(blob_content) + blob_generation = blob.generation + + blob.delete() # Soft-delete the object + storage_restore_object.restore_soft_deleted_object( + bucket_name, blob_name, blob_generation + ) + out, _ = capsys.readouterr() + assert ( + f"Soft-deleted object {blob_name} is restored in the bucket {bucket_name}" + in out + ) + + # Verify the restoration + blob = test_soft_delete_enabled_bucket.get_blob(blob_name) + assert blob is not None diff --git a/storage/samples/snippets/storage_disable_soft_delete.py b/storage/samples/snippets/storage_disable_soft_delete.py new file mode 100644 index 00000000000..dc2447ae873 --- /dev/null +++ b/storage/samples/snippets/storage_disable_soft_delete.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_disable_soft_delete] +from google.cloud import storage + + +def disable_soft_delete(bucket_name): + """Disable soft-delete policy for the bucket.""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + # Setting the retention duration to 0 disables soft-delete. + bucket.soft_delete_policy.retention_duration_seconds = 0 + bucket.patch() + + print(f"Soft-delete policy is disabled for bucket {bucket_name}") + + +# [END storage_disable_soft_delete] + +if __name__ == "__main__": + disable_soft_delete(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_get_soft_delete_policy.py b/storage/samples/snippets/storage_get_soft_delete_policy.py new file mode 100644 index 00000000000..99c4e572a24 --- /dev/null +++ b/storage/samples/snippets/storage_get_soft_delete_policy.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_get_soft_delete_policy] +from google.cloud import storage + + +def get_soft_delete_policy(bucket_name): + """Gets the soft-delete policy of the bucket""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + print(f"Soft-delete policy for {bucket_name}") + if ( + bucket.soft_delete_policy + and bucket.soft_delete_policy.retention_duration_seconds + ): + print("Object soft-delete policy is enabled") + print( + f"Object retention duration: {bucket.soft_delete_policy.retention_duration_seconds} seconds" + ) + print(f"Policy effective time: {bucket.soft_delete_policy.effective_time}") + else: + print("Object soft-delete policy is disabled") + + +# [END storage_get_soft_delete_policy] + +if __name__ == "__main__": + get_soft_delete_policy(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_list_soft_deleted_object_versions.py b/storage/samples/snippets/storage_list_soft_deleted_object_versions.py new file mode 100644 index 00000000000..ecb9851c454 --- /dev/null +++ b/storage/samples/snippets/storage_list_soft_deleted_object_versions.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_list_soft_deleted_object_versions] +from google.cloud import storage + + +def list_soft_deleted_object_versions(bucket_name, blob_name): + """Lists all versions of a soft-deleted object in the bucket.""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + + storage_client = storage.Client() + blobs = storage_client.list_blobs(bucket_name, prefix=blob_name, soft_deleted=True) + + # Note: The call returns a response only when the iterator is consumed. + for blob in blobs: + print( + f"Version ID: {blob.generation}, Soft Delete Time: {blob.soft_delete_time}" + ) + + +# [END storage_list_soft_deleted_object_versions] + +if __name__ == "__main__": + list_soft_deleted_object_versions(bucket_name=sys.argv[1], blob_name=sys.argv[2]) diff --git a/storage/samples/snippets/storage_list_soft_deleted_objects.py b/storage/samples/snippets/storage_list_soft_deleted_objects.py new file mode 100644 index 00000000000..764cac56a6d --- /dev/null +++ b/storage/samples/snippets/storage_list_soft_deleted_objects.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_list_soft_deleted_objects] +from google.cloud import storage + + +def list_soft_deleted_objects(bucket_name): + """Lists all soft-deleted objects in the bucket.""" + # bucket_name = "your-bucket-name" + + storage_client = storage.Client() + blobs = storage_client.list_blobs(bucket_name, soft_deleted=True) + + # Note: The call returns a response only when the iterator is consumed. + for blob in blobs: + print( + f"Name: {blob.name}, Generation: {blob.generation}, Soft Delete Time: {blob.soft_delete_time}" + ) + + +# [END storage_list_soft_deleted_objects] + +if __name__ == "__main__": + list_soft_deleted_objects(bucket_name=sys.argv[1]) diff --git a/storage/samples/snippets/storage_restore_object.py b/storage/samples/snippets/storage_restore_object.py new file mode 100644 index 00000000000..d1e3f29372c --- /dev/null +++ b/storage/samples/snippets/storage_restore_object.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import sys + +# [START storage_restore_object] +from google.cloud import storage + + +def restore_soft_deleted_object(bucket_name, blob_name, blob_generation): + """Restores a soft-deleted object in the bucket.""" + # bucket_name = "your-bucket-name" + # blob_name = "your-object-name" + # blob_generation = "your-object-version-id" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + # Restore function will override if a live object already + # exists with the same name. + bucket.restore_blob(blob_name, generation=blob_generation) + + print( + f"Soft-deleted object {blob_name} is restored in the bucket {bucket_name}" + ) + + +# [END storage_restore_object] + +if __name__ == "__main__": + restore_soft_deleted_object( + bucket_name=sys.argv[1], blob_name=sys.argv[2], blob_generation=sys.argv[3] + ) diff --git a/storage/samples/snippets/storage_set_soft_delete_policy.py b/storage/samples/snippets/storage_set_soft_delete_policy.py new file mode 100644 index 00000000000..26bc5943664 --- /dev/null +++ b/storage/samples/snippets/storage_set_soft_delete_policy.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_set_soft_delete_policy] +from google.cloud import storage + + +def set_soft_delete_policy(bucket_name, duration_in_seconds): + """Sets a soft-delete policy on the bucket""" + # bucket_name = "your-bucket-name" + # duration_in_seconds = "your-soft-delete-retention-duration-in-seconds" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + bucket.soft_delete_policy.retention_duration_seconds = duration_in_seconds + bucket.patch() + + print( + f"Soft delete policy for bucket {bucket_name} was set to {duration_in_seconds} seconds retention period" + ) + + +# [END storage_set_soft_delete_policy] + +if __name__ == "__main__": + set_soft_delete_policy(bucket_name=sys.argv[1], duration_in_seconds=sys.argv[2]) From 231c893638d17e632540cd06d3a730e7f31f4041 Mon Sep 17 00:00:00 2001 From: Pulkit Aggarwal <54775856+Pulkit0110@users.noreply.github.com> Date: Thu, 17 Jul 2025 18:55:55 +0530 Subject: [PATCH 155/172] samples: add samples for move api to rename an object (#1505) * docs: add samples for move api to rename an object * minor change * fix lint errors * minor fix * resolving comments --- storage/samples/snippets/snippets_test.py | 18 +++++++ .../snippets/storage_move_file_atomically.py | 54 +++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100644 storage/samples/snippets/storage_move_file_atomically.py diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 3fe377b6b5b..91018f3dd8b 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -73,6 +73,7 @@ import storage_list_soft_deleted_objects import storage_make_public import storage_move_file +import storage_move_file_atomically import storage_object_get_kms_key import storage_remove_bucket_label import storage_remove_cors_configuration @@ -1037,3 +1038,20 @@ def test_storage_restore_soft_deleted_object(test_soft_delete_enabled_bucket, ca # Verify the restoration blob = test_soft_delete_enabled_bucket.get_blob(blob_name) assert blob is not None + + +def test_move_object(test_blob): + bucket = test_blob.bucket + try: + bucket.delete_blob("test_move_blob_atomic") + except google.cloud.exceptions.NotFound: + print(f"test_move_blob_atomic not found in bucket {bucket.name}") + + storage_move_file_atomically.move_object( + bucket.name, + test_blob.name, + "test_move_blob_atomic", + ) + + assert bucket.get_blob("test_move_blob_atomic") is not None + assert bucket.get_blob(test_blob.name) is None diff --git a/storage/samples/snippets/storage_move_file_atomically.py b/storage/samples/snippets/storage_move_file_atomically.py new file mode 100644 index 00000000000..d659cf3661a --- /dev/null +++ b/storage/samples/snippets/storage_move_file_atomically.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python + +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +# [START storage_move_object] +from google.cloud import storage + + +def move_object(bucket_name: str, blob_name: str, new_blob_name: str) -> None: + """Moves a blob to a new name within the same bucket using the move API.""" + # The name of your GCS bucket + # bucket_name = "your-bucket-name" + + # The name of your GCS object to move + # blob_name = "your-file-name" + + # The new name of the GCS object + # new_blob_name = "new-file-name" + + storage_client = storage.Client() + + bucket = storage_client.bucket(bucket_name) + blob_to_move = bucket.blob(blob_name) + + # Use move_blob to perform an efficient, server-side move. + moved_blob = bucket.move_blob( + blob=blob_to_move, new_name=new_blob_name + ) + + print(f"Blob {blob_to_move.name} has been moved to {moved_blob.name}.") + + +# [END storage_move_object] + +if __name__ == "__main__": + move_object( + bucket_name=sys.argv[1], + blob_name=sys.argv[2], + new_blob_name=sys.argv[3], + ) From 4eb8176cb865e907a7976c37d04400ac1595f430 Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Tue, 29 Jul 2025 11:15:34 +0530 Subject: [PATCH 156/172] chore: improve docs for list_files_with_prefix (#1517) If a user wants to just list prefixes without listing the blob names they can do so by following the updated documentation. --- .../snippets/storage_list_files_with_prefix.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/storage/samples/snippets/storage_list_files_with_prefix.py b/storage/samples/snippets/storage_list_files_with_prefix.py index be7468cba3c..7f877d1d6bc 100644 --- a/storage/samples/snippets/storage_list_files_with_prefix.py +++ b/storage/samples/snippets/storage_list_files_with_prefix.py @@ -46,12 +46,23 @@ def list_blobs_with_prefix(bucket_name, prefix, delimiter=None): that lists the "subfolders" under `a/`: a/b/ + + + Note: If you only want to list prefixes a/b/ and don't want to iterate over + blobs, you can do + + ``` + for page in blobs.pages: + print(page.prefixes) + ``` """ storage_client = storage.Client() # Note: Client.list_blobs requires at least package version 1.17.0. - blobs = storage_client.list_blobs(bucket_name, prefix=prefix, delimiter=delimiter) + blobs = storage_client.list_blobs( + bucket_name, prefix=prefix, delimiter=delimiter + ) # Note: The call returns a response only when the iterator is consumed. print("Blobs:") From d7d5707014e86697194ab3182950080ee2bad6b1 Mon Sep 17 00:00:00 2001 From: shubham-up-47 Date: Mon, 25 Aug 2025 10:46:38 +0530 Subject: [PATCH 157/172] updating signed url samples (#1531) --- storage/samples/snippets/storage_generate_signed_url_v2.py | 4 +--- storage/samples/snippets/storage_generate_signed_url_v4.py | 4 +--- .../samples/snippets/storage_generate_upload_signed_url_v4.py | 4 +--- 3 files changed, 3 insertions(+), 9 deletions(-) diff --git a/storage/samples/snippets/storage_generate_signed_url_v2.py b/storage/samples/snippets/storage_generate_signed_url_v2.py index f1317ea2fbf..9d34630f115 100644 --- a/storage/samples/snippets/storage_generate_signed_url_v2.py +++ b/storage/samples/snippets/storage_generate_signed_url_v2.py @@ -26,9 +26,7 @@ def generate_signed_url(bucket_name, blob_name): """Generates a v2 signed URL for downloading a blob. - Note that this method requires a service account key file. You can not use - this if you are using Application Default Credentials from Google Compute - Engine or from the Google Cloud SDK. + Note that this method requires a service account key file. """ # bucket_name = 'your-bucket-name' # blob_name = 'your-object-name' diff --git a/storage/samples/snippets/storage_generate_signed_url_v4.py b/storage/samples/snippets/storage_generate_signed_url_v4.py index 80625a7b34b..8825a7bb525 100644 --- a/storage/samples/snippets/storage_generate_signed_url_v4.py +++ b/storage/samples/snippets/storage_generate_signed_url_v4.py @@ -27,9 +27,7 @@ def generate_download_signed_url_v4(bucket_name, blob_name): """Generates a v4 signed URL for downloading a blob. - Note that this method requires a service account key file. You can not use - this if you are using Application Default Credentials from Google Compute - Engine or from the Google Cloud SDK. + Note that this method requires a service account key file. """ # bucket_name = 'your-bucket-name' # blob_name = 'your-object-name' diff --git a/storage/samples/snippets/storage_generate_upload_signed_url_v4.py b/storage/samples/snippets/storage_generate_upload_signed_url_v4.py index dc1da88647f..b096fe59eb7 100644 --- a/storage/samples/snippets/storage_generate_upload_signed_url_v4.py +++ b/storage/samples/snippets/storage_generate_upload_signed_url_v4.py @@ -27,9 +27,7 @@ def generate_upload_signed_url_v4(bucket_name, blob_name): """Generates a v4 signed URL for uploading a blob using HTTP PUT. - Note that this method requires a service account key file. You can not use - this if you are using Application Default Credentials from Google Compute - Engine or from the Google Cloud SDK. + Note that this method requires a service account key file. """ # bucket_name = 'your-bucket-name' # blob_name = 'your-object-name' From a8089997dc48e9adb08adc161e308649e37d80ed Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Tue, 2 Sep 2025 14:07:54 +0530 Subject: [PATCH 158/172] chore: add argparse to run samples as script (#1538) * add argparse so that the samples can be run as a a standalone script ```python python samples/snippets/storage_transfer_manager_upload_chunks_concurrently.py --bucket_name --source_filename --destination_blob_name ``` --- ...sfer_manager_upload_chunks_concurrently.py | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/storage/samples/snippets/storage_transfer_manager_upload_chunks_concurrently.py b/storage/samples/snippets/storage_transfer_manager_upload_chunks_concurrently.py index 009f0964870..a4abd13b98b 100644 --- a/storage/samples/snippets/storage_transfer_manager_upload_chunks_concurrently.py +++ b/storage/samples/snippets/storage_transfer_manager_upload_chunks_concurrently.py @@ -11,6 +11,8 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +import argparse + # [START storage_transfer_manager_upload_chunks_concurrently] def upload_chunks_concurrently( @@ -54,4 +56,40 @@ def upload_chunks_concurrently( print(f"File {source_filename} uploaded to {destination_blob_name}.") +if __name__ == "__main__": + argparse = argparse.ArgumentParser( + description="Upload a file to GCS in chunks concurrently." + ) + argparse.add_argument( + "--bucket_name", help="The name of the GCS bucket to upload to." + ) + argparse.add_argument( + "--source_filename", help="The local path to the file to upload." + ) + argparse.add_argument( + "--destination_blob_name", help="The name of the object in GCS." + ) + argparse.add_argument( + "--chunk_size", + type=int, + default=32 * 1024 * 1024, + help="The size of each chunk in bytes (default: 32 MiB). The remote\ + service has a minimum of 5 MiB and a maximum of 5 GiB", + ) + argparse.add_argument( + "--workers", + type=int, + default=8, + help="The number of worker processes to use (default: 8).", + ) + args = argparse.parse_args() + upload_chunks_concurrently( + args.bucket_name, + args.source_filename, + args.destination_blob_name, + args.chunk_size, + args.workers, + ) + + # [END storage_transfer_manager_upload_chunks_concurrently] From 647ff3e9b735b045c209619f3d47d6c4540d16e1 Mon Sep 17 00:00:00 2001 From: "gcf-owl-bot[bot]" <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Date: Wed, 29 Oct 2025 10:52:26 +0000 Subject: [PATCH 159/172] chore(python): Add Python 3.14 to python post processor image (#1563) Source-Link: https://togithub.com/googleapis/synthtool/commit/16790a32126759493ba20781e04edd165825ff82 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:543e209e7c1c1ffe720eb4db1a3f045a75099304fb19aa11a47dc717b8aae2a9 --- storage/samples/snippets/noxfile.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/samples/snippets/noxfile.py b/storage/samples/snippets/noxfile.py index a169b5b5b46..69bcaf56de6 100644 --- a/storage/samples/snippets/noxfile.py +++ b/storage/samples/snippets/noxfile.py @@ -89,7 +89,7 @@ def get_pytest_env_vars() -> Dict[str, str]: # DO NOT EDIT - automatically generated. # All versions used to test samples. -ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "3.13"] +ALL_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"] # Any default versions that should be ignored. IGNORED_VERSIONS = TEST_CONFIG["ignored_versions"] From 9cb61c33d1db0661b7e55457a18b4e9bdbda3226 Mon Sep 17 00:00:00 2001 From: Pulkit Aggarwal <54775856+Pulkit0110@users.noreply.github.com> Date: Thu, 20 Nov 2025 11:58:56 +0530 Subject: [PATCH 160/172] samples: add samples for partial list bucket (#1627) Add samples for the partial list bucket feature. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- .../storage_list_buckets_partial_success.py | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 storage/samples/snippets/storage_list_buckets_partial_success.py diff --git a/storage/samples/snippets/storage_list_buckets_partial_success.py b/storage/samples/snippets/storage_list_buckets_partial_success.py new file mode 100644 index 00000000000..bea4c9ed35c --- /dev/null +++ b/storage/samples/snippets/storage_list_buckets_partial_success.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python + +# Copyright 2025 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_list_buckets_partial_success] +from google.cloud import storage + + +def list_buckets_with_partial_success(): + """Lists buckets and includes unreachable buckets in the response.""" + + storage_client = storage.Client() + + buckets_iterator = storage_client.list_buckets(return_partial_success=True) + + for page in buckets_iterator.pages: + if page.unreachable: + print("Unreachable locations in this page:") + for location in page.unreachable: + print(location) + + print("Reachable buckets in this page:") + for bucket in page: + print(bucket.name) + + +# [END storage_list_buckets_partial_success] + + +if __name__ == "__main__": + list_buckets_with_partial_success() From 5ab91769d64a1c16d27ec10ca1ca6ab704fb8518 Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Wed, 17 Dec 2025 16:19:19 +0530 Subject: [PATCH 161/172] chore: skip failing samples due to public access prevention enforcement (#1668) skip failing samples due to public access prevention enforcement. More Details on b/469643064 --- storage/samples/snippets/snippets_test.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 91018f3dd8b..0edba46ca33 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -399,6 +399,7 @@ def test_delete_blob(test_blob): storage_delete_file.delete_blob(test_blob.bucket.name, test_blob.name) +@pytest.mark.xfail(reason="wait until b/469643064 is fixed") def test_make_blob_public(test_public_blob): storage_make_public.make_blob_public( test_public_blob.bucket.name, test_public_blob.name @@ -620,6 +621,7 @@ def test_get_service_account(capsys): assert "@gs-project-accounts.iam.gserviceaccount.com" in out +@pytest.mark.xfail(reason="wait until b/469643064 is fixed") def test_download_public_file(test_public_blob): storage_make_public.make_blob_public( test_public_blob.bucket.name, test_public_blob.name From ac7d49ae18ca5a45b84c8b186c9d37c883d66223 Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Sat, 20 Dec 2025 01:02:00 +0530 Subject: [PATCH 162/172] chore: skip kms tests until b/470276398 (#1690) chore: skip kms tests until b/470276398 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- storage/samples/snippets/snippets_test.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/storage/samples/snippets/snippets_test.py b/storage/samples/snippets/snippets_test.py index 0edba46ca33..1d3c8c1c442 100644 --- a/storage/samples/snippets/snippets_test.py +++ b/storage/samples/snippets/snippets_test.py @@ -18,6 +18,7 @@ import tempfile import time import uuid +import sys from google.cloud import storage import google.cloud.exceptions @@ -99,8 +100,10 @@ import storage_upload_with_kms_key KMS_KEY = os.environ.get("CLOUD_KMS_KEY") +IS_PYTHON_3_14 = sys.version_info[:2] == (3, 14) +@pytest.mark.skipif(IS_PYTHON_3_14, reason="b/470276398") def test_enable_default_kms_key(test_bucket): storage_set_bucket_default_kms_key.enable_default_kms_key( bucket_name=test_bucket.name, kms_key_name=KMS_KEY @@ -305,6 +308,7 @@ def test_upload_blob_from_stream(test_bucket, capsys): assert "Stream data uploaded to test_upload_blob" in out +@pytest.mark.skipif(IS_PYTHON_3_14, reason="b/470276398") def test_upload_blob_with_kms(test_bucket): blob_name = f"test_upload_with_kms_{uuid.uuid4().hex}" with tempfile.NamedTemporaryFile() as source_file: @@ -598,6 +602,7 @@ def test_create_bucket_dual_region(test_bucket_create, capsys): assert "dual-region" in out +@pytest.mark.skipif(IS_PYTHON_3_14, reason="b/470276398") def test_bucket_delete_default_kms_key(test_bucket, capsys): test_bucket.default_kms_key_name = KMS_KEY test_bucket.patch() @@ -646,6 +651,7 @@ def test_define_bucket_website_configuration(test_bucket): assert bucket._properties["website"] == website_val +@pytest.mark.skipif(IS_PYTHON_3_14, reason="b/470276398") def test_object_get_kms_key(test_bucket): with tempfile.NamedTemporaryFile() as source_file: storage_upload_with_kms_key.upload_blob_with_kms( From 06c8ecad5a20a9d73330d8363663bb4e78d6f27a Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Tue, 20 Jan 2026 18:20:24 +0530 Subject: [PATCH 163/172] feat(samples): add samples for appendable objects writes and reads (#1705) feat(samples): add samples for appendable objects writes and reads --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- ...rage_create_and_write_appendable_object.py | 77 ++++++ ...orage_finalize_appendable_object_upload.py | 78 ++++++ ...orage_open_multiple_objects_ranged_read.py | 85 ++++++ ...torage_open_object_multiple_ranged_read.py | 85 ++++++ .../storage_open_object_read_full_object.py | 72 +++++ .../storage_open_object_single_ranged_read.py | 77 ++++++ ...rage_pause_and_resume_appendable_upload.py | 94 +++++++ .../storage_read_appendable_object_tail.py | 141 ++++++++++ .../zonal_buckets/zonal_snippets_test.py | 260 ++++++++++++++++++ 9 files changed, 969 insertions(+) create mode 100644 storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py create mode 100644 storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py create mode 100644 storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py create mode 100644 storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py create mode 100644 storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py create mode 100644 storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py create mode 100644 storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py create mode 100644 storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py create mode 100644 storage/samples/snippets/zonal_buckets/zonal_snippets_test.py diff --git a/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py b/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py new file mode 100644 index 00000000000..87da6cfddc7 --- /dev/null +++ b/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python + +# Copyright 2026 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import asyncio + +from google.cloud.storage._experimental.asyncio.async_appendable_object_writer import ( + AsyncAppendableObjectWriter, +) +from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient + + +# [START storage_create_and_write_appendable_object] + + +async def storage_create_and_write_appendable_object( + bucket_name, object_name, grpc_client=None +): + """Uploads an appendable object to zonal bucket. + + grpc_client: an existing grpc_client to use, this is only for testing. + """ + + if grpc_client is None: + grpc_client = AsyncGrpcClient().grpc_client + writer = AsyncAppendableObjectWriter( + client=grpc_client, + bucket_name=bucket_name, + object_name=object_name, + generation=0, # throws `FailedPrecondition` if object already exists. + ) + # This creates a new appendable object of size 0 and opens it for appending. + await writer.open() + + # appends data to the object + # you can perform `.append` multiple times as needed. Data will be appended + # to the end of the object. + await writer.append(b"Some data") + + # Once all appends are done, close the gRPC bidirectional stream. + await writer.close() + + print( + f"Appended object {object_name} created of size {writer.persisted_size} bytes." + ) + + +# [END storage_create_and_write_appendable_object] + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--bucket_name", help="Your Cloud Storage bucket name.") + parser.add_argument("--object_name", help="Your Cloud Storage object name.") + + args = parser.parse_args() + + asyncio.run( + storage_create_and_write_appendable_object( + bucket_name=args.bucket_name, + object_name=args.object_name, + ) + ) diff --git a/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py b/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py new file mode 100644 index 00000000000..4a4b6428935 --- /dev/null +++ b/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python + +# Copyright 2026 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import asyncio + +from google.cloud.storage._experimental.asyncio.async_appendable_object_writer import ( + AsyncAppendableObjectWriter, +) +from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient + + +# [START storage_finalize_appendable_object_upload] +async def storage_finalize_appendable_object_upload( + bucket_name, object_name, grpc_client=None +): + """Creates, writes to, and finalizes an appendable object. + + grpc_client: an existing grpc_client to use, this is only for testing. + """ + + if grpc_client is None: + grpc_client = AsyncGrpcClient().grpc_client + writer = AsyncAppendableObjectWriter( + client=grpc_client, + bucket_name=bucket_name, + object_name=object_name, + generation=0, # throws `FailedPrecondition` if object already exists. + ) + # This creates a new appendable object of size 0 and opens it for appending. + await writer.open() + + # Appends data to the object. + await writer.append(b"Some data") + + # finalize the appendable object, + # NOTE: + # 1. once finalized no more appends can be done to the object. + # 2. If you don't want to finalize, you can simply call `writer.close` + # 3. calling `.finalize()` also closes the grpc-bidi stream, calling + # `.close` after `.finalize` may lead to undefined behavior. + object_resource = await writer.finalize() + + print(f"Appendable object {object_name} created and finalized.") + print("Object Metadata:") + print(object_resource) + + +# [END storage_finalize_appendable_object_upload] + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--bucket_name", help="Your Cloud Storage bucket name.") + parser.add_argument("--object_name", help="Your Cloud Storage object name.") + + args = parser.parse_args() + + asyncio.run( + storage_finalize_appendable_object_upload( + bucket_name=args.bucket_name, + object_name=args.object_name, + ) + ) diff --git a/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py b/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py new file mode 100644 index 00000000000..c22023a5c89 --- /dev/null +++ b/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python + +# Copyright 2026 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Downloads a range of bytes from multiple objects concurrently.""" +import argparse +import asyncio +from io import BytesIO + +from google.cloud.storage._experimental.asyncio.async_grpc_client import ( + AsyncGrpcClient, +) +from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import ( + AsyncMultiRangeDownloader, +) + + +# [START storage_open_multiple_objects_ranged_read] +async def storage_open_multiple_objects_ranged_read( + bucket_name, object_names, grpc_client=None +): + """Downloads a range of bytes from multiple objects concurrently. + + grpc_client: an existing grpc_client to use, this is only for testing. + """ + if grpc_client is None: + grpc_client = AsyncGrpcClient().grpc_client + + async def _download_range(object_name): + """Helper coroutine to download a range from a single object.""" + mrd = AsyncMultiRangeDownloader(grpc_client, bucket_name, object_name) + try: + # Open the object, mrd always opens in read mode. + await mrd.open() + + # Each object downloads the first 100 bytes. + start_byte = 0 + size = 100 + + # requested range will be downloaded into this buffer, user may provide + # their own buffer or file-like object. + output_buffer = BytesIO() + await mrd.download_ranges([(start_byte, size, output_buffer)]) + finally: + if mrd.is_stream_open: + await mrd.close() + + # Downloaded size can differ from requested size if object is smaller. + # mrd will download at most up to the end of the object. + downloaded_size = output_buffer.getbuffer().nbytes + print(f"Downloaded {downloaded_size} bytes from {object_name}") + + download_tasks = [_download_range(name) for name in object_names] + await asyncio.gather(*download_tasks) + + +# [END storage_open_multiple_objects_ranged_read] + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--bucket_name", help="Your Cloud Storage bucket name.") + parser.add_argument( + "--object_names", nargs="+", help="Your Cloud Storage object name(s)." + ) + + args = parser.parse_args() + + asyncio.run( + storage_open_multiple_objects_ranged_read(args.bucket_name, args.object_names) + ) diff --git a/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py b/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py new file mode 100644 index 00000000000..55d92d9da4c --- /dev/null +++ b/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python + +# Copyright 2026 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import asyncio +from io import BytesIO + +from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import ( + AsyncMultiRangeDownloader, +) + + +# [START storage_open_object_multiple_ranged_read] +async def storage_open_object_multiple_ranged_read( + bucket_name, object_name, grpc_client=None +): + """Downloads multiple ranges of bytes from a single object into different buffers. + + grpc_client: an existing grpc_client to use, this is only for testing. + """ + if grpc_client is None: + grpc_client = AsyncGrpcClient().grpc_client + + mrd = AsyncMultiRangeDownloader(grpc_client, bucket_name, object_name) + + try: + # Open the object, mrd always opens in read mode. + await mrd.open() + + # Specify four different buffers to download ranges into. + buffers = [BytesIO(), BytesIO(), BytesIO(), BytesIO()] + + # Define the ranges to download. Each range is a tuple of (start_byte, size, buffer). + # All ranges will download 10 bytes from different starting positions. + # We choose arbitrary start bytes for this example. An object should be large enough. + # A user can choose any start byte between 0 and `object_size`. + # If `start_bytes` is greater than `object_size`, mrd will throw an error. + ranges = [ + (0, 10, buffers[0]), + (20, 10, buffers[1]), + (40, 10, buffers[2]), + (60, 10, buffers[3]), + ] + + await mrd.download_ranges(ranges) + + finally: + await mrd.close() + + # Print the downloaded content from each buffer. + for i, output_buffer in enumerate(buffers): + downloaded_size = output_buffer.getbuffer().nbytes + print( + f"Downloaded {downloaded_size} bytes into buffer {i + 1} from start byte {ranges[i][0]}: {output_buffer.getvalue()}" + ) + + +# [END storage_open_object_multiple_ranged_read] + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--bucket_name", help="Your Cloud Storage bucket name.") + parser.add_argument("--object_name", help="Your Cloud Storage object name.") + + args = parser.parse_args() + + asyncio.run( + storage_open_object_multiple_ranged_read(args.bucket_name, args.object_name) + ) diff --git a/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py b/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py new file mode 100644 index 00000000000..044c42b71ff --- /dev/null +++ b/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python + +# Copyright 2026 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import asyncio +from io import BytesIO + +from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import ( + AsyncMultiRangeDownloader, +) + + +# [START storage_open_object_read_full_object] +async def storage_open_object_read_full_object( + bucket_name, object_name, grpc_client=None +): + """Downloads the entire content of an object using a multi-range downloader. + + grpc_client: an existing grpc_client to use, this is only for testing. + """ + if grpc_client is None: + grpc_client = AsyncGrpcClient().grpc_client + + # mrd = Multi-Range-Downloader + mrd = AsyncMultiRangeDownloader(grpc_client, bucket_name, object_name) + + try: + # Open the object, mrd always opens in read mode. + await mrd.open() + + # This could be any buffer or file-like object. + output_buffer = BytesIO() + # A download range of (0, 0) means to read from the beginning to the end. + await mrd.download_ranges([(0, 0, output_buffer)]) + finally: + if mrd.is_stream_open: + await mrd.close() + + downloaded_bytes = output_buffer.getvalue() + print( + f"Downloaded all {len(downloaded_bytes)} bytes from object {object_name} in bucket {bucket_name}." + ) + + +# [END storage_open_object_read_full_object] + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--bucket_name", help="Your Cloud Storage bucket name.") + parser.add_argument("--object_name", help="Your Cloud Storage object name.") + + args = parser.parse_args() + + asyncio.run( + storage_open_object_read_full_object(args.bucket_name, args.object_name) + ) diff --git a/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py b/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py new file mode 100644 index 00000000000..41effce36a4 --- /dev/null +++ b/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python + +# Copyright 2026 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import asyncio +from io import BytesIO + +from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import ( + AsyncMultiRangeDownloader, +) + + +# [START storage_open_object_single_ranged_read] +async def storage_open_object_single_ranged_read( + bucket_name, object_name, start_byte, size, grpc_client=None +): + """Downloads a range of bytes from an object. + + grpc_client: an existing grpc_client to use, this is only for testing. + """ + if grpc_client is None: + grpc_client = AsyncGrpcClient().grpc_client + + mrd = AsyncMultiRangeDownloader(grpc_client, bucket_name, object_name) + + try: + # Open the object, mrd always opens in read mode. + await mrd.open() + + # requested range will be downloaded into this buffer, user may provide + # their own buffer or file-like object. + output_buffer = BytesIO() + await mrd.download_ranges([(start_byte, size, output_buffer)]) + finally: + if mrd.is_stream_open: + await mrd.close() + + # Downloaded size can differ from requested size if object is smaller. + # mrd will download at most up to the end of the object. + downloaded_size = output_buffer.getbuffer().nbytes + print(f"Downloaded {downloaded_size} bytes from {object_name}") + + +# [END storage_open_object_single_ranged_read] + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--bucket_name", help="Your Cloud Storage bucket name.") + parser.add_argument("--object_name", help="Your Cloud Storage object name.") + parser.add_argument( + "--start_byte", type=int, help="The starting byte of the range." + ) + parser.add_argument("--size", type=int, help="The number of bytes to download.") + + args = parser.parse_args() + + asyncio.run( + storage_open_object_single_ranged_read( + args.bucket_name, args.object_name, args.start_byte, args.size + ) + ) diff --git a/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py b/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py new file mode 100644 index 00000000000..db54bb821f3 --- /dev/null +++ b/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python + +# Copyright 2026 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import asyncio + +from google.cloud.storage._experimental.asyncio.async_appendable_object_writer import ( + AsyncAppendableObjectWriter, +) +from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient + + +# [START storage_pause_and_resume_appendable_upload] +async def storage_pause_and_resume_appendable_upload( + bucket_name, object_name, grpc_client=None +): + """Demonstrates pausing and resuming an appendable object upload. + + grpc_client: an existing grpc_client to use, this is only for testing. + """ + if grpc_client is None: + grpc_client = AsyncGrpcClient().grpc_client + + writer1 = AsyncAppendableObjectWriter( + client=grpc_client, + bucket_name=bucket_name, + object_name=object_name, + ) + await writer1.open() + await writer1.append(b"First part of the data. ") + print(f"Appended {writer1.persisted_size} bytes with the first writer.") + + # 2. After appending some data, close the writer to "pause" the upload. + # NOTE: you can pause indefinitely and still read the conetent uploaded so far using MRD. + await writer1.close() + + print("First writer closed. Upload is 'paused'.") + + # 3. Create a new writer, passing the generation number from the previous + # writer. This is a precondition to ensure that the object hasn't been + # modified since we last accessed it. + generation_to_resume = writer1.generation + print(f"Generation to resume from is: {generation_to_resume}") + + writer2 = AsyncAppendableObjectWriter( + client=grpc_client, + bucket_name=bucket_name, + object_name=object_name, + generation=generation_to_resume, + ) + # 4. Open the new writer. + try: + await writer2.open() + + # 5. Append some more data using the new writer. + await writer2.append(b"Second part of the data.") + print(f"Appended more data. Total size is now {writer2.persisted_size} bytes.") + finally: + # 6. Finally, close the new writer. + if writer2._is_stream_open: + await writer2.close() + print("Second writer closed. Full object uploaded.") + + +# [END storage_pause_and_resume_appendable_upload] + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--bucket_name", help="Your Cloud Storage bucket name.") + parser.add_argument("--object_name", help="Your Cloud Storage object name.") + + args = parser.parse_args() + + asyncio.run( + storage_pause_and_resume_appendable_upload( + bucket_name=args.bucket_name, + object_name=args.object_name, + ) + ) diff --git a/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py b/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py new file mode 100644 index 00000000000..5875130560c --- /dev/null +++ b/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python + +# Copyright 2026 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the 'License'); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import asyncio +import time +from datetime import datetime +from io import BytesIO + +from google.cloud.storage._experimental.asyncio.async_appendable_object_writer import ( + AsyncAppendableObjectWriter, +) +from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import ( + AsyncMultiRangeDownloader, +) + +BYTES_TO_APPEND = b"fav_bytes." +NUM_BYTES_TO_APPEND_EVERY_SECOND = len(BYTES_TO_APPEND) + + +# [START storage_read_appendable_object_tail] +async def appender(writer: AsyncAppendableObjectWriter, duration: int): + """Appends 10 bytes to the object every second for a given duration.""" + print("Appender started.") + bytes_appended = 0 + for i in range(duration): + await writer.append(BYTES_TO_APPEND) + now = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3] + bytes_appended += NUM_BYTES_TO_APPEND_EVERY_SECOND + print( + f"[{now}] Appended {NUM_BYTES_TO_APPEND_EVERY_SECOND} new bytes. Total appended: {bytes_appended} bytes." + ) + await asyncio.sleep(1) + print("Appender finished.") + + +async def tailer( + bucket_name: str, object_name: str, duration: int, client: AsyncGrpcClient +): + """Tails the object by reading new data as it is appended.""" + print("Tailer started.") + start_byte = 0 + start_time = time.monotonic() + mrd = AsyncMultiRangeDownloader(client, bucket_name, object_name) + try: + await mrd.open() + # Run the tailer for the specified duration. + while time.monotonic() - start_time < duration: + output_buffer = BytesIO() + # A download range of (start, 0) means to read from 'start' to the end. + await mrd.download_ranges([(start_byte, 0, output_buffer)]) + + bytes_downloaded = output_buffer.getbuffer().nbytes + if bytes_downloaded > 0: + now = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3] + print( + f"[{now}] Tailer read {bytes_downloaded} new bytes: {output_buffer.getvalue()}" + ) + start_byte += bytes_downloaded + + await asyncio.sleep(0.1) # Poll for new data every 0.1 seconds. + finally: + if mrd.is_stream_open: + await mrd.close() + print("Tailer finished.") + + +# read_appendable_object_tail simulates a "tail -f" command on a GCS object. It +# repeatedly polls an appendable object for new content. In a real +# application, the object would be written to by a separate process. +async def read_appendable_object_tail( + bucket_name: str, object_name: str, duration: int, grpc_client=None +): + """Main function to create an appendable object and run tasks. + + grpc_client: an existing grpc_client to use, this is only for testing. + """ + if grpc_client is None: + grpc_client = AsyncGrpcClient().grpc_client + writer = AsyncAppendableObjectWriter( + client=grpc_client, + bucket_name=bucket_name, + object_name=object_name, + ) + # 1. Create an empty appendable object. + try: + # 1. Create an empty appendable object. + await writer.open() + print(f"Created empty appendable object: {object_name}") + + # 2. Create the appender and tailer coroutines. + appender_task = asyncio.create_task(appender(writer, duration)) + tailer_task = asyncio.create_task( + tailer(bucket_name, object_name, duration, grpc_client) + ) + + # 3. Execute the coroutines concurrently. + await asyncio.gather(appender_task, tailer_task) + finally: + if writer._is_stream_open: + await writer.close() + print("Writer closed.") + + +# [END storage_read_appendable_object_tail] + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Demonstrates tailing an appendable GCS object.", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument("--bucket_name", help="Your Cloud Storage bucket name.") + parser.add_argument( + "--object_name", help="Your Cloud Storage object name to be created." + ) + parser.add_argument( + "--duration", + type=int, + default=60, + help="Duration in seconds to run the demo.", + ) + + args = parser.parse_args() + + asyncio.run( + read_appendable_object_tail(args.bucket_name, args.object_name, args.duration) + ) diff --git a/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py b/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py new file mode 100644 index 00000000000..e45d21c4773 --- /dev/null +++ b/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py @@ -0,0 +1,260 @@ +# Copyright 2025 Google, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import uuid +import os + +import pytest +from google.cloud.storage import Client +import contextlib + +from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage._experimental.asyncio.async_appendable_object_writer import ( + AsyncAppendableObjectWriter, +) + +# Import all the snippets +import storage_create_and_write_appendable_object +import storage_finalize_appendable_object_upload +import storage_open_multiple_objects_ranged_read +import storage_open_object_multiple_ranged_read +import storage_open_object_read_full_object +import storage_open_object_single_ranged_read +import storage_pause_and_resume_appendable_upload +import storage_read_appendable_object_tail + +pytestmark = pytest.mark.skipif( + os.getenv("RUN_ZONAL_SYSTEM_TESTS") != "True", + reason="Zonal system tests need to be explicitly enabled. This helps scheduling tests in Kokoro and Cloud Build.", +) + + +# TODO: replace this with a fixture once zonal bucket creation / deletion +# is supported in grpc client or json client client. +_ZONAL_BUCKET = os.getenv("ZONAL_BUCKET") + + +async def create_async_grpc_client(): + """Initializes async client and gets the current event loop.""" + return AsyncGrpcClient().grpc_client + + +# Forcing a single event loop for the whole test session +@pytest.fixture(scope="session") +def event_loop(): + """Redefine pytest-asyncio's event_loop fixture to be session-scoped.""" + loop = asyncio.get_event_loop_policy().new_event_loop() + yield loop + loop.close() + + +@pytest.fixture(scope="session") +def async_grpc_client(event_loop): + """Yields a StorageAsyncClient that is closed after the test session.""" + grpc_client = event_loop.run_until_complete(create_async_grpc_client()) + yield grpc_client + + +@pytest.fixture(scope="session") +def json_client(): + client = Client() + with contextlib.closing(client): + yield client + + +async def create_appendable_object(grpc_client, object_name, data): + writer = AsyncAppendableObjectWriter( + client=grpc_client, + bucket_name=_ZONAL_BUCKET, + object_name=object_name, + generation=0, # throws `FailedPrecondition` if object already exists. + ) + await writer.open() + await writer.append(data) + await writer.close() + return writer.generation + + +# TODO: replace this with a fixture once zonal bucket creation / deletion +# is supported in grpc client or json client client. +_ZONAL_BUCKET = os.getenv("ZONAL_BUCKET") + + +def test_storage_create_and_write_appendable_object( + async_grpc_client, json_client, event_loop, capsys +): + object_name = f"zonal-snippets-test-{uuid.uuid4()}" + + event_loop.run_until_complete( + storage_create_and_write_appendable_object.storage_create_and_write_appendable_object( + _ZONAL_BUCKET, object_name, grpc_client=async_grpc_client + ) + ) + out, _ = capsys.readouterr() + assert f"Appended object {object_name} created of size" in out + + blob = json_client.bucket(_ZONAL_BUCKET).blob(object_name) + blob.delete() + + +def test_storage_finalize_appendable_object_upload( + async_grpc_client, json_client, event_loop, capsys +): + object_name = f"test-finalize-appendable-{uuid.uuid4()}" + event_loop.run_until_complete( + storage_finalize_appendable_object_upload.storage_finalize_appendable_object_upload( + _ZONAL_BUCKET, object_name, grpc_client=async_grpc_client + ) + ) + out, _ = capsys.readouterr() + assert f"Appendable object {object_name} created and finalized." in out + blob = json_client.bucket(_ZONAL_BUCKET).get_blob(object_name) + blob.delete() + + +def test_storage_pause_and_resume_appendable_upload( + async_grpc_client, json_client, event_loop, capsys +): + object_name = f"test-pause-resume-{uuid.uuid4()}" + event_loop.run_until_complete( + storage_pause_and_resume_appendable_upload.storage_pause_and_resume_appendable_upload( + _ZONAL_BUCKET, object_name, grpc_client=async_grpc_client + ) + ) + out, _ = capsys.readouterr() + assert "First writer closed. Upload is 'paused'." in out + assert "Second writer closed. Full object uploaded." in out + + blob = json_client.bucket(_ZONAL_BUCKET).get_blob(object_name) + blob.delete() + + +def test_storage_read_appendable_object_tail( + async_grpc_client, json_client, event_loop, capsys +): + object_name = f"test-read-tail-{uuid.uuid4()}" + event_loop.run_until_complete( + storage_read_appendable_object_tail.read_appendable_object_tail( + _ZONAL_BUCKET, object_name, duration=3, grpc_client=async_grpc_client + ) + ) + out, _ = capsys.readouterr() + assert f"Created empty appendable object: {object_name}" in out + assert "Appender started." in out + assert "Tailer started." in out + assert "Tailer read" in out + assert "Tailer finished." in out + assert "Writer closed." in out + + bucket = json_client.bucket(_ZONAL_BUCKET) + blob = bucket.blob(object_name) + blob.delete() + + +def test_storage_open_object_read_full_object( + async_grpc_client, json_client, event_loop, capsys +): + object_name = f"test-open-read-full-{uuid.uuid4()}" + data = b"Hello, is it me you're looking for?" + event_loop.run_until_complete( + create_appendable_object(async_grpc_client, object_name, data) + ) + event_loop.run_until_complete( + storage_open_object_read_full_object.storage_open_object_read_full_object( + _ZONAL_BUCKET, object_name, grpc_client=async_grpc_client + ) + ) + out, _ = capsys.readouterr() + assert ( + f"Downloaded all {len(data)} bytes from object {object_name} in bucket {_ZONAL_BUCKET}." + in out + ) + blob = json_client.bucket(_ZONAL_BUCKET).blob(object_name) + blob.delete() + + +def test_storage_open_object_single_ranged_read( + async_grpc_client, json_client, event_loop, capsys +): + object_name = f"test-open-single-range-{uuid.uuid4()}" + event_loop.run_until_complete( + create_appendable_object( + async_grpc_client, object_name, b"Hello, is it me you're looking for?" + ) + ) + download_size = 5 + event_loop.run_until_complete( + storage_open_object_single_ranged_read.storage_open_object_single_ranged_read( + _ZONAL_BUCKET, + object_name, + start_byte=0, + size=download_size, + grpc_client=async_grpc_client, + ) + ) + out, _ = capsys.readouterr() + assert f"Downloaded {download_size} bytes from {object_name}" in out + blob = json_client.bucket(_ZONAL_BUCKET).blob(object_name) + blob.delete() + + +def test_storage_open_object_multiple_ranged_read( + async_grpc_client, json_client, event_loop, capsys +): + object_name = f"test-open-multi-range-{uuid.uuid4()}" + data = b"a" * 100 + event_loop.run_until_complete( + create_appendable_object(async_grpc_client, object_name, data) + ) + event_loop.run_until_complete( + storage_open_object_multiple_ranged_read.storage_open_object_multiple_ranged_read( + _ZONAL_BUCKET, object_name, grpc_client=async_grpc_client + ) + ) + out, _ = capsys.readouterr() + assert "Downloaded 10 bytes into buffer 1 from start byte 0: b'aaaaaaaaaa'" in out + assert "Downloaded 10 bytes into buffer 2 from start byte 20: b'aaaaaaaaaa'" in out + assert "Downloaded 10 bytes into buffer 3 from start byte 40: b'aaaaaaaaaa'" in out + assert "Downloaded 10 bytes into buffer 4 from start byte 60: b'aaaaaaaaaa'" in out + blob = json_client.bucket(_ZONAL_BUCKET).blob(object_name) + blob.delete() + + +def test_storage_open_multiple_objects_ranged_read( + async_grpc_client, json_client, event_loop, capsys +): + blob1_name = f"multi-obj-1-{uuid.uuid4()}" + blob2_name = f"multi-obj-2-{uuid.uuid4()}" + data1 = b"Content of object 1" + data2 = b"Content of object 2" + event_loop.run_until_complete( + create_appendable_object(async_grpc_client, blob1_name, data1) + ) + event_loop.run_until_complete( + create_appendable_object(async_grpc_client, blob2_name, data2) + ) + + event_loop.run_until_complete( + storage_open_multiple_objects_ranged_read.storage_open_multiple_objects_ranged_read( + _ZONAL_BUCKET, [blob1_name, blob2_name], grpc_client=async_grpc_client + ) + ) + out, _ = capsys.readouterr() + assert f"Downloaded {len(data1)} bytes from {blob1_name}" in out + assert f"Downloaded {len(data2)} bytes from {blob2_name}" in out + blob1 = json_client.bucket(_ZONAL_BUCKET).blob(blob1_name) + blob2 = json_client.bucket(_ZONAL_BUCKET).blob(blob2_name) + blob1.delete() + blob2.delete() From e080b6eae4ccaad53a4e56b5a009d90af0e76d10 Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Wed, 28 Jan 2026 16:54:59 +0530 Subject: [PATCH 164/172] feat: expose finalized_time in blob.py applicable for GET_OBJECT in ZB (#1719) feat: expose finalized_time in blob.py applicable for GET_OBJECT in ZB --- storage/samples/snippets/storage_get_metadata.py | 1 + 1 file changed, 1 insertion(+) diff --git a/storage/samples/snippets/storage_get_metadata.py b/storage/samples/snippets/storage_get_metadata.py index 7216efdb4de..1e332b44565 100644 --- a/storage/samples/snippets/storage_get_metadata.py +++ b/storage/samples/snippets/storage_get_metadata.py @@ -34,6 +34,7 @@ def blob_metadata(bucket_name, blob_name): blob = bucket.get_blob(blob_name) print(f"Blob: {blob.name}") + print(f"Blob finalization: {blob.finalized_time}") print(f"Bucket: {blob.bucket.name}") print(f"Storage class: {blob.storage_class}") print(f"ID: {blob.id}") From 54a76ec4fad6668685d255c7ffad9334e99f9d4c Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Wed, 28 Jan 2026 19:21:08 +0530 Subject: [PATCH 165/172] fix!: Change contructors of MRD and AAOW AsyncGrpcClient.grpc_client to AsyncGrpcClient (#1727) Change contructors of MRD and AAOW `AsyncGrpcClient.grpc_client` to `AsyncGrpcClient`. This will help us extend multiple methods in AsyncGrpcClient in future. More details in b/479030029 --- .../zonal_buckets/storage_create_and_write_appendable_object.py | 2 +- .../zonal_buckets/storage_finalize_appendable_object_upload.py | 2 +- .../zonal_buckets/storage_open_multiple_objects_ranged_read.py | 2 +- .../zonal_buckets/storage_open_object_multiple_ranged_read.py | 2 +- .../zonal_buckets/storage_open_object_read_full_object.py | 2 +- .../zonal_buckets/storage_open_object_single_ranged_read.py | 2 +- .../zonal_buckets/storage_pause_and_resume_appendable_upload.py | 2 +- .../zonal_buckets/storage_read_appendable_object_tail.py | 2 +- storage/samples/snippets/zonal_buckets/zonal_snippets_test.py | 2 +- 9 files changed, 9 insertions(+), 9 deletions(-) diff --git a/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py b/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py index 87da6cfddc7..f00a6ba80ed 100644 --- a/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py +++ b/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py @@ -35,7 +35,7 @@ async def storage_create_and_write_appendable_object( """ if grpc_client is None: - grpc_client = AsyncGrpcClient().grpc_client + grpc_client = AsyncGrpcClient() writer = AsyncAppendableObjectWriter( client=grpc_client, bucket_name=bucket_name, diff --git a/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py b/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py index 4a4b6428935..971658997fa 100644 --- a/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py +++ b/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py @@ -33,7 +33,7 @@ async def storage_finalize_appendable_object_upload( """ if grpc_client is None: - grpc_client = AsyncGrpcClient().grpc_client + grpc_client = AsyncGrpcClient() writer = AsyncAppendableObjectWriter( client=grpc_client, bucket_name=bucket_name, diff --git a/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py b/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py index c22023a5c89..ce2ba678b06 100644 --- a/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py +++ b/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py @@ -36,7 +36,7 @@ async def storage_open_multiple_objects_ranged_read( grpc_client: an existing grpc_client to use, this is only for testing. """ if grpc_client is None: - grpc_client = AsyncGrpcClient().grpc_client + grpc_client = AsyncGrpcClient() async def _download_range(object_name): """Helper coroutine to download a range from a single object.""" diff --git a/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py b/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py index 55d92d9da4c..02c3bace539 100644 --- a/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py +++ b/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py @@ -33,7 +33,7 @@ async def storage_open_object_multiple_ranged_read( grpc_client: an existing grpc_client to use, this is only for testing. """ if grpc_client is None: - grpc_client = AsyncGrpcClient().grpc_client + grpc_client = AsyncGrpcClient() mrd = AsyncMultiRangeDownloader(grpc_client, bucket_name, object_name) diff --git a/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py b/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py index 044c42b71ff..b4cad671825 100644 --- a/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py +++ b/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py @@ -33,7 +33,7 @@ async def storage_open_object_read_full_object( grpc_client: an existing grpc_client to use, this is only for testing. """ if grpc_client is None: - grpc_client = AsyncGrpcClient().grpc_client + grpc_client = AsyncGrpcClient() # mrd = Multi-Range-Downloader mrd = AsyncMultiRangeDownloader(grpc_client, bucket_name, object_name) diff --git a/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py b/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py index 41effce36a4..b013cc93893 100644 --- a/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py +++ b/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py @@ -33,7 +33,7 @@ async def storage_open_object_single_ranged_read( grpc_client: an existing grpc_client to use, this is only for testing. """ if grpc_client is None: - grpc_client = AsyncGrpcClient().grpc_client + grpc_client = AsyncGrpcClient() mrd = AsyncMultiRangeDownloader(grpc_client, bucket_name, object_name) diff --git a/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py b/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py index db54bb821f3..3fb17ceaea8 100644 --- a/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py +++ b/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py @@ -32,7 +32,7 @@ async def storage_pause_and_resume_appendable_upload( grpc_client: an existing grpc_client to use, this is only for testing. """ if grpc_client is None: - grpc_client = AsyncGrpcClient().grpc_client + grpc_client = AsyncGrpcClient() writer1 = AsyncAppendableObjectWriter( client=grpc_client, diff --git a/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py b/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py index 5875130560c..1134f28d6c9 100644 --- a/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py +++ b/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py @@ -90,7 +90,7 @@ async def read_appendable_object_tail( grpc_client: an existing grpc_client to use, this is only for testing. """ if grpc_client is None: - grpc_client = AsyncGrpcClient().grpc_client + grpc_client = AsyncGrpcClient() writer = AsyncAppendableObjectWriter( client=grpc_client, bucket_name=bucket_name, diff --git a/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py b/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py index e45d21c4773..736576eb593 100644 --- a/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py +++ b/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py @@ -48,7 +48,7 @@ async def create_async_grpc_client(): """Initializes async client and gets the current event loop.""" - return AsyncGrpcClient().grpc_client + return AsyncGrpcClient() # Forcing a single event loop for the whole test session From 63731f21d2ab4cae1222e17a114b4fc75efbe674 Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Thu, 29 Jan 2026 14:28:28 +0530 Subject: [PATCH 166/172] feat: Move Zonal Buckets features of `_experimental` (#1728) feat: Move Zonal Buckets features of `_experimental` to --- .../storage_create_and_write_appendable_object.py | 4 ++-- .../storage_finalize_appendable_object_upload.py | 4 ++-- .../storage_open_multiple_objects_ranged_read.py | 11 ++++++++--- .../storage_open_object_multiple_ranged_read.py | 4 ++-- .../storage_open_object_read_full_object.py | 4 ++-- .../storage_open_object_single_ranged_read.py | 4 ++-- .../storage_pause_and_resume_appendable_upload.py | 4 ++-- .../storage_read_appendable_object_tail.py | 6 +++--- .../snippets/zonal_buckets/zonal_snippets_test.py | 4 ++-- 9 files changed, 25 insertions(+), 20 deletions(-) diff --git a/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py b/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py index f00a6ba80ed..725eeb2bd98 100644 --- a/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py +++ b/storage/samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py @@ -17,10 +17,10 @@ import argparse import asyncio -from google.cloud.storage._experimental.asyncio.async_appendable_object_writer import ( +from google.cloud.storage.asyncio.async_appendable_object_writer import ( AsyncAppendableObjectWriter, ) -from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient # [START storage_create_and_write_appendable_object] diff --git a/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py b/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py index 971658997fa..807fe40a58d 100644 --- a/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py +++ b/storage/samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py @@ -17,10 +17,10 @@ import argparse import asyncio -from google.cloud.storage._experimental.asyncio.async_appendable_object_writer import ( +from google.cloud.storage.asyncio.async_appendable_object_writer import ( AsyncAppendableObjectWriter, ) -from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient # [START storage_finalize_appendable_object_upload] diff --git a/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py b/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py index ce2ba678b06..bed580d3662 100644 --- a/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py +++ b/storage/samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py @@ -14,15 +14,20 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Downloads a range of bytes from multiple objects concurrently.""" +"""Downloads a range of bytes from multiple objects concurrently. +Example usage: + ```python samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py \ + --bucket_name \ + --object_names ``` +""" import argparse import asyncio from io import BytesIO -from google.cloud.storage._experimental.asyncio.async_grpc_client import ( +from google.cloud.storage.asyncio.async_grpc_client import ( AsyncGrpcClient, ) -from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import ( +from google.cloud.storage.asyncio.async_multi_range_downloader import ( AsyncMultiRangeDownloader, ) diff --git a/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py b/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py index 02c3bace539..b0f64c48690 100644 --- a/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py +++ b/storage/samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py @@ -18,8 +18,8 @@ import asyncio from io import BytesIO -from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient -from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import ( +from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage.asyncio.async_multi_range_downloader import ( AsyncMultiRangeDownloader, ) diff --git a/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py b/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py index b4cad671825..2e18caabe23 100644 --- a/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py +++ b/storage/samples/snippets/zonal_buckets/storage_open_object_read_full_object.py @@ -18,8 +18,8 @@ import asyncio from io import BytesIO -from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient -from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import ( +from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage.asyncio.async_multi_range_downloader import ( AsyncMultiRangeDownloader, ) diff --git a/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py b/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py index b013cc93893..74bec43f68e 100644 --- a/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py +++ b/storage/samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py @@ -18,8 +18,8 @@ import asyncio from io import BytesIO -from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient -from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import ( +from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage.asyncio.async_multi_range_downloader import ( AsyncMultiRangeDownloader, ) diff --git a/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py b/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py index 3fb17ceaea8..c758dc6419d 100644 --- a/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py +++ b/storage/samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py @@ -17,10 +17,10 @@ import argparse import asyncio -from google.cloud.storage._experimental.asyncio.async_appendable_object_writer import ( +from google.cloud.storage.asyncio.async_appendable_object_writer import ( AsyncAppendableObjectWriter, ) -from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient # [START storage_pause_and_resume_appendable_upload] diff --git a/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py b/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py index 1134f28d6c9..9e4dcd73896 100644 --- a/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py +++ b/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py @@ -20,11 +20,11 @@ from datetime import datetime from io import BytesIO -from google.cloud.storage._experimental.asyncio.async_appendable_object_writer import ( +from google.cloud.storage.asyncio.async_appendable_object_writer import ( AsyncAppendableObjectWriter, ) -from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient -from google.cloud.storage._experimental.asyncio.async_multi_range_downloader import ( +from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage.asyncio.async_multi_range_downloader import ( AsyncMultiRangeDownloader, ) diff --git a/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py b/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py index 736576eb593..6852efe2286 100644 --- a/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py +++ b/storage/samples/snippets/zonal_buckets/zonal_snippets_test.py @@ -20,8 +20,8 @@ from google.cloud.storage import Client import contextlib -from google.cloud.storage._experimental.asyncio.async_grpc_client import AsyncGrpcClient -from google.cloud.storage._experimental.asyncio.async_appendable_object_writer import ( +from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient +from google.cloud.storage.asyncio.async_appendable_object_writer import ( AsyncAppendableObjectWriter, ) From 5e4bceec7268d3a029210c8f68458019b96d3824 Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Mon, 2 Feb 2026 14:40:44 +0530 Subject: [PATCH 167/172] chore: Add README for running zonal buckets samples (#1734) chore: Add README for running zonal buckets samples --- .../samples/snippets/zonal_buckets/README.md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 storage/samples/snippets/zonal_buckets/README.md diff --git a/storage/samples/snippets/zonal_buckets/README.md b/storage/samples/snippets/zonal_buckets/README.md new file mode 100644 index 00000000000..71c17e5c3f1 --- /dev/null +++ b/storage/samples/snippets/zonal_buckets/README.md @@ -0,0 +1,78 @@ +# Google Cloud Storage - Zonal Buckets Snippets + +This directory contains snippets for interacting with Google Cloud Storage zonal buckets. + +## Prerequisites + +- A Google Cloud Platform project with the Cloud Storage API enabled. +- A zonal Google Cloud Storage bucket. + +## Running the snippets + +### Create and write to an appendable object + +This snippet uploads an appendable object to a zonal bucket. + +```bash +python samples/snippets/zonal_buckets/storage_create_and_write_appendable_object.py --bucket_name --object_name +``` + +### Finalize an appendable object upload + +This snippet creates, writes to, and finalizes an appendable object. + +```bash +python samples/snippets/zonal_buckets/storage_finalize_appendable_object_upload.py --bucket_name --object_name +``` + +### Pause and resume an appendable object upload + +This snippet demonstrates pausing and resuming an appendable object upload. + +```bash +python samples/snippets/zonal_buckets/storage_pause_and_resume_appendable_upload.py --bucket_name --object_name +``` + +### Tail an appendable object + +This snippet demonstrates tailing an appendable GCS object, similar to `tail -f`. + +```bash +python samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py --bucket_name --object_name --duration +``` + + +### Download a range of bytes from an object + +This snippet downloads a range of bytes from an object. + +```bash +python samples/snippets/zonal_buckets/storage_open_object_single_ranged_read.py --bucket_name --object_name --start_byte --size +``` + + +### Download multiple ranges of bytes from a single object + +This snippet downloads multiple ranges of bytes from a single object into different buffers. + +```bash +python samples/snippets/zonal_buckets/storage_open_object_multiple_ranged_read.py --bucket_name --object_name +``` + +### Download the entire content of an object + +This snippet downloads the entire content of an object using a multi-range downloader. + +```bash +python samples/snippets/zonal_buckets/storage_open_object_read_full_object.py --bucket_name --object_name +``` + + + +### Download a range of bytes from multiple objects concurrently + +This snippet downloads a range of bytes from multiple objects concurrently. + +```bash +python samples/snippets/zonal_buckets/storage_open_multiple_objects_ranged_read.py --bucket_name --object_names +``` \ No newline at end of file From 2cde838159cfe789a5911033f0a15190d9130a76 Mon Sep 17 00:00:00 2001 From: gurusai-voleti Date: Wed, 4 Feb 2026 13:50:18 +0530 Subject: [PATCH 168/172] chore: Migrate gsutil usage to gcloud storage (#1732) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Automated: Migrate {target_path} from gsutil to gcloud storage This CL is part of the on going effort to migrate from the legacy `gsutil` tool to the new and improved `gcloud storage` command-line interface. `gcloud storage` is the recommended and modern tool for interacting with Google Cloud Storage, offering better performance, unified authentication, and a more consistent command structure with other `gcloud` components. 🚀 ### Automation Details This change was **generated automatically** by an agent that targets users of `gsutil`. The transformations applied are based on the [gsutil to gcloud storage migration guide](http://go/gsutil-gcloud-storage-migration-guide). ### ⚠️ Action Required: Please Review and Test Carefully While we have based the automation on the migration guide, every use case is unique. **It is crucial that you thoroughly test these changes in environments appropriate to your use-case before merging.** Be aware of potential differences between `gsutil` and `gcloud storage` that could impact your workflows. For instance, the structure of command output may have changed, requiring updates to any scripts that parse it. Similarly, command behavior can differ subtly; the `gcloud storage rsync` command has a different file deletion logic than `gsutil rsync`, which could lead to unintended file deletions. Our migration guides can help guide you through a list of mappings and some notable differences between the two tools. Standard presubmit tests are run as part of this CL's workflow. **If you need to target an additional test workflow or require assistance with testing, please let us know.** Please verify that all your Cloud Storage operations continue to work as expected to avoid any potential disruptions in production. ### Support and Collaboration The `GCS CLI` team is here to help! If you encounter any issues, have a complex use case that this automated change doesn't cover, or face any other blockers, please don't hesitate to reach out. We are happy to work with you to test and adjust these changes as needed. **Contact:** `gcs-cli-hyd@google.com` We appreciate your partnership in this important migration effort! #gsutil-migration Co-authored-by: Chandra Shekhar Sirimala --- storage/samples/snippets/notification_polling.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/samples/snippets/notification_polling.py b/storage/samples/snippets/notification_polling.py index 2ee6789c32f..1359c9cfa19 100644 --- a/storage/samples/snippets/notification_polling.py +++ b/storage/samples/snippets/notification_polling.py @@ -32,10 +32,10 @@ https://console.cloud.google.com/flows/enableapi?apiid=pubsub 3. Create a Google Cloud Storage bucket: - $ gsutil mb gs://testbucket + $ gcloud storage buckets create gs://testbucket 4. Create a Cloud Pub/Sub topic and publish bucket notifications there: - $ gsutil notification create -f json -t testtopic gs://testbucket + $ gcloud storage buckets notifications create gs://testbucket --topic=testtopic --payload-format=json 5. Create a subscription for your new topic: $ gcloud pubsub subscriptions create testsubscription --topic=testtopic From 543ec2560742be672a130193f599b6a0bc1de5bb Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Wed, 11 Feb 2026 20:50:22 +0530 Subject: [PATCH 169/172] fix: don't flush at every append, results in bad perf (#1746) fix: don't flush at every append, results in bad perf --- .../storage_read_appendable_object_tail.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py b/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py index 9e4dcd73896..6248980669c 100644 --- a/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py +++ b/storage/samples/snippets/zonal_buckets/storage_read_appendable_object_tail.py @@ -28,7 +28,7 @@ AsyncMultiRangeDownloader, ) -BYTES_TO_APPEND = b"fav_bytes." +BYTES_TO_APPEND = b"fav_bytes." * 100 * 1024 * 1024 NUM_BYTES_TO_APPEND_EVERY_SECOND = len(BYTES_TO_APPEND) @@ -37,14 +37,16 @@ async def appender(writer: AsyncAppendableObjectWriter, duration: int): """Appends 10 bytes to the object every second for a given duration.""" print("Appender started.") bytes_appended = 0 - for i in range(duration): + start_time = time.monotonic() + # Run the appender for the specified duration. + while time.monotonic() - start_time < duration: await writer.append(BYTES_TO_APPEND) now = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3] bytes_appended += NUM_BYTES_TO_APPEND_EVERY_SECOND print( f"[{now}] Appended {NUM_BYTES_TO_APPEND_EVERY_SECOND} new bytes. Total appended: {bytes_appended} bytes." ) - await asyncio.sleep(1) + await asyncio.sleep(0.1) print("Appender finished.") @@ -67,9 +69,7 @@ async def tailer( bytes_downloaded = output_buffer.getbuffer().nbytes if bytes_downloaded > 0: now = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3] - print( - f"[{now}] Tailer read {bytes_downloaded} new bytes: {output_buffer.getvalue()}" - ) + print(f"[{now}] Tailer read {bytes_downloaded} new bytes: ") start_byte += bytes_downloaded await asyncio.sleep(0.1) # Poll for new data every 0.1 seconds. From a53dbb4b5324d6acab9009222f96b8cd3bbb5bf3 Mon Sep 17 00:00:00 2001 From: Nidhi Date: Mon, 16 Mar 2026 17:58:55 +0000 Subject: [PATCH 170/172] chore: skip hmac tests until b/493225655 is fixed (#1771) Skipping hmac tests if they fail with a 412 PreconditionFailed. This is occurring because the testing project python-docs-samples-tests has the constraints/iam.disableServiceAccountKeyCreation Organization Policy enforced. --- storage/samples/snippets/hmac_samples_test.py | 29 ++++++++++++++----- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/storage/samples/snippets/hmac_samples_test.py b/storage/samples/snippets/hmac_samples_test.py index 988b4030502..fbc2e292df6 100644 --- a/storage/samples/snippets/hmac_samples_test.py +++ b/storage/samples/snippets/hmac_samples_test.py @@ -17,7 +17,6 @@ set in order to run. """ - import os import google.api_core.exceptions @@ -51,9 +50,18 @@ def new_hmac_key(): NOTE: Due to the module scope, test order in this file is significant """ - hmac_key, secret = STORAGE_CLIENT.create_hmac_key( - service_account_email=SERVICE_ACCOUNT_EMAIL, project_id=PROJECT_ID - ) + try: + hmac_key, secret = STORAGE_CLIENT.create_hmac_key( + service_account_email=SERVICE_ACCOUNT_EMAIL, project_id=PROJECT_ID + ) + except google.api_core.exceptions.PreconditionFailed as e: + # Check if the failure is due to the Organization Policy constraint + if "constraints/iam.disableServiceAccountKeyCreation" in str(e): + pytest.skip( + "Temporary skip: HMAC key creation is disabled by organization policy " + "on project python-docs-samples-tests. See b/493225655." + ) + raise yield hmac_key # Re-fetch the key metadata in case state has changed during the test. hmac_key = STORAGE_CLIENT.get_hmac_key_metadata( @@ -77,9 +85,16 @@ def test_list_keys(capsys, new_hmac_key): def test_create_key(capsys): - hmac_key = storage_create_hmac_key.create_key( - PROJECT_ID, SERVICE_ACCOUNT_EMAIL - ) + try: + hmac_key = storage_create_hmac_key.create_key(PROJECT_ID, SERVICE_ACCOUNT_EMAIL) + except google.api_core.exceptions.PreconditionFailed as e: + if "constraints/iam.disableServiceAccountKeyCreation" in str(e): + pytest.skip( + "Temporary skip: HMAC key creation is disabled by organization policy " + "on project python-docs-samples-tests. See b/493225655." + ) + raise + hmac_key.state = "INACTIVE" hmac_key.update() hmac_key.delete() From 979cc931adea38b5b648dd14abdb8b89f88f6fc4 Mon Sep 17 00:00:00 2001 From: Chandra Shekhar Sirimala Date: Wed, 18 Mar 2026 20:28:49 +0530 Subject: [PATCH 171/172] feat(samples): add argparse and clarify traversal support in download_many snippet (#1775) This PR adds argparse support to the download_many snippet for CLI testing, and updates the description containing traversal safety. --- .../storage_transfer_manager_download_many.py | 78 ++++++++++++++++--- 1 file changed, 67 insertions(+), 11 deletions(-) diff --git a/storage/samples/snippets/storage_transfer_manager_download_many.py b/storage/samples/snippets/storage_transfer_manager_download_many.py index 02cb9b8877c..447d0869c5b 100644 --- a/storage/samples/snippets/storage_transfer_manager_download_many.py +++ b/storage/samples/snippets/storage_transfer_manager_download_many.py @@ -12,9 +12,17 @@ # See the License for the specific language governing permissions and # limitations under the License. +# Example usage: +# python samples/snippets/storage_transfer_manager_download_many.py \ +# --bucket_name \ +# --blobs \ +# --destination_directory \ +# --blob_name_prefix + + # [START storage_transfer_manager_download_many] def download_many_blobs_with_transfer_manager( - bucket_name, blob_names, destination_directory="", workers=8 + bucket_name, blob_names, destination_directory="", blob_name_prefix="", workers=8 ): """Download blobs in a list by name, concurrently in a process pool. @@ -36,11 +44,11 @@ def download_many_blobs_with_transfer_manager( # blob_names = ["myblob", "myblob2"] # The directory on your computer to which to download all of the files. This - # string is prepended (with os.path.join()) to the name of each blob to form - # the full path. Relative paths and absolute paths are both accepted. An - # empty string means "the current working directory". Note that this - # parameter allows accepts directory traversal ("../" etc.) and is not - # intended for unsanitized end user input. + # string is prepended to the name of each blob to form the full path using + # pathlib. Relative paths and absolute paths are both accepted. An empty + # string means "the current working directory". Note that this parameter + # will NOT allow files to escape the destination_directory and will skip + # downloads that attempt directory traversal outside of it. # destination_directory = "" # The maximum number of processes to use for the operation. The performance @@ -56,15 +64,63 @@ def download_many_blobs_with_transfer_manager( bucket = storage_client.bucket(bucket_name) results = transfer_manager.download_many_to_path( - bucket, blob_names, destination_directory=destination_directory, max_workers=workers + bucket, + blob_names, + destination_directory=destination_directory, + blob_name_prefix=blob_name_prefix, + max_workers=workers, ) for name, result in zip(blob_names, results): - # The results list is either `None` or an exception for each blob in + # The results list is either `None`, an exception, or a warning for each blob in # the input list, in order. - - if isinstance(result, Exception): + if isinstance(result, UserWarning): + print("Skipped download for {} due to warning: {}".format(name, result)) + elif isinstance(result, Exception): print("Failed to download {} due to exception: {}".format(name, result)) else: - print("Downloaded {} to {}.".format(name, destination_directory + name)) + print( + "Downloaded {} inside {} directory.".format(name, destination_directory) + ) + + # [END storage_transfer_manager_download_many] + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser( + description="Download blobs in a list by name, concurrently in a process pool." + ) + parser.add_argument( + "--bucket_name", required=True, help="The name of your GCS bucket" + ) + parser.add_argument( + "--blobs", + nargs="+", + required=True, + help="The list of blob names to download", + ) + parser.add_argument( + "--destination_directory", + default="", + help="The directory on your computer to which to download all of the files", + ) + parser.add_argument( + "--blob_name_prefix", + default="", + help="A string that will be prepended to each blob_name to determine the source blob name", + ) + parser.add_argument( + "--workers", type=int, default=8, help="The maximum number of processes to use" + ) + + args = parser.parse_args() + + download_many_blobs_with_transfer_manager( + bucket_name=args.bucket_name, + blob_names=args.blobs, + destination_directory=args.destination_directory, + blob_name_prefix=args.blob_name_prefix, + workers=args.workers, + ) From 1c292dbda95d5e7f092d50e873c3b1537cc17e9d Mon Sep 17 00:00:00 2001 From: Nidhi Date: Fri, 27 Mar 2026 17:31:40 +0000 Subject: [PATCH 172/172] samples: add samples for bucket encryption enforcement config (#1772) Add Python samples demonstrating how to set, get and update bucket encryption enforcement configuration. Includes an integration test. --- *PR created automatically by Jules for task [3410657303470871774](https://jules.google.com/task/3410657303470871774) started by @nidhiii-27* --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: nidhiii-27 <224584462+nidhiii-27@users.noreply.github.com> --- storage/samples/snippets/encryption_test.py | 113 +++++++++++++++++- ...et_bucket_encryption_enforcement_config.py | 48 ++++++++ ...et_bucket_encryption_enforcement_config.py | 55 +++++++++ ...te_bucket_encryption_enforcement_config.py | 60 ++++++++++ 4 files changed, 271 insertions(+), 5 deletions(-) create mode 100644 storage/samples/snippets/storage_get_bucket_encryption_enforcement_config.py create mode 100644 storage/samples/snippets/storage_set_bucket_encryption_enforcement_config.py create mode 100644 storage/samples/snippets/storage_update_bucket_encryption_enforcement_config.py diff --git a/storage/samples/snippets/encryption_test.py b/storage/samples/snippets/encryption_test.py index 9039b1fad8d..f4d857dd88e 100644 --- a/storage/samples/snippets/encryption_test.py +++ b/storage/samples/snippets/encryption_test.py @@ -27,6 +27,10 @@ import storage_object_csek_to_cmek import storage_rotate_encryption_key import storage_upload_encrypted_file +import storage_get_bucket_encryption_enforcement_config +import storage_set_bucket_encryption_enforcement_config +import storage_update_bucket_encryption_enforcement_config +from google.cloud.storage.bucket import EncryptionEnforcementConfig BUCKET = os.environ["CLOUD_STORAGE_BUCKET"] KMS_KEY = os.environ["MAIN_CLOUD_KMS_KEY"] @@ -85,11 +89,7 @@ def test_blob(): except NotFound as e: # For the case that the rotation succeeded. print(f"Ignoring 404, detail: {e}") - blob = Blob( - blob_name, - bucket, - encryption_key=TEST_ENCRYPTION_KEY_2_DECODED - ) + blob = Blob(blob_name, bucket, encryption_key=TEST_ENCRYPTION_KEY_2_DECODED) blob.delete() @@ -126,3 +126,106 @@ def test_object_csek_to_cmek(test_blob): ) assert cmek_blob.download_as_bytes(), test_blob_content + + +@pytest.fixture +def enforcement_bucket(): + bucket_name = f"test_encryption_enforcement_{uuid.uuid4().hex}" + yield bucket_name + + storage_client = storage.Client() + try: + bucket = storage_client.get_bucket(bucket_name) + bucket.delete(force=True) + except Exception: + pass + + +def create_enforcement_bucket(bucket_name): + """Sets up a bucket with GMEK AND CSEK Restricted""" + client = storage.Client() + bucket = client.bucket(bucket_name) + + bucket.encryption.google_managed_encryption_enforcement_config = ( + EncryptionEnforcementConfig(restriction_mode="FullyRestricted") + ) + bucket.encryption.customer_managed_encryption_enforcement_config = ( + EncryptionEnforcementConfig(restriction_mode="NotRestricted") + ) + bucket.encryption.customer_supplied_encryption_enforcement_config = ( + EncryptionEnforcementConfig(restriction_mode="FullyRestricted") + ) + + bucket.create() + return bucket + + +def test_set_bucket_encryption_enforcement_config(enforcement_bucket): + storage_set_bucket_encryption_enforcement_config.set_bucket_encryption_enforcement_config( + enforcement_bucket + ) + + storage_client = storage.Client() + bucket = storage_client.get_bucket(enforcement_bucket) + + assert ( + bucket.encryption.google_managed_encryption_enforcement_config.restriction_mode + == "FullyRestricted" + ) + assert ( + bucket.encryption.customer_managed_encryption_enforcement_config.restriction_mode + == "NotRestricted" + ) + assert ( + bucket.encryption.customer_supplied_encryption_enforcement_config.restriction_mode + == "FullyRestricted" + ) + + +def test_get_bucket_encryption_enforcement_config(enforcement_bucket, capsys): + # Pre-setup: Creating a bucket + create_enforcement_bucket(enforcement_bucket) + + storage_get_bucket_encryption_enforcement_config.get_bucket_encryption_enforcement_config( + enforcement_bucket + ) + + out, _ = capsys.readouterr() + assert f"Encryption Enforcement Config for bucket {enforcement_bucket}" in out + assert ( + "Customer-managed encryption enforcement config restriction mode: NotRestricted" + in out + ) + assert ( + "Customer-supplied encryption enforcement config restriction mode: FullyRestricted" + in out + ) + assert ( + "Google-managed encryption enforcement config restriction mode: FullyRestricted" + in out + ) + + +def test_update_encryption_enforcement_config(enforcement_bucket): + # Pre-setup: Create a bucket in a different state before update + create_enforcement_bucket(enforcement_bucket) + + storage_update_bucket_encryption_enforcement_config.update_bucket_encryption_enforcement_config( + enforcement_bucket + ) + + storage_client = storage.Client() + bucket = storage_client.get_bucket(enforcement_bucket) + + assert ( + bucket.encryption.google_managed_encryption_enforcement_config.restriction_mode + == "NotRestricted" + ) + assert ( + bucket.encryption.customer_managed_encryption_enforcement_config.restriction_mode + == "FullyRestricted" + ) + assert ( + bucket.encryption.customer_supplied_encryption_enforcement_config.restriction_mode + == "FullyRestricted" + ) diff --git a/storage/samples/snippets/storage_get_bucket_encryption_enforcement_config.py b/storage/samples/snippets/storage_get_bucket_encryption_enforcement_config.py new file mode 100644 index 00000000000..033dcc8224c --- /dev/null +++ b/storage/samples/snippets/storage_get_bucket_encryption_enforcement_config.py @@ -0,0 +1,48 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_get_bucket_encryption_enforcement_config] +from google.cloud import storage + + +def get_bucket_encryption_enforcement_config(bucket_name): + """Gets the bucket encryption enforcement configuration.""" + # The ID of your GCS bucket + # bucket_name = "your-unique-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + print(f"Encryption Enforcement Config for bucket {bucket.name}:") + + cmek_config = bucket.encryption.customer_managed_encryption_enforcement_config + csek_config = bucket.encryption.customer_supplied_encryption_enforcement_config + gmek_config = bucket.encryption.google_managed_encryption_enforcement_config + + print( + f"Customer-managed encryption enforcement config restriction mode: {cmek_config.restriction_mode if cmek_config else None}" + ) + print( + f"Customer-supplied encryption enforcement config restriction mode: {csek_config.restriction_mode if csek_config else None}" + ) + print( + f"Google-managed encryption enforcement config restriction mode: {gmek_config.restriction_mode if gmek_config else None}" + ) + + +# [END storage_get_bucket_encryption_enforcement_config] + + +if __name__ == "__main__": + get_bucket_encryption_enforcement_config(bucket_name="your-unique-bucket-name") diff --git a/storage/samples/snippets/storage_set_bucket_encryption_enforcement_config.py b/storage/samples/snippets/storage_set_bucket_encryption_enforcement_config.py new file mode 100644 index 00000000000..107564e7f6c --- /dev/null +++ b/storage/samples/snippets/storage_set_bucket_encryption_enforcement_config.py @@ -0,0 +1,55 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_set_bucket_encryption_enforcement_config] +from google.cloud import storage +from google.cloud.storage.bucket import EncryptionEnforcementConfig + + +def set_bucket_encryption_enforcement_config(bucket_name): + """Creates a bucket with encryption enforcement configuration.""" + # The ID of your GCS bucket + # bucket_name = "your-unique-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.bucket(bucket_name) + + # Setting restriction_mode to "FullyRestricted" for Google-managed encryption (GMEK) + # means objects cannot be created using the default Google-managed keys. + bucket.encryption.google_managed_encryption_enforcement_config = ( + EncryptionEnforcementConfig(restriction_mode="FullyRestricted") + ) + + # Setting restriction_mode to "NotRestricted" for Customer-managed encryption (CMEK) + # ensures that objects ARE permitted to be created using Cloud KMS keys. + bucket.encryption.customer_managed_encryption_enforcement_config = ( + EncryptionEnforcementConfig(restriction_mode="NotRestricted") + ) + + # Setting restriction_mode to "FullyRestricted" for Customer-supplied encryption (CSEK) + # prevents objects from being created using raw, client-side provided keys. + bucket.encryption.customer_supplied_encryption_enforcement_config = ( + EncryptionEnforcementConfig(restriction_mode="FullyRestricted") + ) + + bucket.create() + + print(f"Created bucket {bucket.name} with Encryption Enforcement Config.") + + +# [END storage_set_bucket_encryption_enforcement_config] + + +if __name__ == "__main__": + set_bucket_encryption_enforcement_config(bucket_name="your-unique-bucket-name") diff --git a/storage/samples/snippets/storage_update_bucket_encryption_enforcement_config.py b/storage/samples/snippets/storage_update_bucket_encryption_enforcement_config.py new file mode 100644 index 00000000000..9b704bc0b8d --- /dev/null +++ b/storage/samples/snippets/storage_update_bucket_encryption_enforcement_config.py @@ -0,0 +1,60 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# [START storage_update_bucket_encryption_enforcement_config] +from google.cloud import storage +from google.cloud.storage.bucket import EncryptionEnforcementConfig + + +def update_bucket_encryption_enforcement_config(bucket_name): + """Updates the encryption enforcement policy for a bucket.""" + # The ID of your GCS bucket with GMEK and CSEK restricted + # bucket_name = "your-unique-bucket-name" + + storage_client = storage.Client() + bucket = storage_client.get_bucket(bucket_name) + + # Update a specific type (e.g., change GMEK to NotRestricted) + bucket.encryption.google_managed_encryption_enforcement_config = ( + EncryptionEnforcementConfig(restriction_mode="NotRestricted") + ) + + # Update another type (e.g., change CMEK to FullyRestricted) + bucket.encryption.customer_managed_encryption_enforcement_config = ( + EncryptionEnforcementConfig(restriction_mode="FullyRestricted") + ) + + # Keeping CSEK unchanged + bucket.encryption.customer_supplied_encryption_enforcement_config = ( + EncryptionEnforcementConfig(restriction_mode="FullyRestricted") + ) + + bucket.patch() + + print(f"Encryption enforcement policy updated for bucket {bucket.name}.") + + gmek = bucket.encryption.google_managed_encryption_enforcement_config + cmek = bucket.encryption.customer_managed_encryption_enforcement_config + csek = bucket.encryption.customer_supplied_encryption_enforcement_config + + print(f"GMEK restriction mode: {gmek.restriction_mode if gmek else 'None'}") + print(f"CMEK restriction mode: {cmek.restriction_mode if cmek else 'None'}") + print(f"CSEK restriction mode: {csek.restriction_mode if csek else 'None'}") + + +# [END storage_update_bucket_encryption_enforcement_config] + + +if __name__ == "__main__": + update_bucket_encryption_enforcement_config(bucket_name="your-unique-bucket-name")