sample migration for google-cloud-bigtable #14904
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* chore(deps): update all dependencies to v2.11.1 * revert Co-authored-by: Anthonios Partheniou <[email protected]>
fix(deps): require proto-plus >= 1.22.0
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* chore: move row value classes out of row_data This is in preparation for extracting row merging into a separate class. See googleapis/python-bigtable#628 Co-authored-by: Anthonios Partheniou <[email protected]>
This PR rewrites the row merging logic to be more correct and improve performance:
- extract row merging logic into its own class to simplify complexity of ReadRows handling
- Use OrderedDict instead of dict() for `{family: { qualifier: [] }}` data, this should maintain serverside ordering (family in creation order and qualifier in lexiographical).
- define an explicit state machine with states implemented as methods
- add various optimizations like:
- __slots__ on hot objects to avoid dict lookups
- avoiding dict lookups for contiguous family and qualifier keys
Overall this improves performance by 20% and in my opinion is a lot more readable
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* chore(deps): update dependency apache-beam to v2.41.0 * fix flaky test * fix flaky test Co-authored-by: Anthonios Partheniou <[email protected]>
…[autoapprove] (#655) Source-Link: googleapis/synthtool@4826337 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:60a63eddf86c87395b4bb394fdddfe30f84a7726ee8fe0b758ea132c2106ac75 Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
* ci(python): fix path to requirements.txt in release script Source-Link: googleapis/synthtool@fdba3ed Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:1f0dbd02745fb7cf255563dab5968345989308544e52b7f460deadd5e78e63b0 * Increase the timeouts. * increase timeout * increase timeout * increase timeouts * increase timeouts * increase timeouts * increase timeouts * increase timeouts * increase timeouts Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Mariatta Wijaya <[email protected]> Co-authored-by: Anthonios Partheniou <[email protected]>
* chore(deps): update all dependencies * revert * increase timeouts * increase timeouts Co-authored-by: Anthonios Partheniou <[email protected]>
* chore(python): exclude setup.py in renovate config Source-Link: googleapis/synthtool@56da63e Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:993a058718e84a82fda04c3177e58f0a43281a996c7c395e0a56ccc4d6d210d7 * increase timeouts Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <[email protected]>
- [ ] Regenerate this pull request now. PiperOrigin-RevId: 472772457 Source-Link: googleapis/googleapis@855b74d Source-Link: googleapis/googleapis-gen@b64b1e7 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYjY0YjFlN2RhM2UxMzhmMTVjYTM2MTU1MmVmMDU0NWU1NDg5MWI0ZiJ9
- [ ] Regenerate this pull request now. PiperOrigin-RevId: 473833416 Source-Link: googleapis/googleapis@565a550 Source-Link: googleapis/googleapis-gen@1ee1a06 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMWVlMWEwNmM2ZGUzY2E4Yjg0MzU3MmMxZmRlMDU0OGY4NDIzNjk4OSJ9
Source-Link: googleapis/synthtool@50db768 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:e09366bdf0fd9c8976592988390b24d53583dd9f002d476934da43725adbb978
…est, and UpdateTable API (#670) * feat: Publish CBT deletion_protection field in Table, UpdateTableRequest, and UpdateTable API in **stable** proto to external customers PiperOrigin-RevId: 474010093 Source-Link: googleapis/googleapis@e210283 Source-Link: googleapis/googleapis-gen@46d5c58 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNDZkNWM1OGI2NDdlYTljMDUwYTAwYzNiNjgyNWEzY2YzMTZmODk0OCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
* docs: Remove unnecessary comment PiperOrigin-RevId: 474807836 Source-Link: googleapis/googleapis@dfd68f7 Source-Link: googleapis/googleapis-gen@fe305e0 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZmUzMDVlMDI1MmJjOWE4ODQ2NmUxYjk2YzJmMjQ2NGRkNzI5Yjk3OCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* feat: publish the RequestStats proto PiperOrigin-RevId: 476399682 Source-Link: googleapis/googleapis@0c4e682 Source-Link: googleapis/googleapis-gen@ff87f69 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZmY4N2Y2OWFhZDFmMmZmMDY3NzJmOTE3ZWVjYmVhNjQ5Nzk3ZDlkNSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
* chore: exclude requirements.txt file from renovate-bot Source-Link: googleapis/synthtool@f58d313 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:7a40313731a7cb1454eef6b33d3446ebb121836738dc3ab3d2d3ded5268c35b6 * update constraints files * fix(deps): require protobuf 3.20.2 Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <[email protected]>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* fix(deps): allow protobuf 3.19.5 * explicitly exclude protobuf 4.21.0
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* feat: allow end users to override the deadline for manipulating column families As a sideeffect this also fixes a bug where the default deadlines were not applied. * limit scope of change to just use the default timeout * lint
🤖 I have created a release *beep* *boop* --- ## [2.13.2](https://togithub.com/googleapis/python-bigtable/compare/v2.13.1...v2.13.2) (2022-10-20) ### Bug Fixes * Respect deadlines for column family operations ([#687](https://togithub.com/googleapis/python-bigtable/issues/687)) ([df2e64a](https://togithub.com/googleapis/python-bigtable/commit/df2e64a79bbd8b28d0991706607af99d539320d1)) --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
* chore: Update gapic-generator-python to 1.25.0 PiperOrigin-RevId: 755914147 Source-Link: googleapis/googleapis@97a83d7 Source-Link: googleapis/googleapis-gen@a9977ef Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYTk5NzdlZmVkYzgzNmNjZWNlMWYwMWQ1MjliMDMxNWUxZWZlNTJhZCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Daniel Sanche <[email protected]>
* chore: update renovate.json Renovate bot should ignore all Github Action workflows. We want tests to use fixed python versions, instead of having them update to latest * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update owlbot.py * Update renovate.json --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <[email protected]>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
…ration (#1177) * chore: Removed old admin_v2 GAPIC layer (#1111) * feat!: Generated Selective GAPIC layer for Admin API (#1112) * chore: Updated service YAML by making all methods in BigtableInstanceAdmin public (#1113) * refactor: Refactored classic client to use new Admin API (#1114) * refactor: Refactored classic client to use new Admin API * added newline after gapic_version files * fix: Made generate_consistency_token and check_consistency public (#1116) methods * feat: Consistency polling + restore table for sync client in admin (#1117) * feat: Prototyped handwritten layer * Added newlines * linting * Added docstrings for restore table and consistency token polling; removed gc_rule * docs: owlbot related changes (#1133) * docs: owlbot related changes * Addressed PR feedback + made changes to toc.yml for docs pipeline * Fixed type hint * linting + added validation for admin section * linting + added noqas to owlbot lines * tests: Tests for sync client + fixes + client library versioning (#1132) * tests: Tests for sync client + fixes + client library versioning * Removed raise exception * linting + name changes in tests + added test for timeout * linting * Fixed tests on Python 3.7 * feat: Proto-plus modifications for enforcing strict oneofs (#1126) * feat: Proto-plus modifications for enforcing strict oneofs * Added template directory + changed unit tests to pytest * Finished README * linting * Added source of truth comment * feat: Reworked the wait_for_consistency call (#1144) * feat: Reworked the wait_for_consistency call * linting * Update google/cloud/bigtable/admin_v2/overlay/services/bigtable_table_admin/client.py Co-authored-by: Mattie Fu <[email protected]> * Improved documentation * linting again * linting --------- Co-authored-by: Mattie Fu <[email protected]> * feat: Async consistency polling harness (#1142) * feat: Async consistency polling harness * Fixed AsyncMock issue in Python 3.7 * Reworked async_consistency and added async client to __init__.py * linting * addressed review feedback * linting * feat: Restore Table LRO rework + async restore table (#1148) * chore(tests): system tests for autogen API (#1151) * tests: system tests for autogen API * Fixed async system tests * addressed review feedback * Fixed system test failure at the end of a test run * Linting * more linting * chore: Moved Admin API from google.cloud.bigtable.admin_v2 back to google.cloud.bigtable_admin_v2 (#1153) * chore: Removed autogenerated files from the feature branch (#1170) * chore: Merged selective GAPIC autogenerated changes into feature branch (#1175) * chore: Merged selective GAPIC owlbot changes into feature branch * linting * changed comment text * Removed redundant items * Fixed owlbot infinitely appending text * Added comments + fixed indentation in Owlbot * Added anonymous credentials to client tests * Fixed project ID issue in system tests * Fixed docstrings and skipped system tests on emulator. --------- Co-authored-by: Mattie Fu <[email protected]>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* test: Unskipped wait_for_consistency system tests in emulator * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Added sync surface * Actual sync surface * Used pytest.mark.skipif --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
* chore(tests): prefer bytes for mutation fields * decode instead of encode
- [ ] Regenerate this pull request now. BEGIN_COMMIT_OVERRIDE feat: add PeerInfo proto in Bigtable API fix: Add ReadRows/SampleRowKeys bindings for materialized views fix: Deprecate credentials_file argument feat: Add Type API updates needed to support structured keys in materialized views feat: Add encodings for STRUCT and the Timestamp type END_COMMIT_OVERRIDE PiperOrigin-RevId: 829585900 Source-Link: googleapis/googleapis@1b5f863 Source-Link: googleapis/googleapis-gen@1a69287 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMWE2OTI4NzUwMDNlMjc1NDcyOWRjMmE0Y2NhODgwNTUwNTFkMmFhZSJ9 BEGIN_NESTED_COMMIT fix: Add ReadRows/SampleRowKeys bindings for materialized views fix: Deprecate credentials_file argument chore: Update gapic-generator-python to 1.28.0 PiperOrigin-RevId: 816753840 Source-Link: googleapis/googleapis@d06cf27 Source-Link: googleapis/googleapis-gen@a524e73 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYTUyNGU3MzEwODgyYmJiOTliZmUxMzk5YjE4YmVkMzI4OTc5MjExYyJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT feat: Add Type API updates needed to support structured keys in materialized views feat: Add encodings for STRUCT and the Timestamp type PiperOrigin-RevId: 805031861 Source-Link: googleapis/googleapis@6d1dca2 Source-Link: googleapis/googleapis-gen@ecd9d88 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZWNkOWQ4ODYwYmFlOGJiMzdiNDUyYmZjNmVlZmJkZDIyZDAyOGYwOSJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT chore: Update gapic-generator-python to 1.26.2 PiperOrigin-RevId: 802200836 Source-Link: googleapis/googleapis@d300b15 Source-Link: googleapis/googleapis-gen@a1ff0ae Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYTFmZjBhZTcyZGRjYjY4YTI1OTIxNWQ4Yzc3NjYxZTJjZGJiOWIwMiJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT chore: update Python generator version to 1.25.1 PiperOrigin-RevId: 800535761 Source-Link: googleapis/googleapis@4cf1f99 Source-Link: googleapis/googleapis-gen@133d25b Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMTMzZDI1YjY4ZTcxMjExNmUxYzVkYzcxZmMzZWIzYzVlNzE3MDIyYSJ9 END_NESTED_COMMIT BEGIN_NESTED_COMMIT fix: Add ReadRows/SampleRowKeys bindings for materialized views PiperOrigin-RevId: 793800781 Source-Link: googleapis/googleapis@fe06a49 Source-Link: googleapis/googleapis-gen@c4550f6 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzQ1NTBmNjA3MjVkYzRjMDdhZGZlNjhjZWUwYWM3MmViNGE1YjFiYiJ9 END_NESTED_COMMIT --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <[email protected]>
… into migration.python-bigtable.migration.2025-11-22.migrate
parthea
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Towards #10992.