Migration progress: include DFSA records in the history log #3039
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…lled. Work in progress.
This field is encoded as a Spark SQL LONG, which has a (signed) range of 64-bits.
THe history will be maintained adjacent to the crawler framework.
…ion, updated to use the Historical record type.
… as default values.
asnare
had a problem deploying
to
account-admin
GitHub Actions
Failure
|
❌ 49/50 passed, 1 failed, 2 skipped, 1h7m16s total ❌ test_running_real_migration_progress_job: AssertionError: Workflow failed: migration-progress-experimental (31m40.612s)Running from acceptance #7013 |
asnare
had a problem deploying
to
account-admin
GitHub Actions
Failure
…ables and UDF refresh. Previously it was implicit: there was a transitive dependency via the grants refresh.
asnare
had a problem deploying
to
account-admin
GitHub Actions
Failure
Contributor
Author
This is currently failing due to a bug in the crawlers that means the snapshots cannot be loaded when the Spark-based runtime is being used; fixed in #3046. |
asnare
had a problem deploying
to
account-admin
GitHub Actions
Failure
asnare
had a problem deploying
to
account-admin
GitHub Actions
Failure
nfx
pushed a commit
that referenced
this pull request
Oct 23, 2024
## Changes This PR fixes an issue with the DFSA and used-table crawlers that could prevent loading of the snapshots. When loading they convert the rows to dictionaries using `.as_dict()` which isn't available on rows provided by the spark-based lsql backend. Instead `.asDict()` needs to be used. Incidental changes: - An existing integration test was updated to also test snapshot loading for these crawlers. - Another test was renamed to fix a typo in the name. ### Linked issues Relates to #3036, #3039. ### Tests - existing unit tests - existing integration tests
asnare
had a problem deploying
to
account-admin
GitHub Actions
Failure
…tances instead of attaching the timezone.
asnare
had a problem deploying
to
account-admin
GitHub Actions
Failure
asnare
had a problem deploying
to
account-admin
GitHub Actions
Failure
Contributor
Author
|
Following a discussion, we've decided not to include DFSA records in their current form in the history table. Each DFSA record corresponds to a problem with another resource (eg. notebook, jobs). As such the intent is to aggregate these records and include them in the list of failures on the resource-specific record. |
nfx
pushed a commit
that referenced
this pull request
Oct 24, 2024
…ons in addition to the normal type-based ones (#3068) ## Changes This PR cherry-picks some changes from #3039 that updated the `HistoryEncoder` to work correctly with databases that are declared with `__future__.__annotations__` in effect. When this annotation is in effect, python converts all type-hints during import/declaration into strings and then performs deferred resolution at a later stage. (This is why forward references work.) Unfortunately the dataclass mechanism captures field types prior to deferred resolution. This PR ensures that our type checking works anyway. ### Linked issues Cherry-picks from #3039. ### Tests - updated unit tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This PR follows on from #2743 by extending the set of updates that we capture to include updated
DirectFsAccesssnapshots for dashboards and jobs.Linked issues
Follows #2743.
Functionality
migration-progress-experimentalTests