gsoc/embedding-ingestion

[i-am-leslie]

Describe changes proposed in this pull request:

• Introduced Python validation layer for embedding data types, including EmbeddingDefinitionValidator and EmbeddingValidator in validateData.py
• Implemented Java ingestion layer for embedding definitions, including EmbeddingDefinition model, DaoEmbeddingDefinition, and ImportEmbeddingDefinition
• Implemented Java ingestion layer for embedding data, including EmbeddingData model, DaoEmbeddingData, and ImportEmbeddingData using ClickHouseBulkLoader for high-volume coordinate ingestion
• Registered EMBEDDING and EMBEDDING_DEFINITION meta types in IMPORTER_CLASSNAME_BY_META_TYPE and IMPORTER_REQUIRES_METADATA (TODO: negative tests and edge cases to be added in a follow-up)
• Added tests for ImportEmbeddingDefinition and ImportEmbeddingData scripts
• Added unit tests for EmbeddingDefinitionValidator and EmbeddingValidator covering the happy path (TODO: negative tests and edge cases to be added in a follow-up)

Checks

• The commit log is comprehensible. It follows 7 rules of great commit messages. We can fix this during merge by using a squash+merge if necessary
• Has tests or has a separate issue that describes the types of test that should be created. If no test is included it should explicitly be mentioned in the PR why there is no test.
• Is this PR adding logic based on one or more clinical attributes?
• Make sure your PR has one of the labels defined in https://github.com/cBioPortal/cbioportal/blob/master/.github/release-drafter.yml

Notify reviewers

Read our Pull request merging
policy. It can help to figure out who worked on the
file before you. Please use git blame <filename> to determine that
and notify them either through slack or by assigning them as a reviewer on the PR

Note future plan

• Rename ambiguous embeddingId fields for clarity on both new models for EmbeddingDefinition and EmbeddingData. Needs to be renamed for clarity
• Write more tests for the importer, which includes the Java layer(import) and Python layer(validation)
• Edit the validation requirements for meta files so that definition no longer reqires cancser study id rather the embedding meta file requires cacncer study id
• Use the log to provide useful outputs throughout the importing process
• Fix a situation where data has been imported to the data embedding table it should not appear twice on the table. It should only appear once, Possible solutions using ReplacingMegeTree as the engine for the table so clickhouse sorts ut out later dont forget the orderyby