Export solution for source tags and comments?

[ajstewart]

I realised that currently if a group of sources were tagged and then the user wanted to extract all these sources to analyse, for example in a vast-tools environment, there's currently no easy way to obtain all these tagged sources.

The DataTables are paginated hence the CSV button won't work for large numbers of sources.

I wondered if the comments and tags should also be exported to the parquet format for the entire database? That could be one potential solution, but relies on always being exported when new tags and comments are made.

Maybe you have some good ideas @marxide?

[marxide]

I wondered if the comments and tags should also be exported to the parquet format for the entire database? That could be one potential solution, but relies on always being exported when new tags and comments are made.

Yeah, I think we'd often end up with stale data with this solution.

Perhaps an API endpoint that will return the source IDs with their tags and comments? Then users can query this endpoint in a notebook and join the result with their source lists. We could implement filtering on the endpoint to include/exclude tags just like the query form UI.

Ideally, once implemented on the pipeline side, we'd add some convenience functions to vast-tools to make this easier for the user.

[ajstewart]

I think you're right and it has to be some sort of API endpoint to be able to obtain the tags given the dynamic nature of tags.

Perhaps just storing an API access key as an env variable in tools would suffice with a wrapper function in the pipeline class for users to make it easy to download and perhaps convert to a dataframe. A slight question of if there will be multiple different pipeline instances to support, perhaps there can be some sort of global key generated that will allow access to any pipeline instance endpoint?

Maybe even a button on the pipeline interface somewhere using the same mechanism to manually download all required tags? Especially to maintain the 'portability' nature of the pipeline results with some users perhaps doing analysis locally.