AB-357: Evaluate both MB database access methods to decide which one is more efficient#401

Draft

amCap1712 wants to merge 129 commits intomasterfrom

musicbrainz-integration-gsoc

Commits on Aug 5, 2018

AB-334: Setup to directly access Musicbrainz Database

rsh7
authored and
paramsingh
committed
Database name as in brainzutils and update requirements.txt

rsh7
authored and
paramsingh
committed
AB-334: Update documentation for setting up MusicBrainz database

rsh7
authored and
paramsingh
committed
Convert documentation to markdown syntax

rsh7
authored and
paramsingh
committed
Add recent BrainzUtils version

rsh7
authored and
paramsingh
committed
AB-338: Create MusicBrainz schema in AB and add tables

rsh7
authored and
paramsingh
committed
AB:338: Add sql commands for MB tables in db testing script and fix test errors

rsh7
authored and
paramsingh
committed
AB-338: Add primary and foreign keys for MusicBrainz tables

rsh7
authored and
paramsingh
committed
Create indexes for columns of MB tables and add update script for schema change

rsh7
authored and
paramsingh
committed
Add a condition to create schema command

rsh7
authored and
paramsingh
committed
Add separate sql files for musicbrainz schema, table, indexes, primary & foreign keys

rsh7
authored and
paramsingh
committed
Add a function to initiate MB db and a command to initialize & create table structure

rsh7
authored and
paramsingh
committed
Correct the create musicbrainz foreign keys file

rsh7
authored and
paramsingh
committed
Add more indexes to musicbrainz indexes file and add an update file

rsh7
authored and
paramsingh
committed
Update the musicbrainz schema update sql file and delete the separate one

rsh7
authored and
paramsingh
committed
AB-340: Import MusicBrainz data into separate schema tables in AB database

rsh7
authored and
paramsingh
committed
Add commands to start the import in readme

rsh7
authored and
paramsingh
committed
Get data in batches, 10000 recordings data at a time and make code modular

rsh7
authored and
paramsingh
committed
Get data for FK referenced tables, write docstrings & comments for every function

rsh7
authored and
paramsingh
committed
Remove repeated MB db initialization command

rsh7
authored and
paramsingh
committed
Write docstrings in imperative mood

rsh7
authored and
paramsingh
committed
AB-344: Add a force recreate musicbrainz schema option

rsh7
authored and
paramsingh
committed
Add a line to print a message of dropping the schema and capitalize first letters of musicbrainz

rsh7
authored and
paramsingh
committed
AB-343: Add number of recording per batch in config.py

rsh7
authored and
paramsingh
committed
AB-342: Improve logging during MusicBrainz data import

rsh7
authored and
paramsingh
committed
AB-346: Import MusicBrainz data for every new recording added to AB db

rsh7
authored and
paramsingh
committed
Include table name in - No Data Found - message

rsh7
authored and
paramsingh
committed
Add new musicbrainz_importer module with a musicbrainz_importer script and modify a message

rsh7
authored and
paramsingh
committed
Import recordings in batches and pick the limit from config file

rsh7
authored and
paramsingh
committed
AB-345: Log the amount of time taken to import MB data in AB

rsh7
authored and
paramsingh
committed
Move the sql query function to db module, remove offset and remove one returning parameter

rsh7
authored and
paramsingh
committed
Change name of the mb importer script

rsh7
authored and
paramsingh
committed
Use % style format in the print message & remove .format

rsh7
authored and
paramsingh
committed
AB-347: Use logging in place of print in db/import_mb_data script

rsh7
authored and
paramsingh
committed
Replace print with logging in time status message

rsh7
authored and
paramsingh
committed
Move a function to db/data.py file

rsh7
authored and
paramsingh
committed
Add tests for getting new recordings from lowlevel

rsh7
authored and
paramsingh
committed
Get data for more foreign key dependent tables

rsh7
authored and
paramsingh
committed
Use key names instead of integer indexes & use set-comprehension for data retrieval

rsh7
authored and
paramsingh
committed
Add more documentations to specify the definition of function arguments and return values

rsh7
authored and
paramsingh
committed
Import data by including more foreign key referenced tables

rsh7
authored and
paramsingh
committed
Functions to import and write data for track gid redirect table

rsh7
authored and
paramsingh
committed
Import data in track table for every new_id in track gid redirect data

rsh7
authored and
paramsingh
committed
Reorder the import of a module

rsh7
authored and
paramsingh
committed
Add more explainations about the arguments in docstrings

rsh7
authored and
paramsingh
committed
Enable logging.info to log messages in the mb importer script

rsh7
authored and
paramsingh
committed
Write correct arguments for load_script function

rsh7
authored and
paramsingh
committed
Reduce import time by removing inner joins with recording table and remove distinct from query

rsh7
authored and
paramsingh
committed
Remove DISTINCT keyword from a track gid redirect function and modify query for gender function

rsh7
authored and
paramsingh
committed
Change order of calling functions to write the data into tables

rsh7
authored and
paramsingh
committed
Remove the redundant variable gids_in_Ab from all function's parameter

rsh7
authored and
paramsingh
committed
Add more documentation for new function's parameters and update some docs

rsh7
authored and
paramsingh
committed
Update some docstrings

rsh7
authored and
paramsingh
committed
Improve some queries whose number of rows in actual MB db is less than 15

rsh7
authored and
paramsingh
committed
Add DISTINCT in queries, otherwise there was a lot duplications in the resultant data

rsh7
authored and
paramsingh
committed
Add detailed docs to artist_credit_from_recording argument in all functions

rsh7
authored and
paramsingh
committed
Change format of select queries

rsh7
authored and
paramsingh
committed
Change docstrings to include details of fetching complete data for some tables

rsh7
authored and
paramsingh
committed
Include a sleep schedule of 5 seconds after every batch import

rsh7
authored and
paramsingh
committed
Add a sleep for every batch import in import_mb_data script as well

rsh7
authored and
paramsingh
committed
Move sleep variables to config file

rsh7
authored and
paramsingh
committed
Change the sleep info message in musicbrainz importer script

rsh7
authored and
paramsingh
committed
Change the sleep info message in import-mb_data script

rsh7
authored and
paramsingh
committed
Tests for writing data into musicbrainz schema tables

rsh7
authored and
paramsingh
committed
Add tests in modular form for area, artist & gid_redirect, artist_credit, artist_credit_name

rsh7
authored and
paramsingh
committed
Add tests to load and write recording and recording_gid_redirect tables

rsh7
authored and
paramsingh
committed
Add tests to load and write release and release_gid_redirect tables

rsh7
authored and
paramsingh
committed
Add tests to load and write release_group and release_group_gid_redirect tables

rsh7
authored and
paramsingh
committed
Add tests to load and write medium table

rsh7
authored and
paramsingh
committed
Add tests to load and write track and track_gid_redirect tables

rsh7
authored and
paramsingh
committed
Change testing order of release and release_group tables

rsh7
authored and
paramsingh
committed
Remove a big function for load and write as it is broken into modules already

rsh7
authored and
paramsingh
committed
Remove unnecessary comments and new lines

rsh7
authored and
paramsingh
committed
Add docstrings for all the test functions

rsh7
authored and
paramsingh
committed
Add release_gid_redirect fetch and write code, missed in the previous commits

rsh7
authored and
paramsingh
committed
redirect mbids to their original entities and returns recordings ids along
rsh7
committed
Add docstrings to the function definitions in get_entities.py script
rsh7
committed
Upgrade to recent BrainzUtils version and import utils.py in the script
rsh7
committed

Commits on Aug 6, 2018

Commits on Aug 7, 2018

Commits on Aug 14, 2018

Commits on Jun 7, 2021

Merge pull request #292 from rsh7/evaluation
amCap1712
authored

Commits on Jun 10, 2021

Commits on Jun 16, 2021

Add commands to test with a dataset
amCap1712
committed