AB-357: Evaluate both MB database access methods to decide which one is more efficient by amCap1712 · Pull Request #401 · metabrainz/acousticbrainz-server

amCap1712 · 2021-06-07T14:52:22Z

AB-357

…est errors

…ema change

…y & foreign keys

… table structure

… one

…abase

…dular

…ery function

…irst letters of musicbrainz

…t and modify a message

AB-350: Update musicbrainz schema whenever actual MusicBrainz database is updated

…and recording

AB-357: Evaluate both MB database access methods to decide which one is more efficient.

pep8speaks · 2021-06-07T14:52:28Z

Hello @amCap1712! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file db/data.py:

Line 1111:49: W291 trailing whitespace
Line 1121:1: E302 expected 2 blank lines, found 1
Line 1131:65: W291 trailing whitespace
Line 1132:27: W291 trailing whitespace
Line 1133:54: W291 trailing whitespace
Line 1141:1: E302 expected 2 blank lines, found 1
Line 1147:65: W291 trailing whitespace
Line 1148:27: W291 trailing whitespace
Line 1149:54: W291 trailing whitespace

In the file db/import_mb_data.py:

Line 8:25: E261 at least two spaces before inline comment
Line 89:9: E124 closing bracket does not match visual indentation
Line 96:131: E501 line too long (148 > 130 characters)
Line 224:28: E222 multiple spaces after operator
Line 258:26: E222 multiple spaces after operator
Line 1196:13: E203 whitespace before ':'
Line 1197:15: E203 whitespace before ':'
Line 1198:23: E203 whitespace before ':'
Line 1199:20: E203 whitespace before ':'
Line 1200:18: E203 whitespace before ':'
Line 1221:15: E203 whitespace before ':'
Line 1222:17: E203 whitespace before ':'
Line 1223:22: E203 whitespace before ':'
Line 1224:22: E203 whitespace before ':'
Line 1225:14: E203 whitespace before ':'
Line 1319:11: E225 missing whitespace around operator
Line 1554:1: E302 expected 2 blank lines, found 1
Line 1718:9: E128 continuation line under-indented for visual indent
Line 1719:9: E128 continuation line under-indented for visual indent
Line 1720:5: E124 closing bracket does not match visual indentation
Line 1983:131: E501 line too long (139 > 130 characters)
Line 2068:131: E501 line too long (149 > 130 characters)
Line 2075:131: E501 line too long (179 > 130 characters)
Line 2218:69: E222 multiple spaces after operator

In the file db/test/test_data.py:

Line 793:5: E303 too many blank lines (2)

In the file db/test/test_import_mb_data.py:

Line 19:5: E303 too many blank lines (2)
Line 23:131: E501 line too long (191 > 130 characters)
Line 24:13: E128 continuation line under-indented for visual indent
Line 24:131: E501 line too long (160 > 130 characters)
Line 33:13: E128 continuation line under-indented for visual indent
Line 33:131: E501 line too long (172 > 130 characters)
Line 34:13: E128 continuation line under-indented for visual indent
Line 35:13: E128 continuation line under-indented for visual indent
Line 35:131: E501 line too long (172 > 130 characters)
Line 37:13: E128 continuation line under-indented for visual indent
Line 37:131: E501 line too long (172 > 130 characters)
Line 45:5: E303 too many blank lines (2)
Line 51:131: E501 line too long (191 > 130 characters)
Line 52:13: E128 continuation line under-indented for visual indent
Line 52:131: E501 line too long (160 > 130 characters)
Line 61:13: E128 continuation line under-indented for visual indent
Line 61:131: E501 line too long (172 > 130 characters)
Line 62:13: E128 continuation line under-indented for visual indent
Line 63:13: E128 continuation line under-indented for visual indent
Line 63:131: E501 line too long (172 > 130 characters)
Line 65:13: E128 continuation line under-indented for visual indent
Line 65:131: E501 line too long (172 > 130 characters)
Line 74:13: E128 continuation line under-indented for visual indent
Line 83:13: E128 continuation line under-indented for visual indent
Line 91:131: E501 line too long (145 > 130 characters)
Line 92:13: E128 continuation line under-indented for visual indent
Line 92:131: E501 line too long (141 > 130 characters)
Line 93:9: E124 closing bracket does not match visual indentation
Line 100:5: E303 too many blank lines (2)
Line 106:131: E501 line too long (191 > 130 characters)
Line 107:13: E128 continuation line under-indented for visual indent
Line 107:131: E501 line too long (160 > 130 characters)
Line 116:13: E128 continuation line under-indented for visual indent
Line 116:131: E501 line too long (172 > 130 characters)
Line 117:13: E128 continuation line under-indented for visual indent
Line 118:13: E128 continuation line under-indented for visual indent
Line 118:131: E501 line too long (172 > 130 characters)
Line 120:13: E128 continuation line under-indented for visual indent
Line 120:131: E501 line too long (172 > 130 characters)
Line 129:13: E128 continuation line under-indented for visual indent
Line 138:13: E128 continuation line under-indented for visual indent
Line 146:131: E501 line too long (145 > 130 characters)
Line 147:13: E128 continuation line under-indented for visual indent
Line 147:131: E501 line too long (141 > 130 characters)
Line 148:9: E124 closing bracket does not match visual indentation
Line 155:131: E501 line too long (186 > 130 characters)
Line 162:5: E303 too many blank lines (2)
Line 166:131: E501 line too long (166 > 130 characters)
Line 167:13: E128 continuation line under-indented for visual indent
Line 167:131: E501 line too long (160 > 130 characters)
Line 168:131: E501 line too long (170 > 130 characters)
Line 169:131: E501 line too long (168 > 130 characters)
Line 170:131: E501 line too long (152 > 130 characters)
Line 171:131: E501 line too long (154 > 130 characters)
Line 179:5: E303 too many blank lines (2)
Line 185:131: E501 line too long (166 > 130 characters)
Line 186:13: E128 continuation line under-indented for visual indent
Line 186:131: E501 line too long (160 > 130 characters)
Line 187:131: E501 line too long (170 > 130 characters)
Line 188:131: E501 line too long (168 > 130 characters)
Line 189:131: E501 line too long (152 > 130 characters)
Line 190:131: E501 line too long (154 > 130 characters)
Line 198:131: E501 line too long (191 > 130 characters)
Line 199:13: E128 continuation line under-indented for visual indent
Line 199:131: E501 line too long (160 > 130 characters)
Line 208:13: E128 continuation line under-indented for visual indent
Line 208:131: E501 line too long (172 > 130 characters)
Line 209:13: E128 continuation line under-indented for visual indent
Line 210:13: E128 continuation line under-indented for visual indent
Line 210:131: E501 line too long (172 > 130 characters)
Line 212:13: E128 continuation line under-indented for visual indent
Line 212:131: E501 line too long (172 > 130 characters)
Line 221:13: E128 continuation line under-indented for visual indent
Line 230:13: E128 continuation line under-indented for visual indent
Line 238:131: E501 line too long (145 > 130 characters)
Line 239:13: E128 continuation line under-indented for visual indent
Line 239:131: E501 line too long (141 > 130 characters)
Line 240:9: E124 closing bracket does not match visual indentation
Line 254:5: E303 too many blank lines (2)
Line 258:131: E501 line too long (166 > 130 characters)
Line 259:13: E128 continuation line under-indented for visual indent
Line 259:131: E501 line too long (160 > 130 characters)
Line 260:131: E501 line too long (170 > 130 characters)
Line 261:131: E501 line too long (168 > 130 characters)
Line 262:131: E501 line too long (152 > 130 characters)
Line 263:131: E501 line too long (154 > 130 characters)
Line 271:131: E501 line too long (173 > 130 characters)
Line 272:13: E128 continuation line under-indented for visual indent
Line 280:5: E303 too many blank lines (2)
Line 286:131: E501 line too long (166 > 130 characters)
Line 287:13: E128 continuation line under-indented for visual indent
Line 287:131: E501 line too long (160 > 130 characters)
Line 288:131: E501 line too long (170 > 130 characters)
Line 289:131: E501 line too long (168 > 130 characters)
Line 290:131: E501 line too long (152 > 130 characters)
Line 291:131: E501 line too long (154 > 130 characters)
Line 299:131: E501 line too long (173 > 130 characters)
Line 300:13: E128 continuation line under-indented for visual indent
Line 308:131: E501 line too long (189 > 130 characters)
Line 315:5: E303 too many blank lines (2)
Line 321:131: E501 line too long (166 > 130 characters)
Line 322:13: E128 continuation line under-indented for visual indent
Line 322:131: E501 line too long (160 > 130 characters)
Line 323:131: E501 line too long (170 > 130 characters)
Line 324:131: E501 line too long (168 > 130 characters)
Line 325:131: E501 line too long (152 > 130 characters)
Line 326:131: E501 line too long (154 > 130 characters)
Line 342:13: E128 continuation line under-indented for visual indent
Line 343:13: E128 continuation line under-indented for visual indent
Line 343:131: E501 line too long (142 > 130 characters)
Line 344:13: E128 continuation line under-indented for visual indent
Line 352:5: E303 too many blank lines (2)
Line 358:131: E501 line too long (166 > 130 characters)
Line 359:13: E128 continuation line under-indented for visual indent
Line 359:131: E501 line too long (160 > 130 characters)
Line 360:131: E501 line too long (170 > 130 characters)
Line 361:131: E501 line too long (168 > 130 characters)
Line 362:131: E501 line too long (152 > 130 characters)
Line 363:131: E501 line too long (154 > 130 characters)
Line 379:13: E128 continuation line under-indented for visual indent
Line 380:13: E128 continuation line under-indented for visual indent
Line 380:131: E501 line too long (142 > 130 characters)
Line 381:13: E128 continuation line under-indented for visual indent
Line 389:131: E501 line too long (188 > 130 characters)
Line 396:5: E303 too many blank lines (2)
Line 403:131: E501 line too long (166 > 130 characters)
Line 404:13: E128 continuation line under-indented for visual indent
Line 404:131: E501 line too long (160 > 130 characters)
Line 405:131: E501 line too long (170 > 130 characters)
Line 406:131: E501 line too long (168 > 130 characters)
Line 407:131: E501 line too long (152 > 130 characters)
Line 408:131: E501 line too long (154 > 130 characters)
Line 424:13: E128 continuation line under-indented for visual indent
Line 425:13: E128 continuation line under-indented for visual indent
Line 425:131: E501 line too long (142 > 130 characters)
Line 426:13: E128 continuation line under-indented for visual indent
Line 441:131: E501 line too long (165 > 130 characters)
Line 442:13: E128 continuation line under-indented for visual indent
Line 450:13: E128 continuation line under-indented for visual indent
Line 451:9: E124 closing bracket does not match visual indentation
Line 465:131: E501 line too long (159 > 130 characters)
Line 466:13: E128 continuation line under-indented for visual indent
Line 467:9: E124 closing bracket does not match visual indentation
Line 475:13: E128 continuation line under-indented for visual indent
Line 483:5: E303 too many blank lines (2)
Line 490:131: E501 line too long (166 > 130 characters)
Line 491:13: E128 continuation line under-indented for visual indent
Line 491:131: E501 line too long (160 > 130 characters)
Line 492:131: E501 line too long (170 > 130 characters)
Line 493:131: E501 line too long (168 > 130 characters)
Line 494:131: E501 line too long (152 > 130 characters)
Line 495:131: E501 line too long (154 > 130 characters)
Line 511:13: E128 continuation line under-indented for visual indent
Line 512:13: E128 continuation line under-indented for visual indent
Line 512:131: E501 line too long (142 > 130 characters)
Line 513:13: E128 continuation line under-indented for visual indent
Line 528:131: E501 line too long (165 > 130 characters)
Line 529:13: E128 continuation line under-indented for visual indent
Line 537:13: E128 continuation line under-indented for visual indent
Line 538:9: E124 closing bracket does not match visual indentation
Line 552:131: E501 line too long (159 > 130 characters)
Line 553:13: E128 continuation line under-indented for visual indent
Line 554:9: E124 closing bracket does not match visual indentation
Line 561:5: E303 too many blank lines (2)
Line 575:131: E501 line too long (166 > 130 characters)
Line 576:13: E128 continuation line under-indented for visual indent
Line 576:131: E501 line too long (160 > 130 characters)
Line 577:131: E501 line too long (170 > 130 characters)
Line 578:131: E501 line too long (168 > 130 characters)
Line 579:131: E501 line too long (152 > 130 characters)
Line 580:131: E501 line too long (154 > 130 characters)
Line 596:13: E128 continuation line under-indented for visual indent
Line 597:13: E128 continuation line under-indented for visual indent
Line 597:131: E501 line too long (142 > 130 characters)
Line 598:13: E128 continuation line under-indented for visual indent
Line 613:131: E501 line too long (165 > 130 characters)
Line 614:13: E128 continuation line under-indented for visual indent
Line 622:13: E128 continuation line under-indented for visual indent
Line 623:9: E124 closing bracket does not match visual indentation
Line 637:131: E501 line too long (159 > 130 characters)
Line 638:13: E128 continuation line under-indented for visual indent
Line 639:9: E124 closing bracket does not match visual indentation
Line 646:131: E501 line too long (180 > 130 characters)
Line 653:5: E303 too many blank lines (2)
Line 660:131: E501 line too long (166 > 130 characters)
Line 661:13: E128 continuation line under-indented for visual indent
Line 661:131: E501 line too long (160 > 130 characters)
Line 662:131: E501 line too long (170 > 130 characters)
Line 663:131: E501 line too long (168 > 130 characters)
Line 664:131: E501 line too long (152 > 130 characters)
Line 665:131: E501 line too long (154 > 130 characters)
Line 673:131: E501 line too long (173 > 130 characters)
Line 674:13: E128 continuation line under-indented for visual indent
Line 697:13: E128 continuation line under-indented for visual indent
Line 698:13: E128 continuation line under-indented for visual indent
Line 698:131: E501 line too long (142 > 130 characters)
Line 699:13: E128 continuation line under-indented for visual indent
Line 714:131: E501 line too long (165 > 130 characters)
Line 715:13: E128 continuation line under-indented for visual indent
Line 723:13: E128 continuation line under-indented for visual indent
Line 724:9: E124 closing bracket does not match visual indentation
Line 738:131: E501 line too long (159 > 130 characters)
Line 739:13: E128 continuation line under-indented for visual indent
Line 740:9: E124 closing bracket does not match visual indentation
Line 747:131: E501 line too long (180 > 130 characters)
Line 754:131: E501 line too long (163 > 130 characters)
Line 755:13: E128 continuation line under-indented for visual indent
Line 755:131: E501 line too long (150 > 130 characters)
Line 762:5: E303 too many blank lines (2)
Line 769:131: E501 line too long (166 > 130 characters)
Line 770:13: E128 continuation line under-indented for visual indent
Line 770:131: E501 line too long (160 > 130 characters)
Line 771:131: E501 line too long (170 > 130 characters)
Line 772:131: E501 line too long (168 > 130 characters)
Line 773:131: E501 line too long (152 > 130 characters)
Line 774:131: E501 line too long (154 > 130 characters)
Line 782:131: E501 line too long (173 > 130 characters)
Line 783:13: E128 continuation line under-indented for visual indent
Line 806:13: E128 continuation line under-indented for visual indent
Line 807:13: E128 continuation line under-indented for visual indent
Line 807:131: E501 line too long (142 > 130 characters)
Line 808:13: E128 continuation line under-indented for visual indent
Line 823:131: E501 line too long (165 > 130 characters)
Line 824:13: E128 continuation line under-indented for visual indent
Line 832:13: E128 continuation line under-indented for visual indent
Line 833:9: E124 closing bracket does not match visual indentation
Line 847:131: E501 line too long (159 > 130 characters)
Line 848:13: E128 continuation line under-indented for visual indent
Line 849:9: E124 closing bracket does not match visual indentation
Line 856:131: E501 line too long (180 > 130 characters)
Line 863:131: E501 line too long (163 > 130 characters)
Line 864:13: E128 continuation line under-indented for visual indent
Line 864:131: E501 line too long (150 > 130 characters)
Line 871:131: E501 line too long (189 > 130 characters)

In the file db/testing.py:

Line 132:13: E124 closing bracket does not match visual indentation

In the file manage.py:

Line 347:1: E302 expected 2 blank lines, found 1

In the file musicbrainz_importer/apply_replication_changes.py:

Line 41:131: E501 line too long (133 > 130 characters)
Line 41:133: E502 the backslash is redundant between brackets
Line 42:131: E501 line too long (139 > 130 characters)
Line 42:139: E502 the backslash is redundant between brackets
Line 43:131: E501 line too long (136 > 130 characters)
Line 43:136: E502 the backslash is redundant between brackets
Line 49:1: E302 expected 2 blank lines, found 1
Line 230:22: E211 whitespace before '('
Line 255:131: E501 line too long (145 > 130 characters)
Line 263:42: E211 whitespace before '('
Line 269:38: E211 whitespace before '('
Line 271:30: E211 whitespace before '('
Line 286:10: E211 whitespace before '('
Line 293:131: E501 line too long (139 > 130 characters)
Line 296:18: E211 whitespace before '('
Line 319:10: E211 whitespace before '('
Line 356:14: E211 whitespace before '('
Line 359:18: E211 whitespace before '('
Line 364:14: E211 whitespace before '('

In the file musicbrainz_importer/musicbrainz_importer.py:

Line 8:25: E261 at least two spaces before inline comment

In the file webserver/external/get_entities.py:

Line 50:81: E222 multiple spaces after operator
Line 58:99: E222 multiple spaces after operator

In the file worker_manage.py:

Line 29:1: E101 indentation contains mixed spaces and tabs
Line 29:1: W191 indentation contains tabs
Line 30:1: W191 indentation contains tabs

Comment last updated at 2021-06-16 06:51:35 UTC

…tion-gsoc

rsh7 added 30 commits August 5, 2018 16:24

AB-334: Setup to directly access Musicbrainz Database

e6c1eab

Database name as in brainzutils and update requirements.txt

cf9bb35

AB-334: Update documentation for setting up MusicBrainz database

42e3802

Convert documentation to markdown syntax

4b3b3d5

Add recent BrainzUtils version

3a3e9c6

AB-338: Create MusicBrainz schema in AB and add tables

6c0bfb9

AB:338: Add sql commands for MB tables in db testing script and fix t…

d84c898

…est errors

AB-338: Add primary and foreign keys for MusicBrainz tables

3a979cd

Create indexes for columns of MB tables and add update script for sch…

67ff744

…ema change

Add a condition to create schema command

fe1186a

Add separate sql files for musicbrainz schema, table, indexes, primar…

ec7e8ca

…y & foreign keys

Add a function to initiate MB db and a command to initialize & create…

ae72050

… table structure

Correct the create musicbrainz foreign keys file

dd80e11

Add more indexes to musicbrainz indexes file and add an update file

98ef544

Update the musicbrainz schema update sql file and delete the separate…

8d9b43a

… one

AB-340: Import MusicBrainz data into separate schema tables in AB dat…

dc1c999

…abase

Add commands to start the import in readme

41e0937

Get data in batches, 10000 recordings data at a time and make code mo…

131b4c4

…dular

Get data for FK referenced tables, write docstrings & comments for ev…

c0931ca

…ery function

Remove repeated MB db initialization command

5b38820

Write docstrings in imperative mood

5f4b1b3

AB-344: Add a force recreate musicbrainz schema option

4853ebc

Add a line to print a message of dropping the schema and capitalize f…

038f204

…irst letters of musicbrainz

AB-343: Add number of recording per batch in config.py

ddaa587

AB-342: Improve logging during MusicBrainz data import

1552006

AB-346: Import MusicBrainz data for every new recording added to AB db

ac856e1

Include table name in - No Data Found - message

4154ff1

Add new musicbrainz_importer module with a musicbrainz_importer scrip…

5bba175

…t and modify a message

Import recordings in batches and pick the limit from config file

39dd8c9

AB-345: Log the amount of time taken to import MB data in AB

e0c644f

rsh7 and others added 16 commits August 12, 2018 19:43

Remove a useless print statement

9f190aa

Move the queries to the db module

4b615bc

Move update and write replication sequence queries to db module

0a43c08

Close the tar file

0566d77

Passing the variables in query and their values in a dictionary

acae8b7

Add docstrings to get sequence functions in data.py

302ab50

Add proper formatting in query functions

8a07943

Merge pull request #282 from rsh7/apply_replication

0253dfa

AB-350: Update musicbrainz schema whenever actual MusicBrainz database is updated

Evaluate both db access methods by simply getting data from lowlevel …

13ada58

…and recording

Add a limit to the number of recordings fetched

8884214

Add logging instead of print and change file name in manage.py

e58f15a

Move queries inside the db module

5adf1a5

Add docstrings for the functions moved to db module

bf6873a

Evaluate a functionality of mbid redirects for both db access methods

f7fa076

Add comments stating the explainations of some lines of code

6c42c4a

Merge pull request #292 from rsh7/evaluation

a815672

AB-357: Evaluate both MB database access methods to decide which one is more efficient.

amCap1712 added 8 commits June 9, 2021 14:07

Merge remote-tracking branch 'origin/master' into musicbrainz-integra…

0f5b4c4

…tion-gsoc

Fix requirements.txt after merge

f64afa3

Fix command name

8f78f60

Add missing keyword argument to command

a3ab871

Fix write_language query

dd3c834

Add :id to language insert query

92f3ab5

Drop constraints before importing data

97f24c8

Create PKs always

f9c80c9

amCap1712 force-pushed the musicbrainz-integration-gsoc branch from 4365129 to f9c80c9 Compare June 9, 2021 15:04

amCap1712 added 4 commits June 10, 2021 17:08

Separate access methods to evaluate separately

8624df1

Add method to fetch data directly from AB as well

afcf826

Add method to evaluate fetch using exists

de998f5

Add commands to test with a dataset

d5f3250

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

AB-357: Evaluate both MB database access methods to decide which one is more efficient#401

AB-357: Evaluate both MB database access methods to decide which one is more efficient#401
amCap1712 wants to merge 129 commits intomasterfrom
musicbrainz-integration-gsoc

amCap1712 commented Jun 7, 2021

Uh oh!

pep8speaks commented Jun 7, 2021 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

amCap1712 commented Jun 7, 2021

Uh oh!

pep8speaks commented Jun 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated at 2021-06-16 06:51:35 UTC

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pep8speaks commented Jun 7, 2021 •

edited

Loading