Releases: Flowminder/FlowKit
Releases · Flowminder/FlowKit
1.26.0
Changed
- FlowDB now enables partitionwise aggregation planning by default
- FlowDB now uses a default fillfactor of 100 for cache table indexes
Fixed
- Queries that have multiple of the same subquery with different parameters no longer cause duplicate scopes in tokens. #6580
- FlowETL QA checks
count_imeis,count_imsis,max_msisdns_per_imeiandmax_msisdns_per_imsinow only count non-null IMEIs/IMSIs. #6619
What's Changed
- Revert "Delete .github/dependabot.yml" by @greenape in #6512
- Bump github/codeql-action from 2 to 3 by @dependabot in #6514
- Exclude flowetl from dependabot entirely by @greenape in #6553
- Bump black from 24.3.0 to 24.4.2 in /flowdb by @dependabot in #6518
- Bump black from 24.3.0 to 24.4.2 in /flowdb/testdata/synthetic_data by @dependabot in #6519
- Bump husky from 8.0.3 to 9.0.11 in /flowauth/frontend by @dependabot in #6520
- Bump black from 24.3.0 to 24.4.2 in /flowauth by @dependabot in #6530
- Bump prance from 0.21.8.0 to 23.6.21.0 in /flowapi by @dependabot in #6537
- Bump pytest from 8.1.1 to 8.2.0 in /flowmachine by @dependabot in #6539
- Bump black from 24.3.0 to 24.4.2 in /flowmachine by @dependabot in #6548
- Bump numpy from 1.26.2 to 1.26.4 in /flowdb/testdata/synthetic_data by @dependabot in #6525
- Bump tar from 6.1.12 to 6.2.1 in /flowauth/frontend by @dependabot in #6489
- Bump sqlalchemy from 1.4.44 to 2.0.29 in /flowauth by @dependabot in #6541
- Bump pytest from 8.1.1 to 8.2.0 in /flowauth by @dependabot in #6552
- Bump black from 24.3.0 to 24.4.2 in /flowapi by @dependabot in #6546
- Bump freezegun from 1.4.0 to 1.5.0 in /flowauth by @dependabot in #6545
- Bump redis from 5.0.3 to 5.0.4 in /flowauth by @dependabot in #6550
- Bump jsonschema from 4.17.3 to 4.22.0 in /flowapi by @dependabot in #6554
- Bump classnames from 2.3.2 to 2.5.1 in /flowauth/frontend by @dependabot in #6555
- Bump flask from 2.2.5 to 3.0.3 in /flowauth by @dependabot in #6549
- Bump flask-sqlalchemy from 3.0.5 to 3.1.1 in /flowauth by @dependabot in #6556
- Bump prettier from 3.1.0 to 3.2.5 in /flowauth/frontend by @dependabot in #6532
- Bump marshmallow from 3.21.1 to 3.21.2 in /flowmachine by @dependabot in #6559
- Bump pyzmq from 26.0.2 to 26.0.3 in /flowmachine by @dependabot in #6558
- Bump pyzmq from 26.0.2 to 26.0.3 in /flowapi by @dependabot in #6560
- Bump approvaltests from 11.2.1 to 12.0.0 in /flowmachine by @dependabot in #6551
- Bump jsonschema from 4.17.3 to 4.22.0 in /flowapi by @dependabot in #6561
- Bump tqdm from 4.66.2 to 4.66.4 in /flowclient by @dependabot in #6562
- Enable partitionwise aggregation in pg config by @greenape in #6511
- Bump cryptography from 42.0.5 to 42.0.6 in /flowauth by @dependabot in #6566
- Bump sqlalchemy from 2.0.29 to 2.0.30 in /flowmachine by @dependabot in #6565
- Bump cryptography from 42.0.5 to 42.0.6 in /flowapi by @dependabot in #6567
- Bump werkzeug from 3.0.2 to 3.0.3 in /flowapi by @dependabot in #6568
- Bump jinja2 from 3.1.3 to 3.1.4 in /flowdb/testdata/synthetic_data by @dependabot in #6570
- Bump sqlalchemy from 2.0.29 to 2.0.30 in /flowdb/testdata/synthetic_data by @dependabot in #6571
- Bump cryptography from 42.0.5 to 42.0.6 in /flowkit_jwt_generator by @dependabot in #6572
- Bump werkzeug from 3.0.2 to 3.0.3 in /flowauth by @dependabot in #6573
- Bump jinja2 from 3.1.3 to 3.1.4 in /flowapi by @dependabot in #6575
- Bump cryptography from 42.0.6 to 42.0.7 in /flowkit_jwt_generator by @dependabot in #6579
- Update GADM download URL by @jc-harrison in #6589
- Bump deps by @jc-harrison in #6587
- Deduped recieved scopes by @Thingus in #6581
- Bump black from 24.3.0 to 24.4.2 in /flowmachine by @dependabot in #6590
- Bump black from 24.3.0 to 24.4.2 in /flowapi by @dependabot in #6591
- Bump black from 24.3.0 to 24.4.2 in /flowdb/testdata/synthetic_data by @dependabot in #6594
- Bump black from 24.3.0 to 24.4.2 in /flowdb by @dependabot in #6595
- Bump prance from 0.21.8.0 to 23.6.21.0 in /flowapi by @dependabot in #6593
- Bump jsonschema from 4.17.3 to 4.22.0 in /flowapi by @dependabot in #6592
- Bump black from 24.3.0 to 24.4.2 in /flowauth by @dependabot in #6597
- Bump jupyter/scipy-notebook from
a7fb226to85bef7eby @dependabot in #6605 - Bump python-rapidjson from 1.16 to 1.17 in /flowmachine by @dependabot in #6611
- Bump quart from 0.19.5 to 0.19.6 in /flowapi by @dependabot in #6606
- Bump jsonschema from 4.17.3 to 4.22.0 in /flowapi by @dependabot in #6596
- Bump python-rapidjson from 1.16 to 1.17 in /flowapi by @dependabot in #6607
- Increase index fillfactor on cache tables by @greenape in #6598
- Bump requests from 2.31.0 to 2.32.2 in /flowkit_jwt_generator by @dependabot in #6620
- Bump pytest-asyncio from 0.23.6 to 0.23.7 in /flowmachine by @dependabot in #6609
- Bump requests from 2.31.0 to 2.32.0 in /flowmachine by @dependabot in #6616
- Bump requests from 2.31.0 to 2.32.0 in /flowauth by @dependabot in #6618
- Bump requests from 2.31.0 to 2.32.0 in /flowapi by @dependabot in #6617
- Bump approvaltests from 12.1.0 to 12.2.0 in /flowmachine by @dependabot in #6610
- Bump jupyter/scipy-notebook from
85bef7eto053aea4by @dependabot in #6614 - Ignore null IMEI/IMSI in QA checks by @jc-harrison in #6621
- Bump pytest from 8.2.0 to 8.2.1 in /flowkit_jwt_generator by @dependabot in #6623
- Bump watchdog from 4.0.0 to 4.0.1 in /flowmachine by @dependabot in #6624
- Bump structlog from 24.1.0 to 24.2.0 in /flowdb/testdata/synthetic_data by @dependabot in #6629
- Bump structlog from 24.1.0 to 24.2.0 in /flowapi by @dependabot in #6630
- Bump hypercorn from 0.16.0 to 0.17.2 in /flowapi by @dependabot in #6631
- Bump structlog from 24.1.0 to 24.2.0 in /flowauth by @dependabot in #6633
- Bump hypercorn from 0.17.2 to 0.17.3 in /flowapi by @dependabot in #6635
- Bump requests from 2.32.2 to 2.32.3 in /flowkit_jwt_generator by @dependabot in #6636
- Bump cryptography from 42.0.7 to 42.0.8 in /flowauth by @dependabot in #6640
- Bump cryptography from 42.0.7 to 42.0.8 in /flowkit_jwt_generator by @dependabot in #6641
- Bump pytest from 8.2.1 to 8.2.2 in /flowkit_jwt_generator by @dependabot in #6642
- Bump prettier from 3.2.5 to 3.3.1 in /flowauth/frontend by @dependabot in #6643
- Bump marshmallow from 3.21.2 to 3.21.3 in /flowmachine by @dependabot in #6644
- Bump tornado from 6.4 to 6.4.1 in /flowmachine by @dependabot in #6645
- Bump redis from 5.0.4 to 5.0.5 in /flowauth by @dependabot in #6646
- Bump redis from 5.0.4 to 5.0.5 in /flowmachine by @dependabot in https://github.com/Flowminder/Flow...
1.25.0
Fixed
- FlowETL
get_qa_checksno longer attempts to create duplicate tasks for QA checks defined in the DAG folder. #6494
Removed
- Removed
flowpyter-taskfrom the FlowETL Docker image. For a Docker image withflowpyter-taskincluded, see flowminder/flowbot.
1.24.0
Added
- Test and synthetic data generators now perform QA checks on the generated data. #6467
- Added new
/qaendpoint to FlowAPI and FlowClient, which supports getting the results of QA checks run by FlowETL #2704 - Added new
available_qa_checksproperty to flowmachineConnectionobjects #2704 - Added new
get_qa_checksmethod to flowmachineConnectionobjects #2704
Fixed
- Test QA check IDs are now of the same format as those produced by FlowETL. #6472
- FlowAuth now runs migrations correctly on startup. #6480
What's Changed
- Mock qa data by @Thingus in #6465
- No more before_first_request by @greenape in #6266
- Test qa data rename by @Thingus in #6470
- Bump black from 24.2.0 to 24.3.0 in /flowdb by @dependabot in #6473
- Bump black from 24.2.0 to 24.3.0 in /flowdb/tests by @dependabot in #6476
- Bump express from 4.17.1 to 4.19.2 in /flowauth/frontend by @dependabot in #6481
- Fix missing migrations by @greenape in #6483
- Flowapi qa checks by @Thingus in #6471
Full Changelog: 1.23.0...1.24.0
1.23.0
Changed
-
MostFrequentLocationnow breaks ties based on the last used location, instead of by arbitrary Postgres sort order. #6268 -
Users no longer have write access to the public schema in FlowDB following a change introduced in PostgreSQL 15
-
FlowDB is now built on PostgreSQL 16, debian bullseye
Warning
You may need to update your docker version to use newer releases of FlowDB. You will also need to create a fresh database and reimport data if you are upgrading from a previous FlowDB release.
What's Changed
- Break ties in MostFrequentLocation using last event by @greenape in #6466
- Postgres 16 by @greenape in #6468
- Bump follow-redirects from 1.15.4 to 1.15.6 in /flowauth/frontend by @dependabot in #6469
Full Changelog: 1.22.0...1.23.0
1.22.0
Added
- FlowETL sensor
NRowsPresentSensorwhich checks for a specified minimum number of rows.
Changed
ForeignStagingTableOperatorwill now error if the underlying file cannot be read or the command returns an error. #5763- Flowmachine now requires SQLAlchemy >= 2.0.0 #6066
What's Changed
- Bump cryptography from 42.0.0 to 42.0.2 in /flowkit_jwt_generator by @dependabot in #6450
- Bump cryptography from 42.0.0 to 42.0.2 in /deployment by @dependabot in #6453
- Fix no poke by @greenape in #6448
- Bump cryptography from 42.0.2 to 42.0.4 in /flowkit_jwt_generator by @dependabot in #6457
- Bump cryptography from 42.0.2 to 42.0.4 in /deployment by @dependabot in #6460
- Bump es5-ext from 0.10.53 to 0.10.63 in /flowauth/frontend by @dependabot in #6462
- Sqlalachemy 2 by @greenape in #6461
- Bump ip from 1.1.5 to 1.1.9 in /flowauth/frontend by @dependabot in #6456
- Bump deps by @greenape in #6463
- Unpin examples from old jupyter by @greenape in #6464
Full Changelog: 1.21.1...1.22.0
1.21.1
Changed
- Upgraded Python dependencies
What's Changed
- Bump deps by @jc-harrison in #6442
- Release 1.21.1 by @jc-harrison in #6443
- Bump cryptography from 41.0.7 to 42.0.0 in /flowkit_jwt_generator by @dependabot in #6444
- Bump cryptography from 41.0.7 to 42.0.0 in /deployment by @dependabot in #6446
Full Changelog: 1.21.0...1.21.1
1.21.0
Added
- Added new FlowDB tables
infrastructure.cell_infoandinfrastructure.cells_table_versionsto keep track of changes to the cell info over time (note: the new tables have not yet replacedinfrastructure.cellsas the source of cell information for FlowKit queries). #6184
What's Changed
- Bump tiangolo/uwsgi-nginx-flask from
0ceb6c7tof918c2eby @dependabot in #6412 - Bump deps by @jc-harrison in #6413
- Remove AutoFlow references from docs by @jc-harrison in #6410
- Bump cryptography from 41.0.5 to 41.0.6 in /flowapi by @dependabot in #6415
- Bump cryptography from 41.0.5 to 41.0.6 in /flowkit_jwt_generator by @dependabot in #6414
- Bump cryptography from 41.0.5 to 41.0.6 in /deployment by @dependabot in #6416
- Bump black from 23.7.0 to 23.11.0 in /flowkit_jwt_generator by @dependabot in #6418
- Bump cryptography from 41.0.6 to 41.0.7 in /flowkit_jwt_generator by @dependabot in #6417
- Bump sqlalchemy from 1.4.50 to 2.0.23 in /flowmachine by @dependabot in #6419
- Bump ipykernel from 6.26.0 to 6.27.1 in /flowmachine by @dependabot in #6421
- Bump pyyaml from 5.3.1 to 6.0.1 in /flowapi by @dependabot in #6423
- Bump prance from 0.21.8.0 to 23.6.21.0 in /flowapi by @dependabot in #6425
- Bump black from 23.9.1 to 23.11.0 in /flowmachine by @dependabot in #6420
- Bump jsonschema from 4.17.3 to 4.20.0 in /flowapi by @dependabot in #6424
- Bump deps by @jc-harrison in #6426
- Bump deps by @jc-harrison in #6430
- Bump deps by @jc-harrison in #6432
- Bump follow-redirects from 1.14.9 to 1.15.4 in /flowauth/frontend by @dependabot in #6434
- Bump gitpython from 3.1.40 to 3.1.41 in /flowdb/tests by @dependabot in #6436
- Bump jinja2 from 3.1.2 to 3.1.3 in /flowauth by @dependabot in #6437
- Bump jinja2 from 3.1.2 to 3.1.3 in /deployment by @dependabot in #6438
- Cell info table by @jc-harrison in #6433
Full Changelog: 1.20.0...1.21.0
1.20.0
Changed
- Updated flowpyter-task to 1.1.0
Removed
- Removed AutoFlow. #6394
What's Changed
- Bump prettier from 3.0.3 to 3.1.0 in /flowauth/frontend by @dependabot in #6400
- Bump numpy from 1.25.2 to 1.26.2 in /flowdb/testdata/synthetic_data by @dependabot in #6403
- Bump pandas from 2.1.2 to 2.1.3 in /flowdb/testdata/synthetic_data by @dependabot in #6402
- Bump cypress from 13.5.0 to 13.5.1 in /flowauth/frontend by @dependabot in #6404
- Bump deps and pin base images by @jc-harrison in #6406
- Bump cypress from 13.5.1 to 13.6.0 in /flowauth/frontend by @dependabot in #6408
- Bump flowpyter-task to 1.1.0 by @greenape in #6409
Full Changelog: 1.19.1...1.20.0
1.19.1
Added
- flowpyter-task now included in the flowetl image
What's Changed
- Bump black from 23.10.1 to 23.11.0 in /flowdb by @dependabot in #6396
- Bump cypress from 13.3.3 to 13.5.0 in /flowauth/frontend by @dependabot in #6397
- Remove AutoFlow by @jc-harrison in #6395
- Bump actions/checkout from 2 to 4 by @dependabot in #6398
- Bump python deps and Airflow by @jc-harrison in #6393
- Adding flowpytertask to flowetl requirements by @Thingus in #6286
Full Changelog: 1.19.0...1.19.1
1.19.0
Added
- FlowETL now updates a new table
events.location_idseach time a new day of CDR data is ingested, to record the first and last date that each location ID appears in the data. #5376 - New FlowETL QA check "count_locatable_events", which counts the number of added rows with location ID corresponding to a cell with a known location. #5289
- flowkit_jwt_generator is now published as a wheel via pypi
What's Changed
- Bump numpy from 1.24.2 to 1.24.3 in /flowmachine by @dependabot in #6077
- Bump pandas from 2.0.0 to 2.0.1 in /flowdb/testdata/synthetic_data by @dependabot in #6081
- Bump pandas from 2.0.0 to 2.0.1 in /flowmachine by @dependabot in #6083
- Bump pandas from 2.0.0 to 2.0.1 in /flowclient by @dependabot in #6082
- Remove quotes from default data path by @Thingus in #6079
- Bump cypress from 12.10.0 to 12.11.0 in /flowauth/frontend by @dependabot in #6085
- Bump prettier from 2.8.7 to 2.8.8 in /flowauth/frontend by @dependabot in #6074
- Bump date-fns from 2.29.3 to 2.30.0 in /flowauth/frontend by @dependabot in #6090
- Bump ipython from 8.12.0 to 8.12.1 in /flowmachine by @dependabot in #6093
- Bump pre-commit from 3.2.2 to 3.3.0 in /flowmachine by @dependabot in #6096
- Bump pre-commit from 3.3.0 to 3.3.1 in /flowmachine by @dependabot in #6097
- Bump flask from 2.2.3 to 2.2.5 in /flowauth by @dependabot in #6098
- Bump ipython from 8.12.1 to 8.12.2 in /flowmachine by @dependabot in #6100
- Bump geopandas from 0.12.2 to 0.13.0 in /flowmachine by @dependabot in #6101
- Bump redis from 4.5.4 to 4.5.5 in /flowmachine by @dependabot in #6103
- Bump ipykernel from 6.22.0 to 6.23.0 in /flowmachine by @dependabot in #6104
- Bump pyjwt from 2.6.0 to 2.7.0 in /flowkit_jwt_generator by @dependabot in #6106
- Bump ipykernel from 6.23.0 to 6.23.1 in /flowmachine by @dependabot in #6112
- Bump pyjwt from 2.6.0 to 2.7.0 in /flowclient by @dependabot in #6107
- Bump pyjwt from 2.6.0 to 2.7.0 in /flowauth by @dependabot in #6108
- Bump pre-commit from 3.3.1 to 3.3.2 in /flowmachine by @dependabot in #6114
- Bump httpx from 0.24.0 to 0.24.1 in /flowclient by @dependabot in #6118
- Bump requests from 2.30.0 to 2.31.0 in /flowmachine by @dependabot in #6122
- Bump cypress from 12.11.0 to 12.13.0 in /flowauth/frontend by @dependabot in #6123
- Bump pytest-cov from 4.0.0 to 4.1.0 in /flowmachine by @dependabot in #6127
- Bump pytest-cov from 4.0.0 to 4.1.0 in /flowclient by @dependabot in #6124
- Bump pytest-cov from 4.0.0 to 4.1.0 in /flowauth by @dependabot in #6126
- Bump pytest-cov from 4.0.0 to 4.1.0 in /flowkit_jwt_generator by @dependabot in #6125
- Bump cachetools from 5.3.0 to 5.3.1 in /flowmachine by @dependabot in #6133
- Bump pyzmq from 25.0.2 to 25.1.0 in /flowmachine by @dependabot in #6134
- Bump pandas from 2.0.1 to 2.0.2 in /flowdb/testdata/synthetic_data by @dependabot in #6136
- Bump pandas from 2.0.1 to 2.0.2 in /flowclient by @dependabot in #6137
- Bump pandas from 2.0.1 to 2.0.2 in /flowmachine by @dependabot in #6138
- Bump cryptography from 40.0.2 to 41.0.0 in /flowkit_jwt_generator by @dependabot in #6140
- Bump cryptography from 40.0.2 to 41.0.0 in /flowauth by @dependabot in #6141
- Bump cryptography from 41.0.0 to 41.0.1 in /flowkit_jwt_generator by @dependabot in #6144
- Bump cryptography from 40.0.2 to 41.0.0 in /deployment by @dependabot in #6145
- Bump cryptography from 41.0.0 to 41.0.1 in /flowauth by @dependabot in #6143
- Bump geopandas from 0.13.0 to 0.13.1 in /flowmachine by @dependabot in #6150
- Fix build issues with synth data and test data flowdb by @greenape in #6163
- Location ids table and locatable events QA check by @jc-harrison in #6164
- Pinning pyyaml in progress by @Thingus in #6259
- Bump click from 8.1.3 to 8.1.6 in /flowkit_jwt_generator by @dependabot in #6227
- Bump cryptography from 41.0.0 to 41.0.3 in /deployment by @dependabot in #6246
- Bump cryptography from 41.0.1 to 41.0.3 in /flowkit_jwt_generator by @dependabot in #6247
- Bump pyotp from 2.8.0 to 2.9.0 in /flowauth by @dependabot in #6241
- Bump pyjwt from 2.7.0 to 2.8.0 in /flowauth by @dependabot in #6228
- Bump certifi from 2023.5.7 to 2023.7.22 in /flowclient by @dependabot in #6237
- Bumping airflow + hash to 2.6.3 by @Thingus in #6258
- Bump pyjwt from 2.7.0 to 2.8.0 in /flowclient by @dependabot in #6229
- Bump respx from 0.20.1 to 0.20.2 in /flowclient by @dependabot in #6231
- Fix inconsistent hashing by @greenape in #6256
- Bump psycopg2 from 2.9.6 to 2.9.7 in /flowauth by @dependabot in #6273
- Bump tornado from 6.3.2 to 6.3.3 in /flowmachine by @dependabot in #6271
- Bump gitpython from 3.1.31 to 3.1.32 in /flowdb/tests by @dependabot in #6262
- Bump deps 20230815 by @greenape in #6274
- Bump argon2-cffi from 21.3.0 to 23.1.0 in /flowauth by @dependabot in #6275
- Bump asciidoctor from 2.0.18 to 2.0.20 in /autoflow by @dependabot in #6116
- Bump redis from 4.6.0 to 5.0.0 in /flowmachine by @dependabot in #6278
- Bump prettier from 2.8.8 to 3.0.1 in /flowauth/frontend by @dependabot in #6252
- Bump cypress from 12.13.0 to 12.17.4 in /flowauth/frontend by @dependabot in #6276
- Bump prettier from 3.0.1 to 3.0.2 in /flowauth/frontend by @dependabot in #6280
- Bump word-wrap from 1.2.3 to 1.2.5 in /flowauth/frontend by @dependabot in #6281
- Bump click from 8.1.6 to 8.1.7 in /flowkit_jwt_generator by @dependabot in #6283
- Bump jest from 29.5.0 to 29.6.3 in /flowauth/frontend by @dependabot in #6287
- Bump jest from 29.6.3 to 29.6.4 in /flowauth/frontend by @dependabot in #6293
- Bump @material-ui/pickers from 3.3.10 to 3.3.11 in /flowauth/frontend by @dependabot in #6294
- Bump pytest from 7.4.0 to 7.4.1 in /flowkit_jwt_generator by @dependabot in #6304
- Bump pandas from 1.3.5 to 2.1.0 in /flowdb/testdata/synthetic_data by @dependabot in #6301
- Bump pre-commit from 3.3.3 to 3.4.0 in /flowmachine by @dependabot in #6306
- Bump gitpython from 3.1.32 to 3.1.34 in /flowdb/tests by @dependabot in #6312
- Bump pytest from 7.4.0 to 7.4.1 in /flowclient by @dependabot in #6305
- Bump ipykernel from 6.25.1 to 6.25.2 in /flowmachine by @dependabot in #6310
- Bump pytest from 7.4.1 to 7.4.2 in /flowkit_jwt_generator by @dependabot in #6314
- Bump pytest from 7.4.1 to 7.4.2 in /flowclient by @dependabot in #6313
- Bump approvaltests from 8.4.1 to 9.0.0 in /flowmachine by @dependabot in #6296
- Bump black from 23.7.0 to 23.9.1 in /flowdb by @dependabot in #6318
- Bump pytz from 2023.3 to 2023.3.post1 in /flowmachine by @dependabot in #6311
- Bump black from 23.7.0 to 23.9.1 in /flowmachine b...