1.18.4

Changed

• docker-compose has been replaced with docker compose in the makefile; this might break builds on machines that haven't updated their docker in a while.

Fixed

• SQLAlchemy version installed in the FlowMachine docker image is now compatible with the flowmachine library. #6052

1.18.3

Added

• Quickstart script now supports arbitrary countries via EXAMPLE_COUNTRY env var. #5796
• FlowDB's maximum locks per transaction setting can now be controlled using the MAX_LOCKS_PER_TRANSACTION env var. #5157

Changed

• Increased FlowDB's default maximum locks per transaction to 365 * 5 * 4 * (1 + 4). #5157

Fixed

• Null values in first column of first row of ingested data no longer cause flowetl to skip ingestion #5090

1.18.2

Fixed

• Fixed migrations being missing from the built FlowAuth docker images #5818

1.18.1

1.18.1

Added

• Added Alembic support via flask-migrate to Flowauth #5799

1.18.0

1.18.0

Added

• Added views etl.ingested_state, etl.available_dates and etl.deduped_post_etl_queries in FlowDB, for convenient extraction of relevant information from the ETL tables. #5641
• Added MajorityLocationWithUnlocatable query class and majority_location function. #5720

Changed

• Important; tokens issued by previous versions of Flowauth are not compatible with this version. Users will need to regenerate tokens using the updated Flowauth.
• Move from groups to roles in flowauth; see here for full details. #5613
• Changed AIRFLOW__CORE__SQL_ALCHEMY_CONN env var to AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
• RoleScopePicker component redesigned and reimplemented.
• Docs now recommend creating a separate bind mount for airflow scheduler logs, and include this in the secrets quickstart. #3622
• jwt tokens now use sub instead of identity for JWT_IDENTITY_CLAIM.
• A majority_location query with include_unlocatable=True will now include rows for all subscribers in the subscriber_location_weights sub-query, including those for whom all weights are negative (previously subscribers with only negative weights were excluded).

Fixed

• Fixed a potential deadlock when using a small connection pool and store-ing queries
• AutoFlow can now be run in a docker container with non-default user. #5574
• Passing an empty list of events tables when creating a query now raises ValueError: Empty tables list. instead of a MissingDateError. #436
• Flowmachine now looks at only the most recent state (per CDR type per CDR date) in etl.etl_records to determine available dates. #5641
• It is now possible to run API queries that include multiple different aggregation units (e.g. joined_spatial_aggregate with displacement metric). #4649
• Demo roles can now be used in worked_examples. #5735

Removed

• Removed the include_unlocatable parameter from MajorityLocation class (the majority_location function should be used instead if include_unlocatable is required). #5720

1.17.2

Note: This is a side-release for accelerating the Ghana project - please use v. 1.18.0 for work

1.17.2

Added

Changed

• Changed AIRFLOW__CORE__SQL_ALCHEMY_CONN env var to AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
• Docs now recommend creating a separate bind mount for airflow scheduler logs, and include this in the secrets quickstart. #3622
• A majority_location query with include_unlocatable=True will now include rows for all subscribers in the subscriber_location_weights sub-query, including those for whom all weights are negative (previously subscribers with only negative weights were excluded).

Fixed

• Fixed a potential deadlock when using a small connection pool and store-ing queries
• Added MajorityLocationWithUnlocatable query class and majority_location function. #5720
• AutoFlow can now be run in a docker container with non-default user. #5574
• Passing an empty list of events tables when creating a query now raises ValueError: Empty tables list. instead of a MissingDateError. #436
• Flowmachine now looks at only the most recent state (per CDR type per CDR date) in etl.etl_records to determine available dates. #5641
• It is now possible to run API queries that include multiple different aggregation units (e.g. joined_spatial_aggregate with displacement metric). #4649

Removed

• Removed the include_unlocatable parameter from MajorityLocation class (the majority_location function should be used instead if include_unlocatable is required). #5720

1.17.1

Added

• Added get_aggregation_unit server action, for getting the aggregation unit associated with a query specification. #5141

Changed

• nocturnal_events now expects a night_hours parameter with nested sub-fields start_hour and end_hour, instead of two parameters night_start_hour and night_end_hour.
• Spatial units with a mapping table now only include cells that appear in the mapping table. #5360

Fixed

• Invalid sub-query specs nested within a modal_location spec now raise appropriate validation errors, instead of being masked by internal flowmachine server errors. #4816

1.17.0

Added

• inflows and outflows exposed via API endpoint + added to flowclient #2029, #4866

Changed

• Action Needed Airflow updated to version 2.3.3; backup flowetl_db before applying update #4940
• Tables created under the cache schema in FlowDB will automatically be set to be owned by the flowmachine user. #4714
• Query.explain will now explain the query even where it is already stored. #1285
• unstored_dependencies_graph no longer blocks until dependencies are in a determinate state. #4949
• In and out flows no longer return location columns with to/from suffix.
• FlowDB now always creates a role named flowmachine.
• Flowmachine will set the state of a query being stored to cancelled if interrupted while the store is running.
• Flowmachine now supports sqlalchemy >=1.4 #5140

Fixed

• Flowmachine now makes the built in flowmachine role owner of cache tables as a post-action when a query is stored. #4714
• TopupBalance now returns the weighted mode when requested instead of weighted median #1412
• Fixed in and out flow geojson for multicolumn location types #5132

Removed

• use_file_flux_sensor removed entirely. #2812
• Model, ModelResult and Louvain have been removed. #5168

1.16.0

Added

• Most frequent locations is now available via FlowAPI. #3165
• Total active periods is now available via FlowAPI.
• Made hour of day slicing available via FlowAPI. #3165
• Added visited on most days reference location query. #4267
• Added unique value from query list query. #4486
• Added mixin for exposing start_date and end_date internally as datetime objects #4497
• Added CombineFirst and CoalescedLocation queries. #4524
• Added MajorityLocation query. #4522
• Added join_type param to Flows class. #4539
• Added PerSubscriberAggregate query. #4559
• Added FlowETL QA checks 'count_imeis', 'count_imsis', 'count_locatable_location_ids', 'count_null_imeis', 'count_null_imsis', 'count_null_location_ids', 'max_msisdns_per_imei', 'max_msisdns_per_imsi', 'count_added_rows_outgoing', 'count_null_counterparts', 'count_null_durations', 'count_onnet_msisdns_incoming', 'count_onnet_msisdns_outgoing', 'count_onnet_msisdns', 'max_duration' and 'median_duration'. #4552
• Added FilteredReferenceLocation query, which returns only rows where a subscriber visited a reference location the required number of times. #4584
• Added LabelledSpatialAggregate query and redaction, which sub-aggregates by subscriber labels. #4668
• Added MobilityClassification query, to classify subscribers by mobility type based on a sequence of locations. #4666
• Exposed CoalescedLocation via FlowAPI, in the specific case where the fallback location is a FilteredReferenceLocation query. #4585
• Added LabelledFlows query, which returns flows disaggregated by label #4679
• Exposed LabelledSpatialAggregate and LabelledFlows via FlowAPI, with a MobilityClassification query accepted as the 'labels' parameter. #4669
• Added RedactedLabelledAggregate and subclasses for redacting labelled data (see ADR 0011). #4671

Changed

• Harmonised FlowAPI parameter names for start and end dates. They are now all start_date and end_date
• Further improvements to token display in FlowAuth. #1124
• Increased the FlowDB quickstart container's timeout to 15 minutes. #782
• Union and Query.union now accept a variable number of queries to concatenate. #4565

Fixed

• Autoflow's prefect version is now current. #2544
• FlowMachine server will now successfully remove cache for queries defined in an interactive flowmachine session during cleanup. #4008

1.15.0

Added

• FlowETL flux check can be turned off by setting use_flux_sensor=False in create_dag. #3603

Changed

• The use_file_flux_sensor argument to create_dag is deprecated. To use the table-based flux check in a file-based DAG, set use_flux_sensor='table'.
• Improvements to token display in FlowAuth. #2812