Update to v0.8.5. Add check for ENV variable to fix corrupt data

mikechambers · mikechambers · commit 4bcbcd1184fc · 2022-08-07T23:13:07.000-07:00
If the `DCLI_FIX_DATA` environment variable is set to `TRUE` then when corrupt or missing data is returned from the Bungie API, and there is not a valid local version, DCLI will attempt to retrieve updated, non-corrupt data from Bungie. (This sometimes happens if a lot of people leave a game, and no player names will be returned from the server).

Setting this to true can significantly slow down sync time, especially the initial sync, and in general, is meant to be used when using DCLI to create datastores for larger applications.
diff --git a/README.md b/README.md
@@ -7,28 +7,28 @@ dcli (Destiny Command Line Interface) is a collection of utilities and apps that
 
 If you run into any issues, have any ideas, or just want to chat, please post in [issues](https://github.com/mikechambers/dcli/issues) or share on [Discord](https://discord.gg/2Y8bV2Mq3p)
 
-
 ## Apps
 
 ### Utilities
-| TOOL | DESCRIPTION |
-| --- | --- |
-| [dclim](https://github.com/mikechambers/dcli/tree/main/src/dclim) | Manages and syncs the remote Destiny 2 API manifest database |
-| [dclisync](https://github.com/mikechambers/dcli/tree/main/src/dclisync) | Downloads and syncs Destiny 2 Crucible activity history into a local sqlite3 database file |
-| [dclitime](https://github.com/mikechambers/dcli/tree/main/src/dclitime) | Generates date / time stamps for Destiny 2 weekly event moments |
 
+| TOOL                                                                    | DESCRIPTION                                                                                |
+| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
+| [dclim](https://github.com/mikechambers/dcli/tree/main/src/dclim)       | Manages and syncs the remote Destiny 2 API manifest database                               |
+| [dclisync](https://github.com/mikechambers/dcli/tree/main/src/dclisync) | Downloads and syncs Destiny 2 Crucible activity history into a local sqlite3 database file |
+| [dclitime](https://github.com/mikechambers/dcli/tree/main/src/dclitime) | Generates date / time stamps for Destiny 2 weekly event moments                            |
 
 ### Apps
-| TOOL | DESCRIPTION |
-| --- | --- |
-| [dclia](https://github.com/mikechambers/dcli/tree/main/src/dclia) | Displays information on player's current activity within Destiny 2 |
-| [dcliah](https://github.com/mikechambers/dcli/tree/main/src/dcliah) | Displays Destiny 2 activity history and stats |
-| [dcliad](https://github.com/mikechambers/dcli/tree/main/src/dcliad) | Displays Destiny 2 Crucible activity / match details |
 
+| TOOL                                                                | DESCRIPTION                                                        |
+| ------------------------------------------------------------------- | ------------------------------------------------------------------ |
+| [dclia](https://github.com/mikechambers/dcli/tree/main/src/dclia)   | Displays information on player's current activity within Destiny 2 |
+| [dcliah](https://github.com/mikechambers/dcli/tree/main/src/dcliah) | Displays Destiny 2 activity history and stats                      |
+| [dcliad](https://github.com/mikechambers/dcli/tree/main/src/dcliad) | Displays Destiny 2 Crucible activity / match details               |
 
 ### Libraries
-| TOOL | DESCRIPTION |
-| --- | --- |
+
+| TOOL                                                            | DESCRIPTION                              |
+| --------------------------------------------------------------- | ---------------------------------------- |
 | [dcli](https://github.com/mikechambers/dcli/tree/main/src/dcli) | Library used across all of the dcli apps |
 
 Each tool page contains additional tool specific information and usage examples.
@@ -55,7 +55,6 @@ If you are running on Mac, make sure to [read this article](https://github.com/m
 
 Before viewing your stats you need to first sync the manifest (which contains information about weapons, maps, etc...).
 
-
 ### Download the manifest
 
 You can download the latest Destiny 2 manifest database using dclim. This contains information about all of the items and activities in Destiny 2, and is updated periodically.
@@ -73,7 +72,7 @@ and the manifest file will be downloaded and saved in a system appropriate direc
 Next, lets sync all of our activity history to a local database and view data. This data will be used by other apps, such as dcliah to generate and display stats.
 
 ```
-$ dcliah --name mesh#3230 
+$ dcliah --name mesh#3230
 ```
 
 Replacing mesh#3230 with your own Bungie name.
@@ -106,15 +105,15 @@ These can be useful on their own, but where they can be really powerful is when
 
 There are a couple of examples in the [examples directory](https://github.com/mikechambers/dcli/tree/main/examples):
 
-* Send a notification when you load into a new activity (particularly useful when playing crucible so you can see which map you are loading into)
-* Automatically generate weekly reports on your Crucible stats and email them to yourself
-* Track your Crucible stats per game play session
+- Send a notification when you load into a new activity (particularly useful when playing crucible so you can see which map you are loading into)
+- Automatically generate weekly reports on your Crucible stats and email them to yourself
+- Track your Crucible stats per game play session
 
 As you can see, right now, a lot of the functionality is Crucible based. If you would like to see other functionality, make sure you requests it in the [issues](https://github.com/mikechambers/dcli/issues), or [Discord](https://discord.gg/2Y8bV2Mq3p).
 
 ### Learning More
 
-At anytime, you can see which arguments and options are avaliable by passing the *--help* argument:
+At anytime, you can see which arguments and options are avaliable by passing the _--help_ argument:
 
 ```
 $ dcliah --help
@@ -128,6 +127,12 @@ If you have any questions, feature requests, need help, or just want to chat, jo
 
 You can also log bugs and features requests on the [issues page](https://github.com/mikechambers/dcli/issues).
 
+### Environment Variables
+
+If the `DCLI_FIX_DATA` environment variable is set to `TRUE` then when corrupt or missing data is returned from the Bungie API, and there is not a valid local version, DCLI will attempt to retrieve updated, non-corrupt data from Bungie. (This sometimes happens if a lot of people leave a game, and no player names will be returned from the server).
+
+Setting this to true can significantly slow down sync time, especially the initial sync, and in general, is meant to be used when using DCLI to create datastores for larger applications.
+
 ## Compiling
 
 Tools are written and compiled in [Rust](https://www.rust-lang.org/). The 1.50.0 rust version is required, which can be installed and set via:
@@ -147,11 +152,11 @@ To compile, switch to the `src/` directory and run:
 $ cargo build --release
 ```
 
-which will place the compiled tools in *src/target/release*
+which will place the compiled tools in _src/target/release_
 
 ## Known Issues
 
-* Tied matches are not displayed correctly, and are treated as a Victory.
+- Tied matches are not displayed correctly, and are treated as a Victory.
 
 ## License
 
diff --git a/RELEASE.md b/RELEASE.md
@@ -1,5 +1,9 @@
 # dcli Release Notes
 
+## v0.8.5 August 7, 2022
+
+- Improved how corrupted data from Bungie is handled. Now, if data is corrupted, we check for existing data before updating. If the DCLI_FIX_DATA environment variable is set to true, then if there is not already a local, non-corrupt version of the data, we will go back to Bungie to retrieve updated data. Note, this can significantly slow down initial syncs.
+
 ## v0.8.4 August 1, 2022
 
 - Fixed data parsing error that could occur due to wrong data type in code
diff --git a/src/Cargo.lock b/src/Cargo.lock
diff --git a/src/dcli/Cargo.toml b/src/dcli/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "dcli"
 #version
-version = "0.8.4"
+version = "0.8.5"
 authors = ["Mike Chambers <mikechambers@gmail.com>"]
 edition = "2018"
 description = "Library for the dcli collection of command line tools for Destiny 2."
diff --git a/src/dcli/src/activitystoreinterface.rs b/src/dcli/src/activitystoreinterface.rs
@@ -26,7 +26,6 @@ use std::{collections::HashMap, path::Path};
 use chrono::{DateTime, Utc};
 
 use crate::response::character::CharacterData;
-use crate::response::gmd::UserMembershipData;
 use crate::utils::calculate_percent;
 use crate::{
     crucible::{CrucibleActivity, Member, PlayerName, Team},
@@ -64,9 +63,13 @@ use crate::{
     },
 };
 
+use std::env;
+
 const STORE_FILE_NAME: &str = "dcli.sqlite3";
 const STORE_DB_SCHEMA: &str = include_str!("../actitvity_store_schema.sql");
 
+const DCLI_FIX_DATA: &str = "DCLI_FIX_DATA";
+
 //numer of simultaneous requests we make to server when retrieving activity history
 const PGCR_REQUEST_CHUNK_AMOUNT: usize = 24;
 
@@ -78,6 +81,7 @@ pub struct ActivityStoreInterface {
     db: SqliteConnection,
     path: String,
     api_interface: ApiInterface,
+    fix_corrupt_data: bool,
 }
 
 impl ActivityStoreInterface {
@@ -92,6 +96,18 @@ impl ActivityStoreInterface {
     ) -> Result<ActivityStoreInterface, Error> {
         let path = store_dir.join(STORE_FILE_NAME).display().to_string();
 
+        let fix_corrupt_data = match env::var(DCLI_FIX_DATA) {
+            Ok(v) => v.to_lowercase() == "true",
+            Err(_e) => false,
+        };
+
+        if verbose {
+            println!(
+                "DCLI_FIX_DATA environment variable enabled : {}",
+                fix_corrupt_data
+            );
+        }
+
         let read_only = false;
         let connection_string: &str = &path;
 
@@ -135,6 +151,7 @@ impl ActivityStoreInterface {
             verbose,
             path: path.to_string(),
             api_interface,
+            fix_corrupt_data,
         })
     }
 
@@ -1125,11 +1142,11 @@ impl ActivityStoreInterface {
         let mut member = mem;
         let mut m = None;
         //check and see if data is missing (happens a lot with the API)
-        if member.name.display_name.is_none()
+        if member.name.bungie_display_name.is_none()
             || member.name.bungie_display_name_code.is_none()
             || member.platform == Platform::Unknown
         {
-            //if some data is missing, the see if we already have info on
+            //if some data is missing, we see if we already have info on
             //this member stored.
 
             let row_id: i32 = match self.get_member_row_id(member).await {
@@ -1143,27 +1160,33 @@ impl ActivityStoreInterface {
                 return Ok(row_id);
             }
 
-            //need to catch and ignore error here
-            let card_result = self
-                .api_interface
-                .retrieve_player_info_by_id(&member.id)
-                .await;
-
-            if card_result.is_ok() {
-                let c = card_result.unwrap();
-                let cards = c.destiny_memberships;
-
-                for card in cards.iter() {
-                    if card.cross_save_override == card.membership_type {
-                        m = Some(card.to_member());
-                        break;
+            //if the DCLI_FIX_DATA environemnt variable is set to true
+            //the code will try and fix corrupt data with additional API
+            //calls. This can dramatically slow down the first time data is
+            //synced, but can help fix missing data
+            if self.fix_corrupt_data {
+                //need to catch and ignore error here
+                let card_result = self
+                    .api_interface
+                    .retrieve_player_info_by_id(&member.id)
+                    .await;
+
+                if card_result.is_ok() {
+                    let c = card_result.unwrap();
+                    let cards = c.destiny_memberships;
+
+                    for card in cards.iter() {
+                        if card.cross_save_override == card.membership_type {
+                            m = Some(card.to_member());
+                            break;
+                        }
                     }
-                }
 
-                //if we get here, then we need to go to the server to try and get the
-                //data
-                if m.is_some() {
-                    member = m.as_ref().unwrap();
+                    //if we get here, then we need to go to the server to try and get the
+                    //data
+                    if m.is_some() {
+                        member = m.as_ref().unwrap();
+                    }
                 }
             }
         }
@@ -1172,7 +1195,7 @@ impl ActivityStoreInterface {
             r#"
             INSERT into "member" ("member_id", "platform_id", "display_name", "bungie_display_name", "bungie_display_name_code") VALUES (?, ?, ?, ?, ?)
             ON CONFLICT(member_id) DO UPDATE
-            set display_name = ?, bungie_display_name = ?, bungie_display_name_code = ?
+            set display_name = ?, bungie_display_name = ?, bungie_display_name_code = ?, platform_id = ?
         "#,
         )
         .bind(member.id.to_string())
@@ -1183,6 +1206,7 @@ impl ActivityStoreInterface {
         .bind(&member.name.display_name)
         .bind(&member.name.bungie_display_name)
         .bind(&member.name.bungie_display_name_code)
+        .bind(member.platform.as_id().to_string())
         .execute(&mut self.db)
         .await?;
 
diff --git a/src/dclia/Cargo.toml b/src/dclia/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "dclia"
 #version
-version = "0.8.4"
+version = "0.8.5"
 authors = ["Mike Chambers <mikechambers@gmail.com>"]
 description = "Command line tool for retrieving information on current activity for specified player character."
 homepage = "https://www.mikechambers.com"
diff --git a/src/dcliad/Cargo.toml b/src/dcliad/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "dcliad"
 #version
-version = "0.8.4"
+version = "0.8.5"
 authors = ["Mike Chambers <mikechambers@gmail.com>"]
 edition = "2018"
 description = "Command line tool for viewing Destiny 2 activity details."
diff --git a/src/dcliah/Cargo.toml b/src/dcliah/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "dcliah"
 #version
-version = "0.8.4"
+version = "0.8.5"
 authors = ["Mike Chambers <mikechambers@gmail.com>"]
 edition = "2018"
 description = "Command line tool for viewing Destiny 2 activity history."
diff --git a/src/dclim/Cargo.toml b/src/dclim/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "dclim"
 #version
-version = "0.8.4"
+version = "0.8.5"
 authors = ["Mike Chambers <mikechambers@gmail.com>"]
 edition = "2018"
 description = "Command line tool for managing and syncing the remote Destiny 2 API manifest database."
diff --git a/src/dclisync/Cargo.toml b/src/dclisync/Cargo.toml
@@ -1,7 +1,7 @@
 [package]
 name = "dclisync"
 #version
-version = "0.8.4"
+version = "0.8.5"
 authors = ["Mike Chambers <mikechambers@gmail.com>"]
 edition = "2018"
 description = "Command line tool for downloading and syncing Destiny 2 Crucible activity history."
diff --git a/src/dclisync/README.md b/src/dclisync/README.md
diff --git a/src/dclitime/Cargo.toml b/src/dclitime/Cargo.toml
