How to better data map from dataSource to our data?

[jduan00]

Issues

We’ve identified many data mapping issues. Notable: #67 #68 #70.

Plan

Let’s proactively work on improving our data mapping process.

1. Use AI to help.
2. Go through the many tracking numbers (listed below) and test.

To get started: To identify which major and minor statuses are missing.

For example, the below is missing 3300 (status code definition)

curl -s "https://api.eg1.io/v0/whereis/sfex-SF3144980189509?phonenum=4868" \
     -H "Authorization: Bearer xxxx" | jq | grep status | sort
      "status": 3001,
      "status": 3001,
      "status": 3001,
      "status": 3001,
      "status": 3002,
      "status": 3002,
      "status": 3002,
      "status": 3002,
      "status": 3004,
      "status": 3004,
      "status": 3004,
      "status": 3100,
      "status": 3250,
      "status": 3350,
      "status": 3350,
      "status": 3350,
      "status": 3350,
      "status": 3400,
      "status": 3450,
      "status": 3450,
      "status": 3500,

curl -s "https://api.eg1.io/v0/whereis/sfex-SF3144980189509?phonenum=4868&refresh=true" \
     -H "Authorization: Bearer xxx" | jq | grep status | grep "0,$"
      "status": 3100,
      "status": 3250,
      "status": 3350,
      "status": 3350,
      "status": 3350,
      "status": 3350,
      "status": 3400,
      "status": 3450,
      "status": 3450,
      "status": 3500,

[jduan00]

Showing all events; missing 3100 (Received by carrier)

curl -s "https://api.eg1.io/v0/whereis/fdx-880576926686?refresh=true" \
     -H "Authorization: Bearer xxxx" | jq | grep status

      "status": 3000,
      "status": 3050,
      "status": 3002,
      "status": 3001,
      "status": 3250,
      "status": 3300,
      "status": 3001,
      "status": 3400,
      "status": 3300,
      "status": 3450,
      "status": 3500,

Showing only major and minor events

curl -s "https://api.eg1.io/v0/whereis/fdx-880576926686?refresh=true" \
     -H "Authorization: Bearer xxxx" | jq | grep status | grep "0,$"

      "status": 3000,
      "status": 3050,
      "status": 3250,
      "status": 3300,
      "status": 3400,
      "status": 3300,
      "status": 3450,
      "status": 3500,

[jduan00]

A good example

With most events, the sequence is not as expected. Import clearance is done (3400( before the shipment arrived at destination (3300).

When we look at the last major event, we need to sort by statusCode, not by time.

curl -s "https://api.eg1.io/v0/whereis/fdx-772892610064?refresh=true" \
     -H "Authorization: Bearer xxxx" | jq | grep status             
      "status": 3000,
      "status": 3050,
      "status": 3100,
      "status": 3002,
      "status": 3001,
      "status": 3250,
      "status": 3200,
      "status": 3001,
      "status": 3001,
      "status": 3001,
      "status": 3001,
      "status": 3001,
      "status": 3350,
      "status": 3400,
      "status": 3400, <--
      "status": 3300, <--
      "status": 3300, <--
      "status": 3450,
      "status": 3500,

With all major events

curl -s "https://api.eg1.io/v0/whereis/fdx-772892610064?refresh=true" \
     -H "Authorization: Bearer xxx" | jq | grep status | grep "0,$"
      "status": 3000,
      "status": 3050,
      "status": 3100,
      "status": 3250,
      "status": 3200,
      "status": 3350,
      "status": 3400,
      "status": 3400,
      "status": 3300,
      "status": 3300,
      "status": 3450,
      "status": 3500,

[jduan00]

Ideas

How can we leverage AI LLMs to enhance our mapping capabilities and identify areas we might have overlooked? This aligns with the discussion in #36.

More tracking numbers

FedEx

772892610064 (SG)
779761249721 (SG)
772873053639 (SG)
880657123042 (CH)
880150644080 (CH)
880203713161 (FR)
880150569025 (JP)
880203183551 (JP)
880150812930 (HK)
880090278103 (HK)
880121867322 (UK)
771195139980 (TW)
880300780195 (PR)
880264269736 (PR)
880701922622 (FR) - Master
  880701922677
  880701922633 

Sfex

More to come...

[jduan00]

This has a very long journey with most events

Observations

1. The sequence is not always as expected.
2. 3400 - 3250 - 3400 - 2350 - 3300

So for the API user:

1. Look for the latest major event (sorted by statusCode).
2. Also, look for the status code 3400 (import clearance done). However, this is not the latest event as it is likely to be out of sequence.

All events sorted by when time

curl --request GET \
  --url "https://api.eg1.io/v0/whereis/fdx-880640895620?refresh=true" \
  --header 'Authorization: Bearer xxxx' -s | jq | grep status

      "status": 3000,
      "status": 3050,
      "status": 3100,
      "status": 3002,
      "status": 3250,
      "status": 3002,
      "status": 3250,
      "status": 3002,
      "status": 3001,
      "status": 3400, <--
      "status": 3250, <--
      "status": 3001,
      "status": 3002,
      "status": 3001,
      "status": 3400, <--
      "status": 3250, <--
      "status": 3300, <--
      "status": 3450,
      "status": 3500,

All events sorted by statusCode

curl --request GET \
  --url "https://api.eg1.io/v0/whereis/fdx-880640895620" \             
  --header 'Authorization: Bearer xxxx' -s | jq | grep status | sort

      "status": 3000,
      "status": 3001,
      "status": 3001,
      "status": 3001,
      "status": 3002,
      "status": 3002,
      "status": 3002,
      "status": 3002,
      "status": 3050,
      "status": 3100,
      "status": 3250,
      "status": 3250,
      "status": 3250,
      "status": 3250,
      "status": 3300,
      "status": 3400,
      "status": 3400,
      "status": 3450,
      "status": 3500,

Sorted major and minor events

curl --request GET \
  --url "https://api.eg1.io/v0/whereis/fdx-880640895620" \
  --header 'Authorization: Bearer xxxx' -s | jq | grep status | grep "0,$" | sort

      "status": 3000,
      "status": 3050,
      "status": 3100,
      "status": 3250,
      "status": 3250,
      "status": 3250,
      "status": 3250,
      "status": 3300,
      "status": 3400,
      "status": 3400,
      "status": 3450,
      "status": 3500,