users.messages.list and users.threads.list does NOT return messages/threads in reverse-chronological order

[wdanilo]

Description

users.messages.list returns messages in a little bit random order.
I also reported the issue here, but knowing Google's issue tracker, I will get a sensible reply there in a few years, so trying my luck here as well :(

Code

I'm using the following code:

async test() {
    console.log(
      "Performing test if 'users.messages.list' returns messages sorted by decreasing dates and historyIDs.",
    )
    const count = 100
    const cfg: GmailApi.Params$Resource$Users$Messages$List = {
      userId: 'me',
      maxResults: count,
    }
    const resp = await Auth.catchAllOAuth(() => this.api.users.messages.list(cfg))
    if (resp.isErr()) {
      console.error('GMail API query error.', resp.error)
      return
    }
    const list = resp.value.data.messages
    if (list == null) {
      console.error('GMail API invalid response.')
      return
    }
    let isHistoryIdDecreasing = true
    let isDateDecreasing = true
    let prevHistoryId: null | number = null
    let prevDate: null | number = null
    let i = 0
    for (const message of list) {
      i++
      const id = message.id
      if (id == null) {
        console.error('Message is missing the "id" field.')
        return
      }
      const msgQuery = {userId: 'me', id}
      const msgResp = await Auth.catchAllOAuth(() => this.api.users.messages.get(msgQuery))
      if (msgResp.isErr()) {
        console.error('GMail API query error.', msgResp.error)
        return
      }
      const historyIdStr = msgResp.value.data.historyId
      const dateStr = msgResp.value.data.internalDate
      if (!historyIdStr || !dateStr) {
        console.error('Message is missing the "historyId" or "internalDate" field.')
        return
      }
      const historyId = parseInt(historyIdStr, 10)
      const date = parseInt(dateStr, 10)
      let localIsHistoryIdDecreasing = true
      let localIsDateDecreasing = true
      if (prevHistoryId != null && prevHistoryId < historyId) {
        isHistoryIdDecreasing = false
        localIsHistoryIdDecreasing = false
      }
      if (prevDate != null && prevDate < date) {
        isDateDecreasing = false
        localIsDateDecreasing = false
      }
      prevHistoryId = historyId
      prevDate = date
      console.log(
        `${i}/${count}: historyId: ${historyId}, date: ${date}, isHistoryIdDecreasing: ${localIsHistoryIdDecreasing}, isDateDecreasing: ${localIsDateDecreasing}`,
      )
    }
    console.log({isHistoryIdDecreasing, isDateDecreasing})
  }

Output

Performing test if 'users.messages.list' returns messages sorted by decreasing dates and historyIDs.
1/100: historyId: 15297863, date: 1719191761000, isHistoryIdDecreasing: true, isDateDecreasing: true
2/100: historyId: 15297367, date: 1719170805000, isHistoryIdDecreasing: true, isDateDecreasing: true
3/100: historyId: 15297189, date: 1719159861000, isHistoryIdDecreasing: true, isDateDecreasing: true
4/100: historyId: 15297049, date: 1719150301000, isHistoryIdDecreasing: true, isDateDecreasing: true
5/100: historyId: 15297132, date: 1719149574000, isHistoryIdDecreasing: false, isDateDecreasing: true
6/100: historyId: 15296597, date: 1719147670000, isHistoryIdDecreasing: true, isDateDecreasing: true
7/100: historyId: 15297036, date: 1719122788000, isHistoryIdDecreasing: false, isDateDecreasing: true
8/100: historyId: 15296855, date: 1719120755000, isHistoryIdDecreasing: true, isDateDecreasing: true
9/100: historyId: 15296852, date: 1719120714000, isHistoryIdDecreasing: true, isDateDecreasing: true
10/100: historyId: 15297511, date: 1719118502000, isHistoryIdDecreasing: false, isDateDecreasing: true
11/100: historyId: 15296422, date: 1719108184000, isHistoryIdDecreasing: true, isDateDecreasing: true
12/100: historyId: 15296007, date: 1719103434000, isHistoryIdDecreasing: true, isDateDecreasing: true
13/100: historyId: 15297035, date: 1719094619000, isHistoryIdDecreasing: false, isDateDecreasing: true
14/100: historyId: 15297035, date: 1719094618000, isHistoryIdDecreasing: true, isDateDecreasing: true
15/100: historyId: 15295930, date: 1719088891000, isHistoryIdDecreasing: true, isDateDecreasing: true
16/100: historyId: 15295489, date: 1719077834000, isHistoryIdDecreasing: true, isDateDecreasing: true
17/100: historyId: 15297034, date: 1719077423000, isHistoryIdDecreasing: false, isDateDecreasing: true
18/100: historyId: 15295374, date: 1719076577000, isHistoryIdDecreasing: true, isDateDecreasing: true
19/100: historyId: 15297544, date: 1719070375000, isHistoryIdDecreasing: false, isDateDecreasing: true
20/100: historyId: 15297032, date: 1719069634000, isHistoryIdDecreasing: true, isDateDecreasing: true
21/100: historyId: 15297569, date: 1719058421000, isHistoryIdDecreasing: false, isDateDecreasing: true
22/100: historyId: 15295569, date: 1719057377000, isHistoryIdDecreasing: true, isDateDecreasing: true
23/100: historyId: 15297030, date: 1719053485000, isHistoryIdDecreasing: false, isDateDecreasing: true
24/100: historyId: 15295971, date: 1719044954000, isHistoryIdDecreasing: true, isDateDecreasing: true
25/100: historyId: 15297029, date: 1719039969000, isHistoryIdDecreasing: false, isDateDecreasing: true
26/100: historyId: 15297429, date: 1719039871000, isHistoryIdDecreasing: false, isDateDecreasing: true
27/100: historyId: 15297027, date: 1719033506000, isHistoryIdDecreasing: true, isDateDecreasing: true
28/100: historyId: 15294309, date: 1719031329000, isHistoryIdDecreasing: true, isDateDecreasing: true
29/100: historyId: 15295111, date: 1719027583000, isHistoryIdDecreasing: false, isDateDecreasing: true
30/100: historyId: 15297593, date: 1719026659000, isHistoryIdDecreasing: false, isDateDecreasing: true
31/100: historyId: 15297025, date: 1719025075000, isHistoryIdDecreasing: true, isDateDecreasing: true
32/100: historyId: 15297024, date: 1719024799000, isHistoryIdDecreasing: true, isDateDecreasing: true
33/100: historyId: 15297618, date: 1719024388000, isHistoryIdDecreasing: false, isDateDecreasing: true
34/100: historyId: 15297022, date: 1719021288000, isHistoryIdDecreasing: true, isDateDecreasing: true
35/100: historyId: 15297642, date: 1719021206000, isHistoryIdDecreasing: false, isDateDecreasing: true
36/100: historyId: 15293469, date: 1719021069000, isHistoryIdDecreasing: true, isDateDecreasing: true
37/100: historyId: 15297642, date: 1719021008000, isHistoryIdDecreasing: false, isDateDecreasing: true
38/100: historyId: 15297671, date: 1719008653000, isHistoryIdDecreasing: false, isDateDecreasing: true
39/100: historyId: 15293217, date: 1719008487000, isHistoryIdDecreasing: true, isDateDecreasing: true
40/100: historyId: 15297705, date: 1719008444000, isHistoryIdDecreasing: false, isDateDecreasing: true
41/100: historyId: 15293049, date: 1719007821000, isHistoryIdDecreasing: true, isDateDecreasing: true
42/100: historyId: 15292962, date: 1719005719000, isHistoryIdDecreasing: true, isDateDecreasing: true
43/100: historyId: 15297739, date: 1719004759000, isHistoryIdDecreasing: false, isDateDecreasing: true
44/100: historyId: 15293296, date: 1719004141000, isHistoryIdDecreasing: true, isDateDecreasing: true
45/100: historyId: 15292645, date: 1719002715000, isHistoryIdDecreasing: true, isDateDecreasing: true
46/100: historyId: 15292586, date: 1719001722000, isHistoryIdDecreasing: true, isDateDecreasing: true
47/100: historyId: 15292507, date: 1718994153000, isHistoryIdDecreasing: true, isDateDecreasing: true
48/100: historyId: 15296270, date: 1719000410000, isHistoryIdDecreasing: false, isDateDecreasing: false
49/100: historyId: 15296304, date: 1719000210000, isHistoryIdDecreasing: false, isDateDecreasing: true
50/100: historyId: 15292265, date: 1719000212000, isHistoryIdDecreasing: true, isDateDecreasing: false
51/100: historyId: 15293696, date: 1719000158000, isHistoryIdDecreasing: false, isDateDecreasing: true
52/100: historyId: 15292133, date: 1719000132000, isHistoryIdDecreasing: true, isDateDecreasing: true
53/100: historyId: 15291968, date: 1718999917000, isHistoryIdDecreasing: true, isDateDecreasing: true
54/100: historyId: 15291886, date: 1718999562000, isHistoryIdDecreasing: true, isDateDecreasing: true
55/100: historyId: 15291826, date: 1718999475000, isHistoryIdDecreasing: true, isDateDecreasing: true
56/100: historyId: 15291744, date: 1718999417000, isHistoryIdDecreasing: true, isDateDecreasing: true
57/100: historyId: 15291662, date: 1718999355000, isHistoryIdDecreasing: true, isDateDecreasing: true
58/100: historyId: 15291604, date: 1718999152000, isHistoryIdDecreasing: true, isDateDecreasing: true
59/100: historyId: 15291544, date: 1718999072000, isHistoryIdDecreasing: true, isDateDecreasing: true
60/100: historyId: 15291461, date: 1718998798000, isHistoryIdDecreasing: true, isDateDecreasing: true
61/100: historyId: 15291384, date: 1718998480000, isHistoryIdDecreasing: true, isDateDecreasing: true
62/100: historyId: 15291302, date: 1718998079000, isHistoryIdDecreasing: true, isDateDecreasing: true
63/100: historyId: 15291242, date: 1718997877000, isHistoryIdDecreasing: true, isDateDecreasing: true
64/100: historyId: 15291164, date: 1718997874000, isHistoryIdDecreasing: true, isDateDecreasing: true
65/100: historyId: 15291101, date: 1718997051000, isHistoryIdDecreasing: true, isDateDecreasing: true
66/100: historyId: 15291019, date: 1718997022000, isHistoryIdDecreasing: true, isDateDecreasing: true
67/100: historyId: 15290937, date: 1718996810000, isHistoryIdDecreasing: true, isDateDecreasing: true
68/100: historyId: 15290879, date: 1718996476000, isHistoryIdDecreasing: true, isDateDecreasing: true
69/100: historyId: 15290819, date: 1718996035000, isHistoryIdDecreasing: true, isDateDecreasing: true
70/100: historyId: 15290759, date: 1718995732000, isHistoryIdDecreasing: true, isDateDecreasing: true
71/100: historyId: 15290677, date: 1718995321000, isHistoryIdDecreasing: true, isDateDecreasing: true
72/100: historyId: 15290621, date: 1718995295000, isHistoryIdDecreasing: true, isDateDecreasing: true
73/100: historyId: 15290510, date: 1718994605000, isHistoryIdDecreasing: true, isDateDecreasing: true
74/100: historyId: 15290450, date: 1718994479000, isHistoryIdDecreasing: true, isDateDecreasing: true
75/100: historyId: 15290371, date: 1718994196000, isHistoryIdDecreasing: true, isDateDecreasing: true
76/100: historyId: 15290592, date: 1718994167000, isHistoryIdDecreasing: false, isDateDecreasing: true
77/100: historyId: 15290141, date: 1718993384000, isHistoryIdDecreasing: true, isDateDecreasing: true
78/100: historyId: 15290059, date: 1718993022000, isHistoryIdDecreasing: true, isDateDecreasing: true
79/100: historyId: 15289999, date: 1718992719000, isHistoryIdDecreasing: true, isDateDecreasing: true
80/100: historyId: 15289917, date: 1718992717000, isHistoryIdDecreasing: true, isDateDecreasing: true
81/100: historyId: 15289836, date: 1718992611000, isHistoryIdDecreasing: true, isDateDecreasing: true
82/100: historyId: 15289755, date: 1718992550000, isHistoryIdDecreasing: true, isDateDecreasing: true
83/100: historyId: 15289673, date: 1718992117000, isHistoryIdDecreasing: true, isDateDecreasing: true
84/100: historyId: 15289614, date: 1718992106000, isHistoryIdDecreasing: true, isDateDecreasing: true
85/100: historyId: 15289532, date: 1718992105000, isHistoryIdDecreasing: true, isDateDecreasing: true
86/100: historyId: 15289468, date: 1718991762000, isHistoryIdDecreasing: true, isDateDecreasing: true
87/100: historyId: 15289402, date: 1718991761000, isHistoryIdDecreasing: true, isDateDecreasing: true
88/100: historyId: 15289338, date: 1718991702000, isHistoryIdDecreasing: true, isDateDecreasing: true
89/100: historyId: 15289249, date: 1718991700000, isHistoryIdDecreasing: true, isDateDecreasing: true
90/100: historyId: 15289156, date: 1718991369000, isHistoryIdDecreasing: true, isDateDecreasing: true
91/100: historyId: 15297775, date: 1718991164000, isHistoryIdDecreasing: false, isDateDecreasing: true
92/100: historyId: 15297396, date: 1718991124000, isHistoryIdDecreasing: true, isDateDecreasing: true
93/100: historyId: 15297463, date: 1718989132000, isHistoryIdDecreasing: false, isDateDecreasing: true
94/100: historyId: 15297463, date: 1718989129000, isHistoryIdDecreasing: true, isDateDecreasing: true
95/100: historyId: 15288682, date: 1718987737000, isHistoryIdDecreasing: true, isDateDecreasing: true
96/100: historyId: 15288603, date: 1718986405000, isHistoryIdDecreasing: true, isDateDecreasing: true
97/100: historyId: 15288488, date: 1718982653000, isHistoryIdDecreasing: true, isDateDecreasing: true
98/100: historyId: 15288464, date: 1718982414000, isHistoryIdDecreasing: true, isDateDecreasing: true
99/100: historyId: 15288464, date: 1718982411000, isHistoryIdDecreasing: true, isDateDecreasing: true
100/100: historyId: 15297373, date: 1718980504000, isHistoryIdDecreasing: false, isDateDecreasing: true
{ isHistoryIdDecreasing: false, isDateDecreasing: false }

Summary

As you can see, the message internalDate is sometimes ... increasing. I've observed the same behavior for users.threads.list. In such a scenario:

1. What is described here: https://developers.google.com/gmail/api/guides/sync is not true, I can't query for the first message of users.messages.list, remember its historyId and assume this is the newest historyId, as this might NOT be the newest message.

2. What is described here: https://developers.google.com/gmail/api/reference/rest/v1/users.history/list (History IDs increase chronologically but are not contiguous with random gaps in between valid IDs) is not true, as you can see, historyIDs are sometimes increasing for older messages.

[wdanilo]

Also, here is a code checking the same broken behavior for users.threads.list:

Code

async test2() {
    console.log(
      "Performing test if 'users.threads.list' returns threads sorted by decreasing dates and historyIDs of their first messages.",
    )
    const count = 100
    const cfg: GmailApi.Params$Resource$Users$Threads$List = {
      userId: 'me',
      maxResults: count,
    }
    const resp = await Auth.catchAllOAuth(() => this.api.users.threads.list(cfg))
    if (resp.isErr()) {
      console.error('GMail API query error.', resp.error)
      return
    }
    const list = resp.value.data.threads
    if (list == null) {
      console.error('GMail API invalid response.')
      return
    }
    let isHistoryIdDecreasing = true
    let isDateDecreasing = true
    let prevHistoryId: null | number = null
    let prevDate: null | number = null
    let i = 0
    for (const thread of list) {
      i++
      const id = thread.id
      if (id == null) {
        console.error('Thread is missing the "id" field.')
        return
      }
      const threadQuery = {userId: 'me', id}
      const threadResp = await Auth.catchAllOAuth(() => this.api.users.threads.get(threadQuery))
      if (threadResp.isErr()) {
        console.error('GMail API query error.', threadResp.error)
        return
      }
      if (threadResp.value.data.messages == null) {
        console.error('Thread is missing the "messages" field.')
        return
      }
      const firstMessage = threadResp.value.data.messages[0]
      if (firstMessage == null) {
        console.error('Thread is missing the first message.')
        return
      }
      const historyIdStr = firstMessage.historyId
      const dateStr = firstMessage.internalDate
      if (!historyIdStr || !dateStr) {
        console.error('First message is missing the "historyId" or "internalDate" field.')
        return
      }
      const historyId = parseInt(historyIdStr, 10)
      const date = parseInt(dateStr, 10)
      let localIsHistoryIdDecreasing = true
      let localIsDateDecreasing = true
      if (prevHistoryId != null && prevHistoryId < historyId) {
        isHistoryIdDecreasing = false
        localIsHistoryIdDecreasing = false
      }
      if (prevDate != null && prevDate < date) {
        isDateDecreasing = false
        localIsDateDecreasing = false
      }
      prevHistoryId = historyId
      prevDate = date
      console.log(
        `${i}/${count}: historyId: ${historyId}, date: ${date}, isHistoryIdDecreasing: ${localIsHistoryIdDecreasing}, isDateDecreasing: ${localIsDateDecreasing}`,
      )
    }
    console.log({isHistoryIdDecreasing, isDateDecreasing})
  }

Output:

Performing test if 'users.threads.list' returns threads sorted by decreasing dates and historyIDs of their first messages.
1/100: historyId: 15297863, date: 1719191761000, isHistoryIdDecreasing: true, isDateDecreasing: true
2/100: historyId: 15297367, date: 1719170805000, isHistoryIdDecreasing: true, isDateDecreasing: true
3/100: historyId: 15297189, date: 1719159861000, isHistoryIdDecreasing: true, isDateDecreasing: true
4/100: historyId: 15297049, date: 1719150301000, isHistoryIdDecreasing: true, isDateDecreasing: true
5/100: historyId: 15297132, date: 1719149574000, isHistoryIdDecreasing: false, isDateDecreasing: true
6/100: historyId: 15296597, date: 1719147670000, isHistoryIdDecreasing: true, isDateDecreasing: true
7/100: historyId: 15297036, date: 1719122788000, isHistoryIdDecreasing: false, isDateDecreasing: true
8/100: historyId: 15296855, date: 1719120755000, isHistoryIdDecreasing: true, isDateDecreasing: true
9/100: historyId: 15297511, date: 1719118502000, isHistoryIdDecreasing: false, isDateDecreasing: true
10/100: historyId: 15295111, date: 1719027583000, isHistoryIdDecreasing: true, isDateDecreasing: true
11/100: historyId: 15296007, date: 1719103434000, isHistoryIdDecreasing: false, isDateDecreasing: false
12/100: historyId: 15297035, date: 1719094618000, isHistoryIdDecreasing: false, isDateDecreasing: true
13/100: historyId: 15295569, date: 1719057377000, isHistoryIdDecreasing: true, isDateDecreasing: true
14/100: historyId: 15295489, date: 1719077834000, isHistoryIdDecreasing: true, isDateDecreasing: false
15/100: historyId: 15297034, date: 1719077423000, isHistoryIdDecreasing: false, isDateDecreasing: true
16/100: historyId: 15295374, date: 1719076577000, isHistoryIdDecreasing: true, isDateDecreasing: true
17/100: historyId: 15297544, date: 1719070375000, isHistoryIdDecreasing: false, isDateDecreasing: true
18/100: historyId: 15297032, date: 1719069634000, isHistoryIdDecreasing: true, isDateDecreasing: true
19/100: historyId: 15297569, date: 1719058421000, isHistoryIdDecreasing: false, isDateDecreasing: true
20/100: historyId: 15297030, date: 1719053485000, isHistoryIdDecreasing: true, isDateDecreasing: true
21/100: historyId: 15295971, date: 1719044954000, isHistoryIdDecreasing: true, isDateDecreasing: true
22/100: historyId: 15297029, date: 1719039969000, isHistoryIdDecreasing: false, isDateDecreasing: true
23/100: historyId: 15297429, date: 1719039871000, isHistoryIdDecreasing: false, isDateDecreasing: true
24/100: historyId: 15297027, date: 1719033506000, isHistoryIdDecreasing: true, isDateDecreasing: true
25/100: historyId: 15294309, date: 1719031329000, isHistoryIdDecreasing: true, isDateDecreasing: true
26/100: historyId: 15297593, date: 1719026659000, isHistoryIdDecreasing: false, isDateDecreasing: true
27/100: historyId: 15297025, date: 1719025075000, isHistoryIdDecreasing: true, isDateDecreasing: true
28/100: historyId: 15297024, date: 1719024799000, isHistoryIdDecreasing: true, isDateDecreasing: true
29/100: historyId: 15297618, date: 1719024388000, isHistoryIdDecreasing: false, isDateDecreasing: true
30/100: historyId: 15297022, date: 1719021288000, isHistoryIdDecreasing: true, isDateDecreasing: true
31/100: historyId: 15297642, date: 1719021008000, isHistoryIdDecreasing: false, isDateDecreasing: true
32/100: historyId: 15293469, date: 1719021069000, isHistoryIdDecreasing: true, isDateDecreasing: false
33/100: historyId: 15297671, date: 1719008653000, isHistoryIdDecreasing: false, isDateDecreasing: true
34/100: historyId: 15293217, date: 1719008487000, isHistoryIdDecreasing: true, isDateDecreasing: true
35/100: historyId: 15297705, date: 1719008444000, isHistoryIdDecreasing: false, isDateDecreasing: true
36/100: historyId: 15293049, date: 1719007821000, isHistoryIdDecreasing: true, isDateDecreasing: true
37/100: historyId: 15292962, date: 1719005719000, isHistoryIdDecreasing: true, isDateDecreasing: true
38/100: historyId: 15297739, date: 1719004759000, isHistoryIdDecreasing: false, isDateDecreasing: true
39/100: historyId: 15293296, date: 1719004141000, isHistoryIdDecreasing: true, isDateDecreasing: true
40/100: historyId: 15292645, date: 1719002715000, isHistoryIdDecreasing: true, isDateDecreasing: true
41/100: historyId: 15290059, date: 1718993022000, isHistoryIdDecreasing: true, isDateDecreasing: true
42/100: historyId: 15292507, date: 1718994153000, isHistoryIdDecreasing: false, isDateDecreasing: false
43/100: historyId: 15296270, date: 1719000410000, isHistoryIdDecreasing: false, isDateDecreasing: false
44/100: historyId: 15296304, date: 1719000210000, isHistoryIdDecreasing: false, isDateDecreasing: true
45/100: historyId: 15291662, date: 1718999355000, isHistoryIdDecreasing: true, isDateDecreasing: true
46/100: historyId: 15293696, date: 1719000158000, isHistoryIdDecreasing: false, isDateDecreasing: false
47/100: historyId: 15292133, date: 1719000132000, isHistoryIdDecreasing: true, isDateDecreasing: true
48/100: historyId: 15291968, date: 1718999917000, isHistoryIdDecreasing: true, isDateDecreasing: true
49/100: historyId: 15291886, date: 1718999562000, isHistoryIdDecreasing: true, isDateDecreasing: true
50/100: historyId: 15291744, date: 1718999417000, isHistoryIdDecreasing: true, isDateDecreasing: true
51/100: historyId: 15291461, date: 1718998798000, isHistoryIdDecreasing: true, isDateDecreasing: true
52/100: historyId: 15291384, date: 1718998480000, isHistoryIdDecreasing: true, isDateDecreasing: true
53/100: historyId: 15291302, date: 1718998079000, isHistoryIdDecreasing: true, isDateDecreasing: true
54/100: historyId: 15291164, date: 1718997874000, isHistoryIdDecreasing: true, isDateDecreasing: true
55/100: historyId: 15291019, date: 1718997022000, isHistoryIdDecreasing: true, isDateDecreasing: true
56/100: historyId: 15290937, date: 1718996810000, isHistoryIdDecreasing: true, isDateDecreasing: true
57/100: historyId: 15290677, date: 1718995321000, isHistoryIdDecreasing: true, isDateDecreasing: true
58/100: historyId: 15290510, date: 1718994605000, isHistoryIdDecreasing: true, isDateDecreasing: true
59/100: historyId: 15290371, date: 1718994196000, isHistoryIdDecreasing: true, isDateDecreasing: true
60/100: historyId: 15290592, date: 1718994167000, isHistoryIdDecreasing: false, isDateDecreasing: true
61/100: historyId: 15290141, date: 1718993384000, isHistoryIdDecreasing: true, isDateDecreasing: true
62/100: historyId: 15289917, date: 1718992717000, isHistoryIdDecreasing: true, isDateDecreasing: true
63/100: historyId: 15289836, date: 1718992611000, isHistoryIdDecreasing: true, isDateDecreasing: true
64/100: historyId: 15289755, date: 1718992550000, isHistoryIdDecreasing: true, isDateDecreasing: true
65/100: historyId: 15289673, date: 1718992117000, isHistoryIdDecreasing: true, isDateDecreasing: true
66/100: historyId: 15289532, date: 1718992105000, isHistoryIdDecreasing: true, isDateDecreasing: true
67/100: historyId: 15289249, date: 1718991700000, isHistoryIdDecreasing: true, isDateDecreasing: true
68/100: historyId: 15289156, date: 1718991369000, isHistoryIdDecreasing: true, isDateDecreasing: true
69/100: historyId: 15297775, date: 1718991164000, isHistoryIdDecreasing: false, isDateDecreasing: true
70/100: historyId: 15297396, date: 1718991124000, isHistoryIdDecreasing: true, isDateDecreasing: true
71/100: historyId: 15297463, date: 1718989129000, isHistoryIdDecreasing: false, isDateDecreasing: true
72/100: historyId: 15288682, date: 1718987737000, isHistoryIdDecreasing: true, isDateDecreasing: true
73/100: historyId: 15288603, date: 1718986405000, isHistoryIdDecreasing: true, isDateDecreasing: true
74/100: historyId: 15288488, date: 1718982653000, isHistoryIdDecreasing: true, isDateDecreasing: true
75/100: historyId: 15288464, date: 1718982411000, isHistoryIdDecreasing: true, isDateDecreasing: true
76/100: historyId: 15297373, date: 1718980504000, isHistoryIdDecreasing: false, isDateDecreasing: true
77/100: historyId: 15288268, date: 1718978327000, isHistoryIdDecreasing: true, isDateDecreasing: true
78/100: historyId: 15288267, date: 1718971574000, isHistoryIdDecreasing: true, isDateDecreasing: true
79/100: historyId: 15293754, date: 1718909485000, isHistoryIdDecreasing: false, isDateDecreasing: true
80/100: historyId: 15287783, date: 1718956922000, isHistoryIdDecreasing: true, isDateDecreasing: false
81/100: historyId: 15287862, date: 1718956695000, isHistoryIdDecreasing: false, isDateDecreasing: true
82/100: historyId: 15288266, date: 1718954346000, isHistoryIdDecreasing: false, isDateDecreasing: true
83/100: historyId: 15287564, date: 1718953202000, isHistoryIdDecreasing: true, isDateDecreasing: true
84/100: historyId: 15287478, date: 1718935413000, isHistoryIdDecreasing: true, isDateDecreasing: true
85/100: historyId: 15287253, date: 1718924448000, isHistoryIdDecreasing: true, isDateDecreasing: true
86/100: historyId: 15287477, date: 1718920072000, isHistoryIdDecreasing: false, isDateDecreasing: true
87/100: historyId: 15287061, date: 1718917993000, isHistoryIdDecreasing: true, isDateDecreasing: true
88/100: historyId: 15286956, date: 1718909913000, isHistoryIdDecreasing: true, isDateDecreasing: true
89/100: historyId: 15286786, date: 1718904398000, isHistoryIdDecreasing: true, isDateDecreasing: true
90/100: historyId: 15286705, date: 1718898024000, isHistoryIdDecreasing: true, isDateDecreasing: true
91/100: historyId: 15286624, date: 1718900423000, isHistoryIdDecreasing: true, isDateDecreasing: false
92/100: historyId: 15295890, date: 1718857229000, isHistoryIdDecreasing: false, isDateDecreasing: true
93/100: historyId: 15286444, date: 1718883913000, isHistoryIdDecreasing: true, isDateDecreasing: false
94/100: historyId: 15287475, date: 1718879255000, isHistoryIdDecreasing: false, isDateDecreasing: true
95/100: historyId: 15286284, date: 1718876932000, isHistoryIdDecreasing: true, isDateDecreasing: true
96/100: historyId: 15287493, date: 1718859740000, isHistoryIdDecreasing: false, isDateDecreasing: true
97/100: historyId: 15286009, date: 1718853779000, isHistoryIdDecreasing: true, isDateDecreasing: true
98/100: historyId: 15285920, date: 1718839427000, isHistoryIdDecreasing: true, isDateDecreasing: true
99/100: historyId: 15285840, date: 1718831857000, isHistoryIdDecreasing: true, isDateDecreasing: true
100/100: historyId: 15285787, date: 1718829287000, isHistoryIdDecreasing: true, isDateDecreasing: true
{ isHistoryIdDecreasing: false, isDateDecreasing: false }

[wdanilo]

Also, to make it more visual, I plotted the time I get from users.messages.list. As you can see, there is a drop, so we got 2 messages with date way earlier than messages we get after it. X-axis is the message # from the users.messages.list. Y-axis is its internalDate:

And here is plot of historyId. X-axis is the message # from the users.messages.list. Y-axis is its historyId. So, in contrast to what Google docs state, historyId does not increase chronologically.

[wdanilo]

What is even worse, it appears that users.history.list does not provide results in chronological order neither, despite Google claims so (https://developers.google.com/gmail/api/reference/rest/v1/users.history/list) (citation: Lists the history of all changes to the given mailbox. History results are returned in chronological order (increasing historyId).). Below is a code to test it:

Code

async test_invariants_history_list() {
    console.log(
      "Performing test if 'users.history.list' returns history records sorted by increasing update date.",
    )
    const viz = (t: boolean) => (t ? '✅' : '❌')
    const count = 100
    const cfg: GmailApi.Params$Resource$Users$Messages$List = {
      userId: 'me',
      maxResults: count,
    }
    const resp = await Auth.catchAllOAuth(() => this.api.users.messages.list(cfg))
    if (resp.isErr()) {
      console.error('GMail API query error.', resp.error)
      return
    }
    const list = resp.value.data.messages
    if (list == null) {
      console.error('GMail API invalid response.')
      return
    }
    const lastMessage = list[list.length - 1]
    if (lastMessage == null) {
      console.error('Last message is missing.')
      return
    }
    const lastMessageId = lastMessage.id
    if (lastMessageId == null) {
      console.error('Last message is missing the "id" field.')
      return
    }
    const lastMessageQuery = {userId: 'me', id: lastMessageId}
    const lastMessageResp = await Auth.catchAllOAuth(() =>
      this.api.users.messages.get(lastMessageQuery),
    )
    if (lastMessageResp.isErr()) {
      console.error('GMail API query error.', lastMessageResp.error)
      return
    }
    const startHistoryId = lastMessageResp.value.data.historyId
    if (startHistoryId == null) {
      console.error('Last message is missing the "historyId" field.')
      return
    }
    const historyCfg: GmailApi.Params$Resource$Users$History$List = {
      userId: 'me',
      startHistoryId,
      maxResults: 500,
    }
    const historyResp = await Auth.catchAllOAuth(() => this.api.users.history.list(historyCfg))
    if (historyResp.isErr()) {
      console.error('GMail API query error.', historyResp.error)
      return
    }
    const historyList = historyResp.value.data.history
    if (historyList == null) {
      console.error('GMail API invalid response.')
      return
    }

    let localDateIncreasing = true
    let prevDate: null | number = null
    const updateCount = historyList.length
    let i = 0
    for (const update of historyList) {
      i++
      if (update.messages == null) {
        console.error('Update is missing the "messages" field.')
        return
      }
      let m = 0
      for (const message of update.messages) {
        m++
        if (message.id == null) {
          console.error('Message is missing the "id" field.')
          return
        }
        const messageQuery = {userId: 'me', id: message.id}
        const messageResp = await Auth.catchAllOAuth(() =>
          this.api.users.messages.get(messageQuery),
        )
        if (messageResp.isErr()) {
          console.error('GMail API query error.', messageResp.error)
          continue
        }
        const dateStr = messageResp.value.data.internalDate
        if (!dateStr) {
          console.error('Message is missing the "internalDate" field.')
          return
        }
        const date = parseInt(dateStr, 10)
        if (prevDate != null && prevDate > date) {
          localDateIncreasing = false
        } else {
          localDateIncreasing = true
        }
        prevDate = date
        console.log(
          `${i}/${updateCount}[${m}]: date: ${date}, dateIncreasing: ${viz(localDateIncreasing)}`,
        )
      }
    }
  }

Output

1/232[1]: date: 1718992106000, dateIncreasing: ✅
2/232[1]: date: 1718992117000, dateIncreasing: ✅
3/232[1]: date: 1718992550000, dateIncreasing: ✅
4/232[1]: date: 1718992611000, dateIncreasing: ✅
5/232[1]: date: 1718992717000, dateIncreasing: ✅
6/232[1]: date: 1718992719000, dateIncreasing: ✅
7/232[1]: date: 1718993022000, dateIncreasing: ✅
8/232[1]: date: 1718993384000, dateIncreasing: ✅
9/232[1]: date: 1718994167000, dateIncreasing: ✅
10/232[1]: date: 1718994167000, dateIncreasing: ✅
11/232[1]: date: 1718994167000, dateIncreasing: ✅
12/232[1]: date: 1718994167000, dateIncreasing: ✅
13/232[1]: date: 1718994196000, dateIncreasing: ✅
14/232[1]: date: 1718994479000, dateIncreasing: ✅
15/232[1]: date: 1718994605000, dateIncreasing: ✅
16/232[1]: date: 1718994167000, dateIncreasing: ❌
17/232[1]: date: 1718995295000, dateIncreasing: ✅
18/232[1]: date: 1718995321000, dateIncreasing: ✅
19/232[1]: date: 1718995732000, dateIncreasing: ✅
20/232[1]: date: 1718996035000, dateIncreasing: ✅
21/232[1]: date: 1718996476000, dateIncreasing: ✅
22/232[1]: date: 1718996810000, dateIncreasing: ✅
23/232[1]: date: 1718997022000, dateIncreasing: ✅
24/232[1]: date: 1718997051000, dateIncreasing: ✅
25/232[1]: date: 1718997874000, dateIncreasing: ✅
26/232[1]: date: 1718997877000, dateIncreasing: ✅
27/232[1]: date: 1718998079000, dateIncreasing: ✅
28/232[1]: date: 1718998480000, dateIncreasing: ✅
29/232[1]: date: 1718998798000, dateIncreasing: ✅
30/232[1]: date: 1718999072000, dateIncreasing: ✅
31/232[1]: date: 1718999152000, dateIncreasing: ✅
32/232[1]: date: 1718999355000, dateIncreasing: ✅
33/232[1]: date: 1718999417000, dateIncreasing: ✅
34/232[1]: date: 1718999475000, dateIncreasing: ✅
35/232[1]: date: 1718999562000, dateIncreasing: ✅
36/232[1]: date: 1718999917000, dateIncreasing: ✅
37/232[1]: date: 1719000132000, dateIncreasing: ✅
38/232[1]: date: 1719000132000, dateIncreasing: ✅
39/232[1]: date: 1719000132000, dateIncreasing: ✅
40/232[1]: date: 1719000132000, dateIncreasing: ✅
41/232[1]: date: 1719000132000, dateIncreasing: ✅
42/232[1]: date: 1719000132000, dateIncreasing: ✅
43/232[1]: date: 1719000158000, dateIncreasing: ✅
44/232[1]: date: 1719000212000, dateIncreasing: ✅
45/232[1]: date: 1719000210000, dateIncreasing: ❌
GMail API query error. {
  level: 4,
  message: 'OAuth call failed.',
  data: {
    error: GaxiosError: Requested entity was not found.
        at Gaxios._request (<redacted>)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async UserRefreshClient.requestAsync (<redacted>)
        at async catchAllOAuth (<redacted>)
        at async Sync.test_invariants_history_list (<redacted>)
      response: [Object],
      config: [Object],
      code: 404,
      errors: [Array]
    }
  },
  flags: {}
}
47/232[1]: date: 1719000410000, dateIncreasing: ✅
48/232[1]: date: 1718994153000, dateIncreasing: ❌
49/232[1]: date: 1719001722000, dateIncreasing: ✅
50/232[1]: date: 1719002715000, dateIncreasing: ✅
51/232[1]: date: 1719004141000, dateIncreasing: ✅
52/232[1]: date: 1719004141000, dateIncreasing: ✅
53/232[1]: date: 1719004141000, dateIncreasing: ✅
54/232[1]: date: 1719004141000, dateIncreasing: ✅
55/232[1]: date: 1719004759000, dateIncreasing: ✅
56/232[1]: date: 1719005719000, dateIncreasing: ✅
57/232[1]: date: 1719007821000, dateIncreasing: ✅
58/232[1]: date: 1719008444000, dateIncreasing: ✅
59/232[1]: date: 1719008487000, dateIncreasing: ✅
60/232[1]: date: 1719008653000, dateIncreasing: ✅
61/232[1]: date: 1719021008000, dateIncreasing: ✅
62/232[1]: date: 1719021069000, dateIncreasing: ✅
63/232[1]: date: 1719021206000, dateIncreasing: ✅
64/232[1]: date: 1719021288000, dateIncreasing: ✅
65/232[1]: date: 1719000158000, dateIncreasing: ❌
66/232[1]: date: 1719000158000, dateIncreasing: ✅
67/232[1]: date: 1719000158000, dateIncreasing: ✅
68/232[1]: date: 1718909485000, dateIncreasing: ❌

[wdanilo]

Moreover, as you can see in the above test, users.history.list can return records, that contain invalid message ids (the GMail API query error indicates so and I can reproduce it every time).

[ego]

I also got the same case, and my sync process was broken due to these inconvenients.
We periodically have missed this kind of messages.
Any updates? Any API param for order?

[sofisl]

Hi there! Thanks for the writeup @wdanilo! As you mentioned this is likely an API issue. As this is mostly a generated library unfortunately we don't have much pull to help. I'll convert this to a discussion so people can follow along!