Full Imports failing when lots of external Domain Shared Contacts

[IAmStevenJohnson]

Ryan,

We have tenants that have a lot of external domain shared contacts and we've been unable to do full imports on those MAs for the past week or so. Gemini thinks the solution is to request fewer contacts at a time. More detail below, but my question is how feasible is this? Let me know if you want more detail on the problem, or Gemini's suggestions. Thanks so much.

We get this error in the event viewer:
The extensible extension returned an unsupported error.
The stack trace is:
"System.AggregateException: One or more errors occurred. ---> Google.GData.Client.GDataRequestException: Execution of request failed: https://www.google.com/m8/feeds/contacts/DOMAIN/full?max-results=1000 ---> System.Net.WebException: The remote server returned an error: (504) Gateway Timeout.
at System.Net.HttpWebRequest.GetResponse()
at Google.GData.Client.GDataRequest.Execute()
--- End of inner exception stack trace ---
at Lithnet.GoogleApps.MA.ManagementAgent.<>c__DisplayClass40_0.b__0(Task1 _) in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\ManagementAgent.cs:line 380 at System.Threading.Tasks.Task.Execute() --- End of inner exception stack trace --- at Lithnet.GoogleApps.MA.ManagementAgent.GetImportEntriesFull() in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\ManagementAgent.cs:line 481 at Lithnet.GoogleApps.MA.ManagementAgent.GetImportEntries(GetImportEntriesRunStep importRunStep) in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\ManagementAgent.cs:line 399 ---> (Inner Exception #0) Google.GData.Client.GDataRequestException: Execution of request failed: https://www.google.com/m8/feeds/contacts/DOMAIN/full?max-results=1000 ---> System.Net.WebException: The remote server returned an error: (504) Gateway Timeout. at System.Net.HttpWebRequest.GetResponse() at Google.GData.Client.GDataRequest.Execute() --- End of inner exception stack trace --- at Lithnet.GoogleApps.MA.ManagementAgent.<>c__DisplayClass40_0.<OpenImportConnectionFull>b__0(Task1 _) in D:\dev\git\lithnet\googleapps-managementagent\src\Lithnet.GoogleApps.MA\ManagementAgent.cs:line 380
at System.Threading.Tasks.Task.Execute()<---
Forefront Identity Manager 4.6.607.0"

Gemini has some input on the cause being the old contact API infrastructure. It has a lot more detail on the solution but, in summary:
The Real Problem: Legacy Infrastructure Timeout
The error 504 Gateway Timeout with max-results=1000 indicates a performance failure on Google's backend, not a configuration error on your side.
The Bottleneck: You are hitting the legacy GData endpoint (https://www.google.com/m8/feeds/contacts/DOMAIN/full).
The Cause: The Lithnet MA is requesting 1,000 contacts per page (the default maximum). Because this API is running on older, less optimized infrastructure, it frequently struggles to return 1,000 full XML entries within the standard timeout window (usually 100 seconds), especially if the contacts contain photos or extended properties.
The Result: Google's frontend (the "Gateway") waits for the backend to gather the data. The backend takes too long, and the Gateway cuts the connection, returning the 504 error.
How to Fix It
Since you cannot force Google to speed up their legacy API, you must force the Management Agent to be "gentler" with its requests.
...
Recommended Value: Change this from 1000 to 100 or 200.
Why: Requesting 100 entries is much faster for the legacy backend to process, avoiding the timeout. It will result in more round-trips, but the import will actually complete.

[ryannewington]

Hi Steve,

The page size is hardcoded at the moment. Is this just another transient google issue thought? We see weird stuff all the time that eventually resolves itself?

If it's causing production problems, I can look at adding a parameter for this setting, but I'll need a few days at least to sort this out.

[IAmStevenJohnson]

Ryan,

It hasn't caused any production issues yet that I know of. We normally run a full import once/day and, fortunately, our tenants that have a lot of contacts don't have a lot of managed user accounts. Which would be more important, as user objects have more changes done outside of the MA that we'd need to pick up with a full import. I don't think anyone makes manual changes to contact objects.

And I know what you mean about transient Google issues eventually resolving themselves. But when I see them they are usually somewhat intermittent. This issue isn't intermittent.

Like I said, this hasn't caused any issues yet but I wanted to let you know about it and see how feasible/difficult it would be to possibly make a change for it. I check it daily and will definitely let you know if it corrects itself.

And obviously I can't say for sure that Gemini is correct in its assessment of the problem and solution. But since it's not intermittent, I can definitely test a solution and will know for sure if it works.

Thanks so much for your consideration of this issue.

[ryannewington]

Added a config option for this in v2.2.2

You'll need to set the page size limit in the miiserver.exe.config file.

Add the new <section...> value shown below within the existing <configSections> element, or create the <configSections> element if it doesnt exist. Then add the complete <lithnet-google-ma> section as shown below if it doesn't exist, or modify to include the <contacts-api> element and the new page-size value. The default is 1000 if not set.

<configuration>
  <configSections>
    <section name="lithnet-google-ma" type="Lithnet.GoogleApps.MA.MAConfigurationSection, Lithnet.GoogleApps.MA" />
  </configSections>

  <lithnet-google-ma>
    <contacts-api page-size="100"/>
  </lithnet-google-ma>

... 
/<configuration>

Full instructions are here https://docs.lithnet.io/google-workspace-ma/configuration/advanced-configuration

[IAmStevenJohnson]

Ryan, thanks so much for doing that. I tested it and found that the page size isn't the problem/solution. I hope it wasn't too much trouble for you to make the changes. These advanced options are a nice feature. I'm curious, do these options, like rate-limit, work during exports, or just imports?

I'll post Gemini's follow-up idea below as an FYI, but we've opened an incident with Google and hoping we get a definitive diagnosis from them. We also asked on the GAM Google group if anyone else has this problem and one person noted that he has started having issues with his 50k contacts using GCDS. I'll post here what comes of the Google incident.

After eliminating paging as the problem, here's what Gemini thinks the problem is now:

The Diagnosis:
The Google Legacy API you are forced to use (GData) is extremely inefficient at handling deleted objects. When you request "10 contacts," the Google backend doesn't just grab the first 10 live contacts. It scans through your entire history of contacts—including potentially thousands of deleted ones—to check if they match your query.

If your organization has deleted or updated Shared Contacts frequently over the years, the backend has to scan so many "tombstones" (deleted records) that it hits the 60-second limit before it finds the 10 live records you asked for.

The Code Fix: Disable Deleted Objects
To fix this, we must explicitly tell Google's backend not to scan the trash. This will make the query exponentially faster, but it means MIM will not see deletions during this import (which is usually an acceptable trade-off for Shared Contacts).

Go back to your code and add query.ShowDeleted = false; You need to find where the query object is created and explicitly set ShowDeleted to false.

[ryannewington]

The rate limits apply to all API operations. They are set to known safe values and haven't really had a reason to tweak them in the past, but they have always been there as options in the backend.

I'd be cautious about following Geminis lead. It's just vomiting out words at this point. It's too neiche of an issue for it to have any internal context as to what's actually happening.

There's likely a broken endpoint impacting your region specifically. We have 100k contacts and not seeing anything over here.

We've seen these things before and they tend to resolve themselves before Google makes any meaningful progress on identifying the cause.