502 Bad Gateway: Registered endpoint failed to handle the request

[mm225022]

I have been using the API for some time to access the publicly available data.

About a month or so ago, I noticed that I would sometimes get a 502 Bad Gateway response.

However, I have been unable to pull data yesterday (4/7/21); I keep getting a 502 Bad Gateway: Registered endpoint failed to handle the request.

The specific URL that I am trying to hit is:
https://api.gsa.gov/analytics/dap/v1.1/domain/healthcare.gov/reports/domain/data?api_key=MY_API_KEY&after=2019-01-01&limit=10000

I have also tried it with api_key=DEMO_KEY and without the date and limit parameters.

I also tried a different government website and I got the 502 error as well.

Today (4/8/21), I tried again. I hit the domain Healthcare.gov this morning and it retrieved data without a hitch.

However, I then tried cuidadodesalud.gov and it came back with the 502 gateway error. I tried it a second time and it bounced with the 502 error again.

To test, I went back and tried to hit healthcare.gov and this time it also returned the 502 gateway error.

[tdlowden]

@ryanwoldatwork tagging you for our discussion later today to possibly troubleshoot this.

[tdlowden]

So far, we haven't been able to pinpoint why this is happening. The 502 definitely seems to be a result of a timeout. But we don't understand why some queries are quick to resolve and others not. Trying to round up help to examine more deeply.

[mm225022]

Thanks Tim. I tried again this morning (just now) and am still getting the 502 error. When you are testing on your end, is the resolution time independent of the domain? I would assume so, since it's bouncing off your backend data and no connection to the underlying domain, but I haven't tested against a broad number of different domains.

[mm225022]

I access the API via R. I closed my RStudio session and restarted and so far I have been able to pull the data without a hitch. I don't know if there was something cached or it has to do with the fact that it's now Friday afternoon and traffic on the API is probably extremely low.

[ryanwoldatwork]

Hi @jonathanmburns, thanks for the update.

Please report if this happens again.
An update to the database has been made to improve performance.
The queries are reliably resolving. I expect this to continue, but let us know next week if you experience otherwise.

[mm225022]

Hi @tdlowden and @ryanwoldatwork

I wanted to let you know that the issue is happening again. I had no problem last week, but I am trying to access the API today and no luck - 502 Bad Gateway issues.

[mm225022]

Hello @tdlowden and @ryanwoldatwork

I wanted to check in with you. I was wondering if you had a chance to look into this. I am trying again to access the API and getting the 502 error.

Thanks

[tdlowden]

Hi Jonathan. We're still trying to assess this with the resources we have. Apologies for the delay, but it's definitely on the radar. Trying to line up time for someone who can examine it in depth.

[mm225022]

Hi @tdlowden and @ryanwoldatwork
I hope you don't mind; I wanted to check back in on this. I tried accessing the API again today and am still getting the 502 error.

Thanks

[mm225022]

Hello @tdlowden and @ryanwoldatwork and @nick-jones-gov:

I've noticed @nick-jones-gov has had a lot of activity related to this on posts #161 and #172. I wanted to give you some feedback on my end.
Yesterday, I ran the process. I set it up in a loop; the first time I tried it errored out with the 502 about 25 times until it finally returned data. Today, I ran it 4 times and returned data 4/4.
So, if @nick-jones-gov, if you did implement the changes that you proposed in #172, thank you! It looks like at least from my end they did the trick. The queries still run slow, but at least now I am consistently getting data.

[nick-jones-gov]

Hi @jonathanmburns! Unfortunately, I can't take credit for that - we haven't changed anything in the production environment yet, so I suspect it was just random chance or a case of the database having some of the data you were requesting recently cached. We are hoping to try something out in the next few days (or early next week latest) that should hopefully both reduce the frequency at which you see 502s, and speed up all requests in general. Please let us know if you keep having issues, and if my proposed fix doesn't do it, we'll continue investigating on our end. Thank you!

[tdlowden]

Hi @jonathanmburns. We have some good news, finally! We think we have a solution to improve performance on the API and therefore eliminate the errors, thanks to what @nick-jones-gov has done in #172. The only hiccup is that when running it locally, we found certain reports no longer list the data in perfect metric descending order (because of the way datapoints are assigned ids). The endpoint is still sorted by date (recent to oldest), but within the date, there may be variation in the sort order.

From our perspective, it is more important to have a functioning API than one that sorts in the same fashion as it initially did, so we intend to make this merge a week from today (I am going to notify others via email). We just wanted to give you a heads up so that you wouldn't be caught off guard next week.

I am also adding to our backlog to create a more long-term fix that will restore sort by date, then value, but I don't have a timetable for when we'll be able to get that done, unfortunately.

[mm225022]

Thanks @tdlowden. That is great news!
I certainly agree with your prioritization. I'm not so concerned about the potential issue with the sort order; it's good to have that heads up but I can obviously can sort the data on my own once I've retrieved it.
I appreciate the work that you and @ryanwoldatwork and @nick-jones-gov have done on this issue!
Thanks again

[nick-jones-gov]

Hi @jonathanmburns - we merged a change this morning that should, in general, speed up most of the API endpoints. The endpoints that filter by domain (like you're using for healthcare.gov) won't see as drastic of a performance increase, but should still be faster than before. There are still longer term things we hope to do to improve the performance in general - but I'm not sure about the timeline there.

In the meantime, you will likely see faster response times if you use a smaller limit - I see that you specified limit=10000, I'm guessing so that you can fetch all the data you need at once. Alternatively, you could specify limit=100, and use the page parameter (explained in more detail in the docs here) to page through the results. In testing on my laptop, I see this endpoint respond in ~5 seconds: https://api.gsa.gov/analytics/dap/v1.1/domain/healthcare.gov/reports/domain/data\?api_key\=DEMO_KEY1\&after\=2019-01-01\&limit\=100

Hope this helps!