Official MQTT broker having connection issues

[brianshea2]

The meshmap.net connection to the official MQTT broker has been disconnecting and reconnecting every couple minutes for days now, causing map updates to be delayed or missed entirely, with many nodes no longer showing. This is not the first occurrence of the public broker being unstable for an extended period of time. What can we do to fix this and prevent it from happening again?

[fsgh42]

• first and foremost: running public services like this one obviously requires some dedication. Servers must be paid and kept up to date, outages should be addressed and for this you need humans that have the time to do these things. Ensure enough people and monitoring are in place and have a overall strategy on how to run the system. If either one is a problem, ask the community for help.
• spread out the deployment across multiple data centers. might not only make the message broker resilient, helps improve client connectivity stability as well and avoids sending data around the globe. requires a distributed software setup, though. not easy.
• I doubt that the actual load on the server was too terribly high, but the system could of course go down due to resource exhaustion. Over time, the system might need more suitably sized machines, as meshtastic grows in device numbers using the message broker.
• redesign the system to be a true distributed network of MQTT brokers, mostly run by community members. Not easy, but not impossible either.

All of the above can be also be tackled by simply paying a company for handling all of these, by using a SaaS solution. This is usually of course significantly easier, but has a price tag, of course.

I can offer to talk to whoever is running the message broker and see if and how the community could help with the above.

[blue4world]

I’m also experiencing MQTT connection issues—both on the iOS app and when connecting from a desktop using the acidvegas/meshtastic-mqtt-json client. The frequent disconnects are making the system difficult to use reliably.

I’m happy to help out. It would be great to get in touch with whoever is currently maintaining the MQTT server so we can coordinate efforts and determine what kind of support is needed—whether it’s infrastructure, monitoring, or other forms of community involvement.

[frantsch]

... for quite good money, of course!

[RCDudefpv]

My Two main nodes went of the maps few days ago, i seen most of the local nodes in Scotland all disappear from the maps a few days later, any help i can give , give me a shout

[spmedia]

If I can help in any way, like spinning up a server in a certain geo for it, let me know.

Lets get these maps 🗺️ fixed!

[fscheps]

Hi there, exactly we can buy a lifetime Zap Hosting Server and use that

[gkowalski]

Well switching to https://meshtastic.liamcottle.net/ as that system has no issues

[garthvh]

Messaging does not work and your data is stored there permanently, so there are some issues.

[RCGV1]

Data is deleted within 1 week on his server

[drveni]

mqtt server is down. just ping and no response

[evenstephen]

CloudFlare has their new Pub/Sub service currently in private beta and it is actually MQTT under the hood. Global scaling is their bread and butter. With the visibility of the Meshtastic project I wonder if they’d be willing to become a partner/sponsor?

[fsgh42]

good idea

[fifieldt]

@evenstephen , just to keep you updated - we did take this suggestion and have recently become connected with them. Not sure anything will come of it, but just wanted to let you know.

[Marusame00]

meshmap.net redirects to this very thread, thought I was going insane for a moment but it seems to be the case.

[brianshea2]

I have the website redirecting here while the MQTT downtime persists. Otherwise you'd be looking at an empty map right now 😆

[brianshea2]

Can anyone from Meshtastic chime in here? It's been well over a week now since the MQTT service has gone down.

[thebentern]

Can anyone from Meshtastic chime in here? It's been well over a week now since the MQTT service has gone down.

Hi Brian,

After migrating to a new Droplet in Digital Ocean, ironically with both more compute and memory resources we have seen a lot of network connectivity issues with the new host. Everything is configured identically, and the broker is running and pub/subbing hundreds of thousands of packets a day. As of this moment, we are puzzled why the connections have been more sluggish / unreliable on this new infra, because everything is configured identically. I will continue to investigate as I get time, but the default public broker has always been something we have hosted for free as a starting point for folks to test and experiment. I would not rely on it for anything mission critical. I may attempt to create a new droplot on another region to rule out any issues there.

[geeksville]

hi @thebentern (I know I'm not an active dev by any means but...)

I forget how the MQTT server was setup when I handed it off to MC. And I'm not sure how ya'll have evolved it since. Is the meshtastic MQTT server already 'dockerized' (i.e. an easy to configure deploy docker image)?

If not, I've been doing a lot lately with docker and I'd probably be able to sign up for this. I could make a nice clean image that is guaranteed deployable (with all dependencies and nice volume support for critical state) with a single command. Then it would probably be easier for you (or anyone else who wants to host a meshtastic configed MQTT broker.

I have some time this week if it would help. If not useful no pressure.

You rock btw!

[brianshea2]

Just to reiterate, since I was surprised to hear the broker is up and running in a new instance, I've been unable to connect at all for several days. Each attempt times out.

[andrewthetechie]

Hi @thebentern, I'd suggest reaching out to DigitalOcean support to have then check their networking or to check for noisy neighbors.

Feel free to reach out if you need some assistance, I've got some expertise in running MQTT and other message queues in the cloud.

[TheITJuggler]

Long time UNIX admin, Docker aficionado, and MQTT mid-level.. Happy to help out if needed.

[brianshea2]

MeshMap.net has always been a fun side project for me, originally started to fill my own want of having an online map when there was none (besides self-reported), and continuing to serve primarily, at least in my mind, as a way to promote adoption.

That being said, I have no strong opinions about how the map should be run. My goal was to create a simple, live, low-noise map of active nodes around the world. That last part, I think, is where my and Liam's goals differ the most, as nodes on MeshMap.net completely disappear after no location updates for 24 hours, and I've considered even lowering that. I think having two (or more) different maps to choose from is great, but I do wish there was a single source of truth for such a global node map. I'm not personally a fan of Liam's choice to stand up a separate MQTT broker for his map, as that has led to disjunct data sets. I don't blame him, our maps have different goals, and the recent privacy enhancements to the official MQTT service certainly did limit the usefulness of these maps. I decided to follow along with Meshtastic's and, hopefully, the wider community's wishes.

So now I wonder, what are Meshtastic's and the community's wants for an online map? Should a global map pulling from the project's "starting point for folks to test and experiment" MQTT broker even exist? Do people want a separate MQTT broker for the map, similar to Liam's? Should I refocus the MeshMap project to be more easily self-hosted for smaller community meshes? Heck, I'd even consider handing over the site to the Meshtastic project if wanted, though I fully believe they'd have no problem creating their own, and one would already exist, if that were the case. Or, should MeshMap continue as-is, assuming the official MQTT broker gets fixed?

[CamFlyerCH]

I would realy like to have the default MQTT broker back running again and the map was fine exactly as it was. What I think is a real good differnece of Liam's implementation of the MQTT broker, was that it was only there for the map and you can not subscribe and read all the messages. For the probably only few people that used the default MQTT broker to connect different networks, I think a self-hosted dedicated MQTT broker is the better solution. And you can still push your local MQTT data to the public one to find your nodes also on the map. Perhaps this would also reduce the load on the public broker a lot and make it more stable.

[cpatulea]

I believe also a significant difference between the two maps is the handling of MAP_REPORT_APP packets (*). I'm not confident of my reading of the code, so I don't want to say something possibly wrong, but there are differences.

(*) I believe this is the feature which allows sending map reports over MQTT (https://meshtastic.org/docs/configuration/module/mqtt/#map-reporting-enabled) without otherwise enabling general uplink of all packets. Personally I prefer a mode like this for my nodes.

[frantsch]

I am with @CamFlyerCH, @brianshea2! Get meshmap up and running again as it was as soon as possible, please! And subsequently you might lead a serious discussion on omitting the network messaging functionality as suggested, which I agree on.

[brianshea2]

To the Meshtastic folks, maybe it's possible to add a special map-only root topic that does not relay messages?

I still think there's general confusion around the Map Reporting toggle, as in it does not act as a gateway node for other nodes to be mapped.

[jp-bennett]

@brianshea2 Can you check if you're still having problems? I rolled some Linux tuning parameters out at about noon Central time today (an hour and a half ago) and the server seems to be much more responsive now. I think there is absolutely some thinking to be done about how the MQTT server needs to evolve in the future, but I want to get this current technical showstopper fixed first.

[jp-bennett]

I'm racking up my blocked node list. Is there anything that can be done on your side to kick off these nodes sending multiples per second?

Some sort of fail2ban style plugin seems appropriate here. I'll see if we can do anything about it.

[brianshea2]

Quick update, still getting disconnected and timing out frequently. When it connects, a flood of Map Reports come in (only from a handful of nodes), then generally loses connection a few minutes later.

[tsalisbury0]

I deployed the MeshSTL MQTT server to help offload from the public one. Still working on getting the map working (currently looking for a cheap Docker host to run the scripts needed to run the push to the map). If you need connection details I can get them to you.

[garthvh]

I deployed the MeshSTL MQTT server to help offload from the public one. Still working on getting the map working (currently looking for a cheap Docker host to run the scripts needed to run the push to the map). If you need connection details I can get them to you.

The official server is running, if you are not zero hopping packets the Lora mesh will quickly be flooded.

[tsalisbury0]

I deployed the MeshSTL MQTT server to help offload from the public one. Still working on getting the map working (currently looking for a cheap Docker host to run the scripts needed to run the push to the map). If you need connection details I can get them to you.

The official server is running, if you are not zero hopping packets the Lora mesh will quickly be flooded.

I'm getting errors connecting to the main on via MQTT explorer.

[brianshea2]

Website is temporarily redirecting here again. I've been unable to connect to the MQTT service at all for a couple days.

[trickv]

Still in a connect-retry loop now an hour later. Not been able to establish a single connection.

LMK if I can help more!

[jp-bennett]

We just swung the DNS entry to a new droplet that's running the daemon outside of Docker. So far it appears to be performing much better. Very interested to hear if it's actually working as expected now.

[brianshea2]

Looking promising so far. I'll monitor for a bit and switch the map back on if it holds steady.

[trickv]

Anecdotally seems OK too. My little subscriber app has registered 833k messages in ten hours which is a lot more than I was getting, which would indicate lots of nodes in the world have connected. But that’s anecdotal.

here’s something encouraging: uptimerobot says it’s been available for a solid ten hours straight: https://stats.uptimerobot.com/KB1QJHDnJx/800415559

this doesn’t perform a full MQTT subscribe and read any data, but this means that at least the base TCP layer of the app is reasonably healthy.

Looks like good progress @jp-bennett !

(still happy to lend a hand if you want - getting it running inside Docker shouldn’t be a constraint…)

[brianshea2]

Looking stable over the last day from MeshMap's side. Thank you, Meshtastic folks :)

Circling back to a previous thought, any comments around having a special root topic that does not relay text messages? Or maybe a new firmware option? Would people find this useful? I personally set up a gateway node with channel uplink only to populate the map. As it is now, all messages get uplinked with no chance of seeing any response.

[Marusame00]

MQTT is seemingly up right now, longer than it has been in two weeks so maybe the issue is finally fixed fingers crossed?

[moored57]

Question: will the meshtastic.org mqtt server only work with the longfast lora modem preset settings? If I were to set up a group of friends on the lora LongMod modem presets, would it still provide mqtt service?

[garthvh]

Yep

[moored57]

thanks!!!

[brianshea2]

This is occurring again, unstable connections over the last week or so.

[fifieldt]

@brianshea2 Public server update: about 12 hours ago we updated a bunch of major dependencies, after discovering one claimed some core performance improvements. It's looking good so far...

[brianshea2]

meshmap.net hasn't seen a disconnect in ~16 hours, looking good!

[brianshea2]

unstable again the last couple days

[fifieldt]

@brianshea2 , thanks for the report - I've restarted the broker for you.

We think we've identified the source of something akin to a memory leak and have coded a workaround. Going to try applying that shortly.

[fifieldt]

FYI meshtastic/dotnet-MQTTnet#1