MQTT with GSM module

[tadejales]

I am trying to use TTGO T-Call board as MQTT client. I will have this board positioned in remote location where WiFi access is not possible. Is there a known existng MQTT library which I could use. I was searching for it but only found https://github.com/peterhinch/micropython-mqtt/tree/master which to my knowlage is only avaliable with WiFi interface.

I tought it was worth asking if there is a library in the wild or if @peterhinch work can be customized for this need/used with GSM module instead of Wifi.

Thank you upfront!

[glenn20]

You can install a simple mqtt library from micropython-lib using

import mip
mip.install("mqtt.simple")

There is also a variant called "mqtt.robust" - read the docs to see which is right for you.

These libraries assume the device is network connected and don't try to re-establish a wifi connection if network connectivity is lost (as Peter's excellent libraries do).

[tadejales]

Thank you. I have found that in the mean time. The problem that I am facing now is that I am unable to connect to MQTT server(Adafruit) over gprs. I am facing OSError -202 . I have tried with the same credentials with cpython sandbox test and I was succesfull so the problem should be somewhere else.

Print from terminal:
File "umqtt/simple.py", line 67, in connect OSError: -202

My mqtt postman:

class MQTT:

    def __init__(self) -> None:
        self.config = config.get_config("mqtt")
        self.client = None

        # Exec on init
        self.connect()

    def connect(self) -> None:
        self.client = MQTTClient(client_id=self.config["client_id"], 
                                 server=self.config["broker"], 
                                 port=self.config["port"], 
                                 user=self.config["username"], 
                                 password=self.config["password"])
        
        self.client.connect()

But if I understand it right I should be using gsm modem to transmit data.

[Tico06]

Hi @tadejales,

I had a similar issue, I need to leave a node in an isolate area. I found a USB dongle which is LTE capable. I created a WIFI router with a Raspberry PI 4 I was holding around.

If you go this way, you have to double check the dongle is compatible with the bands you plan to use.

Another option is to go with an LTE/WIFI router.

[tadejales]

I do not have that option as I am also constrained by the space.

Another option that I am thinking of is to use pure http requests and just post data that way to the server...but I am not able to authenticate myself to adafruit server because library that I have for sim800L does not have ability to include headers to post request.

This is the library I am using: https://github.com/pythings/Drivers

[tadejales]

@glenn20 Any idea what might be the problem with gprs and mqtt.simple? I even dont have idea where to look at.

[Pawelinode]

Have you established a network connection via PPP ?

[ThinkTransit]

mqtt.robust is an excellent library if you have wifi access or can get a stable ppp connection.

Considering the SIM800 module has inbuilt support for MQTT I would suggest trying this.

https://microchip.ua/simcom/2G/Application%20Notes/SIM800%20Series_MQTT_Application%20Note_V1.03.pdf

[tadejales]

@Pawelinode , @ThinkTransit I am sorry but I have no idea what ppp connection is. I am new to any operation with GSM module. I am using library mentioned above and that is what I am working with(I followed the example in the repo and it works: https://github.com/pythings/Drivers). I would love to use mqtt.robust if possible.

Can you give me an example what I would need to do to make it work?

[Pawelinode]

You have two options:

1. Establish a network connection via PPP. All MQTT features will work similarly to when connected via WiFi.
2. Don't use Micropython libraries for MQTT. Send data using GSM modem AT commands.

[tadejales]

Are you able to give me example how I can connect to PPP. I have find some examples but I was unable to sucesfully establish ppp connection.

I am able to send data via API post request to the Adafruit Io server to simulate publish mqtt functionality, but I am not sure hot to simulate subscribe with API calls.

So if possible I would preffer MQTT library.

[Pawelinode]

How did you try to connect via PPP ?
Show code.

[tadejales]

Sure. This is what I use.

        gsm = UART(1, 9600, timeout=1000, rx=MODEM_TX_PIN, tx=MODEM_RX_PIN)
        time.sleep(1)
        gsm.write("at\r\n")
        print(gsm.readline())
        print(gsm.readline())
        time.sleep(1)
        gsm.write("AT+CPIN?\r\n")
        print(gsm.readline())
        print(gsm.readline())
        time.sleep(1)
        #gsm.write("ate0\r\n")
        #print(gsm.readline())
        #print(gsm.readline())
        #time.sleep(1)
        gsm.write("AT+CREG?\r\n")
        print(gsm.readline())
        print(gsm.readline())
        time.sleep(1)
        gsm.write("AT+CNMI=0,0,0,0,0\r\n")
        print(gsm.readline())
        print(gsm.readline())
        time.sleep(1)
        gsm.write('AT+CGDCONT=1,"IP","globul"\r\n')
        print(gsm.readline())
        print(gsm.readline())
        time.sleep(1)
        gsm.write('AT+CGDATA="PPP",1\r\n')
        time.sleep(1)
        print(gsm.readline())
        print(gsm.readline())

        time.sleep(5)

        import network
        
        ppp=network.PPP(gsm)
        ppp.active(True)
        time.sleep(15)

        print(ppp.ifconfig())

And when running it it does not connect.

I am using TTGO T-Call module.

[Pawelinode]

After induction:
gsm.write('AT+CGDCONT=1,"IP","globul"\r\n')
send:
gsm.write('ATD*99#\r\n')
The modem should respond 'CONNECT'

time.sleep(5)

import network
ppp=network.PPP(gsm)
ppp.active(True)
ppp.connect(authmode=ppp.AUTH_PAP, username="", password="")
time.sleep(15)
while ppp.isconnected()==False:
time.sleep(1)
print(ppp.ifconfig())

[tadejales]

So if I understand correctly complete code should loomk like this:

...
gsm.write("AT+CNMI=0,0,0,0,0\r\n")
print(gsm.readline())
print(gsm.readline())
time.sleep(1)
gsm.write('AT+CGDCONT=1,"IP","globul"\r\n')

gsm.write('ATD*99#\r\n')

print(gsm.readline())
print(gsm.readline())
time.sleep(1)
gsm.write('AT+CGDATA="PPP",1\r\n')
time.sleep(1)
print(gsm.readline())
print(gsm.readline())

time.sleep(5)

import network
...

[Pawelinode]

After issuing the gsm.write('ATD*99#\r\n') command, another AT command cannot be sent.
Delete gsm.write('AT+CGDATA="PPP",1\r\n')

I don't know if it's enough to call print(gsm.readline()) twice for each AT command.

[peterhinch]

I think you need to issue response.close() to close the socket. My guess is that the host has rejected the second connection because the first is still open (ECONNABORTED means a rejection by the host).

[tadejales]

I think it is not that. I have also tries with only executing from buttom press and the result was the same.

I am only adding bottom part of the code that I have changed:

...
      print("----------  PPP ---------- ")
      ppp=network.PPP(gsm)
      ppp.active(True)
      ppp.connect(authmode=ppp.AUTH_PAP, username="", password="")
      time.sleep(15)
      while ppp.isconnected()==False:
          time.sleep(1)

      print(ppp.ifconfig())

      # Try GET request
      test_get_req("PROCEDURE GET CALL")

def test_get_req(label) -> None:
  print(f"---------- {label} ----------")
  ntp_api_url = 'http://worldtimeapi.org/api/timezone/Etc/UTC'
  try:
      response = requests.get(ntp_api_url)
      print(response.status_code)
      print(json.loads(response.content))
      response.close()
  except Exception as e:
      print(e)

# ------------------------------ MAIN -------------------------------------------

# Pin definitions.
position_2_pin = Pin(32, Pin.IN, Pin.PULL_UP)

# Buttons
position_2_pb = Pushbutton(position_2_pin)

# Create instance
postman = Postman()

async def main():
  position_2_pb.press_func(test_get_req, ("BUTTON PRESS GET CALL",))

# Start the main coroutine.
loop = asyncio.get_event_loop()
loop.create_task(main())  # Schedule the main coroutine to run
loop.run_forever()  # Run main application code

Outcome:

----------  PPP ---------- 
('10.40.232.63', '192.168.254.254', '255.255.255.255', '84.255.209.79')
---------- PROCEDURE GET CALL ----------
200
{'timezone': 'Etc/UTC', 'unixtime': 1713337960, 'dst_offset': 0, 'day_of_year': 108, 'abbreviation': 'UTC', 'raw_offset': 0, 'client_ip': '188.230.129.6', 'dst_from': None, 'utc_datetime': '2024-04-17T07:12:40.429138+00:00', 'week_number': 16, 'dst': False, 'datetime': '2024-04-17T07:12:40.429138+00:00', 'utc_offset': '+00:00', 'dst_until': None, 'day_of_week': 3}
---------- BUTTON PRESS GET CALL ----------
[Errno 113] ECONNABORTED

My question would be if I can even run WiFi and PPP at the same time as I need WiFi for ESP-Now, or do they have problems running together. I tried disabling WiFi init procedure shown in the previous post but it made no difference.

[peterhinch]

I'm afraid I have no experience of PPP.

You could try changing your invocation of asyncio. The following is a more usual approach:

async def main():
  # Instantiate asyncio objects after asyncio has started
  position_2_pb = Pushbutton(position_2_pin)
  position_2_pb.press_func(test_get_req, ("BUTTON PRESS GET CALL",))
  while True:  # Don't let asyncio terminate
    await asyncio.sleep(1)  # In a real app you'd do some work here

asyncio.run(main())  # Start the main coroutine, which runs forever.

If this doesn't help, consider that ECONNABORTED means connection reset by host. So the question is why does the host respond correctly on the first connection, then reject the second. You could try running the same code on a WiFi-connected board to see if it's something related to PPP (this seems unlikely to me, but you never know). You could even run it on a PC (replacing the button press with an asyncio sleep).

Incidentally I would put the response.close() in a finally clause so it's called even when an exception is thrown. It won't fix this problem, but it is wise to close sockets.

[tadejales]

So first point is that if I put response.close() in finally I get NameError: local variable referenced before assignment error.

And sure, your suggestion about using WiFi should come to my ming earlier. But it works, with WiFi it works both ways, with procedure call and with pushbutton. So then it should be something related to PPP.

Do you see any potential problem with async approach with pushbuttons, that that might cause problems with PPP?

from machine import Pin
import network
import uasyncio as asyncio
import urequests as requests
import json
import time
from lib.primitives.pushbutton import Pushbutton

wifi = network.WLAN(network.STA_IF)
wifi.active(True)
wifi.connect('UUID', 'PASS')
while not wifi.isconnected():  # Wait until connected...
    time.sleep(0.1)
wifi.status()

# Pin definitions.
position_2_pin = Pin(32, Pin.IN, Pin.PULL_UP)

def test_get_req(label) -> None:
    print(f"---------- {label} ----------")
    ntp_api_url = 'http://worldtimeapi.org/api/timezone/Etc/UTC'
    try:
        response = requests.get(ntp_api_url)
        print(response.status_code)
        print(json.loads(response.content))
        response.close()
    except Exception as e:
        print(e)

test_get_req("PROCEDURE GET CALL")

async def main():
    # Buttons
    position_2_pb = Pushbutton(position_2_pin)
    position_2_pb.press_func(test_get_req, ("BUTTON PRESS GET CALL",))
    while True:
        await asyncio.sleep(1)

asyncio.run(main())

Output:

---------- PROCEDURE GET CALL ----------
200
{'timezone': 'Etc/UTC', 'unixtime': 1713341521, 'dst_offset': 0, 'day_of_year': 108, 'abbreviation': 'UTC', 'raw_offset': 0, 'client_ip': '89.212.243.251', 'dst_from': None, 'utc_datetime': '2024-04-17T08:12:01.798131+00:00', 'week_number': 16, 'dst': False, 'datetime': '2024-04-17T08:12:01.798131+00:00', 'utc_offset': '+00:00', 'dst_until': None, 'day_of_week': 3}
---------- BUTTON PRESS GET CALL ----------
200
{'timezone': 'Etc/UTC', 'unixtime': 1713341542, 'dst_offset': 0, 'day_of_year': 108, 'abbreviation': 'UTC', 'raw_offset': 0, 'client_ip': '89.212.243.251', 'dst_from': None, 'utc_datetime': '2024-04-17T08:12:22.346011+00:00', 'week_number': 16, 'dst': False, 'datetime': '2024-04-17T08:12:22.346011+00:00', 'utc_offset': '+00:00', 'dst_until': None, 'day_of_week': 3}

[peterhinch]

Is this a problem with objects going out of scope? You define the ppp and UART instance in the class constructor, but the scope of those objects is limited to the constructor. I would try making these objects bound variables and ensuring that test_get_req is a method.

[Pawelinode]

You set a low UART speed and did not define the RTS/CTS line in the modem. There will be a problem with downloading more than ~1.5kB of data (TCP MTU). PPP does not provide flow control. Micropython may lose data sent from the modem.

[tadejales]

Can you show me an example. I dont know what you mean by RTS/CTS. By speed you probably mean baudrate?

For your reference, I am using TTGO T-Call v1.3 board.

[Pawelinode]

Modify this line of code:
gsm = UART(1, 9600, timeout=1000, rx=MODEM_TX_PIN, tx=MODEM_RX_PIN)
9600 bps is very low. To communicate with the modem, use the RxD, TxD, CTS, RTS lines and enable hardware flow control RTS/CTS.

[tadejales]

I know which line I have to correct, but I hardly know what this naming(RTS, CTS) mean and how this should be done or for what it is used.

Are you able to explain a bit more and give me an exact example how this would look like in my case. I have made basicaly zero progres. I am not experienced in this and getting stuck this hard is not very fun 😖

[Pawelinode]

The problem that occurred may result from the use of requests and async.

See the board schematic. Modems are usually connected using 4 lines RxD, TxD, RTS, CTS.
In UART.init() you provided only RxD, TxD (rx=MODEM_TX_PIN, tx=MODEM_RX_PIN).

[tadejales]

Based on the TTGO T-Call v 1.3 there is no exposed RTS, CTS lines that I am aware of.

I tried with 115200 baudrate and it did not made any difference.

[chris35469]

OSError: -202 may be linked to a firmware issue. See note regarding downgrading firmware in #13603