list of detected wifi access points is empty

[bbinet]

We often have the following case: when the rpi starts, it correctly connects to the wifi access point (that was previously configured using wifi-connect captive portal), but after a few minutes or a few hours, the wifi-connect is starting (probably because the connection was temporarily lost), and the captive portal shows an empty list of detected wifis.

But we have a lot of wifi access points around, so there must be an issue here as the list should not be empty...

The rpi never connects back to our initial wifi access point, until we manually reboot the device.

Here is my start.sh script:

$ cat /usr/src/app/start.sh
#!/usr/bin/env bash
export DBUS_SYSTEM_BUS_ADDRESS=unix:path=/host/run/dbus/system_bus_socket
while true
do
    iwgetid -r
    if [ $? -ne 0 ]; then
        printf 'Starting WiFi Connect\n'
        ./wifi-connect --portal-ssid helioslite-tracker --portal-passphrase XXX --activity-timeout 600
    fi
    sleep 10
done

Here are the logs of NetworkManager service:
https://gist.github.com/bbinet/d0ed2e5465824ef2594589b383cc5f93#file-networkmanager-log

Here are the logs of the container that runs wifi-connect:
https://gist.github.com/bbinet/d0ed2e5465824ef2594589b383cc5f93#file-wifi-connect-log

Front conversations

[bbinet]

I'm happy to provide more details as I can connect to the rpi through the wifi-connect access point, so if think some other logs might be helpful to investigate this issue, please tell me.

Also do you know a temporary workaround that I can run to force network manager to reconnect to my initial wifi access point, so that I don't need to hard reboot the device?

[majorz]

Hi, I will check the logs now and let you know if I find anything.

[majorz]

The wifi-connect logs definitely reveal an issue: it should not start the access point when no SSIDs are available.

The empty SSIDs list is another issue that we need to look into. I am thinking about elevating the log level of NetworkManager and wpa_supplicant, and running some diagnostic code that we can try a few things with it. I will do some research tomorrow on this and let you know how that goes.

I had a non-related to WiFi Connect report about RPi 3 and NetworkManager not reporting any networks, so I am definitely interested in figuring out how to workaround this issue without reboots or driver reloads.

[bbinet]

Ok, thanks

[majorz]

@bbinet I started implementing this here: https://github.com/resin-io-playground/networkmanager-empty-ssids

It elevates the log levels of NetworkManager and wpa_supplicant, and then starts to check whether the SSID list is empty each 30 seconds. If the list is empty it triggers a rescan and waits 10 seconds to check whether they are available.

I am going to do local tests with it tomorrow and hopefully I will be able to reproduce an empty list. I am thinking to try it using a router and also a hotspot on my phone.

If I am not able to reproduce, I may ask you to run the script on a device that has this issue. But let's see first what will happen with the local testing. I will keep you posted.

[bbinet]

Sure, I'm also running your ssids.py script on my rpi zero w since this morning, so I let you know the results too.

[majorz]

I was able to reproduce zero access points available locally. The RequestScan method brought back the access point list, which is good news. I will check the logs now and do more testing.

[bbinet]

I've also been able to reproduce the issue.

Here are the logs of the ssids.py script:
https://gist.github.com/bbinet/e53e33236a9862bb2c70a4c6de4bbf36#file-ssids-log

Here are the logs of NetworkManager service:
https://gist.github.com/bbinet/e53e33236a9862bb2c70a4c6de4bbf36#file-nm-log

Here are the logs of the container that runs wifi-connect:
https://gist.github.com/bbinet/e53e33236a9862bb2c70a4c6de4bbf36#file-container-log

[majorz]

@bbinet that is really useful! In your case the RequestScan call did not bring back the list with SSIDs, which is more like the actual problem, than what I was able to reproduce here.

I still have not got to chance to check the NM logs you attached for the specifics, but I will do that first thing tomorrow and will follow up with you. Thanks again!

[majorz]

@bbinet I did different can on the instrumentation script: if the RequestScan does not work, it now tells NM to unmanage and then manage the device again. If that does not work, it then restarts the NM service. I also added timestamped logging.

Can you please run this again, but this time do not start WiFi Connect together with the script (assuming that the connection profile is defined)? Also you may reboot the device after the application update, so that we get only fresher logs.

If you reproduce something, please attach the logs of wpa_supplicant as well. Thanks!

[bbinet]

@majorz but if I don't run WiFi Connect together with the script, I won't be able to log in to get the logs...
Why not run WiFi Connect together with the script?

[majorz]

Actually now on second thought, you may leave it running, it should not be a problem.

[bbinet]

@majorz Let's run it