Failed LoTW Downloads

[ve3hls]

Is there a solution to this issue yet? I still can't download anything from LoTW; I have to logon myself and update Cloudlog from LoTW manually, which takes forever. Uploads work fine because I can see them in LoTW, and of course I can see which QSOs have been verified, but Cloudlog still can't download them from LoTW.

Is anybody working on this, or has it been solved and I just can't see it?

73,
Ken VE3HLS

[gx1400]

#1948

[ve3hls]

Are you referring to this?
_I've found it!

Cloudlog tries to write to the file uploads/lotwreport_download.adi . The www-data group lost write permission on that file. I've added the write permission again, and now my lotw up- and download works as intended.

I susspect that this happend on March 1st, as I do the update cloudlog script once per month. Maybe something similar has happend to others.
To be fair, this may be a setup problem on my side..._

I don't know what that means. Something about write permission? I have no control over that because Peter hosts my Cloudlog. Besides, I don't have any problem writing logs to LoTW. I have trouble getting verifications from LoTW. I guess you could say I have problems writing the verifications to my Cloudlog, but I don't think that's what write permission means...

73 and thanks for getting back to me,

Ken VE3HLS

[magicbug]

If your getting that error Ken, have you recently changed your lotw login details? i'd suggest reentering them in your account settings

[ve3hls]

Hi Peter, I did as a matter of fact. The strange thing is it logs into LoTW and uploads logs without a problem, which leads me to believe that my logins are fine. I also requested and received a new TQSL key and installed that successfully according to the instructions in Cloudlog. It's almost bedtime. I'll try it all again tomorrow morning and let you know what happens. Thanks and 73, Ken VE3HLS

…

On Thu, Apr 27, 2023, 17:40 Peter Goodhall ***@***.***> wrote: If your getting that error Ken, have you recently changed your lotw login details? i'd suggest reentering them in your account settings — Reply to this email directly, view it on GitHub <#2080 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATQT75Q2FSZHJGEHMAJAAGLXDJEIZANCNFSM6AAAAAAXKVYEUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[gx1400]

As a data point, I was recently assigned a vanity and uploaded a new LOTW p12 certificate, when I try to do a manual LOTW Import I also get a

LoTW Downloading failed either due to it being down or incorrect logins.

[gx1400]

And btw. Do you get the same error message on sync now with the latest code base in dev branch?

Yes same error. I need to see how the LAMP services are executing now: as root or as my logged in user or something.

Maybe im missing something being in the www-data user group.

Its been a long time since I've managed linux permissions. Those folders were explicitly set to 'chown root:www-data' and verified but I still had read/write issues until i did a 'chmod 777'

Could you save me some research and point me to any log files or systemctl commands or something that might show why those folders are having write issues and what user account is running the service?

[gx1400]

@phl0

gx1400@ubuntu-22:~$ sudo systemctl status apache2
● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-04-27 20:52:26 UTC; 3h 19min ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 696 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
    Process: 3175 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
   Main PID: 827 (apache2)
      Tasks: 9 (limit: 1115)
     Memory: 56.7M
        CPU: 3.832s
     CGroup: /system.slice/apache2.service
             ├─ 827 /usr/sbin/apache2 -k start
             ├─3218 /usr/sbin/apache2 -k start
             ├─3219 /usr/sbin/apache2 -k start
             ├─3220 /usr/sbin/apache2 -k start
             ├─3221 /usr/sbin/apache2 -k start
             ├─3222 /usr/sbin/apache2 -k start
             ├─3245 /usr/sbin/apache2 -k start
             ├─3246 /usr/sbin/apache2 -k start
             └─3247 /usr/sbin/apache2 -k start

Apr 27 20:52:25 ubuntu-22 systemd[1]: Starting The Apache HTTP Server...
Apr 27 20:52:26 ubuntu-22 systemd[1]: Started The Apache HTTP Server.
Apr 28 00:00:00 ubuntu-22 systemd[1]: Reloading The Apache HTTP Server...
Apr 28 00:00:00 ubuntu-22 systemd[1]: Reloaded The Apache HTTP Server.
gx1400@ubuntu-22:~$ sudo systemctl show -pUser,UID apache2
UID=[not set]
User=
gx1400@ubuntu-22:~$ 

looks like apache2 is running as root

[phl0]

I don't think so. It would be a great security issue if apache2 would run as root. Try executing:

$ ps aux | grep "apache2"

and it should show the user in the first column. I expect this to be www-data rather than root (at least for the sub-processes it spawned.

[gx1400]

I don't think so. It would be a great security issue if apache2 would run as root. Try executing:

$ ps aux | grep "apache2"

and it should show the user in the first column. I expect this to be www-data rather than root (at least for the sub-processes it spawned.

gx1400@ubuntu-22:~$ ps aux | grep "apache2"
root         827  0.0  2.7 220692 27704 ?        Ss   Apr27   0:04 /usr/sbin/apache2 -k start
www-data    8294  0.0  2.9 295748 29616 ?        S    10:20   0:00 /usr/sbin/apache2 -k start
www-data    8312  0.0  2.8 221844 28148 ?        S    10:20   0:00 /usr/sbin/apache2 -k start
www-data    8316  0.0  2.9 221996 29004 ?        S    10:20   0:00 /usr/sbin/apache2 -k start
www-data    8324  0.0  2.7 221916 27064 ?        S    10:20   0:00 /usr/sbin/apache2 -k start
www-data    8360  0.0  2.9 222152 29020 ?        S    10:22   0:00 /usr/sbin/apache2 -k start
www-data    8362  0.0  2.8 221924 28640 ?        S    10:22   0:00 /usr/sbin/apache2 -k start
www-data    8363  0.0  2.6 221612 26672 ?        S    10:22   0:00 /usr/sbin/apache2 -k start
www-data    8364  0.0  2.8 222012 28288 ?        S    10:22   0:00 /usr/sbin/apache2 -k start
www-data   10203  0.0  2.6 221644 26124 ?        S    13:03   0:00 /usr/sbin/apache2 -k start
www-data   10208  0.0  2.6 221572 26676 ?        S    13:03   0:00 /usr/sbin/apache2 -k start
gx1400     12792  0.0  0.2   7004  2128 pts/0    S+   15:51   0:00 grep --color=auto apache2

[gx1400]

And btw. Do you get the same error message on sync now with the latest code base in dev branch?

I see where you asked about dev branch, This is 100% miscommunication on my end, I apologize for throwing you for a loop.

[ve3hls]

It's all Greek to me! Maybe somebody has to sudo some suppositories or git some chrons in my account? I dunno, this totally confuses me. 73, Ken VE3HLS

…

On Thu, Apr 27, 2023, 22:57 gx1400 ***@***.***> wrote: with 'sudo grep CRON /var/log/syslog': Apr 26 18:00:01 ubuntu-22 CRON[813821]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/clublog/upload &>/dev/null) Apr 26 18:00:01 ubuntu-22 CRON[813822]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/lotw/lotw_upload &>/dev/null) Apr 26 18:00:01 ubuntu-22 CRON[813824]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/qrz/upload &>/dev/null) Apr 26 18:09:01 ubuntu-22 CRON[813926]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 18:17:01 ubuntu-22 CRON[814052]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 26 18:39:01 ubuntu-22 CRON[814300]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 19:00:02 ubuntu-22 CRON[814382]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/lotw/lotw_upload &>/dev/null) Apr 26 19:09:02 ubuntu-22 CRON[814419]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 19:17:01 ubuntu-22 CRON[814496]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 26 19:39:01 ubuntu-22 CRON[814605]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 20:00:01 ubuntu-22 CRON[814767]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/lotw/lotw_upload &>/dev/null) Apr 26 20:09:01 ubuntu-22 CRON[814789]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 20:17:01 ubuntu-22 CRON[814871]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 26 20:39:01 ubuntu-22 CRON[815168]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 21:00:01 ubuntu-22 CRON[815466]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/lotw/lotw_upload &>/dev/null) Apr 26 21:09:01 ubuntu-22 CRON[815547]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 21:17:01 ubuntu-22 CRON[815684]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 26 21:39:01 ubuntu-22 CRON[815839]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 22:00:01 ubuntu-22 CRON[815917]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/lotw/lotw_upload &>/dev/null) Apr 26 22:09:01 ubuntu-22 CRON[815927]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 22:17:01 ubuntu-22 CRON[815994]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 26 22:39:01 ubuntu-22 CRON[816134]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 23:00:02 ubuntu-22 CRON[816288]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/lotw/lotw_upload &>/dev/null) Apr 26 23:09:01 ubuntu-22 CRON[816305]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 26 23:17:01 ubuntu-22 CRON[816371]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 26 23:39:01 ubuntu-22 CRON[816530]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) **Apr 27 00:00:01 ubuntu-22 CRON[816688]: (root) CMD (/bin/bash -c "/var/www/cloudlog.sh")** Apr 27 00:00:01 ubuntu-22 CRON[816687]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/lotw/lotw_upload &>/dev/null) Apr 27 00:00:01 ubuntu-22 CRON[816691]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/qrz/upload &>/dev/null) Apr 27 00:00:01 ubuntu-22 CRON[816689]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/clublog/upload &>/dev/null) Apr 27 00:00:01 ubuntu-22 CRON[816684]: (CRON) info (No MTA installed, discarding output) Apr 27 00:09:01 ubuntu-22 CRON[816774]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 27 00:17:01 ubuntu-22 CRON[816853]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 27 00:39:01 ubuntu-22 CRON[816930]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 27 01:00:01 ubuntu-22 CRON[817131]: (root) CMD (curl --silent https://cloudlog.[redacted].com/index.php/lotw/lotw_upload &>/dev/null) Apr 27 01:09:01 ubuntu-22 CRON[817200]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Apr 27 01:17:01 ubuntu-22 CRON[817221]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 27 01:39:01 ubuntu-22 CRON[817935]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) looks like my crontab is running, but maybe it's not running, I need to look at that.. but what info can I provide? or if you'd like me to manually pull the newest git or run the chown scripts; I'll hold off for now — Reply to this email directly, view it on GitHub <#2080 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATQT75WV4YPRCLCD52ZHI5DXDKJOVANCNFSM6AAAAAAXKVYEUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ve3hls]

I'm being hosted on cloudlog.co.uk. I don't know about branches. You'll have to ask Peter. 73, Ken VE3HLS

…

On Thu, Apr 27, 2023, 22:13 Florian (DF2ET) ***@***.***> wrote: As a data point, I was recently assigned a vanity and uploaded a new LOTW p12 certificate, when I try to do a manual LOTW Import I also get a LoTW Downloading failed either due to it being down or incorrect logins. Are you hosting yourself or on cloudlog.co.uk? And which branch of Cloudlog are you using? I would really like to get to the bottom of this... — Reply to this email directly, view it on GitHub <#2080 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATQT75TX6I43O6ZS6VOGW7DXDKEI3ANCNFSM6AAAAAAXKVYEUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ve3hls]

Florian, Are you writing to me or GX1400? It's hard to tell since my thread got taken over! I don't know anything about git pulls or commits. You really should be talking to Peter about updates. I always assumed updates were automatic. 73, Ken VE3HLS

…

On Fri, Apr 28, 2023, 03:11 Florian (DF2ET) ***@***.***> wrote: Yeah. It looks like your installation is not up to date. I bet local changes prevent a simple git pull? Commit 7218961 <7218961> is state as of Feb this year ... Quite old :) Perform a git pull on the console and let us know what it says. — Reply to this email directly, view it on GitHub <#2080 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATQT75T2DPUFMOVPZEL774TXDLHIDANCNFSM6AAAAAAXKVYEUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[phl0]

I am trying to answer @gx1400 because he has access to the console of his system. That is needed to track down the problem here.

[phl0]

I am afraid the issue on cloudlog.co.uk has to be handled by @magicbug because neither you nor me have access to logs etc.

[phl0]

Ok so as this thread gets a little out of control I will answer below to hopefully not confuse anyone.

@gx1400 Could you please try to patch the file application/controllers/Lotw.php and comment out line 567 with the unlink($filepath);? This should prevent Cloudlog from deleting the file downloaded from LotW so that we can see if something is downloaded at all. If the download of the data itself succeeds there is a file lotwreport_download.adi in the uploads folder. Please test and report.

Patch as here:

diff --git a/application/controllers/Lotw.php b/application/controllers/Lotw.php
index db40eb35..33621a18 100644
--- a/application/controllers/Lotw.php
+++ b/application/controllers/Lotw.php
@@ -564,7 +564,7 @@ class Lotw extends CI_Controller {
                                $data['lotw_table'] = $table;
                }
 
-               unlink($filepath);
+               //unlink($filepath);
 
                if(isset($data['lotw_table_headers'])) {
                        echo "LoTW Matches<br>";

[phl0]

That would mean you are on master branch. I am referring to dev because I commited a fix there. Would you be able to switch? (I thought you had done that a per one of the previous posts)

[gx1400]

No I never switched to dev. I can do that now, sorry for the confusion

[gx1400]

@phl0 I never switched to dev. Just did so and did NOT make the modification you requested above, manual import received: "Downloaded LoTW report contains no matches."

[phl0]

Ok that is the expected response. I changed the error handling messages to be more appropriate. So your file is simply empty and the nee error message does not put you on the wrong way.

[phl0]

So all correct for @gx1400 now. LotW results are simply empty.

[ve3hls]

OK, where does that leave me? Ken VE3HLS

…

On Fri, Apr 28, 2023 at 10:45 AM Florian (DF2ET) ***@***.***> wrote: I am trying to answer @gx1400 <https://github.com/gx1400> because he has access to the console of his system. That is needed to track down the problem here. — Reply to this email directly, view it on GitHub <#2080 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATQT75UT25JXZY6E5WMNLPLXDONVVANCNFSM6AAAAAAXKVYEUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[phl0]

I hope that we can get to the source of this problem and find a solution that may also work for you. But as long as we do not have the data to base the analysis on I cannot help. Sri

[ve3hls]

Thanks flo, I can see I need Peter to get involved. Peter, Flo says it looks like my copy of Cloudlog may be out of date. I thought updates would be automatic? He would also need access to things it looks like only you can access in order to help me. I created and uploaded a new certificate just a short time ago and also entered my new LoTW password again but it didn't make any difference. Please have a look at this ASAP. It's been like this for over a month. Thanks and 73, Ken VE3HLS

…

On Fri, Apr 28, 2023 at 11:39 AM Florian (DF2ET) ***@***.***> wrote: I hope that we can get to the source of this problem and find a solution that may also work for you. But as long as we do not have the data to base the analysis on I cannot help. Sri — Reply to this email directly, view it on GitHub <#2080 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATQT75RQAAHQHFHS25ZFGALXDOT7JANCNFSM6AAAAAAXKVYEUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[phl0]

Hi Ken,

so the issue with @gx1400 was that he simply had no confirmed LoTW QSOs in his account and thus no QSLs could be imported. It was just the error message that is misleading in current version of Cloudlog. This will be fixed with the next release.

But I think this is not the issue you are facing as you say you have confirmed QSOs within your LoTW account. So I think you have another issue. Either the download of the report from LoTW fails (because the file is not writable by the web server) or it does not find confirmed QSOs. Both of which is hard to debug at the moment based on the current release code. And I am afraid I cannot look into the details cause you are hosting on cloudlog.co.uk.

[ve3hls]

Thanks Flo, Yes, I went to LoTW and manually updated about 75 or 80 Cloudlog QSOs a week or so back. I've been keeping Peter up to date on what I am seeing but unfortunately he hasn't responded even once. It's like he has lost interest in the project. Luckily, conditions are still rather poor so I won't have much to upload, but there's a big contest next month and I hope to be very busy. I will be checking for new verifications later today and will update Cloudlog (manually) again. Thanks for getting back to me. 73, Ken VE3HLS

…

On Sun, Apr 30, 2023, 19:25 Florian (DF2ET) ***@***.***> wrote: Hi Ken, so the issue with @gx1400 <https://github.com/gx1400> was that he simply had no confirmed LoTW QSOs in his account and thus no QSLs could be imported. It was just the error message that is misleading in current version of Cloudlog. This will be fixed with the next release. But I think this is not the issue you are facing as you say you have confirmed QSOs within your LoTW account. So I think you have another issue. Either the download of the report from LoTW fails (because the file is not writable by the web server) or it does not find confirmed QSOs. Both of which is hard to debug at the moment based on the current release code. And I am afraid I cannot look into the details cause you are hosting on cloudlog.co.uk. — Reply to this email directly, view it on GitHub <#2080 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATQT75RZWU4VOBIHKPNAL3DXDZK2LANCNFSM6AAAAAAXKVYEUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[magicbug]

checked again on the server for @ve3hls and as I stated earlier its failed due to incorrect login information for LoTW

This can be resolved by clicking callsign top left then "Accounts"

[phl0]

I guess we should see a little further with the coming release. It would at least show if it is wrong login really or empty LoTW report (which for now causes the same error message).

[ve3hls]

Ok. I guess I'll go back and recheck for the fourth time. Though I don't know how many more times I can enter the same password and upload the same TQSL key. Both the password and key work fine when I go to the LoTW website and login manually. I'll do it in the morning. 73, Ken VE3HLS

…

On Mon, May 1, 2023, 18:12 Peter Goodhall ***@***.***> wrote: checked again on the server for @ve3hls <https://github.com/ve3hls> and as I stated earlier its failed due to incorrect login information for LoTW This can be resolved by clicking callsign top left then "Accounts" — Reply to this email directly, view it on GitHub <#2080 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATQT75QKMT7QLFUUDDLQFPTXD6LDFANCNFSM6AAAAAAXKVYEUA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ve3hls]

Isn't this rhe definition of insanity? Doing the same thing over and over again, and expecting a different result? Ken VE3HLS

…

On Mon, May 1, 2023, 21:04 Kenneth P Alexander ***@***.***> wrote: Ok. I guess I'll go back and recheck for the fourth time. Though I don't know how many more times I can enter the same password and upload the same TQSL key. Both the password and key work fine when I go to the LoTW website and login manually. I'll do it in the morning. 73, Ken VE3HLS On Mon, May 1, 2023, 18:12 Peter Goodhall ***@***.***> wrote: > checked again on the server for @ve3hls <https://github.com/ve3hls> and > as I stated earlier its failed due to incorrect login information for LoTW > > This can be resolved by clicking callsign top left then "Accounts" > > — > Reply to this email directly, view it on GitHub > <#2080 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ATQT75QKMT7QLFUUDDLQFPTXD6LDFANCNFSM6AAAAAAXKVYEUA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >