Default appVersion/image tag fails to initialize postgres DB

[holysoles]

The used docker image runs database migrations on startup. While this mostly seems fine, there was a bug in one of the database migration scripts that made it incompatible with postgresql. This was fixed in pixelfed/pixelfed@9289792, but that was only merged into the dev branch and pixelfed hasn't generated an official release since November :/, meaning there is no release tag with the fix commit. Since the docker image does also build dev, using that seems to work fine.

I don't know of the best way to address this since I'm not sure that it should be recommended to run the dev image by default, but wanted to make sure it was known. I did raise the problem over on the pixelfed project.

[jessebot]

You bring up a good point! I went ahead and asked for when the next official version is planned:
pixelfed/pixelfed#5524 (comment)

I don't really know what dictates the release schedule at this time 🤷

Doh! I didn't realize you had raised this over at the pixelfed project. Welp. Now we have both a discussion and an Issue comment 🤦

[jessebot]

There's also the larger question of which image do we use? Since pixelfed is written in PHP, this will pretty much always require php-fpm if we're going to use nginx, which is the case for nextcloud as well, however nextcloud runs nginx as a sidecar instead of within the image itself. The reason this is maybe preferable is that then you can isolate functionality and which user everything runs as.

Right now, if I just try to run the image as is, it looks like based on your PR, mattlqx/docker-pixelfed#2 we should run as the www-data user (UID 33) in the container level securityContext, but if they were two separate containers, one could run as 33 and the other could run as 991 (pixelfed user I think).

There's only two other regularly updated docker images already being published to my knowledge:

• https://quay.io/repository/zknt/pixelfed?tab=tags
• https://github.com/jippi/docker-pixelfed/pkgs/container/docker-pixelfed/versions

If we switched to either of those, it would require a small refactor to put all the env vars into a .env file that gets mounted into the pixelfed container (not insurmountable, but a little annoying). Maybe I'll do some work to get the bulk of the non-sensitive values into a configMap or Secret and then do an envFrom with those, so that if we do switch to an image that uses a .env file, the switch is less painful 🤔

[jessebot]

btw, if you're at all curious, I have been testing this chart via Argo CD here (it'll move from the demo directory to the root directory when it's working properly):
https://github.com/small-hack/argocd-apps/tree/main/demo/pixelfed

The actual helm chart deployment is here (it uses an externalDatabase (CNPG), and externalValkey (still just bitnami valkey)):
https://github.com/small-hack/argocd-apps/blob/main/demo/pixelfed/app_of_apps/posgres_appset.yaml

Each thing we test, gets updated by me, or renovatebot, which runs hourly and updates that appset to the latest version of this chart (at least until we are stable enough for me to tackle getting the ci pipeline in this repo good to go).

Currently getting 502 Bad Gateway when I go to the pixelfed URL in my browser and the logs in the pod say:

ERROR The [public/storage] link already exists.
WARN Horizon no longer publishes its assets. You may stop calling the `horizon:publish` command.
INFO Routes cached successfully.
INFO Blade templates cached successfully.
INFO Configuration cached successfully.
INFO Nothing to migrate.
Instance actor succesfully generated. You do not need to run this command again.
[19-Jan-2025 08:47:24] NOTICE: fpm is running, pid 45
[19-Jan-2025 08:47:24] NOTICE: ready to handle connections
ERROR There are no commands defined in the "websockets" namespace.
2025/01/19 08:52:18 [crit] 37#37: *1 connect() to unix:/run/php-fpm/php-fpm.sock failed (13: Permission denied) while connecting to upstream, client: 10.42.0.197, server: my.anonymizeddomain.tld, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm/php-fpm.sock:", host: "my.anonymizeddomain.tld"
10.42.0.197 - - [19/Jan/2025:08:52:18 +0000] "GET / HTTP/1.1" 502 166 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0" "192.168.1.1"
2025/01/19 08:52:32 [crit] 37#37: *1 connect() to unix:/run/php-fpm/php-fpm.sock failed (13: Permission denied) while connecting to upstream, client: 10.42.0.197, server: my.anonymizeddomain.tld, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm/php-fpm.sock:", host: "my.anonymizeddomain.tld"
10.42.0.197 - - [19/Jan/2025:08:52:32 +0000] "GET / HTTP/1.1" 502 166 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0" "192.168.1.1"

[jessebot]

Update after a number of updates to this chart for additional features and fixing little bugs. I am still not able to reach the web interface 🤔 (currently testing version 0.11.1 of this chart):

ERROR The [public/storage] link already exists.
INFO Routes cached successfully.
INFO Blade templates cached successfully.
INFO Configuration cached successfully.
INFO Nothing to migrate.
Instance actor succesfully generated. You do not need to run this command again.
[19-Jan-2025 13:09:57] NOTICE: fpm is running, pid 41
[19-Jan-2025 13:09:57] NOTICE: ready to handle connections
2025/01/19 13:10:07 [crit] 33#33: *1 connect() to unix:/run/php-fpm/php-fpm.sock failed (13: Permission denied) while connecting to upstream, client: 10.42.0.62, server: my.anonymizeddomain.tld, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm/php-fpm.sock:", host: "my.anonymizeddomain.tld"
10.42.0.62 - - [19/Jan/2025:13:10:07 +0000] "GET / HTTP/1.1" 502 166 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0" "192.168.1.1"

I guess I need to research the following:

(13: Permission denied) while connecting to upstream

Here's my securityContext settings in values.yaml right now:

podSecurityContext:
  runAsUser: 33
  runAsGroup: 33
  fsGroup: 33

securityContext: {}

[jessebot]

OK, update AGAIN: I finally got it working, but I don't think it's taking any of my settings done through the helm chart except the APP_KEY 🤦 . Here's my working settings if you're curious (everything after the valuesObject key):

https://github.com/small-hack/argocd-apps/blob/0c0d9c03c956feacd37c3adebc54f4573ef0e243/demo/pixelfed/app_of_apps/pixelfed_argocd_appset.yaml#L57

Update 1

I actually tried to spin up with that configuration again fresh and it didn't work. I got the migrations issue again...

Update 2

But as you said, using dev-nginx image tag for our current build does work for temporarily fixing the problem. For now, we could also wait till this image rebuilds tonight, and then we grab that sha to point directly to that instance of dev, so that at least we know it's pinned in some way. I think there's also a staging branch we could check?

[jessebot]

On the original question of postgres, I'm open to adding another database type if there's already a helm chart for it for us to test with, e.g. bitnami mysql/mariadb, but I have the most experience with postgresql generally. mysql is present in the nextcloud/helm chart as a subchart though, so we could add mysql as a subchart if we think that would help.

We could also offer sqlite without a subchart, which seems fine, but only for testing.

[jessebot]

Might also be worth trying to use https://github.com/pixelfed-glitch/pixelfed. It looks like they publish a docker image, and they have a tag for nginx:
https://github.com/pixelfed-glitch/pixelfed/pkgs/container/pixelfed

[holysoles]

I was able to get this deployed with flux the other day with the various fixes I left on the docker image project (merged now), and here, but also used a branch I had started to handle the APP_KEY value, but looks like you got that solved now. I haven't turned on federation yet but things seem to work otherwise, I'll probably do that in a bit.

As for DB selection, I do like postgres as well (in my limited experience). I'm hoping they'll better support it but it seems like it has had, and continues to experience issues with them trying to support both options.

i'm not too familiar with the pixelfed-glitch fork, but i did see that as an option. pinning the hash also is an option. It seems like the pixelfed folks have a lot going on based on the chatter in their Matrix chat, and image release generation probably wont be a priority for a bit..

[jessebot]

Awesome, really appreciate all your hard work on this chart generally!

For now, I think the best quick action will be to switch the appVersion to a pinned sha from matt's image, which I'm doing in #39.

I'm going to open a new issue to investigate using pixelfed-glitch to be downstream of pixelfed (#41), as it looks like pixelfed-glitch releases much more regularly. I'll also open an issue for supporting mysql or mariadb to allow users more freedom generally (#40).

I'll close this issue for now, and if you still have issues with migrations after #39 is merged, please open a new issue so we can take a look together! :)