Deployment on K8s

[nithin1995]

Hi Team,

When I try to deploy langfuse on a K8 cluster, I get the following error logs on my webserver pod

Script executed successfully.
Prisma schema loaded from packages/shared/prisma/schema.prisma
Datasource "db": PostgreSQL database "postgres", schema "public" at "rahul-postgres-postgresql:5432"

225 migrations found in prisma/migrations

Error: P3009

migrate found failed migrations in the target database, new migrations will not be applied. Read more about how to resolve migration issues in a production database: https://pris.ly/d/migrate-resolve
The 20240528214728_add_cursor_index_07 migration started at 2024-07-24 09:08:03.995844 UTC failed


Applying database migrations failed. This is mostly caused by the database being unavailable.
Exiting...

On the db pod, I get the following log message

2024-07-25 04:28:09.400 GMT [213262] LOG:  could not receive data from client: Connection reset by peer
2024-07-25 04:28:52.364 GMT [213321] LOG:  could not receive data from client: Connection reset by peer
2024-07-25 04:30:39.517 GMT [213479] LOG:  could not receive data from client: Connection reset by peer

I created my own postgres db using bitnami and am trying to deploy langfuse webserver without deploying db. My db pod is running.
My langfuse yaml file

langfuse:
  nextauth:
    url: localhost:3000
    secret: changeme
  salt: changeme
  telemetryEnabled: true
service:
  type: ClusterIP
  additionalLabels: []
ingress:
  enabled: false
  annotations: []
postgresql:
  deploy: false
  auth:
    username: postgres
    password: <password>
    database: postgres
  host: rahul-postgres-postgresql
  directUrl: postgresql://postgres:<password>@<db pod_name>:5432/postgres
  shadowDatabaseUrl: postgresql://postgres:<password>@<db pod_name>:5432/postgres

Both the pods are running on the same cluster. I couldn't figure out the reason for this error, can anyone help me?

[dosubot[bot]]

Hey @nithin1995! I'm here to help you with the langfuse deployment issue. I'm a bot that can assist with bug fixes, answering questions, and guiding you to become a contributor. Let's figure this out together!

To resolve the database migration issues you're encountering when deploying Langfuse on your Kubernetes cluster, you can follow these steps:

1. Check and Resolve Failed Migrations:
   You need to manually resolve the failed migrations in your PostgreSQL database. Connect to your PostgreSQL database and execute the following SQL script to delete the specific failed migrations from the _prisma_migrations table:

   DO $$
   BEGIN
    IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = '_prisma_migrations') THEN
     DELETE FROM _prisma_migrations
     WHERE migration_name IN ('20240528214728_add_cursor_index_07');
    END IF;
   END $$;

2. Configure Environment Variables:
   Ensure that you have configured the DIRECT_URL and SHADOW_DATABASE_URL environment variables correctly in your Kubernetes deployment configuration. This allows you to use a different user for migrations or use connection pooling on DATABASE_URL.

   Here is an example of how to set these environment variables in your Kubernetes deployment configuration:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: langfuse
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: langfuse
     template:
       metadata:
         labels:
           app: langfuse
       spec:
         containers:
         - name: langfuse
           image: langfuse/langfuse:latest
           env:
           - name: DATABASE_URL
             value: "postgresql://postgres:<password>@rahul-postgres-postgresql:5432/postgres"
           - name: DIRECT_URL
             value: "postgresql://postgres:<password>@rahul-postgres-postgresql:5432/postgres"
           - name: NEXTAUTH_URL
             value: "http://localhost:3000"
           - name: NEXTAUTH_SECRET
             value: "changeme"
           - name: SALT
             value: "changeme"
           - name: SHADOW_DATABASE_URL
             value: "postgresql://postgres:<password>@rahul-postgres-postgresql:5432/postgres"
           ports:
           - containerPort: 3000

3. Verify Connectivity and Network Policies:
   Ensure that the Langfuse webserver pod can connect to the PostgreSQL database pod. You can use a simple connectivity test from the Langfuse webserver pod to the PostgreSQL database pod:

   kubectl exec -it <langfuse-webserver-pod> -- curl -s -o /dev/null -w "%{http_code}" postgresql://rahul-postgres-postgresql:5432

   Additionally, verify that the network policies allow traffic between the Langfuse webserver pod and the PostgreSQL database pod. Here is an example of a network policy:

   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: allow-langfuse-to-postgres
     namespace: <your-namespace>
   spec:
     podSelector:
       matchLabels:
         app: postgresql
     ingress:
     - from:
       - podSelector:
           matchLabels:
             app: langfuse-webserver
       ports:
       - protocol: TCP
         port: 5432

By following these steps, you should be able to resolve the database migration issues and ensure proper connectivity between the Langfuse webserver pod and the PostgreSQL database pod ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[nithin1995]

It specifically failing in the migrations, I think its not a database connection issue. I can see some migrations happenning and then this one failed. I also tried spinning up a fresh postgres instance with no data in it. still facing the same issue, can anyone help?

[dosubot[bot]]

To resolve the migration issue with Langfuse on your Kubernetes cluster, specifically the failure of the 20240528214728_add_cursor_index_07 migration, you can follow these steps:

1. Check Database Connection and Permissions:
   Ensure that the database user has the necessary permissions to perform migrations. You might need to use a different user for migrations or configure connection pooling.

2. Use DIRECT_URL for Migrations:
   Set the DIRECT_URL environment variable to use a different user for migrations if needed. This user should have the necessary permissions and long timeouts for large deployments.

   env:
     - name: DIRECT_URL
       value: "your_migration_user_connection_string"

3. Disable Automatic Migrations Temporarily:
   If the migration is failing repeatedly, you can disable automatic migrations temporarily by setting the LANGFUSE_AUTO_POSTGRES_MIGRATION_DISABLED environment variable to true.

   env:
     - name: LANGFUSE_AUTO_POSTGRES_MIGRATION_DISABLED
       value: "true"

4. Manually Apply the Migration:
   Manually apply the migration using the steps provided in the documentation. This involves cloning the repository, setting up the environment, and running the migration script.

   # Clone the Langfuse repository
   git clone https://github.com/langfuse/langfuse.git
   
   # Navigate to the Langfuse directory
   cd langfuse
   
   # Install all dependencies
   pnpm i
   
   # Create an .env file
   cp .env.dev.example .env
   
   # Edit the .env to connect to your database from your machine
   # Replace with your database connection string
   DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
   
   # Execute the migration
   pnpm run models:migrate

5. Re-enable Automatic Migrations:
   Once the migration is successfully applied, re-enable automatic migrations by setting LANGFUSE_AUTO_POSTGRES_MIGRATION_DISABLED back to false.

   env:
     - name: LANGFUSE_AUTO_POSTGRES_MIGRATION_DISABLED
       value: "false"

By following these steps, you should be able to resolve the migration issue with Langfuse on your Kubernetes cluster ^[1].

To continue talking to Dosu, mention @dosu.

[nithin1995]

Can you also help me understand what migrations are happening, why is it happening from the webserver? since its a fresh instance what is the need of migrations. I think its not just applying a schema right?

[dosubot[bot]]

1. What migrations are happening during the deployment?

   The migrations being applied are Prisma migrations. The error message indicates that there are 225 migrations found in the prisma/migrations directory, and the specific migration that failed is 20240528214728_add_cursor_index_07. These migrations are defined in the Prisma schema file located at packages/shared/prisma/schema.prisma ^[1].

2. Why are these migrations being triggered from the webserver pod?

   During the container startup, all database migrations are applied automatically. This behavior can be optionally disabled via environment variables. The webserver pod is responsible for applying these migrations to ensure the database schema is up-to-date with the application requirements. The Prisma schema file specifies the database connection details and the migrations that need to be applied ^[2].

3. Since this is a fresh instance, why is there a need for migrations? Is it not just applying a schema?

   Even for a fresh instance, migrations are necessary to set up the initial database schema. Migrations ensure that the database schema matches the expected state defined by the application. The error message indicates that a previous migration attempt failed, which is why the system is trying to reapply the migrations. The Prisma schema file defines the structure of the database, including models like User, Account, Session, and Project, which need to be created in the database ^[2].

To continue talking to Dosu, mention @dosu.

[nithin1995]

Can anyone help apart from the ai ? I m not able to do the db migrations on a kubernetes cluster.

[marcklingen]

Hi @nithin1995, sorry for the slow response here!

This issue generally hints at that this migration (file) was already applied but failed while being applied. As this migration adds an index to the database, this might happen due to short session timeouts configured on the postgres user that is used to run the migrations.

You can add a separate user that is just used for migrations, quoting from the docs

Connection string of your Postgres database used for database migrations. Use this if you want to use a different user for migrations or use connection pooling on DATABASE_URL. For large deployments, configure the database user with long timeouts as migrations might need a while to complete.

Can you check whether this index exist on the table? If not, you can remove the row from the _prisma_migrations table and restart your container. On startup it will then reapply all outstanding migrations.

[nithin1995]

When I try do the indexing manually I get this message -

This is my _prisma_migrations table, I have deleted the last row which failed

All my timeouts are pretty large

After this when I restart my webserver, I still end up with the same error. What am i missing?

[nithin1995]

I started again with a fresh instance of database and it got fixed. thanks.