[BUG] Processing logs is super slow when the UI is open

[aszurnasirpal]

Describe the bug
Processing logs is super slow

Expected behavior
Able to parse the logs in a reasonable time

Screenshots

Desktop (please complete the following information):
I'm using the Oracle Free Tier (not ampere) - so VPS with one 1GB ram

  Model name:                AMD EPYC 7551 32-Core Processor
    CPU family:              23
    Model:                   1
    Thread(s) per core:      2
    Core(s) per socket:      1
    Socket(s):               1
    Stepping:                2
    BogoMIPS:                3992.50

Additional context
I run the loglynx as a part of my Pagolin setup (+crowdsec)

[github-actions]

🎉 Welcome to LogLynx!

Thank you for opening your first issue in the LogLynx project! We appreciate you taking the time to help improve our advanced log analytics platform.

📝 What to expect

• Our maintainers will review your issue shortly
• We may ask for additional information to better understand the issue
• We'll label the issue appropriately to track its progress

🔍 Useful Resources

• Project Documentation
• Contributing Guidelines
• Code of Conduct
• Security Policy

🚀 Getting Started with LogLynx

LogLynx is a powerful log analytics platform that provides real-time insights into your web traffic. If you're new to the project, you might want to:

• Read about our features
• Learn how to get started

Thanks again for your contribution! 🙏

The LogLynx Team

[K0lin]

Thanks for letting me know, it seems there are issues when importing datasets on smaller machines, can I ask you to run this command (sudo cat /proc/$(pidof loglynx)/status | grep Vm) so I can get more information about what the program is using too much in terms of hardware?

As a stopgap you can skip importing the data using INITIAL_IMPORT_ENABLE=false, or limit the data to import via INITIAL_IMPORT_DAYS=10, this will allow you to unblock you while we troubleshoot this.

If you want to make the parsing process even faster, you could temporarily set GEOIP_ENABLED=false.

So you can temporarily update the docker image like this:

loglynx:
    image: k0lin/loglynx:latest
    container_name: loglynx
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - ./loglynx-data:/data
      - ./config:/app/geoip                 
      - ./config/traefik/logs:/traefik/logs
    environment:
      -INITIAL_IMPORT_ENABLE=false
      -INITIAL_IMPORT_DAYS=10 #even less than 10 days
     # your other configurations....

You can see all the tunables in the .env.example file.

[aszurnasirpal]

Sure, will try with limiting the import days first

output of the command

VmPeak:	 2640536 kB
VmSize:	 2575000 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	  166620 kB
VmRSS:	   25580 kB
VmData:	  481928 kB
VmStk:	     132 kB
VmExe:	   12192 kB
VmLib:	    1528 kB
VmPTE:	     624 kB
VmSwap:	  127312 kB

[K0lin]

I installed an instance on a VPS with one CPU and, while analysing the data loading phase, I noticed that keeping the splash screen open triggers calls to the database, slowing down the programme's execution.

[aszurnasirpal]

I limited the number of ingest days to 10 (seems to be faster, but still it's processing that on my end for the last hour)
In the Docker logs, I noticed entries like

2026-01-04 20:37:07 WARN  ⚠️  Connection pool high utilization detected
                      ├ utilization: 100.0%
                      ├ in_use: 25
                      ├ max_open: 25
                      ├ threshold: 85.0%
                      └ wait_count: 1266
2026-01-04 20:37:07 ERROR 🚨 Connection pool SATURATED - all connections in use!
                      ├ in_use: 25
                      ├ max_open: 25
                      ├ wait_count: 1266
                      └ avg_wait_time: 536.076564ms

[aszurnasirpal]

It seems that it is stuck at 99 per cent (at least no progress since last hour)

recent logs shows (among normal processing information) data like

026-01-04 22:24:20 ERROR Database query error
                      ├ error: context deadline exceeded
                      ├ duration_ms: 30003
                      └ sql: SELECT 
			COUNT(*) as total_requests,
			COUNT(CASE WHEN status_code >= 200 AND status_code < 400 THEN 1 END) as valid_requests,
			COUNT(CASE WHEN status_code >= 400 THEN 1 END) as failed_requests,
			COUNT(DISTINCT client_ip) as unique_visitors,
			COUNT(DISTINCT path) as unique_files,
			COUNT(DISTINCT CASE WHEN status_code = 404 THEN path END) as unique_404,
			COALESCE(SUM(response_size), 0) as total_bandwidth,
			COALESCE(AVG(CASE WHEN response_time_ms > 0 THEN response_time_ms END), 0) as avg_response_time,
			COUNT(CASE WHEN status_code = 404 THEN 1 END) as not_found_count,
			COUNT(CASE WHEN status_code >= 500 AND status_code < 600 THEN 1 END) as server_error_count
		 FROM `http_requests` WHERE timestamp > "2025-12-28 22:23:50.599"
2026-01-04 22:24:20 ERROR Failed to get summary stats
                      ├ error: context deadline exceeded
                      └ caller: /src/internal/database/repositories/stats.go:437
2026-01-04 22:24:20 ERROR Failed to get summary
                      ├ error: context deadline exceeded
                      └ caller: /src/internal/api/handlers/dashboard.go:168

[K0lin]

To speed up the process, I recommend you temporarily close the site window

[aszurnasirpal]

ok, you're right. Closing the tab significantly reduced the CPU-intensive operations, and finally, it parsed all the data.

[K0lin]

Nice, we are continuing to work to resolve the issues, thank you for your patience.