A New LOG_TYPE: `NPM_SEPARATED` (or maybe `NPM_SPLIT`)

[xavier-hernandez]

Discussed in https://github.com/xavier-hernandez/goaccess-for-nginxproxymanager/discussions/161

^{Originally posted by wernerm September 18, 2023}
I would like to suggest a new feature please: The support for a new LOG_TYPE option called NPM_SEPARATED (or maybe NPM_SPLIT) or something similar.

What does it do?

It allows you to see separate statistics for each individual NPM proxy-host. (Instead of aggregated logs.)

How will it work?

An instance of GOACCESS is created for every NPM proxy-host.

• append "/host/n" to the url to access the instance, for example http://localhost:7880/host/7
  • for every host, the following file(s) are read and parsed.
    • proxy-host-*_access.log.gz
    • proxy-host-*_access.log
    • proxyhost-.log

How about an Example?

Suppose you visit http://localhost:7880/host/42, you will only see results for the following parsed logs:

• proxy-host-42_access.log.gz
• proxy-host-42_access.log
• proxy*host-42.log

PS: If you would consider such a feature, one could also consider a mechanism to extract the hostname(s) of the Proxy Hosts from NPM's SQLite DB, so that it can be displayed (instead of a number).

[wernerm]

Based on the feature request above, just another idea for possible future features...

Maybe a new environment variable could be introduced, with a name like LOG_SCOPE. That would allow the user to compose the log scope (for systems that support that) to be composed of separate arguments.

So LOG_TYPE would only define the type of logs - but LOG_SCOPE would define which log files are included, and which "routes" (like "/redirection" ) will be generated.

For example:

The user sets the LOG_TYPE = NPM. That only implies we are dealing with NPM-style logs.

The user can then set LOG_SCOPE to any of the following, since the value is composed of flags for an NPM system:

• A = "A"ccess logs, aggregated. It makes an /access route available. (Works like the current NPM LOG_TYPE.)
• E = "E"rror logs, aggregated. It makes an /error route available. (Works like current NPM+ALL LOG_TYPE, but without redirection logs.)
• R = "R"edirection logs, aggregated. It makes a /redirection route available. (Works like the current NPM+ALL LOG_TYPE, but without error logs.)
• a = "a"ccess logs, separate per host. It makes an /access-n route(s) available, e.g. access-42 for the access logs of host with id of 42. Also generates /access-fallback for the error logs of the NPM fallback host.
• e = "e"rror logs, separate per host. It makes an /error-n route(s) available, e.g. error-42 for the error logs of host with id of 42. Also generates /error-fallback for the error logs of the NPM fallback host.
• r = "r"edirection logs, separate per host. It makes an /redirection-n route(s) available, e.g. redirection-42 for the redirection logs of redirection-host with id of 42.

So, then the user can set LOG_SCOPE to any combination of flags such as:

• LOG_SCOPE = A (the same as the current NPM setting)
• LOG_SCOPE = A,R (the current NPM+R setting)
• LOG_SCOPE = A,R,E (the current NPM+ALL setting)
• LOG_SCOPE = a,E (separate access logs per host - but just one aggregate error logs report)
• ...and so forth, any valid combination of eg A, R, E, a, r, e

That would give users complete control of which logs get processed (for NPM, at least).

[wernerm]

@xavier-hernandez Hello, I think you mentioned a while ago that you had a chance to implement some of this (I think you mentioned it in the Discussions).

Have you made a release with a similar feature yet, or maybe have another repo available that I can test?

[xavier-hernandez]

@wernerm currently there is nothing that divides it up like your asking but I'll look into what you're asking. Creating that LOG_SCOPE for A,R,E shouldn't be that difficult however separating them would. I'm not sure who that would impact a PC though :) running all those instances.

[xavier-hernandez]

@wernerm I can spinoff a branch/tag for you test if you'd like. We can see how it turns out?

[wernerm]

@wernerm I can spinoff a branch/tag for you test if you'd like. We can see how it turns out?

Hi @xavier-hernandez that is a good idea, if you can find the time. We will help test it!

@wernerm currently there is nothing that divides it up like your asking but I'll look into what you're asking. Creating that LOG_SCOPE for A,R,E shouldn't be that difficult however separating them would. I'm not sure who that would impact a PC though :) running all those instances.

Well, the "A","R","E" would work almost like the existing options, just including or excluding some log types. So it's just more specific.

However, yes, I totally agree that the proposed things like "a","r","e" could get intense if there are many busy hosts... Maybe something like "host filters" could be introduced, but that would complicate things even more... (Where does one stop the madness? Everyone will need to read the README more carefully :) ) On the other hand that's the nice thing - the end-user can decide how to configure the instances and what options to choose to enable, depending on their deployment scenario? (The new options we propose should exactly give more fine-grained control in some instances).