[RFC]: Removal of Lua

[LaurenceJJones]

When I first started this project I wanted it to be completely independent of Lua based on our previous experience of cs-haproxy-bouncer. However, one glaring issue that arouse was "How do we render templates for ban and captcha?" well first thought was we could use the spoe protocol to pass back a full template string for haproxy to send back to clients, however, we hit a roadblock that the spoe protocol can only pass 4kb per key inside its value.

This is probably fine for our default template but if users wanted to get fancy with the templating engine then simply this would not scale. So to get a version out the door we fell back to our good ole pal Lua to render the template. At first this was okay, however, with more and more users using the spoa remediation its clear it wont scale to our minimum performance requirements as Lua is single threaded and rather an engine within haproxy.

We kind of mitigated the issue to only allow the Lua to run when needed within #62 but still any amount of Lua is too much in my opinion.

So what this RFC for?

Well we have an idea for an experimental feature which may in turn become the default to allow the spoa process to expose a http server that can render the templates. Then instead of haproxy offloading the rendering to Lua you instead direct the request to go to the http server instead of your traditional backend.

The pros:

• No lua needed anymore
• No mounting of files shared between containers
• Deployments inside kubernetes become simpler because no shared files
• Clear separation between what spoe should handle and the http server should handle (captcha validation)

The cons:

• More configuration for http server, must allow (m)TLS in case requests means sending outside of the primary haproxy server
• More code debt for experimental stages until RFC is accepted

[github-actions]

@LaurenceJJones: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Documentation to see if your issue can be self resolved.
2. You can also join our Discord

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@LaurenceJJones: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[diroots]

hi @LaurenceJJones one question, mainly for the ban template.

iirc, a ban action should throw a 403
why need to load via lua whereas is can just be called with something like

http-request deny deny_status 403 errorfile path/to/custom/ban-template.http if { var(txn.crowdsec.remediation) -m str "ban" }

as it seems the only dynamic part of this template is the contact-us address, and all the rest is a static html page ?

i get that, for the captcha, there is a bit more dynamic things to inject to the template and more complex, but for the ban one at least, should it be enough?

it seems it could even use internal templates ;: https://www.haproxy.com/documentation/haproxy-configuration-tutorials/alerts-and-monitoring/error-pages/#use-an-error-template-file

[LaurenceJJones]

Thank you @diroots perfect example why I like to create these RFC's as I used haproxy but not extensively to know every little part.

You are right with that template system we could do it for both ban and captcha technically if the system has access to the txn we could do %[var(txn.crowdsec.X)] to inject into the template.

will investigate this soon just flushing out 0.2.1

[diroots]

ok i confirm that i could load our custom 403 error page for the ban action(the one we use for now) with that simple call instead of the lua.crowdsec_handle, no lua needed.

also, by adding some acl on the crowdsec.cfg speo engine definition, we can trigger the spoe checks following the rules we want, like any other haproxy definition (or avoid triggering it for un-necessary calls, like whitelisted ips or anything else)

spoe-message crowdsec-ip
    acl is_signup path_beg /path/to/signup
    args id=unique-id src-ip=src src-port=src_port
    event on-client-session if is_signup

[LaurenceJJones]

ok i confirm that i could load our custom 403 error page for the ban action(the one we use for now) with that simple call instead of the lua.crowdsec_handle, no lua needed.

also, by adding some acl on the crowdsec.cfg speo engine definition, we can trigger the spoe checks following the rules we want, like any other haproxy definition (or avoid triggering it for un-necessary calls, like whitelisted ips or anything else)

spoe-message crowdsec-ip
    acl is_signup path_beg /path/to/signup
    args id=unique-id src-ip=src src-port=src_port
    event on-client-session if is_signup

Yeah im currently testing it also! one limitation which may be configurable is lf-file only allows 16kb max file input. Which is totally fine for our default pages, but if a user wants to get fancy they may hit a limitation.

Yeah I know about triggering an if clause on the spoe-messages, in future we will need to move to spoe-groups #97 especially for crowdsec-http message since we need the real IP and spoe-groups actually trigger after ACL checks and overriding the src ip. (makes it easier to bypass for whitelisted ips or routes than editting the crowdsec.cfg)