Add log type include filter #714

jorbaum · 2026-01-05T14:06:49Z

Description

Addresses #711 :

Support both ?include-log-types= and ?exclude-log-types
Throw error in case both are configured. Do not throw an error on misconfigured values for the new config option (consistent with previous HTTP query param usage in binding config), but log a warning (new).

I still need to verify this, but I think a feature flag is not necessary as I tried to make string parsing more efficient by:

running only when source type tag and log type filter is present
strings.IndexByte might take some time, but it is used only with a single char / and strings.IndexByte() should be more efficient than strings.Index()
Looking up content in a map logTypePrefixes rather than in each log line, which should be more efficient

Use both include and exclude log filter

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing performed?

Unit tests
Integration tests
Acceptance tests

Checklist:

This PR is being made against the main branch, or relevant version branch
I have made corresponding changes to the documentation
I have added testing for my changes

chombium

@jorbaum Thanks for the changes.

Generally it looks good, but I have some smaller objections which should be addressed.

src/pkg/egress/syslog/filtering_drain_writer.go

src/pkg/egress/syslog/filtering_drain_writer_test.go

chombium · 2026-01-09T13:55:15Z

src/pkg/egress/syslog/syslog_connector.go

 	DrainData    DrainData `json:"type,omitempty"`
 	OmitMetadata bool
 	InternalTls  bool
+	LogFilter    *LogTypeSet


I don't think that this has to be a pointer.

I tried that at first, but in manager.go line 45 Binding is used as a map key:

sourceDrainMap map[string]map[syslog.Binding]drainHolder

My understanding is that maps in Go are not comparable, so adding LogTypeSet (which is a map[LogType]struct{}) to the Binding struct makes it non-comparable.

AFAICS the alternative to using a pointer would be to implement a custom comparable key for the map, but using a pointer seemed simpler to me.

src/pkg/ingress/bindings/binding_config.go

chombium · 2026-01-09T14:08:56Z

src/pkg/ingress/bindings/binding_config.go

+		default:
+			// Unknown log type, skip it
+			// TODO add unit test for unknown log type
+			d.log.Printf("Unknown log type '%s' in log type filter, ignoring", logType)


If there is a log message here and an app owner wrongly configures the drain, than for every app instance and every app log, a log ,message will be written.

The filter settings should be validated upfront and we should not end up in this situation. At this point we should only have valid drain configuration.
I guess we have to bring in #633 first and than this change.

My understanding is that this code does not run for every log message, but only parsing the syslog drain config. I intentionally left logging out of filtering_drain_writer.go to avoid creating a log for each app log.

After discussing with @chombium we concluded that yhis log gets created for each wrongly configured value in the log types of each syslog drain. This is emitted every few seconds on each VM within cloud foundry. While this is better than for each log line, we would still keep log load at a minimum.

To address this I now made sure to log at most one log line in this method.

src/pkg/ingress/bindings/binding_config.go

chombium · 2026-01-09T16:55:59Z

src/pkg/ingress/bindings/binding_config.go

+	set := make(syslog.LogTypeSet, len(logTypes))
+
+	for _, logType := range logTypes {
+		logType = strings.ToLower(strings.TrimSpace(logType))


The query string in the URL is case sensitive. We should only accept lowercase and not convert eVerY POssibLE string to lower case.

Did so.

Still, I wonder if we should be strict here about this. Not caring about casing might bring us some benefits as this might be something that users accidentally misconfigure. It should not hurt much to support both I think.

src/pkg/ingress/bindings/binding_config_test.go

chombium · 2026-01-09T17:05:15Z

@ZPascal Wdyt about this PR? It addresses #711

jorbaum · 2026-01-12T08:04:26Z

@chombium I left a bunch of replies for points that are not yet clear to me. All the rest look good.

Thanks for the thorough review! You found a bunch of nice things.

jorbaum

Worked in some comments.

src/pkg/egress/syslog/filtering_drain_writer.go

jorbaum · 2026-01-12T16:02:28Z

src/pkg/ingress/bindings/binding_config.go

+	set := make(syslog.LogTypeSet, len(logTypes))
+
+	for _, logType := range logTypes {
+		logType = strings.ToLower(strings.TrimSpace(logType))


Did so.

Still, I wonder if we should be strict here about this. Not caring about casing might bring us some benefits as this might be something that users accidentally misconfigure. It should not hurt much to support both I think.

src/pkg/ingress/bindings/binding_config.go

jorbaum

Worked in most comments. Still 2 left to clarify.

jorbaum · 2026-01-13T10:21:47Z

src/pkg/ingress/bindings/binding_config.go

+		default:
+			// Unknown log type, skip it
+			// TODO add unit test for unknown log type
+			d.log.Printf("Unknown log type '%s' in log type filter, ignoring", logType)


After discussing with @chombium we concluded that yhis log gets created for each wrongly configured value in the log types of each syslog drain. This is emitted every few seconds on each VM within cloud foundry. While this is better than for each log line, we would still keep log load at a minimum.

To address this I now made sure to log at most one log line in this method.

src/pkg/ingress/bindings/binding_config_test.go

jorbaum · 2026-01-19T15:30:27Z

Feel free to take another look at it.

cf-foundation-community-automation bot added this to Application Runtime Platform Working Group Jan 5, 2026

cf-foundation-community-automation bot moved this to Inbox in Application Runtime Platform Working Group Jan 5, 2026

jorbaum added 4 commits January 8, 2026 10:41

Add log type include filter

c100636

Support exclude and include

d21490c

Error when using both include and exclude types

4fb3117

Introduce logging to binding config

dcc096a

jorbaum force-pushed the add-log-type-filter branch from 43b0941 to 9fcc372 Compare January 8, 2026 09:41

Add some more unit tests

bccd92c

jorbaum force-pushed the add-log-type-filter branch from 9fcc372 to bccd92c Compare January 8, 2026 09:43

chombium requested changes Jan 9, 2026

View reviewed changes

Fix wrong word

f5b43ae

jorbaum added 2 commits January 12, 2026 16:43

Create LogType as string, move to package

8487400

Prefix log type constants with LOG_

708301d

jorbaum commented Jan 12, 2026

View reviewed changes

jorbaum added 4 commits January 13, 2026 10:11

Rename value to sourceType

abaf216

Only emit one warning on wrong drain config

b78377d

Rename DrainParameterParser in test

2029984

Add tests for the include and exclude filters

6425efd

jorbaum commented Jan 13, 2026

View reviewed changes

jorbaum added 3 commits January 19, 2026 16:21

Move new log filter test to table test

20364ed

Move next test to table

c98468d

Move drain data for old parameter test to table

5614b36

Add log type include filter #714

Are you sure you want to change the base?

Add log type include filter #714

Conversation

jorbaum commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Testing performed?

Checklist:

Uh oh!

chombium left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorbaum Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chombium commented Jan 9, 2026

Uh oh!

jorbaum commented Jan 12, 2026

Uh oh!

jorbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jorbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jorbaum commented Jan 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jorbaum commented Jan 5, 2026 •

edited

Loading

jorbaum Jan 12, 2026 •

edited

Loading