Skip to content

Pulp tunings #2565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Pulp tunings #2565

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
77b2446
REX pull-based transport
Imaanpreet Oct 31, 2022
4446b96
Merge branch 'theforeman:master' into foreman-3.3.0.17-1.el8sat.noarch
Imaanpreet Apr 19, 2023
ccf2b85
pulp_tunings
Imaanpreet Oct 30, 2023
1e9cd9c
Apply suggestions from code review
Imaanpreet Oct 31, 2023
81c0615
changes discussed over the PR
Imaanpreet Nov 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions guides/common/modules/con_pulp-tuning.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
[id="Pulp_Tunings_{context}"]
= Pulp Tunings

Pulp is a service that handles repository and content management.
Pulp ensures efficient storage space by not duplicating packages even when requested by Content Views in different organizations.
28 changes: 28 additions & 0 deletions guides/common/modules/proc_configuring-pulpcore-api-workers.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
[id="Configuring_Pulpcore_API-Workers_{context}"]
= Configuring Pulpcore API Workers

The pulpcore-api workers respond to incoming API requests.
Copy link
Member

@ekohl ekohl Jan 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And some content types, such as containers.

Fewer API workers consume less memory and results in better performance.
Comment on lines +4 to +5
Copy link
Member

@ekohl ekohl Jan 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is questionable advice. Not enough workers means you can't serve the requests in parallel. Taking your advice to the extreme means I simply have 1 worker. But with enough clients, you can saturate that and performance is lower than it could be.

The API service is used by Foreman (via Katello), but also by container content. If your tests didn't test any container workloads then you're only testing a fraction of what it needs to serve.

Copy link
Contributor

@maximiliankolb maximiliankolb Mar 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Imaanpreet Is this something you've observed? Do you have any numbers on this?


For example, we have compared these two setups:

[width="100%",cols="50%,50%",options="header",]
|===
|{Project} VM with 8 CPUs, 32 GiB RAM, 3 pulpcore-api_workers |{Project} VM with 8 CPUs, 32 GiB RAM, 10 pulpcore-api_workers
Copy link
Member

@ekohl ekohl Jan 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default tuning for this is min(4, $cpu_count). So the user would have 4 workers. There is no way they get 10 unless they know what they're doing. That makes me question this advice.

|===

Outcome looks similar in both cases, so the recommendation would be to lower the number of API workers because of the improvement in memory consumption.

.Procedure
. Update the pulpcore api_service_worker_count on your {ProjectServer} or {SmartProxyServer} in the custom-hiera.yaml file.:
Copy link
Contributor

@maximiliankolb maximiliankolb Oct 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
. Update the pulpcore api_service_worker_count on your {ProjectServer} or {SmartProxyServer} in the custom-hiera.yaml file.:
. Update the pulpcore api_service_worker_count on your {ProjectServer} or {SmartProxyServer} in the `custom-hiera.yaml` file.:

I am unsure if we should recommend users editing this file vs. using foreman-installer.

Copy link
Member

@ekohl ekohl Oct 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've always taken a stance that custom-hiera.yaml is unsupported. Anything that's written in docs, so we can't refer to it.

If there's always a recommendation to lower the number of workers, that should be filed as a bug. Telling users "we have poor defaults, always change this" is a poor experience.

My suggestion is to drop this chapter altogether.

Copy link
Member

@ekohl ekohl Oct 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And if it is common to change it, an RFE to add a parameter as was done with the worker count is also a valid solution.

Copy link
Contributor Author

@Imaanpreet Imaanpreet Oct 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @ekohl: I totally agree on custom-hiera.yaml is unsupported part. Do you mean to dropping off this section or proc_configuring-pulpcore-workers as well?

Copy link
Member

@ekohl ekohl Oct 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm questioning the usefulness of proc_configuring-pulpcore-workers. When I read it, it comes down to "here's a parameter you can change, but we noticed it doesn't make a difference". So as a reader of that, is it adding any value or just confusing me?

Copy link
Contributor Author

@Imaanpreet Imaanpreet Oct 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool, let's drop this section.

Copy link
Contributor Author

@Imaanpreet Imaanpreet Nov 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @ekohl , I have removed this chapter proc_configuring-pulpcore-workers from this PR.

Regarding proc_configuring-pulpcore-api-workers.adoc section, can we suggest updating pulpcore api_service_worker_count with another method, for example - via foreman-installer?

Copy link
Member

@ekohl ekohl Jan 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding proc_configuring-pulpcore-api-workers.adoc section, can we suggest updating pulpcore api_service_worker_count with another method, for example - via foreman-installer?

Currently this is the code that sets it:
https://github.com/theforeman/puppet-pulpcore/blob/297a48acafcbdb9871689b6e267598c7414692c2/manifests/init.pp#L246-L247

https://projects.theforeman.org/issues/36957 was opened by @maximiliankolb to expose a parameter for this and I just opened theforeman/puppet-foreman_proxy_content#467.

I still think that most users shouldn't touch this and if our default tuning is poor, then it should be fixed.

+
[options="nowrap", subs="+quotes,verbatim,attributes"]
----
pulpcore::api_service_worker_count: X
----
. Apply your changes:
+
[options="nowrap", subs="+quotes,verbatim,attributes"]
----
# {foreman-installer}
----
21 changes: 21 additions & 0 deletions guides/common/modules/proc_configuring-pulpcore-workers.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
[id="Configuring_Pulpcore_Workers_{context}"]
= Configuring Pulpcore Workers

The pulpcore workers are the ones that fetch things from the content delivery network, process metadata etc. and the number of those will have an influence on the sync performance.

Decreasing the number does not give any benefit from the sync speed point of view.
Increasing the number gives a marginal benefit from the sync speed point of view, at the expense of almost doubling system load and increasing the PG load.
For example, we have compared these two setups:

[width="100%",cols="50%,50%",options="header",]
|===
|{Project} VM with 8 CPUs, 32 GiB RAM, 4 pulpcore-api_workers |{Project} VM with 8 CPUs, 32 GiB RAM, 16 pulpcore-api_workers
|===

.Procedure
. Update the pulpcore worker_count on your {ProjectServer} or {SmartProxyServer} :
+
[options="nowrap", subs="+quotes,verbatim,attributes"]
----
# {foreman-installer} --foreman-proxy-content-pulpcore-worker-count _My_Pulpcore_Worker_Count_
----
6 changes: 6 additions & 0 deletions guides/doc-Tuning_Performance/master.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,12 @@ include::common/modules/proc_configuring-puma-db-pool.adoc[leveloffset=+3]

include::common/modules/proc_manually-tuning-db-pool.adoc[leveloffset=+3]

include::common/modules/con_pulp-tuning.adoc[leveloffset=+2]

include::common/modules/proc_configuring-pulpcore-api-workers.adoc[leveloffset=+3]

include::common/modules/proc_configuring-pulpcore-workers.adoc[leveloffset=+3]

include::common/modules/con_apache-httpd-performance-tuning.adoc[leveloffset=+2]

include::common/modules/proc_configuring-the-open-files-limit-for-apache-httpd.adoc[leveloffset=+3]
Expand Down