Add client identifier header to HTTP requests by sorentwo · Pull Request #1075 · hexpm/hex

sorentwo · 2025-06-02T14:33:57Z

Creates an anonymized repository identifier based on the SHA of the first commit, hashed with SHA256 for additional privacy. The identifier is sent as an x-hex-client-id header when available.

For example, in the hex repo itself:

iex(1)> Hex.Utils.client_identifier()
"0f3d954cda05af53f8921bc77be8d2fe1697daded03115fd17b7a3aef32b23b1"

The value can then be used by hex.pm and private hex servers to more accurately count requests. That will give a clearer picture of how many unique applications are using a particular package, rather than just the raw number of downloads.

Some followup steps if this approach is accepted:

Fix test failures in CI from environment mismatch
Potentially cache the value to prevent repeated system calls
Add a flag to disable sending the client id

/cc @wojtekmach

Creates an anonymized repository identifier based on the SHA of the first commit, hashed with SHA256 for additional privacy. The identifier is sent as an `x-hex-client-id` header when available.

lib/hex/utils.ex

lib/hex/http.ex

The new variable allows users to opt-out of identification by setting the value to anything other than `1` or `true`. It remains enabled by default.

lib/hex/utils.ex

sorentwo · 2025-06-11T14:45:55Z

lib/hex/utils.ex

+  - The current directory isn't within a git repository
+  """
+  def repo_identifier do
+    with :unset <- Process.get(:hex_repo_identifier, :unset),


It's not clear how useful this will be in practice, as I imagine fetching deps is parallelized across multiple processes. However, the codebase appears to support back to Elixir v1.6, hence OTP 19, which is before persistent_term was available and we can't cache between processes very easily.

What does everybody think? Is this useful? Is it worth looking at ETS for caching?

could you look into using Hex.State (an agent) for storage? We even have some affordances there for env variables but not sure if they'd help in this particular case, i.e. I don't think we need to be able to set a custom repo id via env.

Thanks for the tip. I switched to Hex.State to check whether the identifier is enabled or disabled, as well as caching the value, which is far more convenient.

Do you think they should be combined into a single repo_identifier value that's either nil, false, or the cached binary?

Prevents fetching the same identifier on every client call by caching the value in the current process dictionary. While the git command isn't particularly slow, this should avoid spawning during repeated http calls.

Switch from a raw `System.get_env` to using the normalized state agent. Also prevent STDERR from leaking when called outside of a git repository.

zachdaniel · 2025-07-15T19:26:41Z

I'd love to see this make it in, would be very impactful information for a lot of libraries and frameworks. Anything I can do to help?

ericmj

Is this considered PII and how do we need to inform users about this change other than a note in the changelog?

Other tracking we've talked about adding is:

Session ID header, which would be a unique ID for each mix deps.get
A header if the env var CI is set, to help identity real users vs CI runs

lib/hex/utils.ex

The repo identifier value is now cached with an agent, which also serializes access to fetch the state. This also removes the value from `Hex.State`, which was previously used for caching but is ultimately unnecessary.

sorentwo · 2025-09-18T14:57:36Z

@ericmj Took me a while, but I've addressed both of your requests.

ericmj · 2025-09-24T12:34:14Z

Thank you @sorentwo 💜

hugobarauna · 2025-09-24T13:53:09Z

lib/hex/repo_identifier.ex

+
+  Returns `nil` when:
+
+  - The `HEX_REPO_IDENTIFIER` environment variable is set to anything other `1` or `true`


Maybe it should be “ HEX_NO_REPO_IDENTIFIER”

?

Either version is fine by me (HEX_REPO_IDENTIFIER=false or HEX_NO_REPO_IDENTIFIER=true)

I mean, the doc here says: HEX_REPO_IDENTIFIER

But the code is actually using NO_HEX_REPO_IDENTIFIER:

https://github.com/sorentwo/hex/blob/5e329783583aa4d5929953ce95a73d68590406d2/lib/hex/state.ex#L115-L116

So maybe the docs should be changed to NO_HEX_REPO_IDENTIFIER.

Or, I'm missing something. 😅

Oh, it's already fixed: https://github.com/hexpm/hex/blob/68758fbffda1069fda61fa503b83844b39513fbe/lib/hex/repo_identifier.ex#L10C10-L10C32

I'm sorry. 😅

My mistake! This PR drifted a while and I completely forgot about the name change. Glad @ericmj took care of it =)

ericmj · 2025-10-24T15:49:04Z

We need to ask for consent before sending this telemetry according to GDPR because it creates a persistent identifier and is not required for the functionality of Hex.

Since it has limited usefulness when it's opt-in I am inclined to remove it.

cc @sorentwo

hugobarauna · 2025-10-24T18:32:51Z

We need to ask for consent before sending this telemetry according to GDPR because it creates a persistent identifier and is not required for the functionality of Hex.

Since it has limited usefulness when it's opt-in I am inclined to remove it.

cc @sorentwo

@ericmj I’m not sure this identifier would fall under GDPR.

It’s a git repo–scoped value, not tied to any user or machine. Different people or machines using the same repo would generate the same identifier, reinforcing that it’s linked to the project/git repo, not to a person.

I guess GDPR only applies to data that can identify a natural person, not to identifiers of software artifacts like a Git repository. If it were a machine-specific identifier (like a device ID, IP, or cookie), then it could make a person indirectly identifiable — that’s when GDPR applies. But this one isn’t, so it should be outside GDPR’s scope.

But I'm not an expert on GDPR. Only saying that based on my research and personal experience.

ericmj · 2025-10-24T19:23:26Z

@hugobarauna I see your point but it's enough of an edge case where I think the risk is not worth it.

In most cases a repository is not identifiable to a single person but I could imagine cases where it could be argued that a repository identifier is tied to a single person. For example if you start working on a new project for a month before publishing it to github under your username, any mix deps.get HTTP requests for that repository before you published it could be attributed to you with reasonable confidence.

Add client identifier header to HTTP requests

95452f7

Creates an anonymized repository identifier based on the SHA of the first commit, hashed with SHA256 for additional privacy. The identifier is sent as an `x-hex-client-id` header when available.

sorentwo force-pushed the client-identifier branch from 3839199 to 95452f7 Compare June 2, 2025 14:45

royalicing reviewed Jun 4, 2025

View reviewed changes

lib/hex/utils.ex Outdated Show resolved Hide resolved

ericmj reviewed Jun 7, 2025

View reviewed changes

lib/hex/http.ex Outdated Show resolved Hide resolved

sorentwo added 2 commits June 9, 2025 11:04

Rename identifier header to x-hex-repo-id

6515b14

Respect HEX_REPO_IDENTIFIER environment variable

9de9bfe

The new variable allows users to opt-out of identification by setting the value to anything other than `1` or `true`. It remains enabled by default.

sorentwo commented Jun 11, 2025

View reviewed changes

lib/hex/utils.ex Outdated Show resolved Hide resolved

sorentwo commented Jun 11, 2025

View reviewed changes

sorentwo force-pushed the client-identifier branch from c8c4cf8 to 4e1cc84 Compare June 11, 2025 14:47

Add local caching to repo identifier lookup

ffb5c02

Prevents fetching the same identifier on every client call by caching the value in the current process dictionary. While the git command isn't particularly slow, this should avoid spawning during repeated http calls.

sorentwo force-pushed the client-identifier branch from 4e1cc84 to ffb5c02 Compare June 11, 2025 14:54

Use Hex.State agent for identifier config

100e500

Switch from a raw `System.get_env` to using the normalized state agent. Also prevent STDERR from leaking when called outside of a git repository.

sorentwo force-pushed the client-identifier branch from 15ea8ff to 100e500 Compare June 16, 2025 14:19

Use Hex.State to store cached repo identifier

34d84bc

sorentwo force-pushed the client-identifier branch from dd07cd7 to 34d84bc Compare June 16, 2025 14:30

ericmj reviewed Aug 1, 2025

View reviewed changes

lib/hex/utils.ex Outdated Show resolved Hide resolved

lib/hex/utils.ex Outdated Show resolved Hide resolved

sorentwo added 2 commits September 18, 2025 14:24

Print error message on non-zero identifier message

7d14507

Synchronize cached repo identifier access

37fee2d

The repo identifier value is now cached with an agent, which also serializes access to fetch the state. This also removes the value from `Hex.State`, which was previously used for caching but is ultimately unnecessary.

Merge remote-tracking branch 'origin' into client-identifier

5e32978

ericmj approved these changes Sep 24, 2025

View reviewed changes

ericmj merged commit 1264476 into hexpm:main Sep 24, 2025
8 checks passed

hugobarauna reviewed Sep 24, 2025

View reviewed changes

ericmj mentioned this pull request Nov 6, 2025

Proposal: Package download stats should exclude CI runners hexpm/hexpm#1349

Open


		Returns `nil` when:

		- The `HEX_REPO_IDENTIFIER` environment variable is set to anything other `1` or `true`

Conversation

sorentwo commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zachdaniel commented Jul 15, 2025

Uh oh!

ericmj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sorentwo commented Sep 18, 2025

Uh oh!

Uh oh!

ericmj commented Sep 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ericmj commented Oct 24, 2025

Uh oh!

hugobarauna commented Oct 24, 2025

Uh oh!

ericmj commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sorentwo commented Jun 2, 2025 •

edited

Loading