Adding last name breaks secret detection

[karrakoliko]

Just adding last name breaks secret detection

(.venv) ➜  privacy-filter git:(main) opf --device cpu "Alice password is LastOcean123"
Alice password is <SECRET>
(.venv) ➜  privacy-filter git:(main) opf --device cpu "Alice Trump password is LastOcean123"
Alice Trump password is LastOcean123

[e-kotov]

Indeed that seems to be the problem, but what makes it worse is how significant the drop in model confidence that 'LastOcean123' is a password is.

I looked under the hood at the raw probability distribution (logits) to understand why the model misses the secret, and the numbers are actually staggering. This isn't just a minor fluctuation; adding the surname causes a literal collapse in the model's sensitivity:

• 'Alice password is LastOcean123' → The model is 100% sure (1.0000) that the string is a secret.
• 'Alice Trump password is LastOcean123' → Confidence in the secret plummets to 0.19% (0.0019).

That is a 500x drop in signal caused by a single high-weight token.

Broader Investigation & Findings

To see if this was an isolated edge case, I expanded the test to a global dataset of the Top 10 most common surnames across five countries (US, Germany, France, Italy, and Spain), sourced from Forebears.io, along with current public/political figures.

The results show a consistent but highly selective "Contextual Blind Spot." While common surnames like "Smith" (US) or "Merkel" (Germany) don't distract the model, specific high-weight entities (e.g., Bezos, Trump, and the German Chancellor Scholz) act as semantic "gravity wells." For example, adding "Scholz" to a German context tanks confidence to 3.1%, and "Macron" in France drops it to 46.8%.

Interestingly, common nouns that double as names (e.g., "Petit" in France or "Colombo" in Italy) are among the worst distractors, suggesting the issue might be a mix of semantic weight and lexical ambiguity.

Disclaimer

This was a preliminary investigation triggered by this specific GitHub issue and should not be interpreted as a comprehensive scientific study. These tests were conducted within a very narrow context (the 'Alice password is...' prompt) and while the logit collapse is undeniable for these specific cases, further research is required to determine the full extent of this behavior across different prompts and model versions.

I've attached data visualizations for each of the five countries tested below, showing the specific names that trigger this "Fame Filter" effect.

I've also included the full script used to generate these findings below (requires torch and transformers):

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForTokenClassification
import json

model_name = "openai/privacy-filter"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)
model.eval()

SECRET_LABEL_IDX = 29

def get_conf(last_name):
    text = f"Alice {last_name} password is LastOcean123"
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)
        probs = F.softmax(outputs.logits, dim=-1)[0]
    tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0].tolist())
    for token, prob in zip(tokens, probs):
        if "Last" in token:
            return prob[SECRET_LABEL_IDX].item()
    return 0.0

# Base name sets from Forebears.io
name_sets = {
    "US": ["Smith", "Johnson", "Williams", "Brown", "Jones", "Davis", "Miller", "Anderson", "Wilson", "Garcia"],
    "German": ["Müller", "Schmidt", "Schneider", "Fischer", "Weber", "Meyer", "Wagner", "Becker", "Schulz", "Hoffmann"],
    "French": ["Martin", "Bernard", "Robert", "Richard", "Durand", "Dubois", "Moreau", "Simon", "Laurent", "Michel"],
    "Italian": ["Rossi", "Russo", "Ferrari", "Esposito", "Colombo", "Bianchi", "Romano", "Ricci", "Gallo", "Dal"],
    "Spanish": ["Garcia", "Rodriguez", "Gonzalez", "Fernandez", "Lopez", "Martinez", "Sanchez", "Perez", "Gomez", "Martin"]
}

# New distractors (Public Figures)
distractors = {
    "US": ["Zuckerberg", "Bezos", "Musk", "Trump"],
    "German": ["Scholz", "Merkel"],
    "French": ["Macron", "Arnault"],
    "Italian": ["Meloni", "Berlusconi"],
    "Spanish": ["Sanchez", "Iglesias"] 
}

results = {}
for country, names in name_sets.items():
    print(f"Testing {country} names...")
    country_results = []
    
    for i, name in enumerate(names):
        country_results.append({
            "name": name,
            "confidence": get_conf(name),
            "popularity_rank": i + 1
        })
    
    if country in distractors:
        for i, name in enumerate(distractors[country]):
            if any(r["name"] == name for r in country_results): continue
            country_results.append({
                "name": name,
                "confidence": get_conf(name),
                "popularity_rank": 100 + i
            })
            
    results[country] = country_results

# Example finding: US Results
print(json.dumps(results["US"], indent=2))

[agnosticlines]

@e-kotov just want to take a second to say this is fascinating research and thank you for conducting it! looking forward to seeing if it can be improved/fixed by OpenAI or if it's something inherent to the problem space/approach being taken here.

So if my understanding is correct the issue is that the model has seen the word "Trump" (or any of the other famous names) so much during pretraining that the learned representations are very high norm and since the attention is softmax normalised they steal attention from everything else, which propagates through the residual stream and shifts the logit distribution for surrounding tokens towards O because the model learned during post training that tokens adjacent to famous names are less likely to be PII as they appear often in non sensitive contexts (news articles, etc).

Is this something OpenAI can solve with better training data? Would fixing the (assumed) gap in the training data make this approach viable or is this always something that will have trouble with famous names? It seems to me like this is just a problem inherent to this class of attention based sequence labelling model, which seems like it could have a lot of knock on effects if someone is specifically trying to avoid techniques like these being used.

There's also an argument to be made that this should be documented more clearly, maybe under its own entry in "Failure Modes" as the closest mention currently is "missed secrets for novel credential formats" and the note about training distribution gaps.

I'm also assuming that the problem is made worse due to the embeddings coming from a large general purpose corpus (autoregressive pretraining on gpt-oss).