netrc should be able to ignore some tokens like 'protocol'

[LucHermitte]

Feature or enhancement

Proposal:

netrc._parse() rejects unknown tokens.

So far so good except some tools like git also use netrc files for authentication purposes through git-credential-netrc helper.

The issue is that git-credential-netrc expects a few other tokens like protocol that are currently rejected by netrc._parse()

It would be nice to be able to just ignore those tokens instead of failing the parsing of the netrc file.

I can't tell whether we should just register these exact tokens as "to be ignored", or if it would be better to have a way for netrc users to register the tokens they wish to ignore.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

none found.

[picnixz]

I never used that library and I don't have the code in front of me. But here's what I would have assumed:

• The stdlib netrc client should strictly follow https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html. AFAICT, that's what the docs say, namely "The netrc class parses and encapsulates the netrc file format used by the Unix ftp program and other FTP clients", so it shouldn't do more what it promises (and what the standard describes).

• Librairies that want to parse something that is netrc-like are ... not actually parsing netrc but an extension thereof. This is kind of the same issue when you want to support comments in a JSON file which are not supported by the standard itself I think.

I'll leave the feature opened but I think this kind of proposal should be a third-party package rather than part of the stdlib.

[LucHermitte]

If only non standard entries were rejected instead of the whole file, it would have been enough for me. My request for enhancement is not about extracting these non standard fields.

I perfectly understand that this could/should be fixed in 3rd party packages. Unfortunately, any python tool that wishes to parse netrc files will be glad enough to rely of the stdlib. Then, there would be people wishing to use the same .netrc file to store git credentials (in order to use git through https where ssh is not possible), and credentials for other services. Because of the strong enforcement of the syntax, we'll need to have several netrc files.

[picnixz]

If only non standard entries were rejected instead of the whole file, it would have been enough for me

Would you have something like an error policy in mind where you would by default outright reject the entries? (i.e., raise) but where you could also ignore them? (in which case it would just skip them)? I think by default, the entire file should be rejected because the netrc parser can be used both as a parser and as a validatation tool.

The code is very old though it seems that there were some improvements for users and having an 'ignore' policy could be a small-enough feature.

[LucHermitte]

That would be perfect!

What I'm not sure is: what would be the best granularity?

• option 1: we accept an entry where there are unknown fields, but we just ignore the field
• option 2: we only accept entries where there are no unknown fields.

IOW, given:

machine github.com
  login mylogin
  password my-auth-token
  protocol https

machine dataspace.copernicus.eu
  login myotherlogin
  password my-password

the question is should authenticators('github.com') return something? Returning something would be simpler to implement IMO. But returning None would also make sense -- as only git-credential-netrc would need information for "github.com".

[picnixz]

Personally, I'm in favor of rejecting the entire machine section. This section is not a valid netrc one per se so I would just consider it as a "fake" section. It's always easier to have multiple handle policies in the future.

We have no core dev actively maintaining netrc and we only had a few committers on this part of the stdlib. So I think we can first try to suggest an implementation of something simple and see what reviewers would do (I personally don't have a committer right so I can only suggest and possibly try to find who to @ when it's time).

[picnixz]

By the way, do you want to work on a PR or should I take it? (if you want to familiarize yourself with CPython, this one is a not-that-hard PR to implement I think)

[LucHermitte]

I can work on it. But I'll need a few days before I've enough time to do it correctly.

[picnixz]

In the meantime let's ask someone that works with the configparser in the stdlib for insights, namely @jaraco: do you think my suggestion makes sense?

[jaraco]

In working to understand the problem, I illustrate the reported failure:

 draft 🐚 cat > .netrc
machine foo
login bar
password secret
protocol https
 draft 🐚 chmod 700 .netrc
 draft 🐚 env HOME=. py -m netrc
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/python/lib/python3.13t/netrc.py", line 192, in <module>
    print(netrc())
          ~~~~~^^
  File "/opt/python/lib/python3.13t/netrc.py", line 75, in __init__
    self._parse(file, fp, default_netrc)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.13t/netrc.py", line 140, in _parse
    raise NetrcParseError("bad follower token %r" % tt,
                          file, lexer.lineno)
NetrcParseError: bad follower token 'protocol' (./.netrc, line 4)
 draft [1] 🐚 sed -i -e '/protocol/d' .netrc
 draft 🐚 env HOME=. py -m netrc
machine foo
	login bar
	password secret

I'm on the fence.

On one hand, following the spec strictly for a narrowly-defined, non-extensible format like .netrc is a very appropriate response. After all, if git isn't willing to put in the work to alter or extend the spec, then it's really out of bounds for the format.

On the other hand, the format is pretty straightforward, so it's no surprise that a git developer might have imagined that the spec is extensible, following an obvious pattern for nested key/value pairs. The main difference with the ini format is was intended to generalize and didn't specify a narrow scope of support.

Looking at the netrc implementation, it too is constrained to a narrow scope.

Here's one hot take: Python should remove netrc from the stdlib. I'm actually surprised it didn't get the axe in PEP 594. I'll bet it wasn't considered for removal because ftplib depends on it. Perhaps the module should be deprecated for use outside of the ftplib or deprecated entirely.

After all, its presence encourages a weak security regime - passwords in plain text on the file system. Support for it perpetuates tools like git using it for authentication. I couldn't find any documentation on git-credential-netrc. Even the source is a perl script with not as much as a README. I do see some guides on using git-credential-netrc with GPG in order to avoid storing the credential in plain text, but of course the netrc module doesn't support that either.

Considering the proposed change, "rejecting the entire machine section [when it contains unrecognized follower tokens]," it seems reasonable at first blush. I suspect the implementation will be fairly straightforward, as the error can be replaced by a break. It feels wrong, however, to silently ignore either the machine or machine values without at least logging a warning, something like:

warnings.warn('encountered unrecognized key "protocol", discarding machine "github.com"')

or

warnings.warn('ignoring unrecognized key "protocol"')

Would adding a warning be acceptable for your use-case? Can you say more about your use-case? Are you using netrc directly or from ftplib?

Personally, I prefer to suppress unrecognized keys to suppressing machines containing unrecognized keys and to emit a warning when doing so, but also to encourage users to move away from netrc where possible.

[picnixz]

Would adding a warning be acceptable for your use-case

Not my use case, but I think can indeed emit a warning, say "ignoring section [...] with unrecognized key [...]". I still prefer ignoring the entire section than just the key though but we can have a policy that ignore keys and one that ignore section.

---

Off-topic but looking at the failures, we could also improve the error message (it's a bit cryptic for the end-user).

[LucHermitte]

A warning is fine as far as I'm concerned.

My use case is:

1. I use git-credential-netrc on one hand to work around corporate proxies that disable ssh protocol
2. I'm using sentineleof, a tool that parses .netrc files to extract a pair of login/passwd that is used to download files through REST protocols.

And I'd rather have only one file.
(As you've pointed out, git-credential-netrc is able to decrypt on-the-fly with gpg, and it would have been wonderful to have it also. But it's definitively out-of-scope here IMO.)

[barneygale]

It's been a long time since I've done any perl, but IIUC the git-credential-netrc parser ignores unknown token types, and so:

machine foo
  action login
  protocol https

... would be parsed by ignoring action, parsing login protocol as a key-value pair, and then ignoring https. This doesn't seem very robust, but it's a limitation of the format: there's no way to skip past unknown extensions without making nonstandard assumptions.

As an aside, the popular requests library reads .netrc files and sets http authentication headers for matching domains.