linguist-generated tag not working properly.

[ApoorvGuptaAi]

Describe the bug

linguist-generated tag not working properly.

$ cat .gitattributes 
# Ignore these files for PR and some other processes.
# Based on https://docs.github.com/en/github/administering-a-repository/customizing-how-changed-files-appear-on-github
# Guide: https://github.com/github/linguist

# Protobuf
gui/src/protobuf/truera/protobuf/* linguist-generated
gui/src/protobuf/truera/protobuf/public/model_pb.d.ts linguist-generated
python/truera/protobuf/* linguist-generated

# OpenAPI
python/truera/modelrunner/client/* linguist-generated
gui/src/network/mrc/src/ linguist-generated

gui/package-lock.json linguist-generated

Expected behaviour

Linguist for gui/src/protobuf/truera/protobuf/public/model_pb.d.ts says generated file.
(This does not appear to be true for github either)

Actual behaviour:

$ github-linguist gui/src/protobuf/truera/protobuf/public/model_pb.d.ts
model_pb.d.ts: 156 lines (139 sloc)
  type:      Text
  mime type: video/MP2T
  language:  TypeScript
  appears to be a vendored file

Some files are marked as generated though:

$ github-linguist gui/package-lock.json 
package-lock.json: 0 lines (0 sloc)
  type:      Text
  mime type: application/json
  language:  JSON
  blob is too large to be shown
  appears to be generated source code

Additional notes

Version:

$ github-linguist --version
  Linguist v5.3.3
  Detect language type for a file, or, given a repository, determine language breakdown.

  Usage: linguist <path>
         linguist <path> [--breakdown] [--json]
         linguist [--breakdown] [--json]

[lildude]

Directly querying a file using github-linguist doesn't honour the overrides in .gitattributes as it doesn't pull in all the repo information and calls vendored? from the blob_helper.rb directly which doesn't pull in the overrides:

https://github.com/github/linguist/blob/a3de9b8d54d0b10c41ed80ab2ee9c4bef45c49da/lib/linguist/blob_helper.rb#L219-L232

When querying the repo as a whole, Linguist uses the vendored? from lazy_blob.rb which does honour overrides:

https://github.com/github/linguist/blob/a3de9b8d54d0b10c41ed80ab2ee9c4bef45c49da/lib/linguist/lazy_blob.rb#L56-L62

When it comes to checking whether a file is vendored or generated, Linguist (actually rugged) will query in the order listed:

https://github.com/github/linguist/blob/a3de9b8d54d0b10c41ed80ab2ee9c4bef45c49da/lib/linguist/lazy_blob.rb#L7-L11

... which as you can see, shows vendored before generated.

As your file is considered vendored by Linguist here:

https://github.com/github/linguist/blob/a3de9b8d54d0b10c41ed80ab2ee9c4bef45c49da/lib/linguist/vendor.yml#L179-L180

... it'll use that and not continue onto testing if it is generated as neither count towards the stats. Thus, if you wish for the file to be considered generated and not vendored, you'll need to mark it is "unvendored" and generated in the override, eg:

gui/src/protobuf/truera/protobuf/public/model_pb.d.ts -linguist-vendored linguist-generated

This won't adjust the CLI output, but it will be considered when analysed as part of querying the repo as a whole as happens on GitHub.com.

So in summary, the linguist-generated override is working, it is however being overruled by linguist-vendored in this case.

[ApoorvGuptaAi]

Hi @lildude, thanks for the detailed explanation.
The original reason for me to explore this was because github has stopped marking our files as generated, so the advice on how to fix github is handy.

• The syntax recommended here is different from the one recommended at https://docs.github.com/en/github/administering-a-repository/customizing-how-changed-files-appear-on-github (linguist-generated vs linguist-generated=true), does that make a difference?
• Other files not recognized as vendored are also not picked up as generated, e.g.

github-linguist gui/src/protobuf/truera/protobuf/version_pb.js
version_pb.js: 764 lines (653 sloc)
type:      Text
mime type: text/javascript
language:  JavaScript

Based on your explanation, the override shouldnt work from the command line, but github should be picking them as generated though, any advice on why that would be? Could it be caused by the syntax difference?

[lildude]

* The syntax recommended here is different from the one recommended at https://docs.github.com/en/github/administering-a-repository/customizing-how-changed-files-appear-on-github (`linguist-generated` vs `linguist-generated=true`), does that make a difference?

Nope. It makes no difference. linguist-generated is the method documented in the .gitattributes documentation and linguist-generated=true is the old, yet still supported, Linguist-specific approach.

Based on your explanation, the override shouldnt work from the command line, but github should be picking them as generated though, any advice on why that would be? Could it be caused by the syntax difference?

The override won't show as taking effect when querying the first specifically from the CLI, but it will be considered when checking the repo as a whole from the CLI. ie github-linguist gui/src/protobuf/truera/protobuf/version_pb.js won't consider the override but github-linguist . will and will exclude the file from the language statistics. You can confirm this by appending --breakdown, ie github-linguist . --breakdown`.

$ echo "foo" > gui/src/protobuf/truera/protobuf/version_pb.js
$ echo "foo" > gui/src/protobuf/truera/protobuf/public/another.js
$ git add .
$ git commit -am 'Add more'
[main 4c2d658] Add more
 2 files changed, 2 insertions(+)
 create mode 100644 gui/src/protobuf/truera/protobuf/public/another.js
 create mode 100644 gui/src/protobuf/truera/protobuf/version_pb.js
$ github-linguist .
100.00% JavaScript
$ github-linguist . --breakdown
100.00% JavaScript

JavaScript:
gui/src/protobuf/truera/protobuf/public/another.js
gui/src/protobuf/truera/protobuf/version_pb.js

$ echo 'gui/src/protobuf/truera/protobuf/* linguist-generated' >> .gitattributes
$ git add .gitattributes
$ git commit -m 'Add override'
[main 9297f01] Add override
 1 file changed, 1 insertion(+)
$ github-linguist . --breakdown
100.00% JavaScript

JavaScript:
gui/src/protobuf/truera/protobuf/public/another.js

$

You can also confirm your override is taking effect by taking Linguist out of the equation entirely by querying the attributes with git:

$ git check-attr linguist-generated gui/src/protobuf/truera/protobuf/version_pb.js
gui/src/protobuf/truera/protobuf/version_pb.js: linguist-generated: set    ### <--- Indicates the override is set so will take effect
$ git check-attr linguist-generated gui/src/protobuf/truera/protobuf/public/another.js
gui/src/protobuf/truera/protobuf/public/another.js: linguist-generated: unspecified
$

So, if you've committed your overrides as reported in the OP to the repo (the file must be committed for Linguist to use it), it should be detected as generated as it would be covered by this override rule which applies to all files in that directory, but not within subdirectories as I've illustrated:

gui/src/protobuf/truera/protobuf/* linguist-generated

This rule doesn't apply to files in subdirectories because it's not recursive. For that you need to use ** as detailed in the git docs.

How are you determining your override isn't taking effect? Do you have a repo that I can see this in action?

[lildude]

Oh yes, and you're running a very very old version of Linguist locally. I recommend you upgrade to the latest to be sure you're getting the same behaviour locally as on GitHub.com. The latest is v7.13.0.

[lildude]

At this isn't a but, I'm also moving this to discussions.