Checksums cannot be parsed due to errant carriage returns on JRuby/Windows

[chadlwilson]

Describe the problem as clearly as you can

When trying to use bundler on JRuby/Windows, installs are failing due to issues parsing the checksums for specs off the compact index (I think). There seem to be errant carriage returns in the response that are unexpected and cannot be parsed.

Did you try upgrading rubygems & bundler?

Yes.

• Bundler 2.7.2 via JRuby 10.0
• Bundler 2.6.9 via JRuby 9.4

Post steps to reproduce the problem

• Installed JRuby via Windows installer
• gem update --system if necessary (although same problem with default jruby bundler versions)
• Try and install gems with bundler via jruby, or use the reprod below

Which command did you run?

jruby -rbundler -e "p Bundler::Fetcher.new(Bundler::Source::Rubygems::Remote.new(Gem::URI('https://rubygems.org'))).specs(['test-unit'], Bundler::Source::Rubygems.new)"

(or alternatively installing anything via Bundler with checksum validation enabled)

What were you expecting to happen?

Gems install, or alternatively via the slim reprod, output like

.......#<Bundler::Index:0x4008 sources=[] specs.size=20>

What happened instead?

PS C:\Users\chad> ruby -rbundler -e "p Bundler::Fetcher.new(Bundler::Source::Rubygems::Remote.new(Gem::URI('https://rubygems.org'))).specs(['test-unit'], Bundler::Source::Rubygems.new)"
.......Bundler::GemspecError: There was an error parsing the metadata for the gem test-unit (3.6.9): ArgumentError
Invalid checksum for test-unit-3.6.9: "2be6ae0354623702f004ae626c98337f5e17f2ecc0fc334a9e6fdbb2c7c534da\r" is not a valid SHA256 hex or base64 digest
The metadata was [["checksum", ["2be6ae0354623702f004ae626c98337f5e17f2ecc0fc334a9e6fdbb2c7c534da\r"]]]
  parse_metadata at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/endpoint_specification.rb:167
      initialize at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/endpoint_specification.rb:25
           specs at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher.rb:144
            each at org/jruby/RubyArray.java:2088
           specs at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher.rb:142
          <main> at -e:1
ArgumentError: Invalid checksum for test-unit-3.6.9: "2be6ae0354623702f004ae626c98337f5e17f2ecc0fc334a9e6fdbb2c7c534da\r" is not a valid SHA256 hex or base64 digest
  parse_metadata at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/endpoint_specification.rb:158
            each at org/jruby/RubyArray.java:2088
  parse_metadata at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/endpoint_specification.rb:151
      initialize at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/endpoint_specification.rb:25
           specs at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher.rb:144
            each at org/jruby/RubyArray.java:2088
           specs at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher.rb:142
          <main> at -e:1
ArgumentError: "2be6ae0354623702f004ae626c98337f5e17f2ecc0fc334a9e6fdbb2c7c534da\r" is not a valid SHA256 hex or base64 digest
    to_hexdigest at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/checksum.rb:50
        from_api at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/checksum.rb:34
  parse_metadata at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/endpoint_specification.rb:156
            each at org/jruby/RubyArray.java:2088
  parse_metadata at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/endpoint_specification.rb:151
      initialize at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/endpoint_specification.rb:25
           specs at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher.rb:144
            each at org/jruby/RubyArray.java:2088
           specs at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher.rb:142
          <main> at -e:1

Environment

Via

ruby -rbundler -e "p Bundler::Fetcher.new(Bundler::Source::Rubygems::Remote.new(Gem::URI('https://rubygems.org'))).user_agent"

"bundler/2.7.2 rubygems/3.7.2 ruby/3.4.2 (aarch64-Microsoft-mswin32) command/ jruby/10.0.2.0 options/ 5bb2f02785a0e209"

Bundler       2.7.2
  Platforms   ruby, universal-java-21
Ruby          3.4.2p0 (2025-08-07 revision cba6031bd0c7f7372c69fececce438cad3c77ab0) [universal-java-21]
  Full Path   C:/jruby-10.0.2.0/bin/jruby.exe
  Config Dir  C:/jruby-10.0.2.0/etc
RubyGems      3.7.2
  Gem Home    C:/jruby-10.0.2.0/lib/ruby/gems/shared
  Gem Path    C:\Users\chad/.local/share/gem/jruby/3.4.0;C:/jruby-10.0.2.0/lib/ruby/gems/shared
  User Home   C:\Users\chad
  User Path   C:\Users\chad/.local/share/gem/jruby/3.4.0
  Bin Dir     C:/jruby-10.0.2.0/bin
Tools
  Git         2.51.0.windows.1
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata

Timestamp  2025-09-09
Git SHA    b463ced1459

Workaround

bundle config set --local disable_checksum_validation true

[tenderlove]

I don't have a windows system to test on, but can you try this patch? https://github.com/rubygems/rubygems/compare/master...tenderlove:rubygems:line-separator?expand=1

My guess is this is just Windows specific, not necessarily to do with JRuby. You should be able to test by doing:

$ ruby -I lib:bundler/lib -rbundler -e "p Bundler::Fetcher.new(Bundler::Source::Rubygems::Remote.new(Gem::URI('https://rubygems.org'))).specs(['test-unit'], Bundler::Source::Rubygems.new)"

[simi]

@tenderlove I have Windows system around. Can I help? Should I run the command on JRuby/Windows combination?

[tenderlove]

@tenderlove I have Windows system around. Can I help? Should I run the command on JRuby/Windows combination?

Ya, just give my patch a try. You should be able to reproduce OP's issue, and I think my patch will fix it. This code certainly seems suspicious since line endings on Windows won't be just \n.

[tenderlove]

Also it seems the sha is right before the line ending, so I could see how this code might end up with "sha\r"

[simi]

@tenderlove seems not effective. Same exception. Could be related to fact jruby on windows consider "\n" as newline (same as CRuby).

PS C:\Users\josef> jruby -e "puts ($/).inspect"
"\n"
PS C:\Users\josef> ruby -e "puts ($/).inspect"
"\n"

[simi]

Also it seems the sha is right before the line ending, so I could see how this code might end up with "sha\r"

It depends on metadata present. If there's no ruby constraint, it is the last info, yes.

1.2.1-x86-mingw32 |checksum:31e46657bd193028af1c610fadc403845c7effb7b47754e0a554538cfc65eb87,ruby:>= 1.8.7
1.3.0 |checksum:598adc347a34d2a74ccef68f28f1c744dd7de92a8fb53e740642f5547a524b3b,ruby:>= 1.8.7
1.3.0-java |checksum:623b7783f1d428cf0baf317e98b2a677658e734229785890cb59869cd21aeb06

See https://guides.rubygems.org/rubygems-org-compact-index-api/#get---inforubygem for more details.

[tenderlove]

@tenderlove seems not effective. Same exception. Could be related to fact jruby on windows consider "\n" as newline (same as CRuby).

Ah, great. I didn't know it was the same on Windows. @simi since you can reproduce, maybe try making a fix?

Maybe something like this:

diff --git a/bundler/lib/bundler/compact_index_client/parser.rb b/bundler/lib/bundler/compact_index_client/parser.rb
index 43581fd7ef..aed716764a 100644
--- a/bundler/lib/bundler/compact_index_client/parser.rb
+++ b/bundler/lib/bundler/compact_index_client/parser.rb
@@ -22,7 +22,7 @@ def versions
         @info_checksums = {}
 
         lines(@compact_index.versions).each do |line|
-          name, versions_string, checksum = line.split(" ", 3)
+          name, versions_string, checksum = line.chomp!.split(" ", 3)
           @info_checksums[name] = checksum || ""
           versions_string.split(",") do |version|
             delete = version.delete_prefix!("-")
@@ -40,7 +40,7 @@ def versions
 
       def info(name)
         data = @compact_index.info(name, info_checksums[name])
-        lines(data).map {|line| gem_parser.parse(line).unshift(name) }
+        lines(data).map {|line| gem_parser.parse(line.chomp!).unshift(name) }
       end
 
       def available?
@@ -58,7 +58,7 @@ def info_checksums
 
       def lines(data)
         return [] if data.nil? || data.empty?
-        lines = data.split("\n")
+        lines = data.lines
         header = lines.index("---")
         header ? lines[header + 1..-1] : lines
       end

[simi]

That causes another problem.

PS C:\Users\josef> jruby -rbundler -e "p Bundler::Fetcher.new(Bundler::Source::Rubygems::Remote.new(Gem::URI('https://rubygems.org'))).specs(['test-unit'], Bundler::Source::Rubygems.new)"
NoMethodError: undefined method '+' for nil
  parse_version_checksum at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/compact_index_client/parser.rb:74
          info_checksums at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/compact_index_client/parser.rb:55
                    each at org/jruby/RubyArray.java:2088
        each_with_object at org/jruby/RubyEnumerable.java:1162
          info_checksums at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/compact_index_client/parser.rb:54
              available? at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/compact_index_client/parser.rb:48
              available? at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/compact_index_client.rb:89
              available? at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher/compact_index.rb:55
               bind_call at org/jruby/RubyUnboundMethod.java:174
   compact_index_request at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher/compact_index.rb:13
                fetchers at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher.rb:233
              drop_while at org/jruby/RubyArray.java:4234
                fetchers at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher.rb:233
             fetch_specs at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher.rb:238
                   specs at C:/jruby-10.0.2.0/lib/ruby/stdlib/bundler/fetcher.rb:142
                  <main> at -e:1

Since I can reproduce, I can continue debugging @tenderlove on my own. Feel free to leave it up to me.

[tenderlove]

Thanks @simi!

[chadlwilson]

Hi folks, was overnight for me but can have more of a look later.

I'd be surprised if it affects non-jruby. I've also used bundler 2.6 on jruby 9.4 on windows (11 and server 2022) via a different mechanism (direct jar invocation) many times and not encountered these issues, so I wondered whether there is something environmental or locale sensitive.

[chadlwilson]

Hmm, I cannot replicate this on my Windows machine anymore, after some playing around with CRuby to compare :-( I am very confused!

On this machine I have used both JRuby and regular CRuby over time, and I realise that bundler seems to write these indexes to files which are shared between CRuby and JRuby versions (so the stored file cache doesn't seem separate for different ruby or bundler versions?)

Anyway, to test, I ran the same command I used above via CRuby 3.4.2 and it worked fine. Then I re-ran with JRuby 9.4 and 10.0 and they now both now work fine as well.
--> I infer they are using caches and CRuby has somehow "fixed" the bundler cache, at least for the test-unit dependency?

• I then renamed the cache directory and re-ran with both JRuby 9.4 and 10.0 (upgraded bundler versions as above) and now they both work fine and can bootstrap new cache files that seem to work.
• I also tried with the original bundler versions packaged with 9.4.14.0 and 10.0.2.0 (2.6.3 and 2.6.9 respectively) and they also seem fine bootstrapping new files.
• I also try with the Ruby version I had installed before yesterday (CRuby 3.3.1 / bundler 2.5.9 to see if it had somehow created a file JRuby can't handle, but it seems to work fine.

Unfortunately, I don't have the bundler cache directory exactly as it was when it was broken, so I am not sure exactly what it looked like, however, one version I have still has a mixture of \n and \r\n line endings in the overall index and in various info files, as if different chunks of the file have been written with different implementations/version. There is stuff in there as old as 2023-08-10.

So I am not quite sure what created this situation, or what might need to be done to address it (if anything). Since @simi seems to be able to replicate and presumably knows the internals sufficiently to know what might have caused this, I'll await any instruction as to how I can help or what might have happened.

In any case, it seems clearing/wiping ones bundler cache at C:\Users\blah\.bundle\cache\compact_index might also be a sufficient workaround; but there is presumably still something that JRuby is more sensitive to than CRuby when dealing with the cached files?

[simi]

@tenderlove could that be possible due to non-binary append at https://github.com/rubygems/rubygems/blob/1cfe20f1fb9b74c2254c22e0bfe9cb4d551d42d7/bundler/lib/bundler/compact_index_client/cache_file.rb#L106? I have noticed I have similar files locally corrupted, but only partial (at the end of the file).

See, the first is ok (red), the other pairs are with carriage.

@chadlwilson I was able to fix it by removing my ~/.bundle/cache which will re-download full files again. But not idea if other release and appending will not break it again. To bandaid it is possible to add chomp to lines method in compact index client parser aka lines = data.split("\n").map(&:chomp).