How to develop with decent velocity

[CommanderStorm]

Problem

what is the recomended developement cycle for editing the JS parts of the project?

Description

I tried to use the ruby things that are in the guide, but this does not seem to work well on my system (archlinux).

So the magic incarnation to get any changes to the JS to show up seem to be

docker compose stop web
docker compose rm web
docker volume ls # to list the unnamed volume
docker volume rm ohm-website_web-storage ohm-website_web-tmp  0b494c14f2ce7d8b61bdcc02ce3de0b0840b9f311929f3f0a8d1da8b91d38142
docker compose up web --build

@1ec5 told me that he uses docker-compose run --rm web bundle exec rake assets:precompile, but this does not seem to work for me reliably.

The problem with this takes a bunch of time (~5 minutes) and is also not super intuitive to do.
How are others actually developing changes?
This seems like a pretty non-intitive workflow coming from the web world.

Screenshots

No response

[1ec5]

Cleaning out the entire container and starting from scratch seems like overkill. If you have to do that to get fresh assets, then something is amiss, and yeah, I could see that taking an awfully long time. The command I suggested comes from the test setup documentation (with a minor change to docker-compose for my particular Mac setup). I use it in conjunction with docker-compose build and docker-compose up -d db && docker-compose up web. (Plus the usual clearing the browser cache and all.)

Another alternative is the devcontainer in #6424, if you want to give that a try.

[gravitystorm]

When everything is running smoothly, rails will compile the assets (stylesheets, images and javascripts) on the fly, and take care of everything when the files are changed on disk. This is similar to the ruby code - e.g. you don't need to restart the rails webserver when you change a ruby code file - rails notices automatically, and takes care of everything.

Note the caveat with "running smoothly".

docker-compose run --rm web bundle exec rake assets:precompile

When you precompile the assets, then rails will use those compiled versions in preference to anything else. So if you've run the assets:precompile at any stage, you then need to manually run it again any time you want to see your changes to stylesheets or javascript files.

For this reason, it should only really be used for production, not development. However, there's various edge cases that can stop the automatic asset pipeline from working in the development environment and these situations can be hard to debug. So many people immediately reach for rails assets:precompile, especially if they are not working on anything related to the assets, and the problem (seemingly) goes away. This workaround then gets into our installation notes, into stackoverflow, into LLMs, etc etc and becomes normalised.

That's not to say we shouldn't mention it, because for example it took me months to figure out what was going wrong the last time the automatic assets system broke on my machine (and I relied on repeated assets:precompile until I figured out that something in rails_dir/tmp was causing hash mismatches in the urls).

I'm very open to either changing or removing the mention of assets:precompile from DOCKER.md, I think it's probably unhelpful at present. There might also be something specific to our docker setup that's interfering with the automatic recompilation, in which case that should be investigated and fixed - or it might just be that avoiding precompilation makes everything work fine.

[CommanderStorm]

Thanks, will investigate further

[tomhughes]

Just forget docker and use a local tree. Problem solved.

[CommanderStorm]

Just forget docker and use a local tree

Thanks for the suggestion! I did try working from a local tree instead of Docker, but I haven’t been able to get it into a working state yet.

To be clear, I’m not stuck on or insisting on following DOCKER.md.
In practice I end up running into issues during bundle install, specifically with native extensions failing to build (full output below).

I should also say that I’m quite unfamiliar with the Ruby/Rails side of things (it is not taught in Uni), so it’s very likely this is user error on my part rather than an actual problem with the project. At the moment I’m not really sure how best to debug this further or what a known-good local setup looks like.

If you have any pointers on what your environment, Ruby version, or toolchain tends to work well here, I’d really appreciate the guidance. Happy to try things out and report back.

result of installing dependencys

bundle install   
Fetching gem metadata from https://rubygems.org/........
The following platform specific gems are getting installed, yet the lockfile includes only their generic ruby version:
 * pg-1.6.3-x86_64-linux
 * sass-embedded-1.64.2-x86_64-linux-gnu
 * libv8-node-24.12.0.1-x86_64-linux
 * nokogiri-1.19.0-x86_64-linux-gnu
Please run `bundle lock --normalize-platforms` and commit the resulting lockfile.
Alternatively, you may run `bundle lock --add-platform <list-of-platforms-that-you-want-to-support>`
Fetching libv8-node 24.12.0.1 (x86_64-linux)
Fetching pg 1.6.3 (x86_64-linux)
Fetching webrick 1.9.2
Fetching i18n 1.14.8
Installing date 3.5.1 with native extensions
Installing json 2.18.0 with native extensions
Installing google-protobuf 3.25.8 with native extensions
Installing prism 1.7.0 with native extensions
Installing webrick 1.9.2
Installing i18n 1.14.8
Fetching nokogiri 1.19.0 (x86_64-linux-gnu)
Fetching rackup 2.3.1
Installing pg 1.6.3 (x86_64-linux)
Installing rackup 2.3.1
Fetching brakeman 7.1.2
Fetching reline 0.6.3
Installing reline 0.6.3
Fetching puma 7.1.0
Fetching addressable 2.8.8
Installing puma 7.1.0 with native extensions
Installing addressable 2.8.8
Fetching bootsnap 1.20.1
Installing bootsnap 1.20.1 with native extensions
Installing brakeman 7.1.2
Installing nokogiri 1.19.0 (x86_64-linux-gnu)
Fetching selenium-webdriver 4.39.0
Fetching psych 5.3.1
Fetching faraday-net_http 3.4.2
Installing faraday-net_http 3.4.2
Installing psych 5.3.1 with native extensions
Fetching aws-sdk-core 3.240.0
Installing aws-sdk-core 3.240.0
Fetching oauth 1.1.3
Installing oauth 1.1.3
Fetching opentelemetry-helpers-sql-processor 0.3.1
Installing opentelemetry-helpers-sql-processor 0.3.1
Fetching loofah 2.25.0
Installing loofah 2.25.0
Fetching aws-sdk-kms 1.118.0
Installing aws-sdk-kms 1.118.0
Fetching opentelemetry-instrumentation-ethon 0.26.0
Installing opentelemetry-instrumentation-ethon 0.26.0
Fetching opentelemetry-instrumentation-excon 0.26.1
Installing opentelemetry-instrumentation-excon 0.26.1
Fetching opentelemetry-instrumentation-faraday 0.30.1
Installing opentelemetry-instrumentation-faraday 0.30.1
Fetching opentelemetry-instrumentation-http 0.27.1
Installing opentelemetry-instrumentation-http 0.27.1
Fetching opentelemetry-instrumentation-http_client 0.26.1
Installing opentelemetry-instrumentation-http_client 0.26.1
Fetching opentelemetry-instrumentation-httpx 0.5.1
Installing opentelemetry-instrumentation-httpx 0.5.1
Fetching opentelemetry-instrumentation-mysql2 0.32.1
Installing opentelemetry-instrumentation-mysql2 0.32.1
Fetching opentelemetry-instrumentation-net_http 0.26.1
Installing selenium-webdriver 4.39.0
Installing opentelemetry-instrumentation-net_http 0.26.1
Fetching opentelemetry-instrumentation-pg 0.34.1
Installing opentelemetry-instrumentation-pg 0.34.1
Fetching opentelemetry-instrumentation-que 0.12.0
Installing opentelemetry-instrumentation-que 0.12.0
Fetching opentelemetry-instrumentation-sidekiq 0.28.1
Installing opentelemetry-instrumentation-sidekiq 0.28.1
Fetching opentelemetry-instrumentation-trilogy 0.65.1
Installing opentelemetry-instrumentation-trilogy 0.65.1
Fetching aws-sdk-s3 1.209.0
Fetching opentelemetry-instrumentation-all 0.89.1
Installing opentelemetry-instrumentation-all 0.89.1
Installing aws-sdk-s3 1.209.0
Fetching rdoc 7.0.3
Fetching oauth2 2.0.18
Fetching activesupport 8.1.1
Installing libv8-node 24.12.0.1 (x86_64-linux)
Installing oauth2 2.0.18
Installing rdoc 7.0.3
Installing activesupport 8.1.1
Fetching irb 1.16.0
Fetching activemodel 8.1.1
Fetching actionview 8.1.1
Fetching activejob 8.1.1
Installing irb 1.16.0
Installing activemodel 8.1.1
Installing activejob 8.1.1
Installing actionview 8.1.1
Fetching debug 1.11.1
Fetching activerecord 8.1.1
Installing debug 1.11.1 with native extensions
Fetching actionpack 8.1.1
Installing activerecord 8.1.1
Installing actionpack 8.1.1
Fetching railties 8.1.1
Fetching actioncable 8.1.1
Fetching actionmailer 8.1.1
Fetching omniauth-rails_csrf_protection 2.0.1
Installing railties 8.1.1
Installing actioncable 8.1.1
Installing actionmailer 8.1.1
Installing omniauth-rails_csrf_protection 2.0.1
Fetching rubocop-ast 1.48.0
Fetching activestorage 8.1.1
Fetching strong_migrations 2.5.2
Installing rubocop-ast 1.48.0
Installing activestorage 8.1.1
Installing strong_migrations 2.5.2
Fetching rubocop 1.82.1
Fetching actionmailbox 8.1.1
Fetching action_text-trix 2.1.16
Fetching rails-i18n 8.1.0
Installing actionmailbox 8.1.1
Installing action_text-trix 2.1.16
Fetching actiontext 8.1.1
Installing rails-i18n 8.1.0
Installing rubocop 1.82.1
Installing actiontext 8.1.1
Fetching rails 8.1.1
Installing rails 8.1.1
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /home/frank/dev/ohm-website/vendor/bundle/ruby/3.4.0/gems/google-protobuf-3.25.8/ext/google/protobuf_c
/usr/bin/ruby extconf.rb
creating Makefile

current directory: /home/frank/dev/ohm-website/vendor/bundle/ruby/3.4.0/gems/google-protobuf-3.25.8/ext/google/protobuf_c
make DESTDIR\= sitearchdir\=./.gem.20260105-3642453-gg9p8n sitelibdir\=./.gem.20260105-3642453-gg9p8n clean

current directory: /home/frank/dev/ohm-website/vendor/bundle/ruby/3.4.0/gems/google-protobuf-3.25.8/ext/google/protobuf_c
make DESTDIR\= sitearchdir\=./.gem.20260105-3642453-gg9p8n sitelibdir\=./.gem.20260105-3642453-gg9p8n
compiling protobuf.c
compiling convert.c
compiling defs.c
compiling message.c
compiling repeated_field.c
compiling map.c
map.c: In function ‘Map_delete’:
map.c:447:26: warning: passing argument 1 of ‘upb_Map_Delete’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
  447 |   if (upb_Map_Delete(self->map, key_upb, &val_upb)) {
      |                      ~~~~^~~~~
In file included from defs.h:12,
                 from protobuf.h:29,
                 from convert.h:11,
                 from map.c:8:
ruby-upb.h:1772:38: note: expected ‘upb_Map *’ but argument is of type ‘const upb_Map *’
 1772 | UPB_API bool upb_Map_Delete(upb_Map* map, upb_MessageValue key,
      |                             ~~~~~~~~~^~~
compiling ruby-upb.c
compiling wrap_memcpy.c
compiling ./third_party/utf8_range/naive.c
compiling ./third_party/utf8_range/range2-neon.c
compiling ./third_party/utf8_range/range2-sse.c
compiling shared_convert.c
shared_convert.c: In function ‘shared_Msgval_IsEqual’:
shared_convert.c:39:1: warning: control reaches end of non-void function [-Wreturn-type]
   39 | }
      | ^
shared_convert.c: In function ‘shared_Msgval_GetHash’:
shared_convert.c:64:1: warning: control reaches end of non-void function [-Wreturn-type]
   64 | }
      | ^
compiling shared_message.c
shared_message.c: In function ‘shared_Message_Equal’:
shared_message.c:66:1: warning: control reaches end of non-void function [-Wreturn-type]
   66 | }
      | ^
linking shared-object google/protobuf_c.so
/usr/bin/ld: /tmp/cc0vUE9v.ltrans1.ltrans.o: in function `jsonenc_putbytes':
/usr/include/bits/string_fortified.h:29:(.text+0x366): undefined reference to `__wrap_memcpy'
/usr/bin/ld: /tmp/cc0vUE9v.ltrans1.ltrans.o: relocation R_X86_64_PC32 against undefined hidden symbol `__wrap_memcpy' can not be used when making a shared object
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make: *** [Makefile:268: protobuf_c.so] Error 1

make failed, exit code 2

Gem files will remain installed in /home/frank/dev/ohm-website/vendor/bundle/ruby/3.4.0/gems/google-protobuf-3.25.8 for inspection.
Results logged to /home/frank/dev/ohm-website/vendor/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/google-protobuf-3.25.8/gem_make.out

  /usr/lib/ruby/3.4.0/rubygems/ext/builder.rb:125:in 'Gem::Ext::Builder.run'
  /usr/lib/ruby/3.4.0/rubygems/ext/builder.rb:51:in 'block in Gem::Ext::Builder.make'
  /usr/lib/ruby/3.4.0/rubygems/ext/builder.rb:43:in 'Array#each'
  /usr/lib/ruby/3.4.0/rubygems/ext/builder.rb:43:in 'Gem::Ext::Builder.make'
  /usr/lib/ruby/3.4.0/rubygems/ext/ext_conf_builder.rb:44:in 'Gem::Ext::ExtConfBuilder.build'
  /usr/lib/ruby/3.4.0/rubygems/ext/builder.rb:206:in 'Gem::Ext::Builder#build_extension'
  /usr/lib/ruby/3.4.0/rubygems/ext/builder.rb:240:in 'block in Gem::Ext::Builder#build_extensions'
  /usr/lib/ruby/3.4.0/rubygems/ext/builder.rb:237:in 'Array#each'
  /usr/lib/ruby/3.4.0/rubygems/ext/builder.rb:237:in 'Gem::Ext::Builder#build_extensions'
  /usr/lib/ruby/3.4.0/rubygems/installer.rb:844:in 'Gem::Installer#build_extensions'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/rubygems_gem_installer.rb:111:in 'Bundler::RubyGemsGemInstaller#build_extensions'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/rubygems_gem_installer.rb:30:in 'Bundler::RubyGemsGemInstaller#install'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/source/rubygems.rb:205:in 'Bundler::Source::Rubygems#install'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/installer/gem_installer.rb:55:in 'Bundler::GemInstaller#install'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/installer/gem_installer.rb:17:in 'Bundler::GemInstaller#install_from_spec'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/installer/parallel_installer.rb:133:in 'Bundler::ParallelInstaller#do_install'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/installer/parallel_installer.rb:124:in 'block in Bundler::ParallelInstaller#worker_pool'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/worker.rb:62:in 'Bundler::Worker#apply_func'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/worker.rb:57:in 'block in Bundler::Worker#process_queue'
  <internal:kernel>:168:in 'Kernel#loop'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/worker.rb:54:in 'Bundler::Worker#process_queue'
  /home/frank/.local/share/gem/ruby/3.4.0/gems/bundler-2.6.2/lib/bundler/worker.rb:90:in 'block (2 levels) in Bundler::Worker#create_threads'

An error occurred while installing google-protobuf (3.25.8), and Bundler cannot continue.

In Gemfile:
  dartsass-sprockets was resolved to 3.0.0, which depends on
    dartsass-ruby was resolved to 3.0.2, which depends on
      sass-embedded was resolved to 1.64.2, which depends on
        google-protobuf

[mmd-osm]

The error message looks a bit like the issue discussed over here as option 1: protocolbuffers/protobuf#11935 (comment) - removing the line $LDFLAGS += " -Wl,-wrap,memcpy" in protobuf/ruby/ext/google/protobuf_c/extconf.rb:

I can't really test this, though the linker errors point to an issue regarding `__wrap_memcpy'. Maybe you could raise an issue in that repo describing your specific setup in more detail.

[mmd-osm]

rails assets:precompile

I think it makes sense to include a link to https://guides.rubyonrails.org/asset_pipeline.html, with the hint "You can use the rails assets:clobber command to delete all your precompiled assets". Cleaning up the tmp/ directory might also be required.

I don't know how often I've been searching for a way to get rid of precompilation (+artifacts), only to forget about the exact steps next time I need it.

[gravitystorm]

(it is not taught in Uni),

Don't worry, I didn't get taught any programming at Uni at all, so if you were taught a different language then you're still a step ahead of me!

The error message looks a bit like the issue discussed over here as option 1

I think this might be because we're pinning the version of sass-embedded, which limits the version of google-protobuf, which means we might be running into an old bug there. So the fix might be to release the pin on sass-embedded and we'll find the compilation bug might go away.