OceananigansDocumentation repo is quite large due to SVGs

[ali-ramadhan]

The https://github.com/CliMA/OceananigansDocumentation repository is quite large, which is to be expected as it holds the docs for all versions (including figures and animations) going back to v0.4.0 (which many gaps though).

We probably want to keep its size as reasonable as possible because every time the tests run and the docs build, it git clones the docs.

Running the script from https://stackoverflow.com/a/42544963 and looking at the biggest files (over 10 MiB) I'm seeing SVG files with the internal tide SVG being particularly large at ~20 MiB.

I guess the actionable item I'm suggesting here is to produce PNG figures instead of SVG (at least maybe for heatmaps) to reduce file sizes and repo size.

...
7172adb282e9   10MiB v0.44.1/generated/eady_turbulence.gif
613a8d87655b   10MiB v0.44.1/generated/two_dimensional_turbulence/index.html
2dfdd3c467bd   11MiB v0.45.2/generated/eady_turbulence.gif
988bd69f2ab5   13MiB v0.44.1/generated/eady_turbulence/index.html
0ba1083e8897   14MiB v0.45.2/generated/eady_turbulence/index.html
4d9f6e1aec0c   20MiB v0.91.15/literated/internal_tide-38.svg
fe6333d3478d   20MiB v0.94.3/literated/internal_tide-36.svg
c882e8901513   20MiB v0.93.3/literated/internal_tide-36.svg
6b93a38ac7f7   20MiB v0.90.13/literated/internal_tide-40.svg
94c1158e6bae   20MiB v0.95.29/literated/internal_tide-36.svg
3e274ced7bb5   20MiB v0.96.32/literated/internal_tide-36.svg
31bbd9d5df2b   20MiB v0.96.30/literated/internal_tide-36.svg
099b038dbd30   21MiB dev/literated/internal_tide-36.svg
4d8902759121   21MiB v0.96.35/literated/internal_tide-36.svg
fdcdbd1686ed   21MiB previews/PR4811/literated/internal_tide-36.svg
c9b4cecd25a1   21MiB previews/PR4780/literated/internal_tide-36.svg
e21cd6a172e4   21MiB v0.100.6/literated/internal_tide-36.svg

[glwagner]

I think all we have to do is change this line to png:

Oceananigans.jl/docs/make.jl

Line 12 in 6a539d9

CairoMakie.activate!(type = "svg")

which is something we did on Breeze

cc @giordano

[giordano]

Uhm, that will help for new versions, but we'd need to manually edit all older versions too, which is not fun because we need to convert the SVGs and edit the HTMLs 😭

[ali-ramadhan]

Yeah I was just suggesting to do PNG going forward.

No need to modify git history by converting SVGs to PNG and editing HTML manually, unless someone really has infinite time on their hands lol (or can get an AI agent to do it correctly).

[giordano]

With #5031 merged, going forward we should have smaller images, but I believe it's still important to resize the docs for older versions, because as far as I can tell the docs deployment has been broken for a month now, last time it ran was on November 19.

[giordano]

A quick and dirty script to replace the SVG images with PNGs is the following:

docs_dir = "/tmp/OceananigansDocumentation"
all_versions = filter(x -> tryparse(VersionNumber, basename(x)) isa VersionNumber && !islink(x), readdir(docs_dir; join=true))

for version in all_versions
    for (path, dirs, files) in walkdir(version)
        for file in files
            if file != "index.html"
                continue
            end
            filepath = joinpath(path, file)
            content = readchomp(filepath)
            m = match(r"img src=\"(.*\.svg)\"", content)
            if isnothing(m)
                continue
            end
            svg_img = joinpath(dirname(filepath), m[1])
            png_img = splitext(svg_img)[1] * ".png"
            if !isfile(svg_img)
                continue
            end
            run(`convert $(svg_img) $(png_img)`)
            rm(svg_img)
        end
    end
end

This is incomplete because I still need to patch up the HTML files to replace the references to the SVG files to PNGs. However this seems to save only a bit less than 200 MiB, there are 2 more GiB to go ☹️ Edit: the script only replaces the first image found in each HTML file, it also needs to be changed to replace all of them. Hopefully there's more saving to be gained, I shared the script to save my progress so far.

[giordano]

Better script, now complete:

docs_dir = "/tmp/OceananigansDocumentation" # change this to point to the OceananigansDocumentation directory
all_versions = filter(x -> tryparse(VersionNumber, basename(x)) isa VersionNumber && !islink(x), readdir(docs_dir; join=true))

for version in all_versions
    for (path, dirs, files) in walkdir(version)
        for file in files
            if file != "index.html"
                continue
            end
            filepath = joinpath(path, file)
            content = read(filepath, String)
            matches = eachmatch(r"img src=\"(.*\.svg)\"", content)
            for m in matches
                svg_img = m[1]
                png_img = splitext(svg_img)[1] * ".png"
                dir = dirname(filepath)
                abs_svg_img = joinpath(dir, svg_img)
                abs_png_img = joinpath(dir, png_img)
                if !isfile(abs_svg_img)
                    continue
                end
                run(`convert $(abs_svg_img) $(abs_png_img)`)
                rm(abs_svg_img)
                content = replace(content, "img src=\"$(svg_img)\"" => "img src=\"$(png_img)\"")
            end
            write(filepath, content)
        end
    end
end

This one saves 600 MiB (out of 2.2 GiB, ~25%) 🙂 There's probably more that can be done, but this is already quite some significant saving. Shall we go?

[glwagner]

should we run it on one version to check what happens, just to make sure that it doesn't mangle all the docs versions?

[giordano]

You can run the above script in a local clone of the repo. I did that and it looked ok to me, I checked a few different pages and images were included and rendered correctly, and they look the same as the SVG ones. Maybe independent tests would be good though!

[giordano]

Opinions? 🙂 Has anyone else tried the script above? (to be clear: needs to be run on a checkout of the gh-pages branch of https://github.com/CliMA/OceananigansDocumentation/)

[navidcy]

Please go ahead!

[giordano]

Script above executed, and now largest files in the repo are:

[...]
7172adb282e9   10MiB v0.44.1/generated/eady_turbulence.gif
613a8d87655b   10MiB v0.44.1/generated/two_dimensional_turbulence/index.html
2dfdd3c467bd   11MiB v0.45.2/generated/eady_turbulence.gif
988bd69f2ab5   13MiB v0.44.1/generated/eady_turbulence/index.html
0ba1083e8897   14MiB v0.45.2/generated/eady_turbulence/index.html

• Good news: large SVG files are indeed gone, technically the original issue reported here is addressed
• Bad news: the deployed website is still largish, because all the symlinks v0.X are resolved into full copies, so the website is effectively double the size than the bare repo

One option I had considered was to turn the symlinks v0.X into HTML redirects instead of full copies, this would halve the size of the website, the problem however is that when you see the docs for v0.X you're brought to v0.X.Y, and if you copy the redirected URL you may end up with a broken link in the future because we automatically clean up old patch versions of the docs to reduce the size of the website.

Honestly, only option I see to reduce the size of OceananigansDocumentation is to move old versions (> 3-4 years?) to yet another repo (OceananigansOldDocumentation?), and create HTML redirects in OceananigansDocumentation, not to break any link.

[glwagner]

Honestly, only option I see to reduce the size of OceananigansDocumentation is to move old versions (> 3-4 years?) to yet another repo (OceananigansOldDocumentation?), and create HTML redirects in OceananigansDocumentation, not to break any link.

I think that is fine. or OceananigansMuseum since that code belongs to the dustbins of history.