Skip to content

Improve project organization#48

Open
DanielG wants to merge 4 commits intotoreanderson:masterfrom
DanielG:project-cleanup
Open

Improve project organization#48
DanielG wants to merge 4 commits intotoreanderson:masterfrom
DanielG:project-cleanup

Conversation

@DanielG
Copy link
Contributor

@DanielG DanielG commented Feb 21, 2025

  • MMM: Make Makefile More (useful)

  • Split docs into clatd(8) for system installation and README.md for web presentation

    Currently copies introductory paragraph. Could automate sourcing that from clatd.8 but doesn't seem worth it.

@@ -0,0 +1 @@
/clat.8
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Surely this should be clatd.8 (or clatd.8.gz, see below)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, typo.

Comment on lines +22 to +29
clatd.8: clatd.pod
pod2man \
--name clatd \
--center "clatd - CLAT, SIIT-DC and IPv6-only with many XLAT engines" \
--section 8 \
$< $@
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any particular reason for no longer compressing the generated manual page? Most manual pages on my Fedora workstation are compressed with gzip, so I think that should be kept.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compression is optional as far as man is concerned, it's usually handled by distros. Policy could vary, eg: should xz/lzma/gz be used and what compression settings, and recompressing to comply could be annoying for packagers depending on tooling. So it's just simpler for all involved not to do this upstream unless it's absolutely necessary for some reason. In general upstreams may eg. build tarballs to distribute themselves and size may be a concern there, but since that's not the case with clatd simplicity is the way to go IMO.

Looking at Fedora specifically it seems RPM handles the (re)compression. So removing the gz on the upstream side shouldn't impact them:

https://docs.fedoraproject.org/en-US/packaging-guidelines/#_manpages

When installing man pages, note that RPM will re-compress them into its preferred format. So the %files section MUST reference manpages with a pattern that takes this into account [...]

OpenSUSE on the other hand hardcodes the pod2man call to get an uncompressed file already:

https://build.opensuse.org/projects/openSUSE:Factory/packages/clatd/files/clatd.spec?expand=1

%build
pod2man --name clatd --center "clatd - a CLAT implementation for Linux" --section 8 README.pod clatd.8
sed -i "s,%{_sbindir}/clatd,%{_sbindir}/clatd -c %{_sysconfdir}/clatd.conf," scripts/*

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not entirely sure what you mean by «it's usually handled by distros». As an example, if I, on a Fedora machine drop an uncompressed manual page in /usr/share/man somewhere, there's no periodic job that ensures this file gets compressed, it will remain uncompressed forever. Compression should, the way I see it, be handled by the make install target (or any of its dependencies).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tore. 99% of users are going to be using clatd throught their distros. What your plain make install target does only matters in so far as it works for development and doesn't make distro maintainer's life harder than it already is.

Distro maintainers and/or their build tooling will make the gz/xz/zstd'ing happen. In Debian's case most maintainers use the debhelper toolchain which has dh_compress to handle man page compression (if necessary) just before generating the .deb archive from $DESTDIR. It's complicated^TM ask for details at your own peril.

When was the last time you ran ./configure && make && make install in (for example) a iproute2 tarball unpack? :-)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read the manual page for dh_compress that you linked to. It says:

By default, dh_compress compresses files that Debian policy mandates should be compressed, namely all files in usr/share/info, usr/share/man, (…)

Ok, so it would compress the clatd manual page because it is installed to /usr/share/man/. Makes sense.

The manual page then goes on to say:

(…) (except (…) files that appear to be already compressed based on their extensions)

The current Makefile installs the manual page as clatd.8.gz, so it will appear to be already compressed based on its extension. Therefore, assuming dh_compress's manual page is to be trusted, it should be a no-op as far as the installed clatd.8.gz file is concerned.

I also noted the reference to Debian policy, and looked it up. The policy states:

Manual pages should be installed compressed using gzip -9.

As it happens, installing the manual page compressed with gzip -9 is precisely what the current Makefile does already.

So I do not understand what the actual problem is. As far as I can tell, what the current Makefile does already follows Debian policy to the letter, and does so in a way that the Debhelper build script is documented to deal with gracefully. Could you please explain the problem is, what is it that does not work for you exactly? Otherwise I'm inclined to say that if it ain't broke, don't fix it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tore. You're not looking at this with enough perspective. I've already given you all the information you need to figure out why this makes sense you're not seeing it.

Look at the amount of verbiage we've already spilled over this issue. The cost/benefit ratio is way out of whack here and I don't want to continue this discussion. Either you trust that I'm a distro person deeply versed in the issues (invisible to you) people downstream face or you don't.

If you see no actual problem with changing this why are we arguing about it? Are you just arguing to argue? I'd rather spend my time more productively than that.

I wasn't going to spell it out but I can't help myself: This will help OpenSUSE remove hardcoding from their build recipe and allow them to use the upstream Makefile instead (whithout any other distro feeling any impact due to tooling). This in turn reduces the amount of work they have to do as we introduce new binaries/manpages/features here.

A small thing but goes a long way to cultivating a positive relationship with your distributors, which is key to ubiquitous adoption of software in the Linux ecosystem.

--Daniel

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not heard a peep from the openSUSE packager about this issue, ever. As far as I know, their packages are working well. That said, if compressed manual pages are indeed a problem for them, I expect them to speak up for themselves.

Anyway. What could be done instead, is something like this:

COMPRESS_COMMAND = gzip -f9
…
	pod2man --name clatd --center "clatd - a CLAT implementation for Linux" --section 8 README.pod $(DESTDIR)$(PREFIX)/share/man/man8/clatd.8 && $(COMPRESS_COMMAND) $(DESTDIR)$(PREFIX)/share/man/man8/clatd.8 || echo "pod2man is required to generate manual page"

Or perhaps:

COMPRESS_MANPAGE = true
…
  pod2man --name clatd --center "clatd - a CLAT implementation for Linux" --section 8 README.pod $(DESTDIR)$(PREFIX)/share/man/man8/clatd.8 || echo "pod2man is required to generate manual page"
  if $COMPRESS_MANPAGE; then gzip -f9 $(DESTDIR)$(PREFIX)/share/man/man8/clatd.8; fi

This would allow the openSUSE maintainer to override COMPRESS_COMMAND (in much the same way PREFIX is) during package build to get an uncompressed manual page, while people that install from Git get a compressed manual page as before (which will also mean there will not be an outdated compressed manual page lingering on the system after upgrades).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That said, if compressed manual pages are indeed a problem for them, I expect them to speak up for themselves.

Gentoo Linux compresses man pages themselves. The developer Sam wrote a succinct blog post about it:
https://cmpct.info/~sam/blog/posts/automatic-manpage-compression/

It'd be nice to keep compression optional, since we package the program:
https://gitweb.gentoo.org/repo/proj/guru.git/tree/net-misc/clatd?h=master

Makefile Outdated
if test -x "$(DNF_OR_YUM)"; then $(DNF_OR_YUM) -y install perl perl-IPC-Cmd perl-Net-IP perl-Net-DNS perl-File-Temp iproute nftables; fi
# If necessary, try to install the TAYGA .rpm using dnf/yum. It is unfortunately not available in all .rpm based distros (in particular CentOS/RHEL).
if test -x "$(DNF_OR_YUM)" && test ! -x "$(TAYGA)"; then $(DNF_OR_YUM) -y install tayga || echo "ERROR: Failed to install TAYGA using dnf/yum, the package is probably not included in your distro. Try enabling the EPEL repo <URL: https://fedoraproject.org/wiki/EPEL> and try again, or install TAYGA <URL: http://www.litech.org/tayga> directly from source."; exit 1; fi
@prog_exists () command -v $$@ >/dev/null 2>&1;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work:

/bin/sh: -c: line 1: syntax error near unexpected token `command'
/bin/sh: -c: line 1: `prog_exists () command -v $@ >/dev/null 2>&1;'
make: *** [Makefile:54: installdeps] Error 2

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. BASH is more strict than dash here, but indeed POSIX 2.9.5 requires a "compound command" (i.e. { grouping; }) in function definitions. TIL.

Makefile Outdated
{ PKGS='$(RPM_PACKAGES)'; PKG=dnf; prog_exists $$PKG; } || \
{ PKGS='$(RPM_PACKAGES)'; PKG=yum; prog_exists $$PKG; } || \
{ PKG=false; echo 'ERROR: Failed to detect system package manager.'>&2;}
$(DRY) $$PKG -y $$PKGS
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I didn't get to test this because of the above error, doesn't there need to be an install command line argument here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Forgot. didn't spot it when testing with make DRY=echo :-)

Are you happy with this approach anyway? I admit it's significantly more high-concept than what you had previously. The only difference being that this way detection code (previously $(shell ...) expansions) doesn't run unless the user actually asks us to install system dependencies. Though that could also be alleviated by using = instead of :=.

Also: I don't like /usr/bin/which since it's a bit of a Debianism AFAIK and not necessarily portable though I can't think of a distro that doesn't have an equivalent of it so that's not of overriding importance.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as it works, I'm fine with it...

README.md Outdated
@@ -0,0 +1,84 @@
# clatd - CLAT, SIIT-DC Edge Relay and IPv6-only for Linux
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

«and IPv6-only for Linux» - that doesn't really make any sense, does it?

Just say it's a 464XLAT CLAT and SIIT-DC Edge Relay,

Copy link
Contributor Author

@DanielG DanielG Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still brainstorming better package descriptions for clatd :-)

I want to get the right keywords in there to make it more discoverable. Since I see it as a key enabling component for IPv6-only/-mostly that's what I went with. In the discoverability context it's important to get that into the tagline. It has to capture peoples interest to keep them reading, doesn't matter so much if it makes technical sense or not. The long form docs can do the hard work of making sense :D

On the Debian side I went with the admittedly somewhat cryptic "CLAT and SIIT-DC ER with many XLAT engines" in the hopes it will at least show up for 464XLAT searches, but I'm not really happy with that either.

I would just suggest adopting my "monostack" moniker to communicate the bundle of many technical concepts we deal with here but since I haven't yet done the publicity work to establish that term I don't think it's very useful right now, so I'm looking for other terms to latch onto and currently that's IPv6-only (and to a lesser known degree IPv6-mostly).

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current README.md is titled «clatd - a CLAT / SIIT-DC Edge Relay implementation for Linux». I don't really see why this needs to be changed (well maybe it could say «464XLAT CLAT» instead of just «CLAT»). I am not interested in any "rebranding", to be honest.

Copy link
Contributor Author

@DanielG DanielG Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well.

The thing is I'm considering adding NAT64 support to clatd using our WIP SIIT Linux module so I'm prepping for that. BTW: we finally settled on a name for the module: "ipxlat" (CONFIG_IPV6_IPXLAT).

In that context it make sense for the tagline to be more general. The "IPv6-only" bit makes sense (IMO) because clatd would then represent a full end to end solution to running IPv6-only networks as it would cover both the provider and host sides.

Once NAT64 is supported the tagline could become eg.:

- CLAT / SIIT-DC Edge Relay implementation for Linux
+ CLAT, NAT64, SIIT-DC. In short: IPv6-only for Linux

If you don't want to make those changes (NAT64 support) here that's fine. 'tis a tale as old as source code and why the elders giveth us patch.1.

I can just start maintaining a proper public fork (not just debian/patches). Would also be an opportunity to rename the repo while I'm at it. The future misnomer has been bothering me already :-).

monostackd? Has a nice "*stack-ed" sort of ring to it.

That being said doing this is not my first choice.

--Daniel

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not opposed to considering adding support for NAT64 in clatd if it can be done in a way that makes sense, but let's not get ahead of ourselves – I will not change the documentation based on a feature that is not yet implemented. The documentation should be updated in the same commit (or at least in the same PR) that adds the NAT64 feature.


This will install **clatd**, dependencies, systemd services, NetworkManager
dispatcher integration scripts on your system, enable and start the
clatd.service. Really only recommended for testing.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why «Really only recommended for testing»?

make install is the supported way of installing clatd that should work anywhere. While users are of course free to use distro packages instead, they are often not available or outdated.

(Testing is better done by running clatd directly from the Git repo, there is no need to install it anywhere.)

@toreanderson toreanderson self-assigned this Mar 20, 2025
DanielG added 3 commits April 6, 2025 08:32
BUGS section is for noting down limitations and known bugs in a program,
not bug reporting information.

See

    man 7 man-pages   # conventions for writing Linux man pages
perl-modules now seems to have a version number in it's name. Install the full
perl instead which depends on perl-modules-*.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants