Skip to content

Commit 8f4f91c

Browse files
committed
doc improvements and less verbosity from bsg.
1 parent 3be785e commit 8f4f91c

File tree

4 files changed

+135
-10
lines changed

4 files changed

+135
-10
lines changed

DEVELOPERS.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
1-
# Key Features
1+
# Key Goals
22
- We try to avoid modifying the users system or msys install. We try to make sure everything we download/use is stored under the WLB_BASE_FOLDER for easy cleanup (or alternate bases). Sometimes we will use a users tools (ie cmake binary) but shouldn't require them.
33
- Builds follow a standardized format even if the app itself doesn't. Sometimes this results in us having to manually move files around to match the final folder structure.
4+
- For new builds use the build script generator [BuildScriptGenerator.csx](build/BuildScriptGenerator.csx) to generate the initial build template. See the build [README](build/README.md) for details on its use.
45

56
# Helpers in Detail
67
The helpers are where the bulk of code is consolidated down that we would use for repeatedly. They are broken down into the following

README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@
2222
<!-- /MarkdownTOC -->
2323

2424
# Current Tools
25+
The link to the changes in each row will show the source changes. For build time modifications look at the [build/README](build/README.md) for details on each.
26+
2527
| Our Changes | CI Action Status |
2628
| :---: | :---: |
2729
| [gnulib](repo_notes/gnulib_README.md) | [![GNULIB Patch Tests](https://github.com/mitchcapper/WIN64LinuxBuild/actions/workflows/gnulib_tests.yml/badge.svg)](https://github.com/mitchcapper/WIN64LinuxBuild/actions/workflows/gnulib_tests.yml) |
@@ -68,6 +70,8 @@ It is a few primary components:
6870

6971
# Warnings
7072

73+
DO NOT STAGE CHANGES in build folder. NOTE: During the step in which we apply our patch we assumed any staged work is ours and discard it to a backup file. Technically after we have run our_patch we won't mess with staged items again. Similarly if there is gnulib used we use the same stage behavior when we go to patch it.
74+
7175
This code and the scripts were quickly written without much testing, and often involving poor quality hacks. Things like disabling testing, help generation, direct manipulation of autogenerated files post generation rather than fixing the generators. Often I use _WIN32 gating for changes, this might break cygwin platforms or using gcc/g++ in Windows. Again these code changes are low quality, if upstream projects want to consider including them may be able to clean up those code blocks.
7276

7377
# Why
@@ -144,6 +148,7 @@ Most of the work here was not done by me and there are some great resources out
144148
- Non-makefile based builds (cmake etc) are a bit overcomplicated using wrapper scripts (or for cmake entire other build processes) to be able to capture the build commands and generate build .bat files. This is often not needed and is more prone to breaking. To prevent this set the best cmake style var to "vs" and any `BLD_CONFIG_BUILD_WINDOWS_COMPILE_WRAPPERS` set to 1 are set to 0.
145149
- To build debug builds set the env var `BLD_CONFIG_BUILD_DEBUG=1`
146150
- To generate batch files for building without msys run the build script with the arg "log_full". Note you likely need some generated files (like config.h) so these would need to be added along with the normal sources.
151+
- Generally, when we make changes we try to stage them after so that unstaged work represents things you may have changed. The exception is we don't stage config /build files themselves. See warning under WARNINGS about the stage behavior.
147152

148153
## Why don't you add these patches upstream?
149154

build/BuildScriptGenerator.csx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ void Main() {
7979

8080
var fname=@$"build/f_{BUILD_NAME}_build.sh";
8181
File.WriteAllText(fname,full);
82-
Console.WriteLine(full);
82+
//Console.WriteLine(full);
8383
Console.WriteLine($"DONE wrote: {fname}");
8484

8585
}

build/README.md

Lines changed: 127 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,83 @@ This directory contains the build scripts for all the libraries/apps we currentl
33

44
While we have extracted as much common code out to the helper scripts there is still a similar pattern for the build script for each specific app/library. We now use a template generator [BuildScriptGenerator.csx](BuildScriptGenerator.csx) to generate the initial build script from our template [f_TEMPLATE_build.sbn-sh](f_TEMPLATE_build.sbn-sh). As there may be times we want to re-run the script (say to bulk update the build scripts) below are the commands used to generate the initial template and any custom modifications required. This also makes it very concise what items we change from the defaults. Between this and the patch we have for that repo it should be easy to see what if any modifications were made. Note: if defaults are changed in default_config.ini then things make break / additional options passed for some builds.
55

6+
<!-- MarkdownTOC -->
7+
8+
- [Calling BuildScriptGenerator.csx](#calling-buildscriptgeneratorcsx)
9+
- [Template System](#template-system)
10+
- [What We Template](#what-we-template)
11+
- [Template Calls for Each Lib/App](#template-calls-for-each-libapp)
12+
- [grep](#grep)
13+
- [Template Script Args](#template-script-args)
14+
- [pcre2](#pcre2)
15+
- [Template Script Args](#template-script-args-1)
16+
- [automake](#automake)
17+
- [Template Script Args](#template-script-args-2)
18+
- [Modifications](#modifications)
19+
- [libpsl](#libpsl)
20+
- [Template Script Args](#template-script-args-3)
21+
- [Modifications](#modifications-1)
22+
- [zlib](#zlib)
23+
- [Template Script Args](#template-script-args-4)
24+
- [Modifications](#modifications-2)
25+
- [libhsts](#libhsts)
26+
- [Template Script Args](#template-script-args-5)
27+
- [Modifications](#modifications-3)
28+
- [wolfcrypt](#wolfcrypt)
29+
- [Template Script Args](#template-script-args-6)
30+
- [Modifications](#modifications-4)
31+
- [wget2](#wget2)
32+
- [Template Script Args](#template-script-args-7)
33+
- [Modifications](#modifications-5)
34+
- [Gawk](#gawk)
35+
- [Template Script Args](#template-script-args-8)
36+
- [Modifications](#modifications-6)
37+
- [awk](#awk)
38+
- [Template Script Args](#template-script-args-9)
39+
- [Modifications](#modifications-7)
40+
- [diffutils](#diffutils)
41+
- [Template Script Args](#template-script-args-10)
42+
- [which](#which)
43+
- [Template Script Args](#template-script-args-11)
44+
- [Modifications](#modifications-8)
45+
- [sed](#sed)
46+
- [Template Script Args](#template-script-args-12)
47+
- [zstd](#zstd)
48+
- [Template Script Args](#template-script-args-13)
49+
- [symlinks](#symlinks)
50+
- [Template Script Args](#template-script-args-14)
51+
- [tar](#tar)
52+
- [Template Script Args](#template-script-args-15)
53+
- [Modifications](#modifications-9)
54+
- [wget](#wget)
55+
- [Template Script Args](#template-script-args-16)
56+
- [openssl](#openssl)
57+
- [Template Script Args](#template-script-args-17)
58+
- [Modifications](#modifications-10)
59+
- [pdcurses](#pdcurses)
60+
- [Template Script Args](#template-script-args-18)
61+
- [Modifications](#modifications-11)
62+
- [highlight](#highlight)
63+
- [Template Script Args](#template-script-args-19)
64+
- [Modifications](#modifications-12)
65+
- [findutils](#findutils)
66+
- [Template Script Args](#template-script-args-20)
67+
- [Modifications](#modifications-13)
68+
- [coreutils](#coreutils)
69+
- [Template Script Args](#template-script-args-21)
70+
- [Modifications](#modifications-14)
71+
- [make](#make)
72+
- [Template Script Args](#template-script-args-22)
73+
- [Modifications](#modifications-15)
74+
- [patch](#patch)
75+
- [Template Script Args](#template-script-args-23)
76+
- [gnutls](#gnutls)
77+
- [Template Script Args](#template-script-args-24)
78+
- [Modifications](#modifications-16)
79+
80+
<!-- /MarkdownTOC -->
81+
82+
683
# Calling BuildScriptGenerator.csx
784
It is meant to be called from powershell. It is likely most of the calls below would work in a bash shell as well but any powershell escapes (backticks) below would need to be updated. You can pass `--help` to it to get a brief overview. It has a few CLI options itself, these are essentially options that don't make sense for us to have in default_config.ini. notably:
885

@@ -35,21 +112,22 @@ Below are the template calls and modifications to the produced build script for
35112

36113
## automake
37114
### Template Script Args
38-
`--GitRepo https://github.com/autotools-mirror/automake --BUILD_NAME automake --GNU_LIBS_USED=0 --BUILD_MSVC_RUNTIME_INFO_ADD_TO_C_AND_LDFLAGS 1`
115+
We force no debug build as it has no advantage and causes build failures as there are files left after clean
116+
`--GitRepo https://github.com/autotools-mirror/automake --HaveOurPatch=0 --BUILD_NAME automake --GNU_LIBS_USED=0 --BUILD_MSVC_RUNTIME_INFO_ADD_TO_C_AND_LDFLAGS 1 --BLD_CONFIG_BUILD_DEBUG=0`
39117

40118
### Modifications
41-
near the top add:
42-
```bash
43-
PERL_P=$(convert_to_msys_path "${BLD_CONFIG_SRC_FOLDER}/automake-1.16")
44-
export PERL5LIB="${PERL_P}"
45-
```
46-
47-
for bootstrap step:
119+
After clone step add:
48120
```bash
121+
PERL_P=$(convert_to_msys_path "${BLD_CONFIG_SRC_FOLDER}/automake-1.16")
122+
export PERL5LIB="${PERL_P}"
123+
if [[ -z $SKIP_STEP || $SKIP_STEP == "bootstrap_fix" ]]; then
49124
attrib -r /s "bin/*"
50125
attrib -r /s "t/*"
51126
mv bootstrap bootstrap.in
52127
head bootstrap.in -n -3 > bootstrap #remove lines that remove the temp dir we need for the perl module
128+
./bootstrap
129+
SKIP_STEP=""
130+
fi
53131
```
54132

55133
## libpsl
@@ -132,7 +210,48 @@ before gnulib block:
132210
Right before configure:
133211
`touch ABOUT-NLS`
134212

213+
## Gawk
214+
### Template Script Args
215+
`--BUILD_NAME gawk --GitRepo https://git.savannah.gnu.org/git/gawk.git --GNU_LIBS_ADD_TO_REPO 1 --BUILD_MSVC_RUNTIME_INFO_ADD_TO_C_AND_LDFLAGS=1 --GNU_LIBS_ADDL "mkstemp" "fts" "sys_socket" "strcasestr" "regex" "random" "flexmember" "setlocale" "locale" "dfa" "sleep" "strsignal" "sys_ioctl" "connect" "listen" "accept" "fnmatch-h" "fnmatch-gnu" "recvfrom" "bind" "setsockopt" "getsockopt" "getopt-gnu" "shutdown" "sys_random" "popen" "pclose" "socket" "strcase" "timegm" "setenv" "unsetenv" "usleep" "fprintf-gnu" --BUILD_MSVC_IGNORE_WARNINGS 4068 --CONFIG_CMD_ADDL "ac_cv_search_dlopen=`"none required`"" "--enable-extensions" "--enable-threads=windows" "acl_shlibext=dll" "ac_cv_header_dlfcn_h=yes" --BUILD_MAKE_CMD_ADDL 'DEFPATH="\"./;%%PROGRAMDATA%%/gawk/share\""' 'DEFLIBPATH="\"./;%%PROGRAMDATA%%/gawk/lib\""'`
216+
217+
### Modifications
218+
In the clone step add
219+
```bash
220+
cp gnulib/build-aux/bootstrap .
221+
cp gnulib/build-aux/bootstrap.conf .
222+
echo "gnulib_tool_option_extras=\" --without-tests --symlink --m4-base=m4 --lib=libgawk --source-base=lib --cache-modules\"" >> bootstrap.conf
223+
git mv m4 m4_orig
224+
git rm build-aux/*
225+
mkdir -p m4
226+
mkdir -p pc/old
227+
mv pc/* pc/old/ || true
228+
pushd m4
229+
cp -s -t . ../m4_orig/socket.m4 ../m4_orig/arch.m4 ../m4_orig/noreturn.m4 ../m4_orig/pma.m4 ../m4_orig/triplet-transformation.m4
230+
popd
231+
echo "EXTRA_DIST = " > m4/Makefile.am
232+
add_items_to_gitignore;
233+
git_staging_add bootstrap bootstrap.conf
234+
git_staging_commit #need to commit it up so that the bootstrap files are avail for our gnulib patching by default all local changes are stashed
235+
```
135236

237+
Before our patch block add
238+
```bash
239+
if [[ $BLD_CONFIG_BUILD_DEBUG -eq 1 ]]; then
240+
touch .developing
241+
else
242+
rm .developing &>/dev/null || true
243+
fi
244+
```
245+
After configure add:
246+
`echo "#include <osfixes.h>" > "lib/dlfcn.h"`
247+
After install add:
248+
```bash
249+
mkdir -p $BLD_CONFIG_INSTALL_FOLDER/ProgramData/lib $BLD_CONFIG_INSTALL_FOLDER/ProgramData/share $BLD_CONFIG_INSTALL_FOLDER/ProgramData/libexec
250+
mv $BLD_CONFIG_INSTALL_FOLDER/lib/gawk/* $BLD_CONFIG_INSTALL_FOLDER/ProgramData/lib/
251+
mv $BLD_CONFIG_INSTALL_FOLDER/libexec/awk/* $BLD_CONFIG_INSTALL_FOLDER/ProgramData/libexec/
252+
mv $BLD_CONFIG_INSTALL_FOLDER/share/awk/* $BLD_CONFIG_INSTALL_FOLDER/ProgramData/share/
253+
rmdir $BLD_CONFIG_INSTALL_FOLDER/lib/gawk/ $BLD_CONFIG_INSTALL_FOLDER/libexec/awk/ $BLD_CONFIG_INSTALL_FOLDER/share/awk/ $BLD_CONFIG_INSTALL_FOLDER/lib/ $BLD_CONFIG_INSTALL_FOLDER/libexec/
254+
```
136255
## awk
137256
### Template Script Args
138257
`--BUILD_NAME awk --GitRepo https://github.com/onetrueawk/awk --BUILD_MSVC_RUNTIME_INFO_ADD_TO_C_AND_LDFLAGS=1 --GNU_LIBS_USED=0`

0 commit comments

Comments
 (0)