gap-packages · ssiccha · Jan 10, 2022 · Feb 23, 2022 · Feb 23, 2022 · Feb 24, 2022
diff --git a/TODO-mandarins.md b/TODO-mandarins.md
@@ -0,0 +1,136 @@
+## Make clear where things can be improved
+- comments and how to reproduce
+
+## TODO
+Wenn z.b. in einem Blatt ein SLP berechnet wurde, heißt das in einem inneren
+Knoten `node` nicht, dass der input auch element von Grp(node) ist.
+Insbesondere müssen wir immer das SLP für ein Mandarin auswerten und checken,
+dass der Mandarin rausgekommen ist?
+
+FDPM test:
+  Soll ich mal ein profile anlegen wo ich constructive membership test für An
+  und Sn vergleiche?
+  Ergebnis: ist einfach langsam
+
+There is a regression in bugfix.tst. Clean up and then check again.
+
+    ########> Diff in /home/sergio/projects/pkg/recog/tst/working/slow/bugfix.\
+    tst:25
+    # Input is:
+    for i in [1..50] do
+        ri := RECOG.TestGroup(GL(8,27), false, Size(GL(8,27)));
+    od;
+    # Expected output:
+    # But found:
+    Immediate verification: found extra kernel element(s)!
+    Error, no method found! For debugging hints type ?Recovery from NoMethodFound
+    Error, no 1st choice method found for `Length' on 1 arguments
+    The 1st argument is 'fail' which might point to an earlier problem
+
+    ########
+
+- also use gens of input group as mandarins
+
+- make ImmVer + send 100 elements
+  would this be faster? If so, do a hybrid approach? Namely only work with 10
+  or 20 mandarins during recognition, but with 100 once the tree is finished.
+
+- set mandarin info msgs to level 2
+- revisit 1c19d12 fine tune imprimitive
+- test whether we can make the standard arguments RECOG_FindKernelFastNormalClosureStandardArgumentValues for findgensNmeth smaller and
+  get away with it since we use mandarins.
+
+Could it be that immediate verification doesn't do anything once mandarins are
+enabled? See logs/\*.logs. With master immediate verification prints "found
+extra kernel element".
+ImmVer does something when encountering trivial kernels? Double check this and
+see if performance deteriorates if we disable that.
+
+gap> SetInfoLevel(InfoRecog, 1);
+gap> TestMatDiagonal := function(F, n)
+>     local gens, l, i, m, j, g, ri;
+>     gens := [];
+>     l := ShallowCopy(Elements(F));
+>     RemoveSet(l,Zero(F));
+>     for i in [1..5] do
+>         m := IdentityMat(7,F);
+>         for j in [1..7] do
+>             m[j,j] := Random(l);
+>         od;
+>         Add(gens,m);
+>     od;
+>     g := GroupWithGenerators(gens);
+>     return RecogniseGroup(g);
+> end;;
+gap> while true do TestMatDiagonal(GF(9), 2);; od;
+
+^ there is a call which takes forever when I let it run
+
+Super slow due to crisis:
+- MatDiagonal.tst
+Slow due to crisis:
+- bugfix
+
+## Option to enable/disable mandarins
+set num_mands to 0
+
+## Docs:
+Differences to magma/CompositionTree:
+- We add levels and small/big crises.
+- They during a crisis, the generators of the chopped off is doubled.
+  Then, they claim that nrGens of a kernel is always at least nrGens of the
+  parent assures that all descendants of the new kernel also have a big
+  generatings set.
+  We also do the former. Instead of doing the latter, 
+
+
+## Kernel-safe methods, deterministicKernelCreation
+Let methods say, that they create safe kernels if the current node is safe.
+Try this e.g. with "ThrowAwayFixedPoints"
+
+
+## Performance
+
+Something is weird with the current profiling system. Membership-checking takes
+forever for some groups. But profiling shows that the extra time is spent in
+the SLP functions. Here we can find difference between membership testing and
+mandarins.
+
+    gap-master tst/testall-120-inTests-10-mandarins.g
+    gap-master tst/testall-30-inTests-100-mandarins.g
+
+
+Test performance by manually reducing the arguments for findkernelmeth of
+BalTreeForBlocks and Imprimitive? Or maybe even of the whole
+`RECOG_Find..StandardArguments`?
+- Setting BAL_CONST to 0.1 and running recognition of the wreath product 
+      g := WreathProduct(SymmetricGroup(5),SymmetricGroup(32));
+  triggers lvl 3 crises and works fine. Time for recognition is roughly 5s with
+  no extreme outliers.
+  BUT See how this behaves if I only reduce the "conjugations"
+  argument for findgensNmeth
+
+When crises are raised very often, that may be a hint, that the kernel
+generating size is too small. To an extent, it should be fine to rely on the
+mandarins to catch that.
+
+- ce1f513 master
+  total    591680 ms (34347 ms GC) and 57.3GB allocated
+- 61adc09 but without handling crises and erroring out instead
+  total    982249 ms (78561 ms GC) and 88.7GB allocated
+- b7c0cf0 Mandarins with SLPs but without checking equality
+  total   1043838 ms (84736 ms GC) and 89.6GB allocated
+- faf71d4 Mandarins with SLPs
+  total   3370850 ms (505873 ms GC) and 153GB allocated
+- 65539bf Mandarins without SLPs
+  total   2866155 ms (365469 ms GC) and 188GB allocated
+
+## Performance improvements
+
+According to a few profiles I made a lot of time is spent in the GAP library
+functions in lib/straight.gi:
+- RewriteStraightLineProgram 	
+- ResultOfLineOfStraightLineProgram 	
+- IntermediateResultsOfSLPWithoutOverwriteInner 
+Also, the SLPs seem to use up a lot of storage. That should be reduced by
+implementing an SLP data-type on the kernel level.
diff --git a/TODO.md b/TODO.md
@@ -0,0 +1,14 @@
+- IsReady: only is not set when no recognition method was able to recognise a
+  node in the tree.
+- make in the package RecognizeGeneric only be called with RecogNodes
+  this would enable us to get rid of `depthstring`
+- IsSafeForMandarins: recog method should know whether there is a deterministic
+  function to compute its kernel. Name isFindGensNMethDeterministic?
+  => operation with two methods
+  - IsRecogNode: go check on successmethod
+  - IsRecogMethod: take its component
+  => findGensNmeth: is it ever set by anything but recognition methods? If not,
+  we can store it in the recog method.
+- ValidateHomomInput: everybody must set hasvalidatehomominput! Can we make
+  this a required component of the recognition method? 
+- Enforce that every recog method provides a way to SetNiceGens?
diff --git a/examples/challenges-by-frank-luebeck/README.md b/examples/challenges-by-frank-luebeck/README.md
@@ -0,0 +1,5 @@
+These challenges were taken from Frank Lübeck's homepage:
+
+    http://www.math.rwth-aachen.de/~Frank.Luebeck/data/MatrixChallenges/index.html?LANG=en
+
+Progress can be found in the file `solutions` in the directory above.
diff --git a/examples/example1.g.gz → ...challenges-by-frank-luebeck/example1.g.gz b/examples/example1.g.gz → ...challenges-by-frank-luebeck/example1.g.gz
diff --git a/examples/example10.g.gz → ...hallenges-by-frank-luebeck/example10.g.gz b/examples/example10.g.gz → ...hallenges-by-frank-luebeck/example10.g.gz
diff --git a/examples/example11.g.gz → ...hallenges-by-frank-luebeck/example11.g.gz b/examples/example11.g.gz → ...hallenges-by-frank-luebeck/example11.g.gz
diff --git a/examples/example12.g.gz → ...hallenges-by-frank-luebeck/example12.g.gz b/examples/example12.g.gz → ...hallenges-by-frank-luebeck/example12.g.gz
diff --git a/examples/example2.g.gz → ...challenges-by-frank-luebeck/example2.g.gz b/examples/example2.g.gz → ...challenges-by-frank-luebeck/example2.g.gz
diff --git a/examples/example3.g.gz → ...challenges-by-frank-luebeck/example3.g.gz b/examples/example3.g.gz → ...challenges-by-frank-luebeck/example3.g.gz
diff --git a/examples/example4.g.gz → ...challenges-by-frank-luebeck/example4.g.gz b/examples/example4.g.gz → ...challenges-by-frank-luebeck/example4.g.gz
diff --git a/examples/example5.g.gz → ...challenges-by-frank-luebeck/example5.g.gz b/examples/example5.g.gz → ...challenges-by-frank-luebeck/example5.g.gz
diff --git a/examples/example6.g.gz → ...challenges-by-frank-luebeck/example6.g.gz b/examples/example6.g.gz → ...challenges-by-frank-luebeck/example6.g.gz
diff --git a/examples/example7.g.gz → ...challenges-by-frank-luebeck/example7.g.gz b/examples/example7.g.gz → ...challenges-by-frank-luebeck/example7.g.gz
diff --git a/examples/example8.g.gz → ...challenges-by-frank-luebeck/example8.g.gz b/examples/example8.g.gz → ...challenges-by-frank-luebeck/example8.g.gz
diff --git a/examples/example9.g.gz → ...challenges-by-frank-luebeck/example9.g.gz b/examples/example9.g.gz → ...challenges-by-frank-luebeck/example9.g.gz
diff --git a/gap/base/kernel.gi b/gap/base/kernel.gi
@@ -74,6 +74,7 @@ InstallGlobalFunction( ImmediateVerification,
     fi;
     if verified = true then return true; fi;
     # Now, verified = false.
+    Print("Immediate verification: found extra kernel element(s)!\n");
     Info(InfoRecog,2,
          "Immediate verification: found extra kernel element(s)!");
     if FindKernelFastNormalClosure(ri,5,5) = fail then
@@ -166,6 +167,7 @@ InstallGlobalFunction( FastNormalClosure , function( G, list, n )
   if IsEmpty(list) then
     return [];
   fi;
+  Info(InfoRecog, 2, "Do FastNormalClosure with n=", n, ".");
   list2 := ShallowCopy(list);
   if IsGroup(G) then
     grpgens := GeneratorsOfGroup(G);

diff --git a/gap/base/recognition.gd b/gap/base/recognition.gd
@@ -470,6 +470,30 @@ BindGlobal( "FindHomMethodsGeneric", rec() );
 ## <#/GAPDoc>
 BindGlobal( "SLPforElementFuncsGeneric", rec() );
 
+# TODO Document mandarins
+# Refer to overview paper by Baarnhielm, Holt, Charles, Eamonn, sections "5.2
+# The main algorithm" and "5.4 Crisis management".
+# Explain safe and unsafe nodes.
+# Explain in which situations a crisis can be evoked:
+# - only in a leaf: mandarin can't be written as SLPs in nice gens, this does
+#   not happen in non-leaf nodes
+# - mandarins can't be mapped under reduction homomorphism
+# - 
+# Note that when implementing verification via presentations, that can also
+# evoke crises. Then I'd make sense to remove the "MANDARIN_" prefix from the
+# name.
+# TODO Wait a second, if the generators of the root can be expressed as SLPs in
+# the nice gens, then the kernels must be correct, right?!
+# Crisis objects
+DeclareCategory("IsRecogCrisis", IsObject);
+BindGlobal("RecogCrisisFamily", NewFamily("RecogCrisisFamily", IsRecogCrisis));
+BindGlobal("RecogCrisisType", NewType(RecogCrisisFamily, IsRecogCrisis));
+DeclareOperation( "RecogCrisis", [IsRecogNode]);
+DeclareFilter( "KernelGeneratorsAlreadyEnlargedByCrisis" );
+# TODO remove this:
+BindGlobal("MANDARIN_CRISIS", MakeImmutable("MANDARIN_CRISIS"));
+BindGlobal("NUM_MANDARINS_DEFAULT_VALUE", 100);
+DeclareFilter( "IsSafeForMandarins" );
 
 # Our global functions for the main recursion:
 
@@ -575,7 +599,9 @@ DeclareSynonym("RecognizeGroup", RecogniseGroup);
 ## </Description>
 ## </ManSection>
 ## <#/GAPDoc>
-DeclareGlobalFunction( "RecogniseGeneric" );
+# TODO: change documentation and update references from func to oper
+DeclareOperation( "RecogniseGeneric",
+                  [ IsRecogNode, IsObject, IsString, IsObject, IsBool ] );
 DeclareSynonym("RecognizeGeneric", RecogniseGeneric);
 
 DeclareGlobalFunction( "PrintTreePos" );