Isolate shared C files to a common shared library

[z5146542]

This pull request now puts commonly used .c and .h files in a shared library directory.
Previosuly, when the C code is generated, all relevant files are created within the specified root directory:

root/
root/test.c
root/utils.c
root/utils.h
root/Makefile
root/README
...

With this change, common .c and .h files are now moved to a shared library directory, and Makefile is made aware of this change. The behaviour is different depending on whether the specified output is a tarball.

If the output is is not a tarball, we now have the following directories and files generated:

root/
root/test.c
root/Makefile
root/README
shared_lib/
shared_lib/utils.c
shared_lib/utils.h
...

If the output is a tarball, the tarball will include the shared_lib directory and ultimately comprise the following:

tar.tar/test.c
tar.tar/Makefile
tar.tar/README
tar.tar/shared_lib/
tar.tar/shared_lib/utils.c
tar.tar/shared_lib/utils.h
...

[diaolo01]

Please look into getting this PR to pass make litmus-aarch64-test locally.
I have also attempted to build AArch64/A11.litmus and it fails for me. Can you please investigate.

[diaolo01]

Same argument as for cpy.

[z5146542]

For the previous case, I agree that we can use cpy' but for cpy_platform this needs change because this method does not specify distinct src and dst arguments. I think simply removing cpy_platform and moving cpy_shared_platform to cpy_platform should solve the problem.

[diaolo01]

I see the difference. I don't understand why there is cpy and cpy' - to me, it could have been a single function but whenever the call happened to cpy then dst=src. Am I missing anything?

That would be my preferred approach for `cpy_platform' - we can introduce a new argument that is the destination. I am open to suggestions, but let's not duplicate code.

[z5146542]

No, I also agree -- I think cpy can also be removed. Happy to do so if you think it should be done.

[z5146542]

Please look into getting this PR to pass make litmus-aarch64-test locally. I have also attempted to build AArch64/A11.litmus and it fails for me. Can you please investigate.

Thanks for this. This should now be fixed, and make litmus-aarch64-test should pass all tests. Please let me know if there are further issues.

[maranget]

Hi @z5146542, I have failed to compile a vmsa test.

On my local machine:

% cat CoRR-oa.litmus 
AArch64 CoRR-oa
{
int x=1;
int y=2;
0:X0=x;
1:X0=PTE(x);
1:X1=(oa:PA(y));
}

  P0          |  P1         ;
 LDR W1,[X0]  | STR X1,[X0] ;
 LDR W2,[X0]  |             ;
exists 0:X2=1 /\ 0:X1=2
% litmus7 -mach kvm-aarch64 -o /tmp/A.tar  CoRR-oa.litmus
% scp /tmp/A.tar chianti:

On "chianti" (some AArch64 machine, with kvm-unit-tests installed)

% cd kvm-unit-tests/A
% tar xmf ../../A.tar
% make 
make: *** No rule to make target '/home/maranget/kvm-unit-tests/shared_lib/litmus_rand.o', needed by 'CoRR-oa.elf'.  Stop.

Could you tell me more about the motivation of this PR?

[z5146542]

Hi @maranget ,

On "chianti" (some AArch64 machine, with kvm-unit-tests installed)

% cd kvm-unit-tests/A
% tar xmf ../../A.tar
% make 
make: *** No rule to make target '/home/maranget/kvm-unit-tests/shared_lib/litmus_rand.o', needed by 'CoRR-oa.elf'.  Stop.

When the output is specified to be a tarball, the shared_lib/ directory (which contains common .c and .h files used by the generated C code) should be placed on the parent directory. I believe one could run:

$ tar -xmf ../../A.tar shared_lib/ -C .. && tar -xmf ../../A.tar --exclude="shared_lib*" 

...and then run make in kvm-unit-tests/A.
This step is not necessary if the output is not specified to be a tarball, since the shared_lib/ directory will be placed in the parent directory automatically.

Could you tell me more about the motivation of this PR?

The main motivation behind this PR is to remove unnecessary duplication of commonly used .c and .h files used by many tests (etc. litmus_rand.[ch], utils.[ch], and put such files in a shared directory.

Essentially, when running make in the directory containing the main C code, it is assumed that the shared_lib/ directory exists in the parent directory.

[maranget]

Hi @z5146542. I do think the old behaviour has to be preserved: issuing make in the root directory of the tarball should build the tests. Building the tarball on one machine and compiling it on another is a very frequent standard practice, as litmus may not be installed to the low-end target machine. Notice that this behaviour is documented, for instance here.

I am not convinced that copying the various utility files is too high a price for having self-contained tarballs. Here are some reasons. Utility source code is shared between the tests compiled in the same batch, i.e. by the same invocation of litmus7. Even more, when using the -driver C option, all tests in the same batch reside in the same executable file run.exe and will thus share the same utility object code. If I understand the purpose of this PR correctly, sharing would occur only between different batches, provided the C files are located in a directories that are siblings of shared_lib/. Moreover, the exact "utility" C files may somehow depend on litmus7 options -- Consider for instance -mode std vs. -mode presi. As a consequence, sharing them between various invocations of litmus7 may not be that obvious. Please correct me if my understanding of this PR intent is wrong.

[relokin]

@z5146542 I agree with @maranget and I don't see why we are making this change. I don't think that we want to have a common library across different batches of limtus7 outputs. I don't see the value in doing that.

When we first discussed this, the idea was to try and move functions that now are inlined into a header that can be included and a source .c file that we can compile and link with.

For example, when I do:

$> litmus7 -mach kvm-aarch64 catalogue/aarch64-VMSA/tests/order-via-faults/R-via_fault_STR* -o /tmp/foo

litmus7 produces the following source files:

$> ls -1 /tmp/foo/R-via_fault_STR-*
/tmp/foo/R-via_fault_STR-2.c
/tmp/foo/R-via_fault_STR-mod1.c
/tmp/foo/R-via_fault_STR-mod2.c
/tmp/foo/R-via_fault_STR-mod3.c
/tmp/foo/R-via_fault_STR-with-DSB.c

all of them have a copy of the function fault_handler because litmus7 inlines the code from litmus/libdir/_aarch64/kvm_fault_handler.c in each of the source files above. This creates a complexity in the code of limtus7:

• We have find out precisely the cases where fault_handler is needed and
• Avoid inline it as this would cause gcc to through a used function warning which is treated as an error.
  I think it would be great if we had these common functions in a separate source file that all R-via_fault_*.o would have to link with to create the .flat files.

[maranget]

I agree with @relokin. I know of at least another example of C code that is currently inlined and that can maybe be in an utility source file: the hashtable code used in the presi and kvm modes. Inlining was an easy solution as the table entry type and the hash function vary according to the tests. However, in standard mode, we face a similar situation and nevertheless use a tree structure implemented in the files outs.c and outs.h.