Skip to content

Commit c70c496

Browse files
gflofstgflofst
committed
version 1.1 release
1 parent 9821195 commit c70c496

File tree

6 files changed

+79069
-54284
lines changed

6 files changed

+79069
-54284
lines changed

‎README.rst‎

Lines changed: 179 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,57 +1,196 @@
1-
Additive-IO
2-
===========
1+
Stitch IO
2+
=========
33

4-
by Jay Lofstead, John Mitchell, and Steve Plimpton
4+
by Jay Lofstead (gflofst@sandia.gov) and John Mitchell (jamitch@sandia.gov)
5+
with consulting by Steve Plimpton
56

6-
This project aims to develop an new, efficient IO library optimized for the
7-
style of operation found in the SPPARKS kinetic monte carlo application
8-
(http://spparks.sandia.gov)
7+
This project aims to develop a new, efficient IO library optimized for the
8+
style of operation found in the SPPARKS kinetic monte carlo applications
9+
(http://spparks.sandia.gov). This novel library was motivated by additive
10+
manufacturing applications where material is incrementally added to a part;
11+
using the 'Stitch' libary, data/material is appended to a file much in the same
12+
way material is incrementally added to a part; in this way, simulations of the
13+
additive manufacturing process are conducted locally, ie only on that portion
14+
of the build which is active. This approach to IO is very efficient computationally
15+
since only the active portion of build is computed on; furthermore, the output
16+
database is extremely compact since state is only written to the file when
17+
it changes.
918

10-
The initial design point is using a key-value store for each active region
11-
written with a metadata engine to track all of the basic information and a
12-
generative naming system for finding data later.
1319

14-
The tests and Python interface all assume a current Python 3.x implementation.
20+
Build
21+
-----
1522

16-
The current build setup is fragile, but works. Here is how to make it work:
23+
There are two different entities which may be built.
1724

18-
1. Go to the libstitch directory
19-
2. 'make stitch_module' will create a serial version of the library for Python use and install it in the specified installation directory.
20-
3. 'make stitch.lib' builds a parallel C interface suitable for linking with anything.
21-
4. 'make stitch_test' builds the parallel C test harness that tests the API, but not the data.
25+
#. libstitch.a; for use with c/c++ applications; can be built for serial
26+
or parallel calculations.
2227

23-
To test the library correctness (serial interface):
28+
#. python module; must have python 3, numpy, scipy.
2429

25-
1. build the Python version (serial)
26-
2. go into integrated_test/weld/potts
27-
3. make clean ; make
30+
Below, STITCH denotes the directory where 'Stitch' is cloned.
2831

29-
This should run clean.
32+
Build the 'stitch' python module
33+
--------------------------------
3034

31-
To test the C API correctness (parallel):
35+
The 'stitch' python module allows for creating, reading and writing 'stitch'
36+
files. It is particularly useful for reading 'stitch' files written by
37+
'spparks'.
38+
39+
The python module should be built with gnu gcc. It may be possible to do
40+
otherwise but that has not been tested. In general, the same compiler used to
41+
build python, numpy, and scipy should be used to build the stitch module.
42+
43+
Python 3, numpy, and scipy are required for building the 'stitch' python
44+
module. This documentation was developed with the following versions.
45+
46+
* gnu gcc 9.2.0
47+
* Python 3.6.9
48+
* numpy '1.19.2'
49+
* scipy '1.5.2'
50+
51+
Other versions for all of the above will probably work provided
52+
python 3 is used.
53+
54+
The python module is generally built for serial applications that post process
55+
stitch files; this is the default and is recommended. gcc or other compatible
56+
compilers are used to build the python module. The python module may be built
57+
with an mpi compiler but will remain a serial python interface to a stitch
58+
file.
59+
60+
Change to stitch directory
61+
62+
::
63+
64+
% cd $STITCH
65+
66+
In the root project directory, create a setup.cfg file appropriate for your
67+
system. Look at the other examples for guidance. This is important for the
68+
python build setup. The file 'setup.flamer.cfg' and others are included in
69+
this directory; copy one of these to setup.cfg and edit appropriately; specify
70+
where the python module should be installed -- then add this path to your
71+
PYTHONPATH.
72+
73+
Build and install the python module.
74+
75+
::
76+
77+
% python setup.py build
78+
% python setup.py install
79+
80+
Some build 'warnings' may occur which are generally harmless. On the other
81+
hand 'errors' are not good and must be addressed.
82+
83+
Verify install and PYTHONPATH are correctly set by launching python and
84+
importing the stitch module. Make sure not to do this in the source directory.
85+
86+
If the following command succeeds, python module build and install is
87+
probably good. Start an interactive python session by launching python
88+
and import the stitch module:
89+
90+
::
91+
92+
% python
93+
% from stitch.libstitch import libstitch
94+
95+
As final verification of python module correctness, run the unit test; remove
96+
all stitch files prior to running the test -- otherwise the test will fail.
97+
98+
::
99+
100+
% cd $STITCH/integrated_test/weld/potts/
101+
% rm *.st
32102
33-
1. build the stitch_test test harness
34-
2. mpiexec -np 2 stitch_test 2 1 1
35103

36-
This should just output a bunch of 3-d box corner pairs without any other output. If this happens, it is working correctly.
104+
Run unit test
37105

38-
It is possible to build a serial C library or a Parallel Python library.
106+
::
39107

40-
To build a serial C library:
108+
% python unit_cv_readwrite.py
41109

42-
1. edit the Makefile to remove the -DSTITCH_PARALLEL from both stitch_test and stitch.lib.
43-
2. make stitch.lib ; make stitch_test
44-
3. stitch_test
45110

46-
To build a parallel Python library:
111+
Screen output from test should look something like:
112+
113+
::
114+
115+
.........
116+
----------------------------------------------------------------------
117+
Ran 9 tests in 9.383s
118+
119+
OK
120+
121+
122+
The python module may be built for parallel python applications
123+
using mpi4py -- this works but is not as well
124+
exercised and tested and is not recommended as its still an area of
125+
development. To build a parallel Python module:
126+
127+
* Build and/or install mpi4py; make sure import mpi4py works
128+
* Add -DSTITCH_PARALLEL to the extra_compiler_args list in libstitch/setup.py
129+
* build and test python module as described for the serial python module build
130+
* See Makefile on howto run parallel python tests
131+
* Should pass all tests.
132+
133+
134+
Build the 'stitch' library for c/c++ application development
135+
------------------------------------------------------------
136+
137+
::
138+
139+
% cd $STITCH/libstitch
140+
141+
142+
It is possible to build a serial C library or a Parallel Python library.
143+
144+
Edit path to c compiler by appropriately defining CC in the 'Makefile'. Use
145+
the same mpi compiler to build the stitch library as will be used for the
146+
parallel application. Use -DSTITCH_PARALLEL macro for parallel applications
147+
such as the SPPARKS Kinetic Monte Carlo simulation framework. To build a
148+
serial C library edit the Makefile and remove -DSTITCH_PARALLEL from both
149+
stitch_test and stitch.lib.
150+
151+
Build libstitch.a suitable for linking with c/c++ applications;
152+
153+
::
154+
155+
% make stitch.lib
156+
157+
Build 'stitch_test' written in C; this test exercises the c/c++ parallel API;
158+
159+
::
160+
161+
% make stitch_test
162+
163+
To test the C API correctness (parallel):
47164

48-
1. Figure out your mpi4py import statement.
49-
2. edit setup.py to change the mpi4py import to be what is correct for you.
50-
3. Add -DSTITCH_PARALLEL to the extra_compiler_args list.
51-
4. Make stitch_module
52-
5. go to integrated_tests/weld/potts
53-
6. edit the Makefile
54-
7. comment the serial test case out
55-
8. uncomment the two parallel test cases
56-
9. make
57-
10. This should pass all tests.
165+
::
166+
167+
% mpiexec -np 4 stitch_test
168+
169+
Depending on the mode selected, it will either test (default), generate output
170+
to the console, or write to files to configure a new testing dataset. For the
171+
console output, it should just output a bunch of 3-d box corner pairs and the
172+
values for proc 0. If this happens and it looks right, it is probably working
173+
correctly. The code is configured to be adaptable, but is currently assuming a
174+
2x2x1 process layout per timestep with each process having a 2x2x3 chunk of the
175+
domain. It increments in x and y by 3 and z by 1. Since there are no
176+
incorporated automated testing, it can be scaled and visually tested by
177+
adapting the internal parameters. There are four sets of relevant parameters:
178+
the global (total) domain size, the process count in each dimension, how big
179+
the decomposition is in each dimension, and how far to move when incrementing
180+
in that dimension.
181+
182+
183+
TROUBLESHOOTING
184+
---------------
185+
186+
* Stitch's file format (SQLite3) relies on the file system to properly handle
187+
POSIX locks to manage concurrency. GPFS mostly to completely does this
188+
properly. Lustre does not do this at all. To work around this issue, the
189+
parallel version uses token passing to serialize access. This is created with
190+
-DSTITCH_PARALLEL. If there are issues related to this, please contact us for
191+
an additional workaround.
192+
193+
* The library is written to handle concurrency by retrying. It may fail, but it
194+
should not. There are a lot of error messages that include line numbers,
195+
function names and other information. Please include that info with any bug
196+
report.

0 commit comments

Comments
 (0)