Skip to content

Commit 1b9760e

Browse files
samgibsonmojryankearsley-moj
samgibsonmoj
authored and
ryankearsley-moj
committed
Adds message/process flow to README (#60)
1 parent fa4d34d commit 1b9760e

File tree

1 file changed

+237
-0
lines changed

1 file changed

+237
-0
lines changed

‎README.md‎

Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,3 +59,240 @@ When running via Aspire, the following services are available:
5959
| **MinIO** | S3-compatible file storage | *random port* (check Aspire) | Username: `minioadmin`<br>Password: `minioadmin` |
6060
| **MSSQL** | Application databases (staging, running picture, matching, cluster) | `127.0.0.1,61749` | Username: `sa`<br>Password: `P@ssword123!` |
6161
| **RabbitMQ** | Message broker for inter-service communication | http://localhost:15672 | Username: `guest`<br>Password: `guest` |
62+
63+
## Message Flow Diagram
64+
65+
```
66+
┌─────────────────────────────────────────────────────────────────────────────┐
67+
│ INITIAL FILE DETECTION │
68+
└─────────────────────────────────────────────────────────────────────────────┘
69+
70+
┌──────────────┐
71+
│ FileSync │ (Monitors storage for new files)
72+
└──────┬───────┘
73+
│
74+
├─────────────────────────────────────────┐
75+
│ │
76+
▼ ▼
77+
DeliusDownloadFinishedMessage OfflocDownloadFinished
78+
│ │
79+
│ │
80+
▼ ▼
81+
82+
┌─────────────────────────────────────────────────────────────────────────────┐
83+
│ PARSING & CLEANING STAGE │
84+
└─────────────────────────────────────────────────────────────────────────────┘
85+
86+
┌──────────────────┐ ┌─────────────────┐
87+
│ Delius.Parser │ │ Offloc.Cleaner │
88+
│ │ │ │
89+
│ (Parses Delius │ │ (Cleans Offloc │
90+
│ files into │ │ files, removes │
91+
│ structured │ │ redundant │
92+
│ records) │ │ fields) │
93+
└────────┬─────────┘ └────────┬────────┘
94+
│ │
95+
│ Sends DB requests: │
96+
│ - StartDeliusFileProcessingRequest │
97+
│ │
98+
▼ ▼
99+
DeliusParserFinishedMessage OfflocCleanerFinishedMessage
100+
│ │
101+
│ │
102+
│ ▼
103+
│ ┌─────────────────┐
104+
│ │ Offloc.Parser │
105+
│ │ │
106+
│ │ (Parses cleaned │
107+
│ │ Offloc files │
108+
│ │ into structured│
109+
│ │ records) │
110+
│ └────────┬────────┘
111+
│ │
112+
│ │ Sends DB requests:
113+
│ │ - StartOfflocFileProcessingRequest
114+
│ │
115+
│ ▼
116+
│ OfflocParserFinishedMessage
117+
│ │
118+
└───────────────┬───────────────────────┘
119+
│
120+
▼
121+
122+
┌─────────────────────────────────────────────────────────────────────────────┐
123+
│ STAGING & IMPORT STAGE │
124+
└─────────────────────────────────────────────────────────────────────────────┘
125+
126+
┌────────────────┐
127+
│ Import │
128+
│ │
129+
│ (Coordinates │
130+
│ staging and │
131+
│ merging of │
132+
│ both data │
133+
│ sources) │
134+
└───────┬────────┘
135+
│
136+
│ Sends DB requests:
137+
│ - StageDeliusRequest
138+
│ - MergeDeliusRequest
139+
│ - StageOfflocRequest
140+
│ - MergeOfflocRequest
141+
│
142+
▼
143+
┌────────────────┐
144+
│ DbInteractions │
145+
│ │
146+
│ (Stages data │
147+
│ from parsers, │
148+
│ merges into │
149+
│ running │
150+
│ picture DB) │
151+
└───────┬────────┘
152+
│
153+
│ Sends responses:
154+
│ - StageDeliusResponse
155+
│ - MergeDeliusResponse
156+
│ - StageOfflocResponse
157+
│ - MergeOfflocResponse
158+
│ - DeliusFilesCleanupMessage
159+
│ - OfflocFilesCleanupMessage
160+
│
161+
▼
162+
ImportFinishedMessage
163+
│
164+
│
165+
▼
166+
167+
┌─────────────────────────────────────────────────────────────────────────────┐
168+
│ MATCHING & BLOCKING STAGE │
169+
└─────────────────────────────────────────────────────────────────────────────┘
170+
171+
┌────────────────┐
172+
│ Blocking │
173+
│ │
174+
│ (Generates │
175+
│ candidate │
176+
│ pairs of │
177+
│ records that │
178+
│ may match) │
179+
└───────┬────────┘
180+
│
181+
▼
182+
BlockingFinishedMessage
183+
│
184+
│
185+
▼
186+
┌───────────────────────────────┐
187+
│ Matching.Engine │
188+
│ (ComparatorService) │
189+
│ │
190+
│ (Compares candidate pairs │
191+
│ using matching rules to │
192+
│ identify potential matches) │
193+
└───────────────┬───────────────┘
194+
│
195+
▼
196+
MatchingScoreCandidatesMessage
197+
│
198+
│
199+
▼
200+
┌───────────────────────────────┐
201+
│ Matching.Engine │
202+
│ (ScorerService) │
203+
│ │
204+
│ (Scores comparisons using │
205+
│ Bayesian probability to │
206+
│ determine match likelihood) │
207+
└───────────────┬───────────────┘
208+
│
209+
▼
210+
MatchingScoreCandidatesFinishedMessage
211+
│
212+
│
213+
▼
214+
215+
┌─────────────────────────────────────────────────────────────────────────────┐
216+
│ CLUSTERING STAGE │
217+
└─────────────────────────────────────────────────────────────────────────────┘
218+
219+
┌───────────────────────────────┐
220+
│ Matching.Engine │
221+
│ (ClusteringService) │
222+
│ │
223+
│ (Pre-processes clustering: │
224+
│ prepares data for grouping) │
225+
└───────────────┬───────────────┘
226+
│
227+
▼
228+
ClusteringPreProcessingStartedMessage
229+
│
230+
│
231+
▼
232+
┌───────────────────────────────┐
233+
│ Matching.Engine │
234+
│ (ComparatorService) │
235+
│ │
236+
│ (Compares outstanding edges │
237+
│ for clustering) │
238+
└───────────────┬───────────────┘
239+
│
240+
▼
241+
MatchingScoreOutstandingEdgesMessage
242+
│
243+
│
244+
▼
245+
┌───────────────────────────────┐
246+
│ Matching.Engine │
247+
│ (ScorerService) │
248+
│ │
249+
│ (Scores outstanding edges) │
250+
└───────────────┬───────────────┘
251+
│
252+
▼
253+
ClusteringPreProcessingFinishedMessage
254+
│
255+
│
256+
▼
257+
┌───────────────────────────────┐
258+
│ Matching.Engine │
259+
│ (ClusteringService) │
260+
│ │
261+
│ (Post-processes clustering: │
262+
│ groups related records into │
263+
│ clusters representing │
264+
│ unique individuals) │
265+
└───────────────┬───────────────┘
266+
│
267+
▼
268+
ClusteringPostProcessingFinishedMessage
269+
│
270+
│
271+
▼
272+
┌────────────────┐
273+
│ FileSync │
274+
│ │
275+
│ (Triggers next │
276+
│ processing │
277+
│ cycle if │
278+
│ configured) │
279+
└────────────────┘
280+
281+
┌─────────────────────────────────────────────────────────────────────────────┐
282+
│ DATA CONSUMPTION │
283+
└─────────────────────────────────────────────────────────────────────────────┘
284+
285+
┌────────────────┐
286+
│ API │
287+
│ │
288+
│ (Exposes REST │
289+
│ endpoints for │
290+
│ querying │
291+
│ processed │
292+
│ data) │
293+
└───────┬────────┘
294+
│
295+
▼
296+
External Consumers
297+
(e.g., CATS system)
298+
```

0 commit comments

Comments
 (0)