Commit ecddb9e
committed
File tree
284 files changed
+18853
-951
lines changed- backend
- openapi/specs
- services
- data-cleaning-service
- src/main
- java/com/dataengine/cleaning
- application
- httpclient
- service
- domain
- converter
- model
- infrastructure/persistence/mapper
- resources/mappers
- data-management-service
- src
- main
- java/com/dataengine/datamanagement
- application/service
- infrastructure/persistence/mapper
- interfaces
- dto
- rest
- resources/mappers
- test/java/com/dataengine/datamanagement/interfaces/rest
- deployment
- helm/ray
- ray-cluster
- templates
- kubernetes
- backend
- postgresql
- runtime
- operators
- filter
- file_with_high_repeat_phrase_rate_filter
- resources
- file_with_high_repeat_word_rate_filter
- file_with_high_special_char_rate_filter
- resources
- img_advertisement_images_cleaner
- img_blurred_images_cleaner
- img_duplicated_images_cleaner
- sql
- img_similar_images_cleaner
- sql
- remove_duplicate_file
- sql
- remove_file_with_many_sensitive_words
- resources
- remove_file_with_short_or_long_length
- formatter
- file_exporter
- img_formatter
- slide_formatter
- text_formatter
- word_formatter
- llms
- qa_condition_evaluator
- resources
- text_quality_evaluation
- resources
- mapper
- content_cleaner
- credit_card_number_cleaner
- email_cleaner
- emoji_cleaner
- extra_space_cleaner
- resources
- full_width_characters_cleaner
- garble_characters_cleaner
- resources
- html_tag_cleaner
- id_number_cleaner
- resources
- img_denoise
- img_direction_correct
- img_enhanced_brightness
- img_enhanced_contrast
- img_enhanced_saturation
- img_enhanced_sharpness
- img_perspective_transformation
- img_resize
- img_shadow_remove
- img_type_unify
- img_watermark_remove
- invisible_characters_cleaner
- ip_address_cleaner
- knowledge_relation_slice
- legend_cleaner
- phone_number_cleaner
- political_word_cleaner
- resources
- remove_duplicate_sentences
- sexual_and_violent_word_cleaner
- resources
- text_to_word
- traditional_chinese
- unicode_space_cleaner
- url_cleaner
- xml_tag_cleaner
- slicer
- segmentation
- slide_annotation_slicer
- slide_simple_slicer
- user
- python-executor
- data_platform
- common
- utils
- core
- ops
- scheduler
- sqlite_manager
- sql
- wrappers
- wrappers
- scripts
- db
- images
- data-juicer
- runtime
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
284 files changed
+18853
-951
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
70 | 74 | | |
71 | 75 | | |
72 | 76 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | 62 | | |
82 | 63 | | |
83 | 64 | | |
| |||
99 | 80 | | |
100 | 81 | | |
101 | 82 | | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | 83 | | |
130 | 84 | | |
131 | 85 | | |
| |||
149 | 103 | | |
150 | 104 | | |
151 | 105 | | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | 106 | | |
171 | 107 | | |
172 | 108 | | |
| |||
186 | 122 | | |
187 | 123 | | |
188 | 124 | | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
207 | 125 | | |
208 | 126 | | |
209 | 127 | | |
| |||
223 | 141 | | |
224 | 142 | | |
225 | 143 | | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | 144 | | |
236 | 145 | | |
237 | 146 | | |
| |||
252 | 161 | | |
253 | 162 | | |
254 | 163 | | |
255 | | - | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
264 | | - | |
265 | | - | |
266 | | - | |
267 | | - | |
268 | | - | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | 164 | | |
274 | 165 | | |
275 | 166 | | |
| |||
293 | 184 | | |
294 | 185 | | |
295 | 186 | | |
296 | | - | |
297 | | - | |
298 | | - | |
299 | | - | |
300 | | - | |
301 | | - | |
302 | | - | |
303 | | - | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
308 | | - | |
309 | | - | |
310 | | - | |
311 | | - | |
312 | | - | |
313 | | - | |
314 | 187 | | |
315 | 188 | | |
316 | 189 | | |
| |||
338 | 211 | | |
339 | 212 | | |
340 | 213 | | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | | - | |
360 | | - | |
361 | | - | |
362 | | - | |
363 | | - | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
368 | 214 | | |
369 | 215 | | |
370 | 216 | | |
| |||
384 | 230 | | |
385 | 231 | | |
386 | 232 | | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | | - | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
397 | | - | |
398 | | - | |
399 | | - | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | | - | |
404 | | - | |
405 | | - | |
406 | | - | |
407 | | - | |
408 | | - | |
409 | | - | |
410 | | - | |
411 | | - | |
412 | | - | |
413 | | - | |
414 | 233 | | |
415 | 234 | | |
416 | 235 | | |
| |||
421 | 240 | | |
422 | 241 | | |
423 | 242 | | |
424 | | - | |
425 | | - | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
426 | 246 | | |
427 | 247 | | |
428 | 248 | | |
| |||
449 | 269 | | |
450 | 270 | | |
451 | 271 | | |
452 | | - | |
453 | | - | |
| 272 | + | |
454 | 273 | | |
455 | 274 | | |
456 | 275 | | |
| |||
561 | 380 | | |
562 | 381 | | |
563 | 382 | | |
564 | | - | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
565 | 386 | | |
566 | 387 | | |
567 | 388 | | |
568 | 389 | | |
569 | 390 | | |
570 | 391 | | |
571 | 392 | | |
572 | | - | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
573 | 398 | | |
574 | 399 | | |
575 | 400 | | |
| |||
602 | 427 | | |
603 | 428 | | |
604 | 429 | | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
605 | 442 | | |
606 | 443 | | |
607 | 444 | | |
| |||
0 commit comments