Skip to content

Commit 6172085

Browse files
raczakraczak
committed
full parse feature in fr & en added
1 parent 3375a1c commit 6172085

File tree

4 files changed

+100
-3
lines changed

4 files changed

+100
-3
lines changed

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ In fact Crawlit is a tool :
2828
- [Json file & MongoDB importable file generated](#json-files-also-provided-in-the-repository)
2929
- Resume parsing feature after errors (connection off / 429 error / ...)
3030
- English & French encyclopedia supported (french item, english item and soon all other versions)
31-
- Full Encyclopedia parse available : just execute the `npm run parse-all` command (be patient)
31+
- Full Encyclopedia parse available : just execute the `npm run parse-all-fr` or `npm run parse-all-en` command (**be patient**)
3232

3333
## Interactive prompt
3434

full-parse.sh renamed to full-parse-en.sh

File renamed without changes.

‎full-parse-fr.sh‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
#!/bin/bash
2+
3+
echo "============================================="
4+
echo "Equipments Parse"
5+
echo "============================================="
6+
7+
npm run crawlit && node ./dist/app.js --game dofus --language french --category allequipments --all
8+
sleep 900
9+
10+
echo "============================================="
11+
echo "Weapons Parse"
12+
echo "============================================="
13+
14+
npm run crawlit && node ./dist/app.js --game dofus --language french --category allweapons --all
15+
sleep 900
16+
17+
echo "============================================="
18+
echo "Set Parse"
19+
echo "============================================="
20+
21+
npm run crawlit && node ./dist/app.js --game dofus --language french --category set --all
22+
23+
24+
echo "============================================="
25+
echo "Pet Parse"
26+
echo "============================================="
27+
28+
npm run crawlit && node ./dist/app.js --game dofus --language french --category pet --all
29+
30+
31+
echo "============================================="
32+
echo "Mount Parse"
33+
echo "============================================="
34+
35+
npm run crawlit && node ./dist/app.js --game dofus --language french --category mount --all
36+
sleep 900
37+
38+
echo "============================================="
39+
echo "Resource Parse"
40+
echo "============================================="
41+
42+
npm run crawlit && node ./dist/app.js --game dofus --language french --category resource --all
43+
sleep 900
44+
45+
echo "============================================="
46+
echo "Consumable Parse"
47+
echo "============================================="
48+
49+
npm run crawlit && node ./dist/app.js --game dofus --language french --category consumable --all
50+
sleep 900
51+
52+
echo "============================================="
53+
echo "Monster Parse"
54+
echo "============================================="
55+
56+
npm run crawlit && node ./dist/app.js --game dofus --language french --category monster --all
57+
58+
59+
echo "============================================="
60+
echo "Profession Parse"
61+
echo "============================================="
62+
63+
npm run crawlit && node ./dist/app.js --game dofus --language french --category profession --all
64+
65+
66+
echo "============================================="
67+
echo "Harness Parse"
68+
echo "============================================="
69+
70+
npm run crawlit && node ./dist/app.js --game dofus --language french --category harness --all
71+
72+
73+
echo "============================================="
74+
echo "Class Parse"
75+
echo "============================================="
76+
77+
npm run crawlit && node ./dist/app.js --game dofus --language french --category class --all
78+
sleep 900
79+
80+
echo "============================================="
81+
echo "Idol Parse"
82+
echo "============================================="
83+
84+
npm run crawlit && node ./dist/app.js --game dofus --language french --category idol --all
85+
86+
87+
echo "============================================="
88+
echo "Havenbag Parse"
89+
echo "============================================="
90+
91+
npm run crawlit && node ./dist/app.js --game dofus --language french --category havenbag --all
92+
93+
echo "============================================="
94+
echo "Full parse Finished !"
95+
echo "Now you have the last version of the Encyclopedia in French !"
96+
echo "============================================="

‎package.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "@dofapi/crawlit",
3-
"version": "1.6.12",
3+
"version": "1.6.13",
44
"homepage": "https://github.com/dofapi/crawlit-dofus-encyclopedia-parser",
55
"description": "Crawl structured dofus encyclopedia data to populate the Dofapi app & API",
66
"author": {
@@ -14,7 +14,8 @@
1414
"prestart": "npm run -s build",
1515
"dev": "node lib/app.js",
1616
"crawlit": "npm run -s build",
17-
"parse-all": "sh full-parse.sh",
17+
"parse-all-fr": "sh full-parse-fr.sh",
18+
"parse-all-en": "sh full-parse-en.sh",
1819
"clean": "rm -R ./dist ./data",
1920
"clean-win": "rmdir dist data /s /q",
2021
"build": "babel --presets @babel/preset-env --ignore data -d dist/ lib",

0 commit comments

Comments
 (0)