Skip to content

Commit d7f06f5

Browse files
Abhishek-TAMUAbhishek-TAMU
authored
feat: Handle passing of multiple files, multiple folders, path with patterns, HF Dataset and combination (#424)
Signed-off-by: Abhishek <[email protected]>
1 parent 003041f commit d7f06f5

File tree

11 files changed

+859
-138
lines changed

11 files changed

+859
-138
lines changed

‎tests/artifacts/predefined_data_configs/tokenize_and_apply_input_masking.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,5 +10,5 @@ datasets:
1010
remove_columns: all
1111
batched: false
1212
fn_kwargs:
13-
input_field_name: "INPUT"
14-
output_field_name: "OUTPUT"
13+
input_field_name: input
14+
output_field_name: output

‎tests/artifacts/testdata/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@
2424
ARROW_DATA_DIR = os.path.join(os.path.dirname(__file__), "arrow")
2525
PARQUET_DATA_DIR = os.path.join(os.path.dirname(__file__), "parquet")
2626

27+
TWITTER_COMPLAINTS_DATA_DIR_JSON = os.path.join(DATA_DIR, "datafolder")
28+
2729
TWITTER_COMPLAINTS_DATA_JSON = os.path.join(
2830
JSON_DATA_DIR, "twitter_complaints_small.json"
2931
)

‎tests/artifacts/testdata/datafolder/twitter_complaints_input_output_1.json‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
[
2+
{
3+
"ID": 0,
4+
"Label": 2,
5+
"input": "@HMRCcustomers No this is my first job",
6+
"output": "no complaint"
7+
},
8+
{
9+
"ID": 1,
10+
"Label": 2,
11+
"input": "@KristaMariePark Thank you for your interest! If you decide to cancel, you can call Customer Care at 1-800-NYTIMES.",
12+
"output": "no complaint"
13+
},
14+
{
15+
"ID": 2,
16+
"Label": 1,
17+
"input": "If I can't get my 3rd pair of @beatsbydre powerbeats to work today I'm doneski man. This is a slap in my balls. Your next @Bose @BoseService",
18+
"output": "complaint"
19+
},
20+
{
21+
"ID": 3,
22+
"Label": 1,
23+
"input": "@EE On Rosneath Arial having good upload and download speeds but terrible latency 200ms. Why is this.",
24+
"output": "complaint"
25+
},
26+
{
27+
"ID": 4,
28+
"Label": 2,
29+
"input": "Couples wallpaper, so cute. :) #BrothersAtHome",
30+
"output": "no complaint"
31+
},
32+
{
33+
"ID": 5,
34+
"Label": 2,
35+
"input": "@mckelldogs This might just be me, but-- eyedrops? Artificial tears are so useful when you're sleep-deprived and sp\u2026 https://t.co/WRtNsokblG",
36+
"output": "no complaint"
37+
},
38+
{
39+
"ID": 6,
40+
"Label": 2,
41+
"input": "@Yelp can we get the exact calculations for a business rating (for example if its 4 stars but actually 4.2) or do we use a 3rd party site?",
42+
"output": "no complaint"
43+
},
44+
{
45+
"ID": 7,
46+
"Label": 1,
47+
"input": "@nationalgridus I have no water and the bill is current and paid. Can you do something about this?",
48+
"output": "complaint"
49+
},
50+
{
51+
"ID": 8,
52+
"Label": 1,
53+
"input": "Never shopping at @MACcosmetics again. Every time I go in there, their employees are super rude/condescending. I'll take my $$ to @Sephora",
54+
"output": "complaint"
55+
},
56+
{
57+
"ID": 9,
58+
"Label": 2,
59+
"input": "@JenniferTilly Merry Christmas to as well. You get more stunning every year \ufffd\ufffd",
60+
"output": "no complaint"
61+
},
62+
{
63+
"ID": 10,
64+
"Label": 2,
65+
"input": "@NortonSupport Thanks much.",
66+
"output": "no complaint"
67+
},
68+
{
69+
"ID": 11,
70+
"Label": 2,
71+
"input": "@VerizonSupport all of a sudden I can't connect to my primary wireless network but guest one works",
72+
"output": "no complaint"
73+
},
74+
{
75+
"ID": 12,
76+
"Label": 2,
77+
"input": "Aaaahhhhh!!!! My @Razer @PlayOverwatch d.va meka headset came in!!! I didn't even know it had shipped!!! So excited\u2026 https://t.co/4gXy9xED8d",
78+
"output": "no complaint"
79+
},
80+
{
81+
"ID": 13,
82+
"Label": 2,
83+
"input": "@Lin_Manuel @jmessinaphoto @VAMNit Omg a little squish!!!!! Enjoy and congrats!!!! I miss mine being so young! \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd",
84+
"output": "no complaint"
85+
},
86+
{
87+
"ID": 14,
88+
"Label": 2,
89+
"input": "@IanJamesPoulter What's your secret to poaching eggs? Mine NEVER look that good.",
90+
"output": "no complaint"
91+
},
92+
{
93+
"ID": 15,
94+
"Label": 2,
95+
"input": "@AWSSupport When will be able Kinesis Firehose compatible with Elasticsearch 6.0? Thank you!",
96+
"output": "no complaint"
97+
},
98+
{
99+
"ID": 16,
100+
"Label": 2,
101+
"input": "@NCIS_CBS https://t.co/eeVL9Eu3bE",
102+
"output": "no complaint"
103+
},
104+
{
105+
"ID": 17,
106+
"Label": 2,
107+
"input": "@msetchell Via the settings? That\u2019s how I do it on master T\u2019s",
108+
"output": "no complaint"
109+
},
110+
{
111+
"ID": 18,
112+
"Label": 2,
113+
"input": "Today at work there was a low flying duck heading toward a crowd of people, and I yelled \"watch out! and I'm very disappointed with myself.",
114+
"output": "no complaint"
115+
},
116+
{
117+
"ID": 19,
118+
"Label": 1,
119+
"input": "@NortonSupport @NortonOnline What the hell is a dm 5-10 days to get money back bank account now overdrawn thanks guys",
120+
"output": "complaint"
121+
},
122+
{
123+
"ID": 20,
124+
"Label": 1,
125+
"input": "@united not happy with this delay from Newark to Manchester tonight :( only 30 mins free Wi-fi sucks ...",
126+
"output": "complaint"
127+
},
128+
{
129+
"ID": 21,
130+
"Label": 1,
131+
"input": "@ZARA_Care I've been waiting on a reply to my tweets and DMs for days now?",
132+
"output": "complaint"
133+
},
134+
{
135+
"ID": 22,
136+
"Label": 2,
137+
"input": "New Listing! Large 2 Family Home for Sale in #Passaic Park, #NJ #realestate #homesforsale Great Location!\u2026 https://t.co/IV4OrLXkMk",
138+
"output": "no complaint"
139+
},
140+
{
141+
"ID": 23,
142+
"Label": 1,
143+
"input": "@SouthwestAir I love you but when sending me flight changes please don't use military time #ignoranceisbliss",
144+
"output": "complaint"
145+
},
146+
{
147+
"ID": 24,
148+
"Label": 2,
149+
"input": "@JetBlue Completely understand but would prefer being on time to filling out forms....",
150+
"output": "no complaint"
151+
}
152+
]

‎tests/artifacts/testdata/datafolder/twitter_complaints_input_output_2.json‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
[
2+
{
3+
"ID": 25,
4+
"Label": 2,
5+
"input": "@nvidiacc I own two gtx 460 in sli. I want to try windows 8 dev preview. Which driver should I use. Can I use the windows 7 one.",
6+
"output": "no complaint"
7+
},
8+
{
9+
"ID": 26,
10+
"Label": 2,
11+
"input": "Just posted a photo https://t.co/RShFwCjPHu",
12+
"output": "no complaint"
13+
},
14+
{
15+
"ID": 27,
16+
"Label": 2,
17+
"input": "Love crescent rolls? Try adding pesto @PerdueChicken to them and you\u2019re going to love it! #Promotion #PerdueCrew -\u2026 https://t.co/KBHOfqCukH",
18+
"output": "no complaint"
19+
},
20+
{
21+
"ID": 28,
22+
"Label": 1,
23+
"input": "@TopmanAskUs please just give me my money back.",
24+
"output": "complaint"
25+
},
26+
{
27+
"ID": 29,
28+
"Label": 2,
29+
"input": "I just gave 5 stars to Tracee at @neimanmarcus for the great service I received!",
30+
"output": "no complaint"
31+
},
32+
{
33+
"ID": 30,
34+
"Label": 2,
35+
"input": "@FitbitSupport when are you launching new clock faces for Indian market",
36+
"output": "no complaint"
37+
},
38+
{
39+
"ID": 31,
40+
"Label": 1,
41+
"input": "@HPSupport my printer will not allow me to choose color instead it only prints monochrome #hppsdr #ijkhelp",
42+
"output": "complaint"
43+
},
44+
{
45+
"ID": 32,
46+
"Label": 1,
47+
"input": "@DIRECTV can I get a monthly charge double refund when it sprinkles outside and we lose reception? #IamEmbarrasedForYou",
48+
"output": "complaint"
49+
},
50+
{
51+
"ID": 33,
52+
"Label": 1,
53+
"input": "@AlfaRomeoCares Hi thanks for replying, could be my internet but link doesn't seem to be working",
54+
"output": "complaint"
55+
},
56+
{
57+
"ID": 34,
58+
"Label": 2,
59+
"input": "Looks tasty! Going to share with everyone I know #FebrezeONE #sponsored https://t.co/4AQI53npei",
60+
"output": "no complaint"
61+
},
62+
{
63+
"ID": 35,
64+
"Label": 2,
65+
"input": "@OnePlus_IN can OnePlus 5T do front camera portrait?",
66+
"output": "no complaint"
67+
},
68+
{
69+
"ID": 36,
70+
"Label": 1,
71+
"input": "@sho_help @showtime your arrive is terrible streaming is stop and start every couple mins. Get it together it's xmas",
72+
"output": "complaint"
73+
},
74+
{
75+
"ID": 37,
76+
"Label": 2,
77+
"input": "@KandraKPTV I just witnessed a huge building fire in Santa Monica California",
78+
"output": "no complaint"
79+
},
80+
{
81+
"ID": 38,
82+
"Label": 2,
83+
"input": "@fernrocks most definitely the latter for me",
84+
"output": "no complaint"
85+
},
86+
{
87+
"ID": 39,
88+
"Label": 1,
89+
"input": "@greateranglia Could I ask why the Area in front of BIC Station was not gritted withh all the snow.",
90+
"output": "complaint"
91+
},
92+
{
93+
"ID": 40,
94+
"Label": 2,
95+
"input": "I'm earning points with #CricketRewards https://t.co/GfpGhqqnhE",
96+
"output": "no complaint"
97+
},
98+
{
99+
"ID": 41,
100+
"Label": 2,
101+
"input": "@Schrapnel @comcast RIP me",
102+
"output": "no complaint"
103+
},
104+
{
105+
"ID": 42,
106+
"Label": 2,
107+
"input": "The wait is finally over, just joined @SquareUK, hope to get started real soon!",
108+
"output": "no complaint"
109+
},
110+
{
111+
"ID": 43,
112+
"Label": 2,
113+
"input": "@WholeFoods what's the best way to give feedback on a particular store to the regional/national office?",
114+
"output": "no complaint"
115+
},
116+
{
117+
"ID": 44,
118+
"Label": 2,
119+
"input": "@DanielNewman I honestly would believe anything. People are...too much sometimes.",
120+
"output": "no complaint"
121+
},
122+
{
123+
"ID": 45,
124+
"Label": 2,
125+
"input": "@asblough Yep! It should send you a notification with your driver\u2019s name and what time they\u2019ll be showing up!",
126+
"output": "no complaint"
127+
},
128+
{
129+
"ID": 46,
130+
"Label": 2,
131+
"input": "@Wavy2Timez for real",
132+
"output": "no complaint"
133+
},
134+
{
135+
"ID": 47,
136+
"Label": 1,
137+
"input": "@KenyaPower_Care no power in south b area... is it scheduled.",
138+
"output": "complaint"
139+
},
140+
{
141+
"ID": 48,
142+
"Label": 1,
143+
"input": "Honda won't do anything about water leaking in brand new car. Frustrated! @HondaCustSvc @AmericanHonda",
144+
"output": "complaint"
145+
},
146+
{
147+
"ID": 49,
148+
"Label": 1,
149+
"input": "@CBSNews @Dodge @ChryslerCares My driver side air bag has been recalled and replaced, but what about the passenger side?",
150+
"output": "complaint"
151+
}
152+
]

0 commit comments

Comments
 (0)