Skip to content

Commit 0d56175

Browse files
Oscar-ggOscar-gg
authored
Merge pull request #71 from RoBorregos/hri
Hri
2 parents 5b2064e + fa21d62 commit 0d56175

File tree

4 files changed

+382
-2
lines changed

4 files changed

+382
-2
lines changed

‎docs/2024/Human-Robot Interaction/Areas/Respeaker.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ In addition, the ReSpeaker Mic Array has LED ring support, which can be used to
3131
The `ReSpeaker.py` file needs some permitions to interact with the ReSpeaker hardware (lights). To do so, you need to add the following udev rule:
3232

3333
```bash
34-
echo "SUBSYSTEM=="usb", ATTRS{idVendor}=="2886", MODE="0666"" | sudo tee -a /etc/udev/rules.d/99-usbftdi.rules
34+
echo "SUBSYSTEM=="usb", ATTR{idVendor}=="2886", ATTR{idProduct}=="0018", MODE="0666"" | sudo tee -a /etc/udev/rules.d/99-usb.rules
3535
```
3636

3737
After adding the rule, you need to reload the udev rules:

docs/development/integration/Task Breakdown/GPSR.md renamed to docs/development/integration/Task Breakdown/gpsr/GPSR.md

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,7 @@ command_string = "{find | locate | look for} the {object or single_category} and
131131
- {bring | give | deliver} it to the {gesture or pose} in the {room}"
132132
- {deliver | give} it to the {name} in the {room}"
133133
134-
2. `takeObjFromPlcmt`: The robot is asked to navigate to take an object form a location to another and perform an action.
134+
2. `takeObjFromPlcmt`: The robot is asked to navigate to take an object from a location to another and perform an action.
135135
136136
- The follow up action (self.generate_command_followup("hasObj")) are the same as the follow up for `findObj` discussed above.
137137
@@ -272,3 +272,13 @@ clothe_list = ["t shirt", "shirt", "blouse", "sweater", "coat", "jacket"]
272272
'soccer ball', 'dice', 'tennis ball', 'baseball', 'cleanser', 'sponge'
273273
}
274274
```
275+
276+
# Proposed subtask API for GPSR
277+
278+
## Vision
279+
280+
count_persons(clothe, color, gesture, pose, room)
281+
282+
identify_person_by_gesture()
283+
identify_person_by_pose()
284+
identify_person_by_name()

‎docs/development/integration/Task Breakdown/gpsr/examples.md‎

Lines changed: 311 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,311 @@
1+
# Command Break Down
2+
3+
Go to the dishwasher then find an apple and get it and bring it to the waving person in the office
4+
5+
-> go(dishwasher)
6+
-> find(apple)
7+
----> gaze(n predefined points)
8+
----> detect(apple)
9+
----> approach_point(apple)
10+
-> pick(apple)
11+
-> go(office)
12+
-> find(waving_person)
13+
----> gaze(n predefined points)
14+
----> detect(waving_person)
15+
----> approach_point(waving_person)
16+
-> give(apple)
17+
18+
Locate a waving person in the kitchen and answer a question
19+
20+
-> go(kitchen)
21+
-> find(waving_person)
22+
----> gaze(n predefined points)
23+
----> detect(waving person)
24+
----> approach_point(waving_person)
25+
-> say(ask_question)
26+
-> hear(user)
27+
-> contextual_say(answer)
28+
29+
Meet Axel in the bedroom and answer a quiz
30+
-> go(bedroom)
31+
-> find(Axel)
32+
----> gaze(n predefined points)
33+
----> detect(person)
34+
----> if person[i] == Axel then approach_point(Axel)
35+
----> else: for person in persons: approach_point(person) -> ask if is Axel
36+
----> save_to_context(Axel)
37+
-> say(ask_question)
38+
-> hear(user)
39+
-> contextual_say(answer)
40+
41+
Tell me how many waving persons are in the kitchen
42+
-> go(kitchen)
43+
-> count(waving_person)
44+
----> gaze(n predefined points)
45+
----> detect(waving person)
46+
-> go(task_origin)
47+
-> contextual_say(answer)
48+
49+
Tell me the gesture of the person at the tv stand
50+
-> go(tv_stand)
51+
-> find(person)
52+
----> gaze(n predefined points)
53+
----> detect(person)
54+
-> go(task_origin)
55+
-> contextual_say(gesture)
56+
57+
Say your teams affiliation to the waving person in the bathroom
58+
-> go(bathroom)
59+
-> find(waving_person)
60+
----> gaze(n predefined points)
61+
----> detect(waving person)
62+
----> approach_point(waving_person)
63+
-> contextual_say(affiliation)
64+
65+
Answer the quiz of the person raising their right arm in the living room
66+
-> go(living_room)
67+
-> find(person_raising_right_arm)
68+
----> gaze(n predefined points)
69+
----> detect(person_raising_right_arm)
70+
----> approach_point(person_raising_right_arm)
71+
-> say(ask_quiz)
72+
-> hear(user)
73+
-> contextual_say(answer)
74+
75+
Follow Jane from the desk to the kitchen
76+
-> go(desk)
77+
-> find(Jane)
78+
----> gaze(n predefined points)
79+
----> detect(person)
80+
----> if person[i] == Jane then approach_point(Jane)
81+
----> else: for person in persons: approach_point(person) -> ask if is Jane
82+
----> save_to_context(Jane)
83+
84+
-> say(intent)
85+
-> follow_person_until(Jane, kitchen)
86+
87+
Escort Axel from the trashbin to the pantry
88+
89+
-> go(trashbin)
90+
-> find(Axel)
91+
----> gaze(n predefined points)
92+
----> detect(Axel)
93+
----> if person[i] == Axel then approach_point(Axel)
94+
----> else: for person in persons: approach_point(person) -> ask if is Axel
95+
----> save_to_context(Axel)
96+
97+
-> say(intent)
98+
-> escort_to(pantry)
99+
100+
Take the person pointing to the right from the armchair to the exit
101+
-> go(armchair)
102+
-> find(person_pointing_right)
103+
----> gaze(n predefined points)
104+
----> detect(person_pointing_right)
105+
----> approach_point(person_pointing_right)
106+
-> say(intent)
107+
-> escort_to(exit)
108+
109+
Take the person wearing a yellow t shirt from the side tables to the office
110+
-> go(side_tables)
111+
-> find(person_yellow_tshirt)
112+
----> gaze(n predefined points)
113+
----> detect(person_yellow_tshirt)
114+
----> approach_point(person_yellow_tshirt)
115+
-> say(intent)
116+
-> escort_to(office)
117+
118+
Say hello to the person wearing a gray t shirt in the living room and answer a quiz
119+
-> go(living_room)
120+
-> find(person_gray_tshirt)
121+
----> gaze(n predefined points)
122+
----> detect(person_gray_tshirt)
123+
----> approach_point(person_gray_tshirt)
124+
-> say(hello)
125+
-> say(ask_quiz)
126+
-> hear(user)
127+
-> contextual_say(answer)
128+
129+
Say hello to Morgan in the kitchen and answer a question
130+
-> go(kitchen)
131+
-> find(Morgan)
132+
----> gaze(n predefined points)
133+
----> detect(Morgan)
134+
----> if person[i] == Morgan then approach_point(Morgan)
135+
----> else: for person in persons: approach_point(Morgan) -> ask if is Morgan
136+
----> save_to_context(Morgan)
137+
138+
-> say(hello)
139+
-> say(ask_question)
140+
-> hear(user)
141+
-> contextual_say(answer)
142+
143+
Note: in the second find, the person should be already known i/e found person[i] == Jane
144+
Meet Jane at the entrance then locate them in the kitchen
145+
-> go(entrance)
146+
-> find(Jane)
147+
----> gaze(n predefined points)
148+
----> detect(Jane)
149+
----> if person[i] == Jane then approach_point(Jane)
150+
----> else: for person in persons: approach_point(Jane) -> ask if is Jane
151+
----> save_to_context(Jane)
152+
-> go(kitchen)
153+
-> find(Jane)
154+
----> gaze(n predefined points)
155+
----> detect(Jane)
156+
----> if person[i] == Jane then approach_point(Jane)
157+
----> else: for person in persons: approach_point(Jane) -> ask if is Jane
158+
----> save_to_context(Jane)
159+
-> say(intent finished)
160+
161+
Tell me how many people in the office are wearing yellow coats
162+
-> go(office)
163+
-> count(person_yellow_coat)
164+
----> gaze(n predefined points)
165+
----> detect(person_yellow_coat)
166+
----> save_to_context(count)
167+
168+
-> go(task_origin)
169+
-> contextual_say(answer)
170+
171+
Tell me how many people in the living room are wearing gray coats
172+
-> go(living_room)
173+
-> count(person_gray_coat)
174+
----> gaze(n predefined points)
175+
----> detect(person_gray_coat)
176+
----> save_to_context(count)
177+
-> go(task_origin)
178+
-> contextual_say(answer)
179+
180+
Tell the gesture of the person at the side tables to the person at the tv stand
181+
-> go(side_tables)
182+
-> find(person)
183+
----> gaze(n predefined points)
184+
----> detect(person)
185+
----> save_to_context(gesture)
186+
187+
-> go(tv_stand)
188+
-> find(person)
189+
----> gaze(n predefined points)
190+
----> detect(person)
191+
-> contextual_say(gesture)
192+
193+
Follow the standing person in the living room
194+
-> go(living_room)
195+
-> find(standing_person)
196+
----> gaze(n predefined points)
197+
----> detect(standing_person)
198+
----> approach_point(standing_person)
199+
-> say(intent)
200+
-> follow_person_until_canceled(standing_person)
201+
202+
Go to the kitchen then locate a snack and get it and deliver it to Adel in the bathroom
203+
-> go(kitchen)
204+
-> find(snack)
205+
----> gaze(n predefined points)
206+
----> detect(snack)
207+
----> approach_point(snack)
208+
-> pick(snack)
209+
-> go(bathroom)
210+
-> find(Adel)
211+
----> gaze(n predefined points)
212+
----> detect(Adel)
213+
----> if person[i] == Adel then approach_point(Adel)
214+
----> else: for person in persons: approach_point(Adel) -> ask if is Adel
215+
----> save_to_context(Adel)
216+
-> say(intent)
217+
-> give(snack)
218+
219+
Take a cleaning supply from the kitchen table and bring it to the standing person in the kitchen
220+
-> go(kitchen_table)
221+
-> find(cleaning_supply)
222+
----> gaze(n predefined points)
223+
----> detect(cleaning_supply)
224+
----> approach_point(cleaning_supply)
225+
-> pick(cleaning_supply)
226+
-> find(standing_person)
227+
----> gaze(n predefined points)
228+
----> detect(standing_person)
229+
----> approach_point(standing_person)
230+
-> say(intent)
231+
-> give(cleaning_supply)
232+
233+
Locate a cleaning supply in the bedroom then take it and bring it to the standing person in the bedroom
234+
-> go(bedroom)
235+
-> find(cleaning_supply)
236+
----> gaze(n predefined points)
237+
----> detect(cleaning_supply)
238+
----> approach_point(cleaning_supply)
239+
-> pick(cleaning_supply)
240+
-> find(standing_person)
241+
----> gaze(n predefined points)
242+
----> detect(standing_person)
243+
----> approach_point(standing_person)
244+
-> say(intent)
245+
-> give(cleaning_supply)
246+
247+
Tell me how many fruits there are on the bed
248+
-> go(bed)
249+
-> count(fruits)
250+
----> gaze(n predefined points)
251+
----> detect(fruits)
252+
----> save_to_context(count)
253+
-> go(task_origin)
254+
-> contextual_say(answer)
255+
256+
TODO: how to detect biggest object?
257+
Tell me what is the biggest object on the kitchen table
258+
-> go(kitchen_table)
259+
-> find(biggest_object)
260+
----> gaze(n predefined points)
261+
----> detect(biggest_object)
262+
----> save_to_context(objects)
263+
-> go(task_origin)
264+
-> contextual_say(biggest_object)
265+
266+
Bring me a soccer ball from the dishwasher
267+
-> go(dishwasher)
268+
-> find(soccer_ball)
269+
----> gaze(n predefined points)
270+
----> detect(soccer_ball)
271+
----> approach_point(soccer_ball)
272+
-> pick(soccer_ball)
273+
-> go(task_origin)
274+
-> give(soccer_ball)
275+
276+
TODO: how to detect lightest object that is a toy?
277+
Tell me what is the lightest toy on the bookshelf
278+
-> go(bookshelf)
279+
-> find(lightest_toy)
280+
----> gaze(n predefined points)
281+
----> detect(toys)
282+
----> save_to_context(toys)
283+
284+
-> go(task_origin)
285+
-> contextual_say(lightest_toy)
286+
287+
## Todo ideas
288+
289+
TODO: how to detect lightest object that is a toy?
290+
291+
- When in the find method, insert all objects matching description to a db.
292+
- In the db, store previously know object properties.
293+
- When determining the lightest object, compare the objects in the db and return the lightest one.
294+
- Perhaps, implement a contextual_say method that fetches context to answer queries?
295+
- Let the contextual say choose between the databases
296+
297+
TODO: how to detect biggest object?
298+
299+
- Same idea as previous todo
300+
301+
TODO: how to handle finds with different requirements?
302+
303+
- i.e. find by name, by posture, by object recognizable with model?
304+
305+
TODO: deux ex machina integration
306+
307+
-
308+
309+
TODO: confirmation implementation
310+
311+
- How to add confirmation and fallbacks to the subcommands? add confirmation within the subcommand or before/after the subcommands?

0 commit comments

Comments
 (0)