(The speech bug) Sometimes hears something that was in a grammar loaded earlier

Sometimes the following happens:
Load a grammar to listen to a task description, hear a task; 
load <yes|no>, (I don't know if it happens when nothing is heard here or when yes/no is heard;
load a grammar for a task again, now it hears yes/no