Creating Regression Tests for Multi-task Usecases

[ajyl]

Hi,

I am interested in generating regression tests for a multi-task learning scenario.

While I am able to train or interact with the agents (ex: parlai dd -t task1,task2 ), when I try to create a regression test in a similar manner, I run into an exception while running pytest.

This becomes especially problematic when I have task-specific arguments that I want to include. For instance, in a perfect world, my test would look like the following:

class TestMultitask(AutoTeacherTest):
    task = "task1:arg1=x,arg2=y,task2:arg3=a,arg4=b"

I believe the problem is that commas are overloaded with multiple meanings -- 1) to indicate multiple tasks and 2) to indicate multiple arguments.

For what it's worth, the traceback I have when running pytest is as following:

self = <test.Test...Multitasks object at 0x7f43efb12430>, data_regression = <pytest_regressions.data_regression.DataRegressionFixture object at 0x7f43efb12730>

    def test_test_stream(self, data_regression):
        """
        Test --datatype test:stream.
        """
>       return self._regression(data_regression, 'test')

../../../parlai/utils/testing.py:482:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../parlai/utils/testing.py:454: in _regression
    world.parley()
../../../parlai/core/worlds.py:699: in parley
    self.parley_init()
../../../parlai/core/worlds.py:677: in parley_init
    if self.world_idx >= 0 and self.worlds[self.world_idx].episode_done():
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <parlai.core.worlds.MultiAgentDialogWorld object at 0x7f43ede89fd0>

    def episode_done(self):
        """
        Return if the episode is done for any subagent.
        """
        done = False
        for a in self.agents:
>           if a.episode_done():
E           AttributeError: 'xxxTeacher' object has no attribute 'episode_done'

../../../parlai/core/worlds.py:503: AttributeError

Thanks!

[ajyl]

I suppose I could try removing the task-specific arguments in my regression test to see if multitask regression tests can be made, and that could narrow down the scope.

[github-actions]

This issue has not had activity in 30 days. Please feel free to reopen if you have more issues. You may apply the "never-stale" tag to prevent this from happening.