Query StudentSubmissions by coursework

[dchess]

Rather than querying the StudentSubmission endpoint by looping over Courses, it could be preferable to loop over individual assignments (CourseWork). That way the query could be date-bound by pre-filtering the assignments by creationTime. While this might result in more calls, it may reduce pagination because it would be less data volume overall. Using Courses to query pulls in every assignment ever in that class and many may no longer be relevant to monitor after a certain point. If history is necessary to maintain, one approach would be to handle similar to the StudentUsage data and truncate a subset and append.

[dchess]

@zkagin I think we can accomplish this simply by creating a new method for Courses:

def get_recent_course_ids(self):
    try:
        coursework = pd.read_sql_table("GoogleClassroom_CourseWork", con=self.sql.engine, schema=self.sql.schema)
        courses = coursework.loc[coursework.creationTime >= self.config.SCHOOL_YEAR_START]
        return courses.id.unique()
    except InvalidRequestError as error:
            logging.debug(error)
            return None

and then calling it in main.py:

if config.PULL_SUBMISSIONS:
    course_ids = Courses(classroom_service, sql, config).get_recent_course_ids()
    StudentSubmissions(classroom_service, sql, config).batch_pull_data(course_ids)

[dchess]

Actually, we may want to use a different env var for this date.