Skip to content

Drastically speed up creating rejudgings by using direct queries #2648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 5, 2024
Merged

Drastically speed up creating rejudgings by using direct queries #2648

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions webapp/src/Service/DOMJudgeService.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1202,14 +1202,17 @@ public function maybeCreateJudgeTasks(Judging $judging, int $priority = JudgeTas
// - the new submission would get X+5+60 (since there's only one of their submissions still to be worked on),
// but we want to judge submissions of this team in order, so we take the current max (X+120) and add 1.
$teamPriority = (int)(max($result['max']+1, $submission->getSubmittime() + 60*$result['count']));
$queueTask = new QueueTask();
$queueTask->setJudging($judging)
->setPriority($priority)
->setTeam($team)
->setTeamPriority($teamPriority)
->setStartTime(null);
$this->em->persist($queueTask);
$this->em->flush();
// Use a direct query to speed things up
$this->em->getConnection()->executeQuery(
'INSERT INTO queuetask (judgingid, priority, teamid, teampriority, starttime)
VALUES (:judgingid, :priority, :teamid, :teampriority, null)',
[
'judgingid' => $judging->getJudgingid(),
'priority' => $priority,
'teamid' => $team->getTeamid(),
'teampriority' => $teamPriority,
]
);
}

public function getImmutableCompareExecutable(ContestProblem $problem): ImmutableExecutable
Expand Down
90 changes: 55 additions & 35 deletions webapp/src/Service/RejudgingService.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
use Doctrine\ORM\EntityManagerInterface;
use Doctrine\ORM\NonUniqueResultException;
use Doctrine\ORM\NoResultException;
use Ramsey\Uuid\Uuid;

class RejudgingService
{
Expand Down Expand Up @@ -99,43 +100,62 @@ public function createRejudging(
}


$this->em->wrapInTransaction(function () use (
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you check that this one was also slow?

If I understand the change correctly we still do a transaction but would loose the flush to the entitymanager, wouldn't we need the flush after the last transaction. Wouldn't we want all those transactions in 1 go?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This made it go from ~3x to ~50x speed improvement. The wrap does the exact same thing AND a flush. We don't need the flush if we don't use Doctrine and only direct queries, which we now do.

Doing one transaction might be faster but we also lock the database for some operations for the duration of the transaction, so we better keep it short. I don't think it will make a very big difference anyway.

$priority,
$singleJudging,
$judging,
$rejudging
) {
$this->em->getConnection()->executeStatement(
'UPDATE submission SET rejudgingid = :rejudgingid WHERE submitid = :submitid AND rejudgingid IS NULL',
[
'rejudgingid' => $rejudging->getRejudgingid(),
'submitid' => $judging->getSubmissionId(),
]
);

if ($singleJudging) {
$teamid = $judging->getSubmission()->getTeamId();
if ($teamid) {
$this->em->getConnection()->executeStatement(
'UPDATE team SET judging_last_started = null WHERE teamid = :teamid',
[ 'teamid' => $teamid ]
);
}
// $this->>em->wrapInTransaction flushes the entity manager, which is pretty slow.
// So use the direct connection transaction API here.
$this->em->getConnection()->beginTransaction();

$this->em->getConnection()->executeStatement(
'UPDATE submission SET rejudgingid = :rejudgingid WHERE submitid = :submitid AND rejudgingid IS NULL',
[
'rejudgingid' => $rejudging->getRejudgingid(),
'submitid' => $judging->getSubmissionId(),
]
);

if ($singleJudging) {
$teamid = $judging->getSubmission()->getTeamId();
if ($teamid) {
$this->em->getConnection()->executeStatement(
'UPDATE team SET judging_last_started = null WHERE teamid = :teamid',
[ 'teamid' => $teamid ]
);
}
}

// Give back judging, create a new one.
$newJudging = new Judging();
$newJudging
->setContest($judging->getContest())
->setValid(false)
->setSubmission($judging->getSubmission())
->setOriginalJudging($judging)
->setRejudging($rejudging);
$this->em->persist($newJudging);
$this->em->flush();

$this->dj->maybeCreateJudgeTasks($newJudging, $priority);
});
// Give back judging, create a new one.
// Use a direct query to speed things up.
$this->em->getConnection()->executeStatement(
'INSERT INTO judging (cid, valid, submitid, prevjudgingid, rejudgingid, uuid)
VALUES (:cid, 0, :submitid, :prevjudgingid, :rejudgingid, :uuid)',
[
'cid' => $judging->getContest()->getCid(),
'submitid' => $judging->getSubmissionId(),
'prevjudgingid' => $judging->getJudgingId(),
'rejudgingid' => $rejudging->getRejudgingid(),
'uuid' => Uuid::uuid4()->toString(),
]
);
$newJudgingId = $this->em->getConnection()->lastInsertId();
$newJudging = $this->em->getRepository(Judging::class)
->createQueryBuilder('j')
->join('j.submission', 's')
->join('s.contest_problem', 'cp')
->join('s.language', 'l')
->join('l.compile_executable', 'e')
->join('cp.problem', 'p')
->leftJoin('p.compare_executable', 'ce')
->leftJoin('ce.immutableExecutable', 'ice')
->leftJoin('p.run_executable', 're')
->leftJoin('re.immutableExecutable', 'ire')
->select('j', 's', 'cp', 'l', 'e', 'p', 'ce', 'ice', 're', 'ire')
->andWhere('j.judgingid = :judgingid')
->setParameter('judgingid', $newJudgingId)
->getQuery()
->getSingleResult();

$this->dj->maybeCreateJudgeTasks($newJudging, $priority);

$this->em->getConnection()->commit();

if (!$first) {
$log .= ', ';
Expand Down
Loading