Ever wonder how HackerRank can support evaluation of hundreds of thousands of participants? Codechecker is our in-house online judge built to handle infinite scale. This is the first of a series of posts detailing Codechecker, specifically on scale, performance and security aspects.
Among many performance metrics that we track, the top 3 metrics in priority are: total submission count, queue time and total turn around time.
- Total Submission Count – Total number of code submissions processed.
- Queue Time – Time in seconds taken by the codechecker before the codechecker starts processing the code submission.
- Turn Around Time – Total time in seconds taken by the codechecker to process the submission. This includes queue time, code-compile time, code run time and callback time. We focus a lot on the turn around time and put in all our efforts to make sure we show the results of your code as soon as possible.
One of the interesting challenges ways we’re improving the Codechecker is by creating predictive scaling based on the total number of submissions we receive. Anticipating this should help us iron out peaks happening every now and then. These peaks happen when a contest or a test in HackerRank begins with a lot of users, and the submission queue starts filling up well before the scale up can respond.
Interesting challenge to fix!
For the purpose of this post, we’re documenting the performance of August. In the next post, we’ll analyze month over month trends, and how we plan on tackling any issues that arise. Stay tuned.
