IOI 2020 Minimal E-Proctoring Rules
There is a different set of additional minimal e-proctoring rules that have been circulated to team leaders since 29 June 2020 (v1.0), slightly updated on 18 July 2020 (v1.1) to include the compulsory usage of VM, and with the final update on 08 September 2020 (v1.2) where the video recording requirement is now made optional (plus the official confirmation that we will use VM at maximized/full screen at all times, which indirectly disallow printing outside of VM).
IOI 2020 Rules presented to the GA during IOI 2019
These Competition Rules cover Competition Procedures and Judging Procedures. This draft may be revised to address omissions or inconsistencies, but will not change substantially.
The final version will be presented at the first GA meeting of IOI 2020. Important changes from IOI 2019 are marked in red.
Delegation Leaders have the responsibility to ensure that all members of their delegation fully understand these rules and abide by them.
An updated version of the IOI syllabus can be found here. Current approved IOI Regulations can be found at https://ioinformatics.org/files/regulations19.pdf.
For IOI Code of Conduct, follow this link.
TABLE OF CONTENTS
TASKS
The source program provided by the contestant must be contained in one source file as specified in the task statement.
Submissions must not perform explicit input and output operations; instead data must only be exchanged through the interfaces specified in the task statement. In particular, direct access to any file, including standard input or standard output, is forbidden (though writing to standard error is allowed).
As Java Virtual Machine uses multiple threads internally, using multiple threads is allowed in all programming languages. Note that the running time of the submission will be counted as a sum of running times of all threads. E.g. if there were two threads running for 5 seconds each (thus, the program finishes in 5 seconds), then the running time of the submission will be 10 seconds.
Each task will be divided into several subtasks, each worth a portion of the total points.
Time and memory limits will be specified for every task. In general, time and memory limits will be generous (for example, double those required by the expected solution). The memory limit is on the overall memory usage including executable code size, stack, heap, etc.
For each programming task, the contestants can download a zip file for a programming task from the grading system. The zip file contains interface files, a sample grading program, and a skeleton implementation of a required source file. The skeleton exercises an interface, but it does not solve the problem. The sample grader provided on the workstation would not be the same as the official grader used by the grading system.
PRACTICE SESSION
There will be a 2 hour Practice Competition prior to the first competition day, to familiarize all contestants with the grading system. The practice tasks will be published before the IOI. Contestants may bring printed solutions to the practice tasks, on paper only, during the Practice Competition.
QUARANTINE
In order to protect the confidentiality of the tasks, all direct and indirect contacts and communication between contestants and team leaders are prohibited between the moment where tasks for a competition day are presented to the members of the GA and the start of the competition. During this period the contestants are not allowed to communicate by any means, direct or indirect, with any member of the GA or anyone who knows the tasks. The GA members are not allowed to communicate task-related information to anyone who may not attend GA meetings.
If a contestant violates the quarantine, the contestant can be subject to disqualification. If some other person associated with a national delegation violates the quarantine, then all contestants of that delegation may be subject to disqualification.
COMPETITION EQUIPMENT AND ENVIRONMENT
Please refer to the Contest PC page.
SUPPLIES
In the competition room, blank paper, writing tools, snacks and water should be provided by the proctor. On the competition days, contestants may not bring anything into the competition rooms, except for the following items under the proviso that they cannot transmit or store any data in electronic or printed format (other than the purpose for which they have been designed):
- clothing,
- reasonable jewelry,
- writing utensils,
- keyboards (without wireless, calculation, and/or programmable functions),
- mouse (without wireless and/or calculation functions) and mouse pads,
- small mascots,
- English dictionaries,
- snacks
- medicine and medical equipment
- earplugs and earmuffs
STARTING THE COMPETITION
Please refer to the Contest PC page.
ASSISTANCE AND CLARIFICATION
During the competition, contestants may ask questions concerning competition tasks, rules and/or grading. Such questions would be considered Clarification Requests and should be submitted using the grading system either in English or in the contestant’s preferred language.
In the event that contestants are not able to type the characters required for their preferred language for the Clarification Request in the grading system, contestants should write their clarification on paper and inform their proctor. The proctor should take a photo of the Clarification Request and send it to the Scientific Committee using the communication channel described in the Communications section of the IOI 2020 Contestant PC Manual.
If required, delegation leaders will translate the Clarification Requests into English after they are submitted and before they are sent to the Scientific Committee. Since this might take some time, contestants should continue working while waiting for the answers to their questions. The Scientific Committee will reply to the Clarification Request either through the grading system, or through the communication channel to the proctors who should forward the reply to the contestant.
Contestants should phrase their Clarification Requests so that a yes/no answer will have a clear meaning. Contestants should not ask negative questions such as “Isn’t it true that…?” because the yes/no answer to such questions may cause confusion depending on the native language of the contestants. Instead, positive questions of the form “Is it true that…?” are recommended.
Clarification Requests will be answered with one of the following:
- “YES”
- “NO”
- “ANSWERED IN TASK DESCRIPTION (EXPLICITLY OR IMPLICITLY)” – The task description contains sufficient information. The contestant should read it again carefully.
- “INVALID QUESTION” – The question is most likely not phrased so that a yes/no answer would be meaningful. The contestant is encouraged to rephrase the question.
- “NO COMMENT” – The contestant is asking for information that the Scientific Committee cannot give.
Additional elaboration of the answer may be provided if the Scientific Committee deems it to be necessary.
Questions not concerning competition tasks, rules and/or grading would be considered as Assistance Requests. These Assistance Requests must be made via their proctor. Contestants should not leave their seats until allowed to do so. Contestants should never attempt to fix or debug or even check computer or network problems themselves; instead, they should ask for assistance.
GRADING SYSTEM
There is no restriction on the number of times a program may be edited, compiled, and run on the workstation. The workstations have network access to the grading system, as well as facilities such as printing and solution submission. Grading and evaluation take place on the grading system, which provides a similar execution environment for every solution submission. There is no guarantee that contestant workstations and grading workstations will have a similar execution environment.
SUBMITTING
Contestants must submit their solutions for tasks by using the grading system. To avoid overloading the grading system, there are two restrictions on the number of submissions:
- Contestants may submit a solution to each task at most once per minute. This restriction does not apply in the last 15 minutes of the contest round.
- Contestants may submit at most 50 solutions for each task. This limit may be increased for some task types.
- Each submitted source program must be written in C++ or Java, it must be smaller than 256 KiB, the evaluation server must be able to compile it in less than 10 seconds and at most 512 MiB of memory.
For exceptional circumstances, the Scientific Committee and Technical Committee may provide alternative methods for contestants to submit their solutions instead of the grading system.
SCORING
The scores will be calculated as follows:
- For each submission, the score for each test case is calculated according to your program or output.
- For each submission, the score for each subtask is the minimum of the scores for the test cases in the subtask unless otherwise stated in the task statement.
- The final score for each subtask is the maximum of the scores for this subtask across all submissions.
- The final score for each task is the sum of the scores for its subtasks. This sum is rounded to the nearest 2 decimal places, with ties rounded such that the second decimal place is even. For example, a score of 12.335 and 12.345 will both be rounded to 12.34.
For example, consider a contestant who made two submissions on a task that contains two subtasks. If the first submitted solution got 30 points for the first subtask and 10 points for the second subtask, and the second solution got 0 points for the first subtask and 40 points for the second subtask, then the final score for this task will be 70.
FEEDBACK
Contestants can use the grading system to view the status of their submissions and get a short report on the compilation errors of their source code.
For every submission, the grading system reports the score for each subtask. If a subtask is not fully solved, the grading system gives a feedback only for the first test case among the lowest scored test cases in the subtask. The feedback includes the test case number and one of the following reasons:
- Output is correct
- Output isn’t correct
- Execution timed out
- Execution killed (could be triggered by violating memory limits)
- Execution failed because the return code was nonzero
For tasks with partial scores, instead of “Output is correct” or “Output isn’t correct”, the feedback gives “Accepted” or “Wrong Answer”. “Accepted” means that the submission produced a correct answer. Still, it might not get the full score, due to scoring rules explained in the task statement. “Wrong Answer” means the submission has violated some constraints or produced an incorrect answer. If necessary, the precise meanings of “Accepted” and “Wrong Answer” will be clarified in the task statement.
The test cases are ordered the same way in all the submissions. No information on the actual data, the output produced by the contestant solution or any other execution details will be given to the contestant.
It should be noted that the score reported in the feedback is only provisional. There are two ways how this score may change after it has been reported to the contestant:
- Due to a successful appeal after the contest.
- In some cases the contestants’ submissions may be re-evaluated. This re-evaluation may sometimes lead to a different total score (e.g., if a solution behaves nondeterministically or runs very close to the time or memory limit). In such cases, the final score for the submission is the score for its latest re-evaluation. This change in scoring cannot be appealed. Note that the final score for each subtask is still the maximum score over all submissions.
ENDING THE COMPETITION
Three warnings will be given at 15 minutes, 5 minutes, and 1 minute before the end of the competition, communicated via the proctors. The end of the competition will be announced via the proctors through the communication channel. At the announcement ending the competition, contestants must immediately stop working and wait quietly at their desks without touching the computers or anything on their desks. An additional announcement will be made instructing them to leave their tables.
If a contestant thinks that they should be given extra time for the competition, they should send a Clarification Request either through the grading system or via their proctor. They should not leave their desk or talk to anyone other than their proctor, after the end of the competition. A member of the Scientific Committee will then decide whether to award the extra time, and inform the contestant of the decision.
However, the following issues will not be accepted as grounds for requesting for extra time
- Issues arising from the usage of IDEs and debuggers (such as Code::Blocks and Eclipse). Contestants are to take responsibility for their choice of IDEs and debuggers and be competent in their usage. However, members from the technical committee may provide assistance.
- The contestant loses a significant amount of time when trying to solve technical issues by themselves.
- Instances where the contestant did not inform any support staff or made any clarification request on the issue at all.
CHEATING
Contestants must use only the workstation and account assigned to them on each competition day. In particular:
- contestants must not attempt to submit illegal programs as discussed above, nor try to tamper with or compromise the grading system;
- contestants must not attempt to gain access to root or any account other than the one assigned to them;
- contestants must not attempt to store information in any part of the file system other than the home directory for their account or the /tmp directory;
- contestants must not touch any workstation other than the one assigned to them;
- contestants must not attempt to access any machine on the network or the Internet, other than to access the contest system for usual purposes (e.g. submitting tasks, viewing submission results, downloading sample data, submitting Clarification Requests), call for the support staff through the system, and print documents; even running a single “ping” command is strictly prohibited and may lead to disqualification;
- contestants must not attempt to reboot or alter the boot sequence of any workstation;
- contestants must not communicate with other people during the competition, other than their proctor, and/or Scientific/Technical Committee members;
- contestants must not reverse engineer the test data in order to solve the problems in highly test-data-dependent manners. One example of such behavior is using the feedback system to extract the test data and then applying this knowledge to build solutions adapted to the specific test cases in the grading system. This behavior would be considered cheating only if a contestant submits a solution that would solve significantly fewer test cases correctly if the test data were replaced by an equivalent set of test cases (e.g., one generated with a different random seed).
All of the above actions are considered cheating, and may result in disqualification.
APPEAL PROCESS
Submitted solutions are evaluated using data which conform to the specification given in the problem statement, but which are hidden from contestants during the competition.
Provisional grades, based on these tests, are available immediately to contestants. In the event of an error with the test data, the Scientific Committee will attempt to, but is not obligated to follow the following process:
- Every attempt will be made to fix test data and regrade all solutions as quickly as possible.
- Additional test data may be added only when the grading data does not meet the intention of the Scientific Committee from before the contest.
- Late detections of issues, especially during the last 2 hours of the contest, may be grounds for extending the length of the contest.
This hidden data will be made available online during the scheduled time for analysis after each competition. Contestants and team leaders may use the contestant’s workstations to verify that the grades are assessed correctly.
A Team Leader may file an appeal by completing an online Appeal Form, and submitting it to the Scientific Committee at most 24 hours after the end of each competition day. The GA will be informed of the link to the Appeal Form after the end of each competition day. Every appeal will be reviewed by the Scientific Committee and the team leader will be notified of the committee’s decision. All appeals and their disposition will be summarized 24 hours after the end of each competition day. In the event that every submission of a task should be re-graded and re-scored as a consequence of an accepted appeal, note that re-scoring may result in a higher or lower score for any contestant. Should anyone’s score change after grading results have been published, new results will be published again. Score changes resulting from this are not appealable.
