BOCA Manual for the judges

http://www.ime.usp.br/~cassio/boca/boca/doc/html/manualjudge.en.html

BOCA Online Contest Administrator

BOCA Reference Manual for judges

- version October/2011 (BOCA 1.4.1)

Copyright (c) 2006-2011 Cassio P. de Campos (cassio@ime.usp.br). Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license may be found in http://www.gnu.org /licenses/

BOCA is a software created to control a contest with the ACM International Collegiate Programming Contest rules. It has been developed in PHP and the interaction between judges and the system is done through a web browser. In the following we describe the set of available features for a judge in the system. It is assumed here that the judge has already logged in the system with their username and password in a windows of their browser. Tthe URL to access BOCA will depend on the setup of your server. Usually this URL ends with /boca/, where a simple authentication form will show up. After a successful login, the web page contains the judge identification in the upper-left corner. In the upper-right corner, the clock of the contest is shown, indicating if it has started or not, if it has already ended, stopped, or in progress, in which case the number of minutes to go are displayed. Below that, there is a set of options in a vertical menu, namely Runs, Score, Clarifications, History, As Team, Options e Logout. There is still an extra option named Chief which only appear to the chief judge, who has to be appointed in the admin's interface (with in the Site options).

Runs
In this area, the judge might visualize the runs that are still to be judged and shall be taken care of. In order to process a run, the judge has to click in its number (eventually a message might pop up here indicating the run was already taken; this happens if another judge got the run before you). After clicking on the run to judge, a new set of data appear about it: 1. 2. 3. 4. Site: site number of the run. Number: run number. Time: minutes from the start of the competition until this submission. Problem X: X is the name of problem. There are links to download the input and output files (click on the names), or simply to visualize them in the browser (click on view). These links are mostly used when the autojudge is turned off or is not running properly for a given problem/language. Instead, if the autojudge feature is working correctly, then the judge can directly to go the links in the bottom part of the page in order to see the expected output together with the output generated by the team. Language Y: Y is the language chosen by the team. The scripts that are used to compile and execute the submission with this language are available through links. When using the autojudge, note that is not necessary to use this links, because the results are already shown in the bottom of the page. Source code: here it is possible to view or download the submitted file. An important task is to check if the name and extension of this file are correct with respect to the chosen problem, language, and the specification in the booklet of problem descriptions. Answer: the judge has to choose, among the options available in the system (which were configured by the admin), the correct answer to be sent to the team. Autojudging answer: here it is presented the suggestion from the autojudge about this submission. Usually the autojudge properly identify the answer that has to be sent to the team. However, it may fail to do so. For example, the autojudge might indicate a wrong answer, because the diff procedure identified the output and expected output to be different, while the only mistake happened in the punctuation/accent of a letter (this is usually a case of presentation error, but may
15/04/2012 14:50

5.

6.

7. 8.

1 de 4

BOCA Manual for the judges

http://www.ime.usp.br/~cassio/boca/boca/doc/html/manualjudge.en.html

vary according to the contest). Other issues might happen, and the role of the judge is to check everything for eventual mistakes. 9. Autojudged by: indicates the computer that has acts as autojudge. Usually this is not much relevant for judging a submission. 10. Standard output: links for downloading and visualizing the output generated by the team's code are available. This is the output generated by the team, which is to be compared with the expected output of the given problem. It might also be possible to see some error, in case it happened. 11. Standard error: links for downloading and visualizing the standard error output generated by the team's code. When a error happens (including compilation or runtime errors, but not restricted to them), this is the most probable file to identify it. The judge must always check the content of this file. In the end of it, the judge also finds the output of the diff command that has compared the team's output and the expected one (to facilitate the visual inspection). By pressing the button named Judge, the judge submits their veredict about the run, and then has no access to the this run anymore. It is also possible to give up judging the run by clicking on Cancel, which will send the run back to the pool (and it will be eventually judged by another judge). In the tab Runs, the judge can see all the submissions that are yet to be judged. The colors represent their status, meaning that they are already been judged by others, or that they are waiting you to judge them (red color). You must act with respect to those in red, as only you can judge them (or send them back to the pool).

Chief
Besides acting as a normal judge, there is a designated judge with access to the tab Chief. This tab is used for the chief judge to resolve disputes regarding submissions that received different answers by different judges (they are shown in red). The prodecure is equivalent to that of a normal judge, but the decision made in the tab of the chief is final. The idea is to have the chief judge acting as a tie-breaker between the distinct answers that were assigned to a run. Hence, if the chief judge is acting as a normal judge (and they may do so), it is recommended to use the tab Runs instead, leaving the tab Chief only for resolving issues. (In the very special case where there is a single judge in the competition, then this judge has to be designated as chief and shall use the tab Chief to judge the runs, otherwise they will never receive the judgement of another judge and thus will not be sent to teams.) In this same tab, the chief judge is able to ask the autojudge to be re-executed for some submissions (selected by the boxes besides each of them) or to completely re-open the submission for judging again, which implies in a new round of autojudging and judging by actual judges. The team does not become aware of this rejudging unless the new final result for the run has to be changed (with respect to the first round of judgement).

Score
In this tab the judge can see the scoreboard of the competition. It has to be noted that the scoreboard available for the judges is complete and not subject to the freezing of the final part of the contest. Hence, the judge is expected to keep the scoreboard in secret. In case the scoreboard is consolidated among different sites, the final part of the other sites is not shown (*this fact is for technical reasons, to be discussed later*).

Clarifications
This tab allows the judge to answer clarifications submitted by team regarding a specific problem or any general aspect of the contest. In order to reply to a clarification, the judge must first click on the clarification number. The box available in the bottom part is meant for creating a new clarification, as judges are allowed to do so. The new clarification will only be meaningful if the judge that gets it to

2 de 4

15/04/2012 14:50

BOCA Manual for the judges

http://www.ime.usp.br/~cassio/boca/boca/doc/html/manualjudge.en.html

answer selected the reply to all option (explained in the next paragraph). This is useful for the judges to send a general information to the teams. After clicking on the clarification number, the judge sees information about it, such as site number, clarification number, time in minutes from the start of the competition, and the problem to which the clarification regards. There are two text boxes: one with the question (on top) and one to be filled with the reply. Finally, there is a selection box to indicate if the reply should be sent to all the teams or just the the team that posted the question. The system also allows the judge to select the No response button. In general this is used for questions that are already stated in the booklet of problem descriptions, or that has already been replied, or even that should not be answered at all (for example, about the timelimit of a problem, or about some sensitive information of the input/output).

History
This tab shows the history of Clarifications and Runs that were processed by the judge so far.

Options
This tab shows the information of the judge, such as Username, User full name , User description. It is possible for the judge to update their password, although this is not necessary neither recommended. Instead it is recommended that the admin who created the users already specifies secret and safe password for all users.

Logout
Button to log out from the judge interface.

Important hints for judges

While judging a submission, it is necessary to be very careful. Even if it is possible to alter the judgement of a run afterwards, this has to be avoided as much as possible, because the team might suffer an undesired situation. Every time a judge is about to answer a problem for its first time (that is, the problem has not been submitted before, or has not received yet any YESes), when possible it is interesting to have other judges also checking the submission, which should be analyzed with even greater care. Sometimes this is the moment when an unfortunate issue with the autojudge or with inputs or outputs is discovered (obviously nobody wants it to happen, but it might). In order to reply to clarifications, it is important that the judge has read and understood the question and the problem to which it concerns. When possible, it is interesting to have the opinion of the person or group that created the problem description, its input and output. After that, it is necessary to think if such reply has to be sent only to this user or to all teams in the contest. If it is a relevant issue and the contest is also offered in other sites, then the judge shall find a way to contact other sites about the clarification. Some examples of clarifications that are not disclosured (that is, are replied with a no response) are What is the timelimit for this problem?, Given this input, which is the correct output?, questions that are already explained in the problem description, etc. In case the judge is not certain about it, they must contact the chief judge of the site or the chief judge of the contest for help.

3 de 4

15/04/2012 14:50

BOCA Manual for the judges

http://www.ime.usp.br/~cassio/boca/boca/doc/html/manualjudge.en.html

About BOCA and this document

BOCA System and this document have been created by Cassio Polpo de Campos and can be found at http://www.ime.usp.br/~cassio/boca/.

4 de 4

15/04/2012 14:50