Commons:Jury tools/WLM jury tool requirements


Over the past 6 years, a variety of jury tools have been developed for WLM. While each of these tools have responded to the needs of a subset of the participating countries during their jury process, none are at a stage that can be used by the majority of country's jury and the local coordinators without developer support. The WLM international team understands that a tool developed and maintained by one person at volunteer time cannot respond to the needs of dozens of countries that need to use the tool to meet a deadline. It is our goal to have a broader support system for at least one jury tool during the WLM contest.

To this end, the WLM international team has decided to focus on fully supporting or developing one tool for the 2016 jury process. In order to do this, we have identified 3 major areas of requirements that any tool we support or develop should meet in the coming weeks. We acknowledge that none of the current tools will fully meet all these requirements, we also know that these requirements have different levels of importance for us. We want to emphasize that a tool not meeting these requirements at the moment does not automatically mean it won’t be considered to be officially supported. The plan is to review all the current jury tools in the next few days to assess them based on how much they meet each of these requirements. The team will make a final decision about which tool to support or whether a new tool should be developed based on that assessment.

Below are the list of requirements for the WLM jury tool.

Technical edit

  1. Operational stability: does not crash or go down often.
    1. Uses well known setups for data storage: on tools, this is basically just MySQL.
    2. Follows best practices when interacting with databases: For example, not reading all of a potentially large table into memory (paginate instead), using indexes properly, using transactions properly, etc.
    3. Has a well defined health check endpoint, which should return true if the application is working and false if it isn't and needs a restart.
  2. Maintainability: others can contribute and/or move the tool forward in the future. It is a high priority for us to build a healthy community around the tool we support.
    1. Has a F/OSS license and is publicly hosted in a version control system somewhere (GitHub / Phabricator / Gerrit / BitBucket, etc.)
    2. Has a healthy developing community around the tool
    3. Uses / used some form of code review for merging changes
    4. Has documentation on:
      1. Overall architecture of the setup + rationale
      2. Data model used:database tables and how they are designed, explanation of how this will scale with our scaling requirements (250k images over ~50 contests)
      3. How to set it up locally to start hacking on it
    5. Subjective code quality evaluation by someone trusted

User experience for the jury edit

  1. ease of use: for example, display photos that need ranking, display images' meta data, control image size, low-bandwidth friendly, progress status
  2. control for bias: for example, randomize image orders
  3. voting system flexibility: for example, allow for different voting mechanisms (i.e.: binary, stars, ranking), number of jurors required - per round and/or per country setup
  4. reporting options and exporting results: for example, make reports available to jury/coordinator at the end of each round

User experience for the administrators/coordinators edit

  1. Easy setup for a new country: for example, account creation for jury, setting up the jury process, threshold setting feature by the admin/coordinator
  2. Functional admin interface: for example, export/import options from one round to the next, ability to discard pictures, progress status
  3. Auditability/transparency: export the detailed results, showing detailed calculations.