Skip to content

Latest commit

 

History

History
112 lines (62 loc) · 14.6 KB

TERMS.md

File metadata and controls

112 lines (62 loc) · 14.6 KB
Raw

Individual Terms and Conditions for the WSDM Cup 2023 Visual Question Answering Challenge (hereinafter referred to as the “Contest”)

1. The Contest is organized by Toloka AI, Inc. (the “Organizer”), a Delaware corporation having its registered address at 10 State Street, Newburyport, MA 01950, United States.

Participants in the Contest (the “Participant”) must be a person over the age of eighteen, who has signed up for the participation as described in clause 6 of these conditions. The name, e-mail address and other required data must be entered accurately. If a participant no longer wishes to take part in the Contest, he or she can inform the Organizer by e-mail to the following address: mailto:[email protected].

Participant represents and warrants that neither it, nor any of their respective Affiliates, directors, employees, or other related parties is not listed on any list of prohibited or restricted parties or controlled by such a party, including but not limited to the lists maintained by the United States, European Union (“EU”), Switzerland, the United Nations Security Council, or any other relevant government bodies which prohibit or may prohibit Participant's participation in the Contest ("Sanctions"). The Organizer reserves the right to disqualify without any justification and to exclude from the Contest with immediate effect, or to forfeit the Prizes of any participant, should there be any Sanctions or update of the Sanctions against Participant, or should Toloka reasonably determine that it cannot perform its obligations under this Contest due to Sanctions-related prohibitions. If Participant has a reasonable basis to believe that any of the foregoing warranties, representations, or covenants may no longer be true or has been breached, Participant shall immediately notify the Organizer in writing. Participant shall hold Toloka harmless against all liabilities.

Employees of the Organizer and / or affiliated companies, other persons involved in organizing the Contest, as well as their family members cannot participate in the Contest.

2. The goal of the Contest is to support the research on crowdsourcing and related areas.

3. The Contest is hosted on the Internet: https://codalab.lisn.upsaclay.fr/competitions/7434

4. Key dates of the Contest (end of day Anywhere on Earth, UTC−12):

  • Practice Starts: September 16, 2022
  • Evaluation Starts: September 30, 2022
  • Evaluation Ends: December 16, 2022
  • Reproduction Starts: December 19, 2022
  • Reproduction Ends: January 16, 2023
  • WSDM Cup Workshops: February 27, 2023

5. Important Legal Terms. By participating in the Contest, the Participant confirms that they have read and understood the terms of the Contest (the “Terms and Conditions”) available at https://github.com/Toloka/WSDMCup2023/blob/main/TERMS.md. Each Participant understands and acknowledges the following:

  • YOUR WORK MUST BE YOUR OWN. The work you perform at the Contest will be your own work.

  • YOU MAY NOT SHARE YOUR WORK WITH OTHERS. You will not share your work with anyone else or perform work for others (except the members of your team you are expressly permitted to collaborate).

  • NO CHEATING. You will not engage in any dishonest or improper conduct to improve your results or affect the results of other Participants.

  • ORGANIZER’S CONTENT. The content provided to you at the Contest is owned by Organizer or its affiliates or licensors and may be subject to copyright, trademark and other intellectual property rights under U.S. and international laws. Except as authorized by the Organizer, you agree not to copy, download, stream, capture, archive, upload, publish, broadcast, sell, resell, modify, translate, decompile, disassemble, reverse engineer or exploit for any purposes the Content or any portion of it.

  • YOUR CONTENT. You represent and warrant that: (a) you own all rights in your content; and (b) your content does not infringe the copyright, trademark, patent or other intellectual property rights, privacy rights, publicity rights or any other legal or moral rights of any third party. When you provide to the Organizer with any work result or content of your own, You grant to the Organizer and its affiliates and their successors and assigns, a non-exclusive, fully-paid, royalty-free, transferable, irrevocable, perpetual, worldwide license, with the right to grant sublicenses through multiple tiers of sublicensees, to display, publicly perform, distribute (including, without limitation, through third-party websites), store, transcode, broadcast, transmit, reproduce, edit, modify, create derivative works and otherwise use and reuse your content (or any portions or derivative works thereof) in any manner, in any medium now known or hereinafter created, for any purpose. You acknowledge and agree that you have no expectation of compensation of any nature with respect to any content. You acknowledge and agree that you have no expectation of confidentiality of any nature with respect to any content. The Organizer and its affiliates reserve the right to display advertisements in connection with, and/or based upon, your content.

  • NO WARRANTIES. THE CONTEST AND ANY ORGANIZER’S CONTENT IS PROVIDED ON AN “AS IS” AND “AS AVAILABLE” BASIS. TO THE FULLEST EXTENT PERMITTED BY LAW, ORGANIZER AND ITS AFFILIATES EXPRESSLY DISCLAIM ALL WARRANTIES OF ANY KIND, WHETHER EXPRESS OR IMPLIED, WITH RESPECT TO THE CONTEST OR ORGANIZER’S CONTENT (INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR USE OR PURPOSE AND NON-INFRINGEMENT).

  • DISCLAIMER OF LIABILITIES. TO THE EXTENT PERMITTED UNDER APPLICABLE LAW, IN NO EVENT WILL THE ORGANIZER OR ITS OFFICERS, EMPLOYEES, DIRECTORS, SHAREHOLDERS, SUBSIDIARIES, AFFILIATES, AGENTS OR LICENSORS BE LIABLE UNDER ANY THEORY OF LIABILITY (WHETHER IN CONTRACT, TORT, STATUTORY OR OTHERWISE) FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR EXEMPLARY DAMAGES, INCLUDING, BUT NOT LIMITED TO, DAMAGES FOR LOSS OF REVENUES, PROFITS, GOODWILL, USE, DATA OR OTHER INTANGIBLE LOSSES ARISING OUT OF THE CONTEST OR ORGANIZER’S CONTENT.

  • WAIVER OF CLASS ACTION. BY ENTERING INTO THESE TERMS, EACH PARTICIPANT IRREVOCABLY WAIVES ANY RIGHT IT MAY HAVE TO JOIN CLAIMS WITH THOSE OF OTHERS IN THE FORM OF A CLASS ACTION OR SIMILAR PROCEDURAL DEVICE. ANY CLAIMS ARISING OUT OF, RELATING TO, OR CONNECTED WITH THESE TERMS MUST BE ASSERTED INDIVIDUALLY.

Participation in this Contest implies the acceptance of these Terms and Conditions. The Organizer reserves the right to modify these without giving reasons. Changes will be communicated on the website and will not be sent to participants by e-mail.

The Organizer and its affiliates undertake to handle the personal data collected within the framework of the Contest with due care.

The Organizer reserves the right to disqualify without any justification and to exclude from the Contest with immediate effect, or to forfeit the Prizes of any participant who manipulates or attempts to manipulate the Contest and/or violates the conditions of participation and/or unfairly attempts to influence the Contest.

The Organizer reserves the right to interrupt or terminate the Contest early without prior notice and without stating reasons, in particular if the normal course of the Contest cannot be guaranteed for technical or legal reasons. In such cases, participants may not assert any claims against the Organizer.

Participation in the contest is free of charge and does not give rise to any obligation to purchase.

These Terms shall be governed exclusively by, and construed in accordance with the laws of the State of Massachusetts, without reference to any conflict of laws principles. Any controversy, claim, or dispute arising out of or related to these Terms (or the interpretation, performance, or breach of them) (a “Dispute”) shall be solely and exclusively resolved according to the procedures set forth in this paragraph. If we are unable to resolve any Dispute through informal means, either party may initiate binding arbitration of such Dispute. The arbitration shall be initiated and conducted according to the JAMS/Endispute Comprehensive Arbitration Rules and Procedures in effect as of the date hereof, including the Optional Appeal Procedure provided for in such rules (the “Arbitration Rules”). The arbitration shall be conducted in the State of Massachusetts before a single neutral arbitrator appointed in accordance with the Arbitration Rules. Arbitration can decide only the individual Dispute and the arbitrator may not consolidate or join the claims of other persons or parties who may be similarly situated.

6. The nature of the Contest and selection of Winners.

6.1. To participate in the Contest, an individual must sign up for the competition on https://codalab.lisn.upsaclay.fr/competitions/7434 and join the Contest. Participants can participate in the Contest individually or in Teams of up to 5 members. To form a Team, Participants need to create a separate account for the Team and participate in the Contest using this account. Each participant can either participate individually or be a member of one Team only. In what follows, we use the term Team to refer to both a group of Participants or a single Participant (Team of 1).

6.2. Throughout the duration of the Contest, Teams will work with data provided by the Organizer. The data consists of three parts: train set, public test set, and private test set. Below we discuss the construction of both sets.

  • Train Set. Each entry to the Train set consists of the textual question, image associated with that question, and the ground truth bounding box. The annotations were obtained via crowdsourcing in two steps. First, crowd annotators were asked to draw a bounding box of an interesting object. Second, crowd annotators were asked to compose a textual question for the drawn bounding box.

  • Public Test Set. The Public Test set is constructed in the same manner as the Train set, but the ground truth bounding boxes are not released to the participants (they are available to the Organizer).

  • Private Test Set. The Private Test set is constructed in the same manner as the Train set, but it is not released to the participants (they are available to the Organizer).

6.3. The goal of each Team is to develop a method to reconstruct the bounding boxes from the provided annotations. By the end of the Contest, Teams need to submit their predictions of the ground truth texts which will be evaluated on the public test set and the implementation of their solution which will be evaluated on the private test set. The submission must be generated by the code that only relies on publicly available data or on data collected on the Toloka crowdsourcing platform (see 7.4 for details).

6.4. To compare submissions made by different teams, we use the intersection over union metric (IoU; see definition on Wikipedia https://en.wikipedia.org/wiki/Jaccard_index and on the contest page). Specifically, for each question in the test set, we compute the IoU metric between the ground truth bounding box and the prediction submitted by the team. We then average these values across all questions in the test set and the resulting quantity becomes the score of the team. The larger the score, the better.

6.5. Throughout the Contest, Teams can submit their preliminary predictions to be evaluated on a public part of the public test set. These preliminary evaluations will be maintained on a public leaderboard visible to all Participants of the Contest.

6.6. The Winning Teams of the Contest are determined by evaluating the quality of final predictions made by each Team against the private test set. Teams that achieve the top 3 best scores on these metrics become the Winners of the Contest. In case of a tie between two or more team, the teams that made their submission earlier will be ranked higher.

7. Prizes

7.1. The Winning Teams receive cash prizes in the following amount:

  • 1st place: $3,000
  • 2nd place: $2,000
  • 3rd place: $1,000

If a Team consists of multiple Participants, the price will be divided equally across all members of the Team.

7.2. There are a total of 3 Prizes to be won in the Contest.

7.3. The Prizes are distributed by the Organizer through Association for Computing Machinery (ACM).

7.4. As a condition to being awarded a Prize, a Prize winner must fulfill the following obligations:

7.4.1. Deliver to the competition organizer the final model's software code as used to generate the winning Submission and associated documentation. The delivered software code must rely on publicly available data or on data collected on the Toloka crowdsourcing platform, be capable of generating the winning Submission, and contain a description of resources required to build and/or run the executable code successfully. Provided software will not be used by the Organizer and is only required to ensure that the rules of the contest were not violated.

7.4.2. If the solution uses data obtained on the Toloka crowdsourcing platform, a Prize won must also share the details of the project setup sufficient to reproduce the project. Specifically, the winner should provide a document describing the task interface design, quality control rules used, payment policy, and collected data. This information will not be used by the Organizer and is only required to ensure that the rules of the contest were not violated.

8. Announcement of the Winners and distribution of the Prize.

8.1. The Organizer announces the Winners by emailing them using the email address associated with their account. The list of Winners is also disclosed on the Contest website at the following link: https://toloka.ai/challenges/wsdm2023/

8.2. The Prizes are distributed in equal installments proportionally to the number of Winning teams’ members. Winning teams will be asked to fill out a short form to specify the credentials of the members.

8.3. The payment of the Prize is in no way guarantees you the right to or expectation that you may receive similar Prize with respect to any other similar performance.

8.4. Winners are responsible to pay all applicable taxes.

9. Miscellaneous. These Terms constitute the entire and exclusive and final statement of the agreement between Participants and Organizer. The failure of the Organizer to exercise or enforce any right or provision of these Terms shall not constitute a waiver of such right or provision. If for any reason any provision of these Terms is found to be unenforceable, that provision shall be enforced to the maximum extent permissible so as to give effect to the original intent of that provision, and the remainder of these Terms shall continue in full force and effect.

You may contact us at:

Toloka AI, Inc.
10 State Street
Newburyport, MA 01950
United States
e-mail: [email protected]