Configuration of the Challenge
For the LSC'21 workshop, participating systems will connect to the LSC host server (DRES), which has been developed by Luca Rossetto and team at UZH. There will be a pre-LSC'21 test session in late 2021 for all participating teams to ensure that their systems can interact with the servers. The readme file from the GitHub page for the server gives details of the submission settings for the systems. Specifically the submission format for the LSC'21 competition is http(s)://{server}/submit?item={item} where {item} is the identifier for the retrieved media item.
During the search challenge, participating teams submit an item (image) to a host server for evaluation when a potentially relevant item from the collection is found by a participant. The host server maintains a countdown clock and evaluates in real-time all submissions against a provided groundtruth. For each topic, a score is given based on the time taken to find the relevant content and the number of incorrect items previously submitted by that team to the host server for that topic. There is a maximum of 3 or 5 minutes provided per topic (see below). Throughout the competition, an overall score is maintained for each team, which is the summation of the scores of the topics that had been processed up until that point. The teams will be ranked in terms of overall (expert & novice combined score) and novice score. For more detail on the scoring function and the LSC service, please see the LSC'18 overview paper.
There will be 24 official English language topics prepared which will be textual in nature and, as in previous LSC competitions, may be temporally enhancing queries. A total of either 3 or 5 minutes will be allocated per topic. The topics will be revealed only at the challenge. The LSC'19 topics are available for system testing. During the workshop the topics are first seen on the central screen.
As with the LSC'20 workshop, the LSC'21 workshop is a fully virtual event.
Challenge Rules
Participating teams can develop any type of interactive retrieval system that they wish without restriction. Query input to each system is not restricted (save for the rules below). This means that some systems will accept text queries, others will facilitate faceted queries, others may support spoken input, others relevance feedback. There is no limit on how interaction can be facilitated once it adheres to the rules below.
Each retrieval system is expected to have pre-indexed the LSC'21 dataset prior to arriving at the workshop. All aspects of the dataset can be indexed and the integration of additional semantic and concept detectors is allowed, once details of these are provided The revised rules of participation are intended to facilitate a fair competition and are listed as:
- Every team is allowed to run only one instance of their system (i.e., there is one interface, or one VR glasses) - so that we will have a fair competition.
- However, you may employ as many users as you want for this interface (it should also be visible on the external camera, how many users interact with the system instance).
- No communication is allowed between teams.
- Taking photos of any image (other team's submissions) from the shared screens as query input is prohibited
- Each incorrect submission will be penalised according to the scoring algorithm, so it is recommended not to submit any result to the central server unless the user is confident that the result is likely to be correct.
- The relevance judgements provided with the topic are final and cannot be questioned or queried. Every effort will be made to ensure that relevant judgements are not ambiguous.
- It is prohibited for any system developer (who acts as the expert user for their system) to learn or familiarise themselves sufficiently with the dataset, so as to gain an unfair advantage over other competitors.
- Interaction mechanisms employed should not interfere with, or disturb, other participants. For any participant bringing novel VR-type systems, a dedicated flow space will be prepared for them which is sufficiently distant from the other participants.