Each participating group should contact the organisers to get the collection used in the task. The required forms must be filled in as per the dataset instructions.
Full details of submission will be provided in June 2024.
LSAT Submissions
A submitted run for the LSAT task is in the form of a single CSV file per run. Please note that each group can submit up to 10 runs, each as an individual file. The submission files should be sent (one email per group) to (cathal.gurrin@dcu.ie) by the due date with the title ’NTCIR-Lifelog LSAT Submission’. The submission file should be named as follows: GroupID-RunID-[Interactive or Automatic].txt, where GroupID is the registration ID of your group at NTCIR, RunID is the number of the run (e.g. DCULSAT01 or DCULSAT02, etc..), and the label Automatic or Interactive.
For every topic, every image considered relevant should have one line in the CSV file. For some topics there will be only one relevant item (one line in the submission), for others there will be many relevant items (many lines in the submission), up to 100. It is also possible that no relevant items are found for a topic, so then there should be no entry in the file for the topic.
The format of the CSV file for an automatic run would be as follows:
GROUP-ID, RUN-ID, TOPIC-ID, IMAGE-ID, SECONDS-ELAPSED, SCORE
...
DCU, DCULSAT01, 16001, u1_2016-08-15_112559, 0, 1.0
DCU, DCULSAT01, 16001, u1_2016-08-15_120354, 0, 1.0
...
\end{verbatim}
In total there will be 24 topics for this lifelog LSAT task. There are two types of topics, adhoc and knownitem. All topics are re-used from the LSC'22,23 competitions.
- ADHOC - topics that may have many moments in the collection that are relevant. These topics are all new.
- KNOWNITEM - topics with one (or few) relevant moments in the collection. .
The trec-eval programme will be employed to generate result scores for each run. Relevance judgements will be generated using a pooled approach whereby human judges will be employed to manually evaluate each submitted image for each topic, up to a maximum of 100 images per topic, per run, per participant. The relevance judgements will be binary and will be informed by the immediate context of each image if important.
Details of LIT and LQAT will be updated in June.
Following submission, each participating team must prepare a paper describing their experimental approach and scores. The organisers will prepare their own Overview Paper which should be referenced by all participants.