NTCIR14-Lifelog Task

• HOME
• TASKS
• DATA
• SUBMISSIONS
• CFP
• ORGANISERS

LSAT - Lifelog Semantic Access (sub) Task

A submitted run for the LSAT task is in the form of a single CSV file per run. The submission file should be named as follows: groupID_LSAT_runID.txt, where groupID is the registration ID of your group at NTCIR and runID is an identifier of the submitted run (e.g. DCULSAT01 or DCULSAT02) which identifies the run that you are submitting. Please note that the label given to any segment is not evaluated as part of the official evaluation. The official evaluation will evaluate the number of events detected in a given day (compared to the ground truth) as well as the accuracy of the event-detection process (given a sliding five minute window).

The format of the CSV file is as follows:

GROUP_ID, RUN_ID, TOPIC_ID, IMAGE_ID, SECONDS_ELAPSED, BELIEF_SCORE
DCU, DCULSAT01, LSAT003, u1_2016-08-15_112559, 0, 1.0
DCU, DCULEST01, LSAT004, u1_2016-08-15_120354, 0, 1.0

For every topic, every moment considered relevant should have one line in this CSV file. For some topics there will be only one relevant moment (one line in the file), for others there will be many relevant moments (many lines in the file), up to 100. It is also possible that no relevant moments are found for a topic, so then there should be no entry in the file for the topic.

A moment is identified in the submission file as an image ID that answers the topic. If there are more than one sequential images that answer the topic (i.e. the moment is more than one image in duration), then any image from within that moment is acceptable. GROUP_ID IMAGE_ID is the filename of the image without the .JPG extension. TOPIC_ID is the identifier of the TOPIC for which the result is relevant. SECONDS_ELAPSED and BELIEF_SCORE are described below.

There are two types of run that can be submitted, automatic and interactive:

• Automatic runs assume that there was no user involvement in the search process beyond specifying the query. The search system generates a ranked list of up to 100 moments for each topic. There is no time limit on how long it can take for a manual run. We assume that any human involvement in generating a query from the topic is a once-off process that is not iterative and dependent on the results of a previous execution of a query for that topic. The submission file format includes BELIEF_SCORE to capture the weighting of every moment and the position in the ranked list (in decreasing order). The score values will not be considered in the scoring process. In the automatic run case the SECONDS_ELAPSED column should always have a value equal to 0, since it is only relevant for Interactive runs.

• Interactive runs assume that there is a user involved in the search process that generates a query and selects which moments are considered correct for each topic. In interactive runs, the maximum time allowed for any topic is 5 minutes, after which point, no more moments should be appended to the CSV file for that run. The submission file format includes SECONDS_ELAPSED to capture the time taken to find every moment. In the interactive runs, up to 100 moments may be submitted for any query also, though we expect the number to be significantly lower. In the interactive run, the BELIEF_SCORE value should be equal to 1. For interactive runs, the seconds elapsed should be equal to the number of seconds (from zero) that it took the user to find that particular result. For example, if a user in an interactive run found 3 items that are believed to be relevant, then there should be three different SECONDS_ELAPSED values, one for each entry in the CSV file for that topic. We will use the time elapsed to calculate run performance at different time cutoffs.

It is not necessary to submit your insights task output by the due date. We do, however, request that you send us a short abstract of your LIT subtask visualisations and insights to (ntcir-lifelog@computing.dcu.ie) by the due date. This abstract should be about one paragraph (10 lines) in length and may include the visualisation if ready. It is expected that the submission will be described in the participating group’s NTCIR-14 paper and will be demonstrated at the NTCIR-14 conference.