PHASE 1 TASKS
(currently closed - contact the organisers if you wish to submit LAT runs and we will open submission)
LAT - Lifelog Annotation (sub) Task
The submission for the LAT task is in the form of a single CSV file per run. We can accept a maximum of ten runs per participant. We will release the relevance judgements on the NTCIR website after results are released. The format of the submission file is one line per result in the format as follows:
GROUP_ID, RUN_ID, TOPIC_ID, IMAGE_ID, SCORE (in decreasing order), CONFIDENCE_LEVEL (if no confidence value, include 0)
DCU, DCULAT01, LAT001, u1_2016-08-15_112559_2, 1.0, 0.546352
DCU, DCULAT01, LAT001, u1_2016-08-15_113042_1, 0.98, 0.532413
The submission files should be sent (one email per group) to (ntcir-lifelog@computing.dcu.ie) by the due date with the title ’NTCIR-Lifelog LAT Submission’.
PHASE 2 TASKS
(currently active)
LSAT - Lifelog Semantic Access (sub) Task
A submitted run for the LSAT task is in the form of a single CSV file per run. The submission file should be named as follows: groupID_LSAT_runID.txt, where groupID is the registration ID of your group at NTCIR and runID is an identifier of the submitted run (e.g. DCULSAT01 or DCULSAT02) which identifies the run that you are submitting. Please note that the label given to any segment is not evaluated as part of the official evaluation. The official evaluation will evaluate the number of events detected in a given day (compared to the ground truth) as well as the accuracy of the event-detection process (given a sliding five minute window).
The format of the CSV file is as follows:
GROUP_ID, RUN_ID, TOPIC_ID, IMAGE_ID, SECONDS_ELAPSED, BELIEF_SCORE
DCU, DCULSAT01, LSAT003, u1_2016-08-15_112559, 0, 1.0
DCU, DCULEST01, LSAT004, u1_2016-08-15_120354, 0, 1.0
For every topic, every moment considered relevant should have one line in this CSV file. For some topics there will be only one relevant moment (one line in the file), for others there will be many relevant moments (many lines in the file), up to 100. It is also possible that no relevant moments are found for a topic, so then there should be no entry in the file for the topic.
A moment is identified in the submission file as an image ID that answers the topic. If there are more than one sequential images that answer the topic (i.e. the moment is more than one image in duration), then any image from within that moment is acceptable. GROUP_ID IMAGE_ID is the filename of the image without the .JPG extension. TOPIC_ID is the identifier of the TOPIC for which the result is relevant. SECONDS_ELAPSED and BELIEF_SCORE are described below.
There are two types of run that can be submitted, automatic and interactive:
Automatic runs assume that there was no user involvement in the search process beyond specifying the query. The search system generates a ranked list of up to 100 moments for each topic. There is no time limit on how long it can take for a manual run. We assume that any human involvement in generating a query from the topic is a once-off process that is not iterative and dependent on the results of a previous execution of a query for that topic. The submission file format includes BELIEF_SCORE to capture the weighting of every moment and the position in the ranked list (in decreasing order). The score values will not be considered in the scoring process. In the automatic run case the SECONDS_ELAPSED column should always have a value equal to 0, since it is only relevant for Interactive runs.
Interactive runs assume that there is a user involved in the search process that generates a query and selects which moments are considered correct for each topic. In interactive runs, the maximum time allowed for any topic is 5 minutes, after which point, no more moments should be appended to the CSV file for that run. The submission file format includes SECONDS_ELAPSED to capture the time taken to find every moment. In the interactive runs, up to 100 moments may be submitted for any query also, though we expect the number to be significantly lower. In the interactive run, the BELIEF_SCORE value should be equal to 1. For interactive runs, the seconds elapsed should be equal to the number of seconds (from zero) that it took the user to find that particular result. For example, if a user in an interactive run found 3 items that are believed to be relevant, then there should be three different SECONDS_ELAPSED values, one for each entry in the CSV file for that topic. We will use the time elapsed to calculate run performance at different time cutoffs.
The submission files should be sent (one email per group) to (ntcir-lifelog@computing.dcu.ie) by the due date with the title ’NTCIR-Lifelog LSAT Submission’. The submission file should be named as follows: groupID_LSAT.txt, where groupID is the registration ID of your group at NTCIR.
LEST - Event Segmentation (sub) Task
A submitted run for the LEST task is in the form of a single CSV file per run. Only automatic runs are encouraged for the LEST task. The submission file should be named as follows: groupID_LEST_runID.txt, where groupID is the registration ID of your group at NTCIR and runID is an identifier of the submitted run (e.g. DCULEST01 or DCULEST02) which identifies the run that you are submitting. A maximum of ten officials runs will be allowed per participant. Submissions are requested that identify each segment begin and end time (to the nearest minute). A sliding window of three minutes will be used to provide a fuzzy evaluation. Any image within these three minutes will be considered correct as an identifier of the boundary between segments. The format of the submission should be as follows:
GROUP_ID, RUN_ID, DAY_ID, START_IMAGE_ID, END_IMAGE_ID, LABEL (not evaluated)
DCU, DCULEST01, 2016-08-15, u1_2016-08-15_112559_2, u1_2016-08-15_120324, travel
DCU, DCULEST01, 2016-08-15, u1_2016-08-15_120354, u1_2016-08-15_122501, computer
GROUP_ID is the registration ID of your group at NTCIR. RUN_ID is an identifier of the submitted run (e.g. DCULEST01 or DCULEST02) where there are a maximum of ten official runs allowed per participating group. DAY_ID refers to the date (year-month-day) that is being segmented. START_IMAGE_ID is the filename of the image (without the .JPG extension) that you consider the starting point of the event. END_IMAGE_ID represents the end of the event. LABEL is a tag that we use to identify the type of event (see the tasks page for a list of the 15 tags). While we don’t require you to submit the tag in the official submission, we will evaluate the tags (true or false) for any submissions that include the tag id. Where there is no tag id submitted, please label the event as ‘untagged’.
The submission files (CSV text files) should be sent (one email per group) to (ntcir-lifelog@computing.dcu.ie) by the due date with the title ’NTCIR-Lifelog LEST Submission’. Participants are allowed to submit up to ten runs per group where each run is submitted as a single file. The submission file should be named as follows: groupID_LEST_runID.txt, where groupID is the registration ID of your group at NTCIR and runID is an identifier of the submitted run (e.g. DCULEST01 or DCULEST02) which identifies the run that you are submitting. Please note that the label given to any segment is not evaluated as part of the official evaluation. The official evaluation will evaluate the number of events detected in a given day (compared to the ground truth) as well as the accuracy of the event-detection process (given a sliding five minute window).
LIT - Lifelog Insight Task (sub) Task
The aim of this subtask is to gain insights into the lifelogger's life. It follows the idea of the Quantified Self movement that focuses on the visualization of knowledge mined from self-tracking data to provide "self-knowledge through numbers". Participants are requested to generate new types of visualisations and insights about the life of the lifeloggers by generating a themed diary or themed insights related to some target topics. This task is not evaluated in the traditional sense, but participants will be expected to present their work in a special session at NTCIR.
It is not necessary to submit your insights task output by the due date. We do, however, request that you send us a short abstract of your LIT subtask visualisations and insights to (ntcir-lifelog@computing.dcu.ie) by the due date. This abstract should be about one paragraph (10 lines) in length and may include the visualisation if ready. It is expected that the submission will be described in the participating group’s NTCIR-13 paper and will be demonstrated at the NTCIR-13 conference.