Challenges | Vision-based Assistants in the Real-World

The workshop will host two challenges on tasks that are crucial to enable real-world vision-based assistants. These challenges are designed to test both the low-level visual capabilities and higher-level reasoning skills of vision-based assistants.

The winning teams will receive a prize and are invited to present their solution in a contributed talk.
Deadline: June 1, 2025.

Challenge 1: Interactive Feedback Generation

This challenge focuses on assisting users through a workout session with interactive feedback. Details:

Evaluation Data: We base this challenge on the QEVD dataset, as described here. Specifically, the challenge involves providing timed feedback for a set of evaluation videos. For this challenge, we employ a (private) test set available here.
Training and Validation Data: For training and validation, please use the data provided in the QEVD page.
Evaluation Metrics: We will use the METEOR, ROUGE-L, BERT, LLM-Acc., and T-F-Score as described here. The code for these metrics is available here. If you have any questions contact us here.
Participation:
- Leaderboard: Please email the results here as a json file along with the team name. The json file should contain a list of python dicts with the the following fields:
```
  [{“video_file”: <str: name of the evaluation video file>,
  “feedbacks”: <List[str]: list of predicted feedbacks>,
  “feedback_timestamps”: <List[float]: list of timestamps corresponding to the predicted feedbacks>}, ...]
```
  Each team will be allowed to make five submissions and we will provide the evaluation results of each submission as soon as possible.
- Extended Abstract: The teams submitting to the challenge are also encouraged to submit an extended abstract through CMT. The page limit is a minimum of two pages and a maximum of four pages, excluding references. As subject area please choose “Challenge -> Interactive Feedback Generation”.
- Winner: The winning team will be decided based on the five evaluation metrics described above. The winning team is the one that outperforms others on most metrics. The code of the winning team will be inspected before the workshop. The winner will receive a prize along with a contributed talk at the workshop.

Challenge 2: Interactive Question Answering

This challenge tests the ability of vision-based assistants to converse face-to-face with a human user. This challenge requires models not only to have robust audio-visual perception but also requires reasoning about scenes and events that are unfolding live in front of the camera. Details:

Evaluation Data: We base this challenge on the QIVD dataset, as described here. Specifically, the challenge involves providing (timely) answers to question posed by users in the videos here. The evaluation will be performed based on predicted answers and the associated timestamps. Note that, the questions are not provided and thus need to be infered from the audio associated with each video.
Training and Validation Data: We provide a set of in-context examples here.
Evaluation Metrics: We will use the Correctness, BERT, METEOR, BLEU and ROUGE-L scores described here. If you have any questions contact us here.
Participation:
- Leaderboard: Please email the results here as a json file along with the team name. The json file should contain a list of python dicts with the the following fields (see here for an example):
```
  [{“video”: <str: name of the evaluation video file>,
  “answer”: <str: the predicted answet>,
  “timestamp”: <float: the predicted timestamp assoicated with the answer>}, ...]
```
  Each team will be allowed to make five submissions and we will provide the evaluation results of each submission as soon as possible.
- Extended Abstract: The teams submitting to the challenge are also encouraged to submit an extended abstract through CMT. The page limit is a minimum of two pages and a maximum of four pages, excluding references. As subject area please choose “Challenge -> Interactive Question Answering”.
- Winner: The winning team will be decided based on the five evaluation metrics described above. The winning team is the one that outperforms others on most metrics. The code of the winning team will be inspected before the workshop. The winner will receive a prize along with a contributed talk at the workshop.