The multimodal recognition of eating condition - whether a person is eating or not - and if yes, which food type, is a new research domain in the area of speech and video processing that has many promising applications for future multimodal interfaces such as: adapting speech recognition or lip reading systems to different eating conditions (e.g. dictation systems), health (e.g. ingestive behaviour), or security monitoring (e.g., when eating is not allowed).

We therefore invite for participation in the first open, audio-visual challenge under strictly comparable conditions, namely audio-visual classification of eating conditions and leverage the audio-visual iHEARu-EAT database [1, 2].

This database contains around 1.4 k utterances (2.9 hours of audio/video recordings) taken from 30 German speakers while eating six types of food (Apple, Nectarine, Banana, Haribo Smurfs, Biscuit, and Crisps). The data contains read speech as well as spontaneous speech combined with transcriptions.

We define three Sub-Challenges based on classification tasks in which participants are encouraged to use speech and/or video recordings:

1. Food-type Sub-Challenge: Perform seven-class food classification per utterance
2. Likability Sub-Challenge: Recognize the subjects' food likability rating
3. Chew and Speak Difficulty Sub-Challenge: Recognize the level of difficulty to speak while eating


What are you eating? Does it taste good? Do you have difficulties in eating it?


Participants in the ICMI 2018 EAT Challenge can perform one or more of these Sub-Challenges. For all Sub-Challenges, a target class label has to be predicted per audio-visual clip, where each file contains one full speech utterance. All Sub-Challenges allow participants to find their own acoustic/visual features and/or their own machine learning model. Standard acoustic and visual feature sets will be provided featuring recent end-to-end deep learning and more conventional bags of cross-modal words that may be used by the participants. We encourage both contributions aiming at highest performance, and contributions aiming at fostering brave new ideas in this context.

[1] S. Hantke, F. Weninger, R. Kurle, F. Ringeval, A. Batliner, A. El-Desoky Mousa, and B. Schuller, "I Hear You Eat and Speak: Automatic Recognition of Eating Condition and Food Types, Use-Cases, and Impact on ASR Performance," PLoS ONE, vol. 11, no. 5, pp. 1−24, May 2016.
[2] B. Schuller, S. Steidl, A. Batliner, S. Hantke, F. Hönig, J. R. Orozco-Arroyave, E. Nöth, Y. Zhang, and F. Weninger, "The INTERSPEECH 2015 Computational Paralinguistics Challenge: Degree of Nativeness, Parkinson's & Eating Condition," in Proceedings INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association. Dresden, Germany: ISCA, September 2015, pp. 478−482.


Thank you and welcome to the ICMI EAT Challenge!

Last update: 19 July 2018





19 July 2018	Notifications are send to authors.
17 July 2018	Notifications will be a bit delayed.
11 June 2018	Paper submission is now open
25 May 2018	Paper submission deadline is extended to the 15th of June
16 May 2018	Release of the test data
12 April 2018	The preliminary baseline paper is released
11 April 2018	The second baseline approach is now available
04 April 2018	Training data and first baseline approach are released
22 February 2018	Challenge Website is online

4th April	Release of training data and first evaluation script
11th April	Release of the second evaluation script and prelimary baseline paper
16th May	Release of the test data
14th June	Closing competition
15th June	Paper submission deadline
18th July	Notifications
29th July	Final result submission
31th July	Camera ready

Simone Hantke, Machine Intelligence & Signal Processing Group, Technische Universität München, Germany ➙ website

Maximilian Schmitt, Chair of Embedded Intelligence for Health Care and Wellbeing, University of Augsburg, Germany ➙ website

Panagiotis Tzirakis, Group on Language, Audio, and Music, Imperial College London, U.K. ➙ website

Björn Schuller, audEERING GmbH, Germany ➙ website

We will present you with a baseline paper, including many details about the data, performance measures, baseline features and prediction results. Please note that the content of this paper may change until the camera ready deadline.

To get started

Please contact the organisers by email to register your team. This email should have as a subject line "Registration_TeamName" and include the following:


Team name


Team members, including affiliations


Team leader


Signed EULA


Please obtain the License Agreement here to get a password and further instructions for the download of the dataset: Please fill it out, print, sign, scan, and email accordingly. The agreement has to be signed by a permanent staff industrial or scientific member (e.g., full professor, associate/assistant professor, reader, lecturer).

The list of team members must be the same as those provided on the EULA and it is not allowed to share the data with others including lab mates. Upon registering your team, you will be given a link and a password to access the files.

After downloading the data, you can directly start your experiments with the train set. You can send your best results to the organisers (icmi-eat@ihearu-play.eu) with the subject line "Results_TeamName". We will analyse five result submissions for you and let you know your performance result. See below for more information on the submission process and the way performance is measured.

The organisers provide the following data:


Audio data


Video data


Audio and video features


Labels (food type, difficulty, likability) for the train and development partitions


Individual labels from each subject for difficulty and likability


Averaged labels for difficulty and likability per recording from all subject

Paper on the Challenge

The introductory paper on the Challenge (the paper will be available within the baseline packages) provides extensive descriptions and baseline results. All participants will be asked to avoid repetitions of the challenge itself, the data, or feature descriptions in their submissions - of course, they have to describe shortly the essentials of the databases dealt with - but include the citation of the baseline paper (will be coming soon), as well as the citation of the dataset.

Results submission

Participants may contribute in all Sub-Challenges at a time. A training and development partitioning will allow for tests and results to be reported by the participants apart from their results on the official test set.

Each participant has up to five submission attempts per Sub-Challenge. You can submit results until the final results deadline, which is before the camera ready version deadline. Your best result for each Sub-Challenge will be used in determining the winner of the challenge. Please send submissions by email to organisers (icmi-eat@ihearu-play.eu) with the subject line "Results_TeamName".

Important: we may reserve the right to ask the top performer of the challenge to submit their code to us to verify the results on the original test set.

Participants' results should be sent as a single Zip file to the organisers by email. The zip file should include the name of your team, the Sub-Challenge, and the number of this attempt, e.g. Results_TUM_Likability_1.zip.

The data in the results files themselves should be formatted the same way as the training gold standard label files, that is, one ARFF file per subject, containing three attributes: Instance_name and the prediction in ASCII values. Their filenames should also be formatted in exactly the same way. We might display results obtained by participants on our website.

Paper submission

Please be reminded that a paper submission and at least one upload on the test set are mandatory for the participation in the Challenge. However, paper contributions within the scope are also welcome if the authors do not intend to participate in the Challenge itself.

In any case, please submit your paper until 30th May 2018(and final results by 29th July 2018, the camera ready paper by 31th July 2018) using the standard style info and respecting length limits. Remaining result upload trials can be saved for new Challenge results until 29th July 2018.

The papers will undergo the normal review process. Papers should refer to the baseline paper for details about the dataset and baseline results. This makes for a more readable set of papers, compared to each challenge paper repeating the same information. Please cite the introductive paper which will be available soon.

All papers must be formatted according to ICMI proceedings style, and should be no more than 4 pages in the two-column ACM conference format (excluding references). Papers should be submitted through the ICMI EAT's easychair submission site (submission link will be made available soon). Reviewing will be double-blind, so submissions should be anonymous: do not include the authors' names, affiliations or any clearly identifiable information in the paper. Latex and word templates for this format can be downloaded from the main website.

Registration Information

Please register using the main conference website. Participants simply register using the registration services.

Frequently Asked Questions

• I have already downloaded the iHEARu-EAT database, shall I send another EULA for the EAT challenge?
  Yes please. The shared data will not be the same for accessing the EAT Challenge dataset. Moreover, you may want to change the names of the lab mates that will use the data. Please find the EAT Challenge License Agreement here.


• How will the data be distributed to the participants? Are you sending the test data at the same time with the training partition?
  No, the test data will be released two weeks after the release of the baseline packages. However, labels are only given for training and development partitions. Further, we are not sending the data - you will need to download the data. Please first download, print, and sign the license agreement, scan and mail it to the addresses given on the agreements. You will then receive an email with download instructions.


• Do I have to submit a paper in order to participate?
  Yes, the submission and acceptance of a paper is mandatory.


• May I participate in several Sub-Challenges?
  Yes, of course. Every site may participate in every Sub-Challenge and the five upload trials are per Sub-Challenge. In how many Sub-Challenges one site participates is decided at the moment you submit your paper by the results you include.


• Is it necessary to use both audio-video channels in the audio-video challenge?
  No, unimodal or multimodal input can be used. The challenge data contains audio, video and meta-data. The meta-data is composed of subject identity, age and gender. The participants are welcome to use any combination of modalities.


• Is the use of commercial face detector such as the Google Picasa OK?
  Any face detector whether commercial or academic can be used to participate in the challenge. The paper accompanying the challenge result submission should contain clear details of the detectors/libraries used.


• Can I learn my model on both labelled train and unlabelled test data?
  No, the datasets are subject independent, test data cannot be used for training purposes.


• Can I use external data for training along with the one provided?
  Yes, you are free to use additional data as long as these are well known and accessible to everybody. However, please send an e-mail to the organisers to verify the additional data first. Furthermore, this should be clearly discussed in the accompanying paper.


• Do I have to include test set results for the paper submission?
  Yes. However, you can update these in the final camera ready version. In any case, at least one result submission has to be made before the paper deadline.


• I have submitted a set of results, when will I receive my scores?
  During an active challenge we strive to return scores within 24hrs during typical working days (Monday – Friday), however please try to be patient as this is subject to workload. Under no circumstances should you spam the organisers as this simply delays the process for all teams.


• I have submitted my results on the test partition, can I know my rank now?
  The rank of the participants will be announced at the end of the Challenge day, which means that, in order to know your rank, you need to write a paper describing your system and your results, get it accepted by the Technical Program Committee, and present it during the conference.


• Can I have access to the mailing list of participants? Can you tell me the results of other teams?
  Absolutely not! Registrations and results are not for public view. If there is sufficient demand, we may consider offering a separate opt-in mailing list for teams to discuss the challenge.


• Can we submit one paper to ICMI describing participation in several Sub-Challenges?
  This is possible - obviously the paper should read well and contain all necessary details, as it needs to pass peer-review as condition for successful Challenge participation.


• My results are below the baseline - does it make sense to submit?
  Of course it does. We do not know whether the baseline will be surpassed and different experiences with the tasks on the same dataset will be of interest. Please remember that all submissions to the Challenge go through the normal reviewing process. Although it is very likely that the reviewers do know - and take into account - the baselines, the criteria are the usual, i.e., scientific quality; surpassing any baseline. A paper reporting results above the baseline, but poorly written, runs high risks *not* to be accepted; in contrast, a paper which is well written, contributing to our knowledge, but with results below the baseline, has high chances to be accepted.


• May I include results on other corpora in my paper?
  Yes, of course. As long as it fits the general focus these are of course very welcome.


• How can I cite the baseline paper?
  Please cite as follows: Simone Hantke, Maximilian Schmitt, Panagiotis Tzirakis, and Björn Schuller, "EAT − The ICMI 2018 Eating Analysis and Tracking Challenge", Proceedings ICMI 2018, ACM, Boulder, Colorado, 2018.


• Will the review process be anonymous?
  The review process is double-blind.


• Is it necessary to participate in all three challenges?
  Participants can contribute to all or either of the challenges.


• What are the other formalities with respect to the paper of mine?
  Please make sure to reference the official Paper on the Challenge (soon available) and avoid repetition of the general data and Challenge description.


• Is there any sanity check on the test results?
  The two best performing teams will be asked to share their code/executable/library for checking at our end.


• Are the papers published in the EAT Challenge part of the main proceedings?
  Yes, EAT is a grand challenge in ICMI and hence, EAT papers are part of the ICMI main conference proceedings.

Programme Committee

Eduardo Coutinho, University of Liverpool, U.K.

Anna Esposito, Seconda Università di Napoli (SUN) and IIASS, Italy

Roland Goecke, University of Canberra, Australia

Robert Istepanian, Imperial College London, U.K.

Dongmei Jiang, Northwestern Polytechnical University, China

Erik Marchi, Apple inc., USA

Shri Narayanan, University of Southern California, USA

Fabien Ringeval, Université Grenoble Alpes, France

Stefan Steidl, Friedrich-Alexander University Erlangen-Nuremberg, Germany

Dan Zecha, University of Augsburg, Germany

Sponsors

This Challenge will be mainly sponsored by the industrial partner audEERING GmbH and funded from the European Community's Seventh Framework Programme under grant agreement No. 338164 (ERC Starting Grant iHEARu).