WMT24 Metrics Task: Call for Participation in the Shared Task on Evaluation of Automatic Metrics

The 9th Conference on Machine Translation (WMT24), collocated with EMNLP 2024, is hosting the Shared Task on evaluation of Automatic Metrics. The task is open for both reference-based and reference-free metrics to evaluate the quality of Machine Translation (MT) systems. Expert-based MQM annotations on English-German, English-Spanish, and Japanese-Chinese will be used as the primary gold standard for evaluating metrics.

The task will continue the challenge sets subtask, inviting anyone to submit a new test suite and/or an analysis paper on metric behavior for specific perturbations/phenomena. New this year are the addition of English-Spanish and Japanese-Chinese language pairs, additional 13 typologically diverse African languages in a challenge set, and the use of the Codabench platform to improve the metric submission experience.

Important dates for the task include the challenge sets submission deadline on July 11, metric submission deadline on July 30, and paper submission deadline to WMT on August 20. Participants are required to register their metric submissions “>here so the organizers can keep track of participants.