Join the CLEF 2024 SimpleText Track 4: SOTA? Challenge – Revolutionizing AI Paper Analysis!
The SimpleText Track 4: SOTA? is a novel shared task introduced at CLEF 2024, focusing on developing systems that can determine if an AI paper reports model scores on benchmark datasets and extract relevant (Task, Dataset, Metric, Score) quadruples. The primary goal is to enhance the accessibility and usability of scholarly publications in AI by automatically tracking the state-of-the-art results.
The SOTA? task will consist of two evaluation phases:
- Evaluation Phase 1: Few-shot Testing
- Evaluation Phase 2: Zero-shot Testing
Participating in SimpleText Task 4: SOTA? is a fantastic opportunity to contribute to the advancement of AI in scholarly communications. To join the challenge, follow these steps:
- Register your team through the official CLEF 2024 registration page: CLEF 2024 Registration
- Sign up on the Codalab competition site: Codalab Competition Site
The SimpleText initiative also offers three additional tasks:
- Task 1
- Task 2
- Task 3
Feel free to participate in any or all of these tasks, as there is no dependency between the datasets. Important dates for Task 4 are:
- Training and validation datasets available: March 13, 2024
- Test data available/Evaluation starts: April 23, 2024
- Evaluation ends: May 3, 2024
- Participant paper submissions due: May 31, 2024
- Notification to authors: June 24, 2024
- Camera ready due: July 8, 2024
- CLEF 2024 Workshop, Grenoble, France: 9-12 September 2024
Task organizers include Jennifer D’Souza, Salomon Kabongo, Hamed Babaei Giglou, Yue Zhang, and Sören Auer from TIB Leibniz Information Centre for Science and Technology and Berlin Technical University. For further information and inquiries, please contact the organizing team at sota.task@gmail.com.