#DS4SocietySeminar 2023 <> Towards Evaluating and Understanding the Multilingual Performance of Generative Language Models
Maxamed Axmed/Mohamed Ahmed
Talk Details
Abstract
Generative AI models have impressive performance on many Natural Language Processing tasks such as language understanding, reasoning, and language generation. However, their capabilities and limitation are still not well understood, and evaluating generative AI is very challenging, furthermore, most studies on generative Large Language Models (LLMs) are restricted to English and it is unclear how capable these models are at understanding and generating other languages. We present the first comprehensive benchmarking of generative LLMs - MEGA, which evaluates models on standard NLP benchmarks, covering 8 diverse tasks and 33 typologically diverse languages. We also compare the performance of generative LLMs to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs.
Speaker Bio
Maxamed is a Principal Researcher at the Microsoft Africa Research Institute (MARI) based in Nairobi, Kenya. Before becoming part of Microsoft, he was involved in creating machine learning models for medical chemistry at BenevolentAI and had the pleasure of being a researcher at NEC Labs Europe. Maxamed’s current research interests are in advancing machine learning and NLP to help to scale healthcare, and to overcome the inequities and barriers to healthcare access for most of the world.
Paper - [MEGA: Multilingual Evaluation of Generative AI]
Video, Slides and Notes
- content will be uploaded soon