Indic Eval

A lightweight evaluation suite tailored specifically for assessing Indic LLMs across a diverse range of tasks, aiding in performance assessment and comparison within the Indian language context

Context

Indic Eval is a lightweight LLM evaluation suite built on top of LightEval, which Hugging Face has been using internally. We at cognitivelab.in utilize it for internal testing of LLMs on various tasks.

While early in development, it offers a collaborative space for community-driven advancement in Indic language modeling. Please note that stability is a work in progress. Feel free to contribute or raise issues!

What does it Offer on top of Light eval

• ✅ Intergration with Indic LLM leaderboard
• ✅ Support of the following translated dataset ARC, Hellaswag , Boolq, MMLU , Translate[IND 22] to indian languages
• ✅ Skypilot Integration to run Evals on 15+ CLoud providers with simple configuration
• ✅ Support for Language Base evaluation rather than task based

News

• April 01, 2024: Release of indic_eval

Installation

Clone the repo:

git clone https://github.com/adithya-s-k/indic_eval
cd indic_eval

Create a virtual environment using virtualenv or conda depending on your preferences. We require Python 3.10 or above:

conda create -n indic-eval-venv python=3.10 && conda activate indic-eval-venv

Install the dependencies. For the default installation, you just need:

pip install .

If you want to evaluate models with frameworks like accelerate or peft, you will need to specify the optional dependencies group that fits your use case (accelerate,tgi,optimum,quantization,adapters,nanotron):

pip install .[optional1,optional2]

The setup tested most is:

pip install .[accelerate,quantization,adapters]

If you want to push your results to the Hugging Face Hub, don't forget to add your access token to the environment variable HUGGING_FACE_HUB_TOKEN. You can do this by running:

huggingface-cli login

and pasting your access token.

Run Indic LLM Leaderboard Eval

To conduct an evaluation for the Indic LLM Leaderboard, you can use the following commands:

accelerate launch run_indic_evals_accelerate.py \
    --model_args="pretrained=<>path to model on the hub" \
    --language  \
    --tasks indic_llm_leadeboard \
    --output_dir evals \
    --push_to_leaderboard  

• --model_args: Specifies the arguments for the model. This includes parameters like pretrained, which is the path to the model on the hub that you want to evaluate. Example: --model_args="pretrained=path_to_model"

• --language: Specifies the language you want to evaluate on. This argument determines the language-specific evaluation tasks. Example: --language english Langauges supported are kannada, hindi, tamil, telugu, gujarati, marathi, malayalam , english

• --tasks: Specifies the tasks you want to evaluate the model on. In this case, it's indic_llm_leaderboard, indicating the evaluation tasks specific to the Indic LLM Leaderboard. which includes the following

• --output_dir: Specifies the directory where the evaluation results will be saved. Example: --output_dir evals

• --push_to_leaderboard: Specifies the email address through which we can contact you to verify the scores if needed. Example: --push_to_leaderboard yourname@company.com

Some other parameters include:

• --override_batch_size: You can increase the batch size. If not specified, the library itself picks the most optimal batch size. It is recommended to leave this option out.

to see all the indic benchmarks available you can refer to indic_tasks.txt

Single Benchmark Evaluation for a Single Language

For instance, to run the ARC-Easy benchmark for Kannada:

accelerate launch run_indic_evals_accelerate.py \
    --model_args="pretrained=