Kreyòl-MT

Welcome to our public data repository!

Please download data for any langauge pair via the command load_dataset("jhu-clsp/kreyol-mt", "<langauge-pair-name>"). For example:

from datasets import load_dataset
data = load_dataset("jhu-clsp/kreyol-mt", "acf-eng")

Dataset info

The full dataset we intend to release is not quite here yet, unfortunately. We are still waiting on the LDC release of a portion of it, and the rest we want to release together. What's hosted here now is the exact data set we used to train our models in published work, "Kreyòl-MT: Building Machine Translation for Latin American, Caribbean, and Colonial African Creole Languages" (to be presented at NAACL 2024), with the sentences from the Church of Jesus Christ of Latter-day Saints (CJCLDS) removed from train and dev sets. This is a temporary provision until these data's impending release on LDC.

In the coming weeks and months we will add:

• The CJCLDS data from LDC, upon its release
• NLLB data that we excluded from our model training but decided to include in our public data release
• All releasable monolingual data
• Any additional data that we or others come across and incorporate: we intend this to be a living dataset!

Additional upcoming updates:

• Metadata indicating which aligned sentences came from which sources prior to our data splitting

Since we are still awaiting the public release of CJCLDS data, please contact Nate Robinson at n8rrobinson@gmail.com for the full dataset if needed.

Documentation

Documentation of all our data, including license and release information for data from individual sources, is available at our GitHub repo here.

Cleaning

All dev and test sets are cleaned already. For information on cleaning for train sets, see our GitHub repo here. For unclenaed or additional sets, please contact the authors

Paper and citation information

Please see our paper: 📄 "Kreyòl-MT: Building Machine Translation for Latin American, Caribbean, and Colonial African Creole Languages"

And cite our work:

@article{robinson2024krey,
  title={Krey$\backslash$ol-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages},
  author={Robinson, Nathaniel R and Dabre, Raj and Shurtz, Ammon and Dent, Rasul and Onesi, Onenamiyi and Monroc, Claire Bizon and Grobol, Lo{\"\i}c and Muhammad, Hasan and Garg, Ashi and Etori, Naome A and others},
  journal={arXiv preprint arXiv:2405.05376},
  year={2024}
}