Machine Translation for Vision (MTV)

Given the rise of multimedia content, human translators increasingly focus on culturally adapting not only words but also other modalities such as images to convey the same meaning. While several applications stand to benefit from this, machine translation systems remain confined to dealing with language in speech and text. In this work, we introduce a new task of translating images to make them culturally relevant (image transcreation). For example, a math worksheet teaching children how to count using halloween-themed objects in the US would change to a worksheet using diwali-themed objects to teach the same concept in India.

More details on the dataset and the task can be found in our paper. We also won the Best Paper award at EMNLP 2024 for this work!

Dataset Overview

Our test set contains images paired with concepts that need to be transcreated to different cultural contexts.

Property	Value
Total Images	696
Splits	`concept` (595), `application` (101)
License	MIT

Splits

concept: 595 images focusing on single concepts that are cross-culturally coherent.
application: 101 images sourced from real-world use cases (e.g., educational materials, storybooks). Used for evaluating practical applicability.

Dataset Fields

Field	Type	Description
`id`	string	Unique identifier for each image
`image`	PIL.Image	Image data as a PIL Image object
`category`	string	Category classification (see below)
`text`	string	Descriptive text for the image (see below)
`image_path`	string	URL path to the original image file
`source_country`	string	Country of origin for the image
`target_countries`	string	Comma-separated list of target countries for adaptation (see below)

Field Details by Split

Concept Split:

category: The semantic category of the concept (e.g., food, beverages, housing, clothing, festivals, etc.)
text: The name of the object depicted in the image (e.g., "pancakes", "yerba mate", "kimono")
target_countries: All 7 countries except the source country (6 targets per image)

Application Split:

category: Either "education" or "stories"
text: For education images, the learning concept being taught (e.g., "counting", "addition"). For story images, the accompanying text/caption from the storybook.
target_countries: All 7 countries (brazil, india, japan, nigeria, portugal, turkey, united-states)

Loading the Dataset

from datasets import load_dataset

# Load the entire dataset
dataset = load_dataset("cmu-lti/machine-translation-for-vision")

# Access specific splits
concept_data = dataset["concept"]
application_data = dataset["application"]

# Or load a single split
concept_only = load_dataset("cmu-lti/machine-translation-for-vision", split="concept")

Example Usage

from datasets import load_dataset

dataset = load_dataset("cmu-lti/machine-translation-for-vision")

# View a sample from the concept split
concept_sample = dataset["concept"][0]
print(f"ID: {concept_sample['id']}")
print(f"Category: {concept_sample['category']}")
print(f"Text: {concept_sample['text']}")
print(f"Source Country: {concept_sample['source_country']}")
print(f"Target Countries: {concept_sample['target_countries']}")

# View a sample from the application split
app_sample = dataset["application"][0]
print(f"ID: {app_sample['id']}")
print(f"Category: {app_sample['category']}")
print(f"Text: {app_sample['text']}")
print(f"Source Country: {app_sample['source_country']}")
print(f"Target Countries: {app_sample['target_countries']}")

# Access the image directly (already a PIL Image)
image = concept_sample['image']
image.show()

Key Findings

Evaluation of state-of-the-art generative models on this benchmark reveals:

Current image-editing models perform poorly, with success rates as low as 5% on concept images for certain countries
Complete failure on application images for some regions
Incorporating language models and retrieval systems improves outcomes

Citation

If you use this dataset, please cite:

@inproceedings{khanuja-etal-2024-image,
    title = "An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance",
    author = "Khanuja, Simran  and
      Ramamoorthy, Sathyanarayanan  and
      Song, Yueqi  and
      Neubig, Graham",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.573/",
    doi = "10.18653/v1/2024.emnlp-main.573",
    pages = "10258--10279"
}

License

This dataset is released under the MIT License.