Can We Teach AI to See Through Text? A Fun Exploration with ASCII Art and GPT-3

We’ve all been amazed by the power of large language models (LLMs) like GPT-3. They can write stories, translate languages, and even generate code. But can they see? While LLMs excel at understanding and generating text, they don’t have eyes. So how could we possibly get them to understand images?

In this article, we explore a fun and creative approach: teaching AI to “see” by converting images into ASCII art. The idea is simple yet intriguing: if we can translate images into a text-based format, can LLMs like GPT-3 then perform image classification based on this text representation?

The Hypothesis: Structure Over Semantics

The core of this experiment lies in an interesting hypothesis. We know LLMs are great at few-shot learning – they can learn new tasks from just a few examples. But is it the meaning of the examples that matters, or simply the structure?

Could it be that to get an LLM to perform a task, we just need to provide a prompt with a consistent and recognizable pattern, even if that pattern isn’t something the model encountered during its initial training on human-generated text?

To test this, we’re venturing into the realm of ASCII art – a world of images represented by text characters. If an LLM can learn to classify images based on their ASCII representations, it would suggest that structure and pattern recognition play a significant role in their few-shot learning abilities.

Why Bother with ASCII Art Image Classification?

This might seem like a purely artistic endeavor, but it touches upon some important questions about the future of AI. As we seek to deploy AI in more diverse and specialized settings, we face challenges:

• Data Scarcity: Training new models from scratch often requires vast amounts of data, which might not be available in specific applications.
• Cost of Fine-tuning: Fine-tuning existing models can be computationally expensive and require specialized expertise.
• Adaptability to New Scenarios: Pretrained models might not perform well in unexpected situations or edge cases.

LLMs’ ability to adapt their behavior based on a few examples offers a potential solution. By providing a small set of examples demonstrating the desired behavior, we might be able to guide these models without costly retraining.

But the world isn’t just text! To truly unlock the potential of this approach, we need to extend it to other modalities, like images. Could text serve as a universal bridge to achieve this?

The Experiment: MNIST and CIFAR-10 as ASCII Art

We put our idea to the test using two classic image datasets: MNIST (handwritten digits) and CIFAR-10 (a collection of small color images). Here’s the process:

1. Image Transformation: We downsize the images, center-crop them, convert them to grayscale, and finally, transform them into ASCII art.
2. Prompt Engineering: We create prompts for GPT-3 that follow a simple few-shot learning template:

Input: [flattened_ascii_image]
Output: [class_label]
###
Input: [flattened_ascii_image]
Output: [class_label]
###
...
Input: [flattened_ascii_image]
Output:

We randomly sample a few examples from each class to provide context within the prompt.

GPT-3 Inference: We feed these carefully constructed prompts to GPT-3 (specifically, the text-davinci-003 model) and obtain its predictions.

Evaluation: We compare GPT-3’s predictions to the ground truth labels and measure accuracy.

Results: Better Than Chance!

In our initial experiments, focusing on classifying three classes from MNIST (0, 1, and 2), we achieved results significantly better than random guessing! This suggests that our approach has merit – GPT-3 was able to discern patterns in the ASCII art and associate them with the correct digit classes.

Interestingly, the model performed best on the digit “1,” perhaps because its ASCII representation is more distinct compared to “0” and “2.”

Stepping Up to CIFAR-10 (or CIFAR-3?)

Encouraged by the MNIST results, we moved on to the more challenging CIFAR-10 dataset, focusing on three classes: airplanes, cars, and birds. While the overall accuracy was still above chance, the results were more ambiguous. Notably, the model struggled to predict the “car” class.

Limitations and Future Directions

It’s important to acknowledge that this is a preliminary exploration. The results are sensitive to factors like:

• Prompt Structure: The way we format the prompt can significantly impact performance.
• Sample Selection: The specific examples we choose for the few-shot context can influence the model’s predictions.
• ASCII Conversion: The method we use to convert images to ASCII art might introduce biases or limitations.
• Model Choice: Different LLMs might perform differently on this task.

Conclusion: A Glimpse into the Future?

While our experiment is far from conclusive, it offers a fascinating glimpse into the potential of using text as a bridge to enable LLMs to understand other modalities. By creatively representing images as ASCII art, we’ve shown that GPT-3 can perform few-shot image classification, suggesting that structure and pattern recognition play a crucial role in its learning process.

This is just the beginning. Future work could explore:

• More sophisticated ASCII conversion techniques.
• Different prompt engineering strategies.
• Testing with a wider range of LLMs and datasets.
• Investigating the impact of different hyperparameters.

As we continue to unravel the mysteries of LLMs, this “art project” serves as a reminder that sometimes, the most unconventional approaches can lead to the most intriguing insights.

Download