ViT vs CNN: Which Fits Your Images?

ViT vs CNN: Which Fits Your Images?

Imagine you are building an app that checks photos of apples before they reach a shop. Some apples have tiny bruises. Others are healthy but photographed under a shadow. Your app must spot the difference quickly, and it may have to run on a phone. Would you choose a convolutional neural network, or a Vision Transformer?

That question is more useful than asking which model is “better.” A CNN and a Vision Transformer (ViT) can both recognise images, but they start with different ideas about how to look at a picture. Once you understand those ideas, the choice becomes much less mysterious. We will follow one apple photo, see what each model looks for, and then work out when each approach makes sense.

Start with One Photo, Not a Pile of Jargon

How a CNN reads an image

A CNN looks for small patterns first. Imagine moving a tiny window across the apple photo. In one place it finds an edge. In another, it finds a change in colour. Later layers combine these simple clues into larger ones: a bruise, a stem, or the outline of an apple. Finally, the model uses what it has found to make a prediction.

That small window is called a filter. It uses the same learned pattern in many places. This lets it notice an edge near the top of a photo or near the bottom. CNNs therefore begin with a useful assumption: nearby pixels often belong together, and the same kind of pattern might appear anywhere in an image. Researchers call a built-in assumption like this an inductive bias. In everyday language, the CNN gets a head start on how pictures usually work.

Think about reading a newspaper through a small square cut into a sheet of paper. You first see letters, then words. As you move the square and put the pieces together, you understand the headline. A CNN works in a similar way, moving from small visual clues towards larger ones. Its deeper layers can bring together information from distant parts of the photo, so it does not stay focused on tiny details forever.

This head start can help when labelled photos are limited. If you have only a few thousand apple photos, a CNN can learn useful patterns without discovering every basic rule of images from scratch. It is also a good model to consider when memory and speed are tight. That does not mean every CNN is small or fast; the exact model and device still matter. If filters and layers are new to you, read our CNN basics guide alongside this comparison.

How a Vision Transformer reads the same image

A standard ViT begins differently. It cuts the photo into equal squares called patches. If an image is 224 × 224 pixels and each patch is 16 × 16 pixels, the model gets 196 patches. Each patch is turned into a list of numbers it can process, often called a token. The model also gets information about where each patch came from. If the name sounds unfamiliar, start with Transformers for beginners.

Next comes attention. Each patch can weigh information from other patches. A dark mark on the apple can be considered alongside its shape, stem, and background. In the original ViT, patches can share information across the whole image from the start. The model learns which relationships are useful for its task. Our guide to how self-attention works explains this step in more detail.

Imagine friends solving a jigsaw puzzle. One group begins with pieces that sit next to each other and slowly builds larger sections. Another lays out many pieces and looks for connections across the whole puzzle. The first picture helps explain a CNN; the second helps explain a standard ViT. It is only an analogy, but it captures the starting point: CNNs build from nearby patterns, while a ViT lets distant patches exchange information early.

There is a catch. With little data and no useful pretraining, a ViT may have to learn too many visual rules for itself. The original ViT paper showed strong results after training on large image collections and then adapting the model to smaller tasks. This does not mean you need millions of your own apple photos. You may be able to start with a suitable pretrained ViT. It does mean you should check what it was trained on before expecting it to work well with your small collection.

What “replacing convolutions with attention” means

The phrase sounds dramatic, but it describes a model design. In the original ViT image classifier, image patches pass through Transformer layers instead of the usual stack of convolution layers. Attention is the main way those patches share information. The model still needs other parts to prepare the patches, track their positions, and process what it learns.

Real systems can also mix the two ideas. Some use convolutions before attention. Others let attention work within small windows instead of across every patch at once. So when someone says they are using “a Transformer,” ask which design they mean. A plain ViT, a window-based model, and a mixed model can differ in speed, training needs, and how they handle fine detail. Those differences become much clearer when we put them into real situations.

Where the Difference Shows Up in Practice

Nearby detail versus wider context

Suppose the apple sorter must detect a bruise only a few pixels wide. Small, nearby changes matter a lot. A CNN’s habit of looking for edges, colours, and textures in a small area may be useful here.

Now imagine a different task. The app must decide whether a photo shows a real apple on a market stall or an apple printed on a cereal box. Looking at the fruit alone may not be enough. The surrounding scene gives an important clue. A Vision Transformer can share information between distant patches early on, which may help it learn that relationship.

This is a way to understand the difference, not a rule that predicts every result. A deep CNN can learn about the wider scene, and a ViT can learn small details. Patch size, image size, training data, and the model design all affect what happens. If a patch is too large, a tiny bruise might be hard to see. Making patches smaller keeps more detail, but it also gives the ViT more patches to process. When tiny defects matter, test with the same image size your app will use in real life.

The data question people often miss

Imagine two friends building an app that recognises five kinds of local flowers. They have 1,500 labelled photos and one ordinary computer. It would be risky to assume that a ViT trained from scratch will beat a CNN trained from scratch. The original ViT research showed especially strong results after training on large image collections and then adapting the model to smaller tasks.

For this flower project, a pretrained CNN is a good starting point. A pretrained ViT is worth testing too, if its training images and licence suit the project. Pretrained simply means the model has already learned from other images. Think of a friend who has seen thousands of plants: they may learn the name of a new flower from a few examples. Fine-tuning means taking that existing model and adjusting it with your own flower photos. If you want to explore that step, read fine-tuning an image model.

The research did not stop with the first ViT. The DeiT study showed that a carefully trained Transformer could be competitive using ImageNet alone. Its researchers also explored letting a trained model, including a CNN, help teach the Transformer. So “ViTs always need millions of photos from your own project” is too simple. Ask which pretrained model and training method you can actually use.

There is another trap in a small flower dataset: the test itself. If you take ten almost identical photos of one flower and place some in training and some in testing, the score may look impressive for the wrong reason. Keep near-duplicate photos together. If people will use the app on their phones, test photos from different phones, in bright and dull light, and against different backgrounds. These checks can matter more than whether the model’s name contains “CNN” or “ViT.”

Speed, memory, and the phone in your hand

Return to the apple app. Workers may use it in a warehouse where the internet is unreliable. The model may need to answer on the phone itself. In that case, check how long one answer takes. This waiting time is called latency. Also check how much storage and memory the model needs, what it does to battery life, and whether it stays responsive when workers scan many apples.

A plain ViT lets patches exchange information across the image. When you increase image detail, you usually create more patches, and that exchange can become costly. CNNs have designs made for small devices. MobileNetV3, for example, was developed with phone CPUs in mind. A compact CNN is therefore a useful model to test first on a phone. That does not guarantee it will win: the exact phone, model, and app setup still matter.

Transformer designs have ways to control this cost. Swin Transformer lets patches share information inside small windows, then shifts those windows so information can travel farther. It also works at several image scales, which is helpful when a task involves locating or outlining objects. A product page calling a model “a Transformer” tells you little about its real speed. Test the actual model on the actual device.

Accuracy is more than one score

Suppose two models each get 94% accuracy on your apple photos. They may still make very different mistakes. One might miss bruised apples. The other might reject healthy apples. Missed bruises can lead to customer complaints; rejected healthy apples can create waste. Look at both kinds of error and decide which hurts your project more.

Then test on photos the models have never seen during development: new lighting, different camera angles, messy backgrounds, and apples from another supplier. A score from a familiar benchmark cannot tell you exactly how your app will perform in the warehouse. Whenever you compare a CNN and a ViT, write down the test photos, image size, device, speed, and types of mistake alongside the accuracy score. That gives you something useful for the final choice.

How to Choose Without Guessing

Match the model to the job

Start by describing exactly what you need the model to do. Should it give one label to a whole photo, such as “bruised” or “healthy”? That is image classification. Should it draw a box around each damaged apple? That is object detection. Should it mark the exact damaged area? That is segmentation. These are different jobs. A model that is good at naming a photo may need extra parts before it can locate anything in it. If these terms are new, start with image classification, detection, and segmentation.

For classification with a small collection of photos, begin by testing a suitable pretrained CNN and a suitable pretrained ViT. If the app must work on a phone, include a compact CNN designed for phones. If you need to locate objects or mark their outlines, look at complete systems built for those jobs, including CNN-based models and Transformers that work at several image scales. A plain ViT image classifier cannot be judged as though it were a ready-made detection app.

Consider a small bakery that wants to count pastries in photos of its display trays. Giving each photo one label, such as “pastries,” will not tell the owner how many are there. The system needs to find individual pastries, including ones partly hidden behind others. A large bakery chain may have thousands of labelled photos from many shops; a single bakery may have only a few hundred. The task and the available photos shape the choice before the words “CNN” or “ViT” do.

Run a fair small experiment

First, collect photos that look like real use. Include the awkward ones: blur, poor light, reflections, objects partly out of view, and messy backgrounds. Make sure you have permission to use the images. Give them labels that match the decision you need. “Bad apple” may be too vague if workers need to tell a bruise from a shadow.

Second, set aside test photos before training. Keep photos of the same physical object or scene together. Otherwise, almost identical images can end up in both the training and test groups, making the score look better than it should. If apples from a new supplier will arrive next month, try to reserve that supplier’s photos for a separate check. You want the test to resemble the future, not just the images you already know.

Third, compare realistic models fairly. Use the same training and test groups. Keep the image size and training budget as similar as the models allow. Compare a pretrained CNN with a pretrained ViT if pretraining is what you plan to use. Write down the exact model version, its pretrained weights, and how you prepared the images. If you spend hours improving one model and only minutes on the other, you have mostly measured your own tuning time.

Fourth, examine the wrong answers. Which bruises were missed? Which healthy apples were rejected? Does one camera or lighting condition cause more trouble? Show a few of these photos to the people who will use the app. They may spot a practical problem that a single accuracy number hides.

Fifth, time the complete process on the real device. Include the time taken to resize the photo and prepare it for the model. A model that is slightly more accurate but much slower may frustrate a worker scanning hundreds of apples. Check memory use and whether the app stays responsive after many scans.

Then choose a model that meets your quality and speed needs, and plan how to review mistakes after launch. Real photos change. New packaging, cameras, seasons, and shop layouts can all change what the model sees. A good test before launch is the beginning of the work, not the last time you check it.

So, did attention replace convolution?

It gave us more choices. The first Vit Paper showed that a Transformer working with image patches could be very effective when trained in the right setting. DeiT showed a way to train one with less data than the original approach. Swin Transformer used smaller attention windows and several image scales to support more vision tasks. Meanwhile, the ConvNeXt study showed that a modern CNN could still compete strongly.

So when a friend asks, “Should I use a ViT or a CNN?”, ask four questions back. What must the model do? What suitable data or pretraining is available? Where must it run? Which mistakes hurt most? Test a good candidate from each approach. The answer may be a CNN, a ViT, or a design that uses ideas from both. The useful choice is the one you can explain using your own photos, device, and results.

Frequently Asked Questions

Is a ViT always more accurate than a CNN?

No. The result depends on the images, model size, training method, and test conditions. A good CNN can beat a poorly chosen ViT, and the reverse can also happen. Test both on photos from your own project.

Can I use a Vision Transformer with a small dataset?

Yes, if you start with a suitable pretrained model and then fine-tune it. Training a plain ViT from scratch on a small set of photos is much harder. Keep separate test photos to check whether the model has learned something useful.

Which model should I try first for a phone app?

Try a compact CNN built for phones as your first benchmark. Then test a small ViT if it fits your needs. Measure the full app on the phone: answer time, memory use, battery use, and accuracy all matter.

Does a CNN only see small parts of an image?

No. Its early layers find nearby patterns, but deeper layers combine them and can use information from a much larger area. “Starts with local detail” does not mean “never sees the whole picture.”

Is ViT better for finding objects in an image?

There is no automatic winner. Finding objects requires a complete detection model, not just a model that labels a whole photo. Compare detection systems on the same images and check which objects each system misses.

Have Vision Transformers replaced CNNs?

No. Both are useful, and some models combine ideas from each. Choose based on the task, available data, device, and kinds of mistakes that matter to the people using your app.

To see how CNNs find small visual defects in practice, read our CNN defect detection guide.