How to Use Vision Prompts

A vision prompt is a prompt that includes one or more images alongside text. The model 'sees' the image and reasons over it together with your instruction. Common tasks include reading text from a photo (OCR-style), describing a scene, extracting structured data from a document, comparing two images, answering a question about a chart, or judging whether an image meets some criteria.

The key difference from text-only prompting is that you have two channels. The image carries the visual content; the text carries the intent. Vague text wastes the image — "describe this" gets you a generic caption — while precise text ("List every line item and its price as a table") gets you exactly what you need.

Note that vision prompting (image as input) is the opposite of image generation (image as output). If you want to create images, you need a tool like the DALL-E Prompt Creator or Midjourney Prompt Builder, not a vision prompt.

The reliable, high-value use cases cluster into a few patterns:

**Extraction.** Pull text, numbers, or structured fields out of receipts, invoices, forms, screenshots, and labels — usually returned as JSON or a table.

**Description and accessibility.** Generate alt text, captions, or detailed scene descriptions for images.

**Visual question answering.** Answer specific questions about a chart, diagram, map, or photo — "Which quarter had the highest revenue in this chart?"

**Comparison and inspection.** Spot differences between two images, check whether an image matches a spec, or flag issues in a screenshot of a UI.

**Reasoning over diagrams.** Read a flowchart, schematic, or whiteboard photo and explain or transcribe it.

The image is fixed once you attach it; your leverage is the text. Four moves make the biggest difference.

**State the task explicitly.** "Extract," "compare," "transcribe," "summarize," "judge" — pick the verb and name the target. **Point at regions.** If the answer is in a corner, a column, or a specific row, say so — "read the total in the bottom-right," "focus on the legend."

**Specify the output format.** Ask for JSON, a table, a bullet list, or alt text. For machine-readable results, a strict schema beats prose — see structured output schema design patterns.

**Handle uncertainty.** Tell the model what to do when the image is blurry, cropped, or ambiguous: "If a field is unreadable, return null for it" prevents confident invention. For multi-step visual reasoning, you can also ask the model to describe what it sees before answering, a vision analog of chain-of-thought prompting.

A vague vision prompt produces a caption you can't use:

``` [receipt image] What is this? ```

A precise extraction prompt produces structured, verifiable output:

``` [receipt image] Extract the line items from this receipt. Return JSON with this shape: { "merchant": string, "date": string, "items": [{ "name": string, "price": number }], "total": number }. Read the total from the bottom of the receipt. If any field is unreadable, set it to null — do not guess. ```

The structured version names the task, points at the total's location, fixes the output schema, and forbids guessing on unreadable fields. Always validate extracted numbers against the source image — and never upload documents containing other people's personal data without authorization.

This guide is informational and not legal, medical, or financial advice. Vision models can misread blurry text, transpose digits, and confidently describe things that aren't there, so verify any extracted figure or diagnosis-like output against the original and, for high-stakes use, a licensed professional. Do not upload images containing PHI, PII, or client-confidential information to a consumer chatbot, and be mindful that an image may contain sensitive data (faces, addresses, account numbers) you didn't intend to share.