Building Agents with Vision in Astreus

Modern AI agents can process images just as naturally as they process text. Astreus makes it simple to build vision-enabled agents that analyze visual content, from UI screenshots to data visualizations.

First, install Astreus and set up your environment. You'll need an OpenAI API key with access to vision-capable models.

BashCopy
npm install @astreus-ai/astreus

Create a .env file with your configuration:

OPENAI_API_KEY=sk-your-openai-api-key-here
DB_URL=sqlite://./astreus.db

Enable vision capabilities by setting the vision flag and specifying a vision-capable model. The gpt-4o model provides strong multimodal performance.

JavaScriptCopy
import { Agent } from '@astreus-ai/astreus';

const agent = await Agent.create({
  name: 'VisionBot',
  model: 'gpt-4o',
  visionModel: 'gpt-4o',
  vision: true,
  systemPrompt: 'You can analyze and describe images in detail.'
});

The visionModel parameter specifies which model handles image processing. Using the same model for both text and vision ensures consistent behavior.

Pass images to your agent using the attachments parameter. The agent processes both the text prompt and the image together.

JavaScriptCopy
const result = await agent.ask(
  "Analyze this image and describe what you see",
  {
    attachments: [{
      type: 'image',
      path: './screenshot.png'
    }]
  }
);

console.log(result); // Returns detailed image analysis
more

The agent examines the image and provides a detailed description. You can ask specific questions to focus the analysis on particular aspects.

Vision agents excel at evaluating interface designs. They spot inconsistencies, accessibility issues, and usability problems in mockups and screenshots.

JavaScriptCopy
const designAgent = await Agent.create({
  name: 'UIReviewer',
  model: 'gpt-4o',
  visionModel: 'gpt-4o',
  vision: true,
  systemPrompt: `Analyze UI designs for:
  - Visual hierarchy and balance
  - Color contrast (WCAG standards)
  - Spacing consistency
  - Responsive design considerations`
});

const review = await designAgent.ask(
  'Evaluate this mobile app screen for accessibility issues.',
  {
    attachments: [{
      type: 'image',
      path: './app-screen.png'
    }]
  }
);
more

The system prompt guides how the agent approaches visual analysis. Tailor it to your specific use case for more relevant feedback.

Share error screenshots instead of transcribing stack traces. The agent reads error messages, identifies issues, and suggests fixes.

JavaScriptCopy
const debugAgent = await Agent.create({
  name: 'DebugHelper',
  model: 'gpt-4o',
  visionModel: 'gpt-4o',
  vision: true,
  systemPrompt: 'You are a senior developer helping debug code issues.'
});

const diagnosis = await debugAgent.ask(
  'This error appeared when I tried to save. What is wrong?',
  {
    attachments: [{
      type: 'image',
      path: './error-screenshot.png'
    }]
  }
);
more

This natural workflow accelerates debugging. The agent examines the entire error context visible in the screenshot, often catching details you might miss when manually transcribing.

Extract structured data from invoices, receipts, and forms without complex OCR pipelines. Vision agents understand document layout contextually.

JavaScriptCopy
const extractionAgent = await Agent.create({
  name: 'DataExtractor',
  model: 'gpt-4o',
  visionModel: 'gpt-4o',
  vision: true
});

const data = await extractionAgent.ask(
  'Extract invoice details as JSON: number, date, total, items.',
  {
    attachments: [{
      type: 'image',
      path: './invoice.png'
    }]
  }
);
more

The agent recognizes field types and relationships, handling variations in format. It returns clean structured data ready for processing in your application.

Analyze data visualizations to extract insights and trends. The agent interprets visual encodings like color, position, and size.

JavaScriptCopy
const analysis = await agent.ask(
  'Summarize the key trends shown in this sales chart.',
  {
    attachments: [{
      type: 'image',
      path: './sales-chart.png'
    }]
  }
);

This works even when underlying data isn't available. The agent reads values from axes, identifies patterns, and highlights outliers directly from the visualization.

Process multiple images simultaneously for side-by-side comparison. Pass multiple attachments in a single request.

JavaScriptCopy
const comparison = await agent.ask(
  'Compare these two designs. Which has better visual hierarchy?',
  {
    attachments: [
      { type: 'image', path: './design-v1.png' },
      { type: 'image', path: './design-v2.png' }
    ]
  }
);

The agent analyzes both images together, identifying specific differences and their impact. This enables sophisticated before-after analysis and A/B testing evaluation.

Once you've built your vision agent, run it in your development environment:

BashCopy
npm run dev

The complete example repository is available at astreus-ai/agent-with-vision on GitHub. Clone it to explore the full implementation and experiment with different use cases.

Understanding the configuration options helps you optimize your vision agents:

• name: Agent identifier for tracking and debugging
• model: Primary language model (gpt-4o recommended for vision)
• visionModel: Vision-specific model, typically matches the primary model
• vision: Boolean flag enabling image processing capabilities
• systemPrompt: Instructions that guide agent behavior and analysis approach
• attachments: Array of image references with type and path properties

Astreus supports multiple ways to provide images. Use local file paths for the most straightforward approach:

JavaScriptCopy
attachments: [{
  type: 'image',
  path: '/absolute/path/to/image.png'
}]

Relative paths work too, resolved from your project directory. Choose the method that fits your workflow and file organization.

Specific prompts produce better results. Provide context about the image type and what aspects you want analyzed.

JavaScriptCopy
// Generic (less effective)
await agent.ask('What do you see?', { attachments: [...] });

// Specific (more effective)
await agent.ask(
  'This is a mobile checkout screen. Identify any usability issues that might prevent users from completing their purchase.',
  { attachments: [...] }
);

Frame questions around specific concerns or goals. Mention the target audience or use case to help the agent apply appropriate criteria in its analysis.

Vision-enabled agents unlock powerful workflows across many domains:

Design & UX: Automated design review, accessibility audits, consistency checking across pages and components.

Development: Visual debugging from screenshots, code review from presentation slides, architecture diagram analysis.

Data Processing: Invoice and receipt processing, form data extraction, chart and graph interpretation.

Quality Assurance: Visual regression testing, screenshot comparison, UI compliance verification.

Image processing consumes more tokens than text-only interactions. Start with moderate resolution images and increase quality only when the agent misses important details. Balance quality with cost efficiency for your specific use case.

The gpt-4o model provides strong vision capabilities with reasonable token usage. Monitor your usage patterns to optimize for your workload.

Create domain-specific agents by tailoring system prompts. This focuses analysis on relevant criteria for your use case.

JavaScriptCopy
// E-commerce product reviewer
const productAgent = await Agent.create({
  name: 'ProductReviewer',
  model: 'gpt-4o',
  visionModel: 'gpt-4o',
  vision: true,
  systemPrompt: `Evaluate product photos for e-commerce.
  Check: lighting quality, background cleanliness, product visibility,
  angle appropriateness, color accuracy indicators.`
});

// Medical diagram analyzer
const medicalAgent = await Agent.create({
  name: 'MedicalAnalyzer',
  model: 'gpt-4o',
  visionModel: 'gpt-4o',
  vision: true,
  systemPrompt: `Analyze medical diagrams and charts.
  Focus on: anatomical accuracy, labeling clarity, educational value.`
});
more

Specialized agents provide more relevant insights because they apply domain-appropriate evaluation criteria. The system prompt acts as their expertise and guides their analytical approach.

Start with simple tasks like image description to build intuition. Experiment with prompt phrasing to understand how different approaches affect output quality and focus.

As you gain experience, combine vision with other Astreus capabilities. Build specialized agents for your specific visual analysis needs. The example repository at astreus-ai/agent-with-vision provides a solid foundation to explore and extend.

Vision capabilities open up entirely new interaction patterns. Agents that can see bridge the gap between human visual communication and AI processing, enabling more natural and powerful workflows.

This experiment is written for Astreus v0.5.37. Please ensure you are using a compatible version.