1. Workflow Architecture
+-------------------+ +-------------------+ +--------------------+
| | | | | |
| Cypress E2E Test | ---> | Capture Targeted | ---> | Save Screenshots |
| (Navigate & Act) | | Screenshot | | to Local / Artifact |
| | | | | |
+-------------------+ +-------------------+ +--------------------+
|
v
+-------------------+ +-------------------+ +--------------------+
| | | | | |
| Fail Build or | <--- | AI Agent Generates| <--- | Cypress Node Task |
| Post Pull Request | | Structured UI | | Sends Image + |
| Review Comment | | Inspection Report | | Prompt to LLM API |
+-------------------+ +-------------------+ +--------------------+
2. Prerequisites
Ensure your environment includes:
- Node.js v18+
- Cypress (
npm install cypress --save-dev) - Access to a multimodal AI API key (e.g., OpenAI API or Anthropic API)
3. Step 1: Configure Cypress Tasks for AI Inspection
Cypress runs tests inside a browser sandbox, so API requests that interact with external file systems or require private API keys should run in Node.js via Cypress Tasks.
Edit cypress.config.js to define a custom task that sends captured screenshots to the AI Vision API:
// cypress.config.js
const { defineConfig } = require("cypress");
const fs = require("fs");
const path = require("path");
const OpenAI = require("openai");
module.exports = defineConfig({
e2e: {
setupNodeEvents(on, config) {
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
on("task", {
async analyzeUiScreenshot({ imagePath, instruction, baselinePath = null }) {
const absolutePath = path.resolve(imagePath);
if (!fs.existsSync(absolutePath)) {
throw new Error(`Screenshot not found at ${absolutePath}`);
}
// Convert image to base64
const base64Image = fs.readFileSync(absolutePath, { encoding: "base64" });
const messages = [
{
role: "system",
content: `You are an expert Senior UI/UX QA Automation Engineer.
Your job is to inspect UI screenshots for bugs, layout issues, text truncation, overlap, contrast problems, and unintended visual changes.`
},
{
role: "user",
content: [
{
type: "text",
text: instruction || "Review this UI screenshot for visual bugs, alignment issues, and text overlap."
},
{
type: "image_url",
image_url: {
url: `data:image/png;base64,${base64Image}`
}
}
]
}
];
// If a baseline image is provided for visual regression
if (baselinePath && fs.existsSync(path.resolve(baselinePath))) {
const base64Baseline = fs.readFileSync(path.resolve(baselinePath), { encoding: "base64" });
messages[1].content.push(
{ type: "text", text: "Here is the baseline/expected image for comparison:" },
{ type: "image_url", image_url: { url: `data:image/png;base64,${base64Baseline}` } }
);
}
try {
const response = await openai.chat.completions.create({
model: "gpt-4o",
messages: messages,
response_format: {
type: "json_schema",
json_schema: {
name: "ui_review_report",
schema: {
type: "object",
properties: {
passed: { type: "boolean" },
visual_score: { type: "number", description: "1-10 rating" },
defects_found: {
type: "array",
items: {
type: "object",
properties: {
severity: { type: "string", enum: ["CRITICAL", "WARNING", "INFO"] },
description: { type: "string" },
location: { type: "string" }
},
required: ["severity", "description"]
}
},
summary: { type: "string" }
},
required: ["passed", "visual_score", "defects_found", "summary"]
}
}
}
});
return JSON.parse(response.choices[0].message.content);
} catch (error) {
console.error("AI Analysis Failed:", error);
return { passed: false, summary: "Failed to perform AI analysis due to API error." };
}
}
});
}
}
});
4. Step 2: Create Custom Cypress Commands
To keep test specs clean, create a custom Cypress command cy.reviewUiWithAgent() in cypress/support/commands.js.
// cypress/support/commands.js
Cypress.Commands.add("reviewUiWithAgent", { prevSubject: 'optional' }, (subject, fileName, options = {}) => {
const screenshotName = fileName || `ui_check_${Date.now()}`;
const screenshotPath = `cypress/screenshots/<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi>C</mi><mi>y</mi><mi>p</mi><mi>r</mi><mi>e</mi><mi>s</mi><mi>s</mi><mi mathvariant="normal">.</mi><mi>s</mi><mi>p</mi><mi>e</mi><mi>c</mi><mi mathvariant="normal">.</mi><mi>n</mi><mi>a</mi><mi>m</mi><mi>e</mi></mrow><mi mathvariant="normal">/</mi></mrow><annotation encoding="application/x-tex">{Cypress.spec.name}/</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0715em;">C</span><span class="mord mathnormal" style="margin-right:0.0359em;">y</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">ess</span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">p</span><span class="mord mathnormal">ec</span><span class="mord">.</span><span class="mord mathnormal">nam</span><span class="mord mathnormal">e</span></span><span class="mord">/</span></span></span></span>{screenshotName}.png`;
// Capture screenshot (chainable from element or full page)
if (subject) {
cy.wrap(subject).screenshot(screenshotName, { overwrite: true });
} else {
cy.screenshot(screenshotName, { overwrite: true });
}
// Execute AI analysis task
cy.task("analyzeUiScreenshot", {
imagePath: screenshotPath,
instruction: options.instruction || "Check this UI component for layout bugs, cut-off text, or overlap.",
baselinePath: options.baselinePath || null
}).then((report) => {
// Log output formatted to Cypress Command Log
Cypress.log({
name: "AI UI Review",
message: `Score: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi>r</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>r</mi><mi>t</mi><mi mathvariant="normal">.</mi><mi>v</mi><mi>i</mi><mi>s</mi><mi>u</mi><mi>a</mi><msub><mi>l</mi><mi>s</mi></msub><mi>c</mi><mi>o</mi><mi>r</mi><mi>e</mi></mrow><mi mathvariant="normal">/</mi><mn>10</mn><mi mathvariant="normal">∣</mi><mi>P</mi><mi>a</mi><mi>s</mi><mi>s</mi><mi>e</mi><mi>d</mi><mo>:</mo></mrow><annotation encoding="application/x-tex">{report.visual_score}/10 | Passed:</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.0278em;">or</span><span class="mord mathnormal">t</span><span class="mord">.</span><span class="mord mathnormal" style="margin-right:0.0359em;">v</span><span class="mord mathnormal">i</span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">a</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0197em;">l</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.0197em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="katex-sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">s</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.0278em;">cor</span><span class="mord mathnormal">e</span></span><span class="mord">/10∣</span><span class="mord mathnormal" style="margin-right:0.1389em;">P</span><span class="mord mathnormal">a</span><span class="mord mathnormal">sse</span><span class="mord mathnormal">d</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span></span></span></span>{report.passed}`,
consoleProps: () => report
});
// Assert build pass/fail based on AI response
if (options.failOnDefect && !report.passed) {
const defectsStr = report.defects_found
.map(d => `[<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi>d</mi><mi mathvariant="normal">.</mi><mi>s</mi><mi>e</mi><mi>v</mi><mi>e</mi><mi>r</mi><mi>i</mi><mi>t</mi><mi>y</mi></mrow><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">{d.severity}]</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal">d</span><span class="mord">.</span><span class="mord mathnormal">se</span><span class="mord mathnormal" style="margin-right:0.0359em;">v</span><span class="mord mathnormal" style="margin-right:0.0278em;">er</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.0359em;">y</span></span><span class="mclose">]</span></span></span></span>{d.description} (Location: ${d.location || 'N/A'})`)
.join("\n");
throw new Error(`AI Agent found UI defects:\n${defectsStr}`);
}
return cy.wrap(report);
});
});
5. Step 3: Writing Cypress Spec Files
Now write test files that navigate your app, trigger interaction states (e.g., dark mode toggle, mobile drawer open, dynamic data render), and hand the inspection over to the AI agent.
// cypress/e2e/dashboard_visual_review.cy.js
describe("Dashboard - AI-Driven Visual Review", () => {
beforeEach(() => {
cy.viewport(1280, 720);
cy.visit("/dashboard");
});
it("should verify responsive layout on mobile viewport", () => {
cy.viewport("iphone-x");
// Open mobile sidebar navigation
cy.get("[data-testid='mobile-menu-btn']").click();
// Take screenshot and ask AI agent to analyze specifically for mobile drawer issues
cy.get("[data-testid='sidebar-menu']").reviewUiWithAgent("mobile_sidebar_open", {
instruction: "Verify that all menu items fit within the drawer without horizontal scrollbars or overlapping icons.",
failOnDefect: true
});
});
it("should check data table rendering with dynamic length text", () => {
// Populate dynamic mock data with long string values
cy.intercept("GET", "/api/users", {
body: [
{ id: 1, name: "Alexander Montgomery-Wellington III", status: "Active" },
{ id: 2, name: "Short Name", status: "Pending Approval with Escalated Priority" }
]
});
cy.reload();
// Allow AI agent to verify text truncations and table row heights
cy.get(".user-table").reviewUiWithAgent("user_table_long_text", {
instruction: "Check if the table handles long user names cleanly without pushing status badges out of bounds.",
failOnDefect: false
}).then((report) => {
expect(report.visual_score).to.be.at.least(7);
});
});
});
6. Real-World Advantages vs Standard Pixel Diffing
| Capability | Pixel Diffing (Percy / BackstopJS) | Cypress + AI Vision Agent |
|---|---|---|
| Dynamic Content Handling | Fails due to changing timestamps, avatars, or dynamic values. | Ignores benign content changes; evaluates structural correctness. |
| Flakiness | High (font rendering, sub-pixel differences). | Low (semantic review focuses on layout integrity). |
| Contextual Understanding | None (only compares RGB channels). | High (understands UI patterns, responsiveness, and usability). |
| Maintenance | Requires constantly updating baseline images. | Requires minimal baseline maintenance; prompt guides expected behavior. |
7. Production Best Practices
- Mask Dynamic Dates & User Avatars (Optional): Use
cy.screenshot('name', { blackout: ['.user-avatar', '.timestamp'] })if you want the agent to ignore volatile DOM elements. - Structured JSON Output: Always enforce structured JSON responses (using JSON Schema or Tool Calls) so Cypress assertions can programmatically pass or fail builds based on threshold scores or defect severities.
- Set Viewport Explicitly: Always call
cy.viewport()before capturing screenshots to ensure deterministic dimensions. - CI Artifact Storage: Upload the screenshot folder and the generated JSON reports as build artifacts in GitHub Actions or GitLab CI for auditability when tests fail.