agent-web-tools - v0.3.0

agent-web-tools

Plug-and-play SDK that lets AI agents interact with any web page: navigate, fill forms, click, read content, and more. Works with any framework (React, Vue, Angular, Next.js, Nuxt, SvelteKit, Astro, vanilla) with zero configuration.

npm version License: MIT TypeScript Zero Dependencies

Features

18 DOM tools — snapshot, click, type, select, scroll, fill forms, and more
Advanced form handling — analyzes and fills complex forms including multi-step wizards, custom selects, date pickers, rich text editors, and file inputs
Page perception — spatial layout analysis, visual state detection, and action inventory from DOM data alone (zero API cost)
Session memory — per-session context tracking of visited pages, executed actions, and known elements
Smart intent analysis — tiered classifier with regex fast-path and LLM fallback for ambiguous commands
Action shortcuts — common operations (navigate, search, CRUD) execute immediately without LLM reasoning
Auto-discovery — automatically maps site structure via <nav>, sitemap.xml, links, JSON-LD, and meta tags
SPA navigation — native router integration for React, Vue, Angular, Next.js, Nuxt, SvelteKit, and Astro
Framework detection — identifies the frontend framework and router with confidence scores
3 transport options — WebSocket (with reconnection & heartbeat), postMessage (for iframes), and custom transports
Security built-in — tool allowlists, selector blocklists, rate limiting, user confirmation, and audit logging
Data providers — optional extension to connect agents with backend search/data APIs
0 production dependencies — fully self-contained (client SDK)
Tree-shakeable — import only what you need
TypeScript strict — complete type definitions included

Quick Start

Script Tag (UMD)

<script src="https://unpkg.com/agent-web-tools/dist/umd/agent-web-tools.min.js"></script>
<script>
  const dom = new AgentWebTools.AgentDOM();

  // Take a snapshot of the page
  const snapshot = dom.snapshot();
  console.log(snapshot.data);

  // Click a button
  dom.click({ selector: '#submit-btn' });

  // Fill a form
  dom.fillForm({
    selector: '#contact-form',
    data: {
      name: 'Jane Doe',
      email: 'jane@example.com',
      message: 'Hello from the agent!'
    }
  });
</script>

ES Modules / npm

npm install agent-web-tools

import { AgentDOM, FormHandler, SiteDiscovery } from 'agent-web-tools';

const dom = new AgentDOM();
const forms = new FormHandler();
const discovery = new SiteDiscovery();

// Snapshot the DOM
const { data: tree } = dom.snapshot({ interactableOnly: true });

// Type into an input (works with React/Vue controlled components)
dom.type({ selector: '#search', text: 'agent-web-tools', pressEnter: true });

// Analyze a form
const analysis = forms.analyzeForm('#checkout-form');
console.log(analysis.data.fields);

// Auto-discover site routes
const siteMap = await discovery.discover();
console.log(siteMap.data.routes);

Installation

# npm
npm install agent-web-tools

# pnpm
pnpm add agent-web-tools

# yarn
yarn add agent-web-tools

Or load directly from a CDN (served automatically from npm):

<!-- unpkg — latest version -->
<script src="https://unpkg.com/agent-web-tools"></script>

<!-- unpkg — pinned version (recommended for production) -->
<script src="https://unpkg.com/agent-web-tools@0.1.0/dist/umd/agent-web-tools.min.js"></script>

<!-- jsdelivr — latest version -->
<script src="https://cdn.jsdelivr.net/npm/agent-web-tools"></script>

<!-- jsdelivr — pinned version (recommended for production) -->
<script src="https://cdn.jsdelivr.net/npm/agent-web-tools@0.1.0/dist/umd/agent-web-tools.min.js"></script>

Both CDNs serve the package automatically once it's published to npm — no extra configuration needed.

Integration with external LLM backends (Direct Mode)

For agents that already run their own LLM (e.g. an Ocular Agent OS backend), the SDK can be driven directly without going through the bundled WebSocket server. This eliminates a second-LLM round trip per page action — your LLM emits structured tool calls, the SDK executes them locally via AgentDOM.

import { AgentBrain } from 'agent-web-tools';

const brain = new AgentBrain({ serverUrl: 'ws://unused-in-direct-mode' });

// 1. Per chat request, gather context for the remote LLM:
const init = brain.buildInitPayload();
// init.pageContext   – live PagePerception output
// init.domTools      – 18 web_* tools as ChatCompletionTool[] (OpenAI-shaped)
// init.routes        – discovered routes from SiteDiscovery
// init.currentUrl    – visitor's pathname
// init.framework     – detected SPA framework, if any
// init.sdkVersion    – '0.2.0+' for compat checks

await fetch('/your-agent-backend', {
  method: 'POST',
  body: JSON.stringify({ messages, ...init }),
});

// 2. When the backend streams a structured tool call back, run it:
const result = await brain.executeStructuredTool('web_click', {
  selector: '#cta-btn',
});
// result.success / result.data / result.error / result.durationMs

// 3. Ship the result to the backend in the NEXT request as a `role:'tool'`
//    message (standard OpenAI tool-calling protocol). The backend's LLM
//    sees it and continues its reasoning loop.

buildInitPayload() and executeStructuredTool() are pure local DOM — they never touch the network. The bundled server (under server/) remains available as a free-form natural-language fallback for cases where the backend agent doesn't have selectors handy. See CHANGELOG → 0.2.0.

API Reference

AgentDOM

The core class for interacting with page elements. Every method returns a ToolResult<T> with { success, data, error? }.

const dom = new AgentDOM(document?, window?);

Perception Tools

Method	Description
`snapshot(options?)`	Capture a structured tree of DOM elements
`readText(options)`	Read text content from an element
`inspect(selector)`	Get detailed info about a specific element
`findByText(options)`	Find elements containing specific text

// Snapshot with options
const tree = dom.snapshot({
  root: '#app',
  maxDepth: 5,
  includeHidden: false,
  interactableOnly: true,
});

// Read text
const text = dom.readText({ selector: '.article-body' });

// Inspect element
const info = dom.inspect('#login-button');
// => { tag, id, classes, attributes, computedStyles, rect, visible, interactable, ... }

// Find by text
const matches = dom.findByText({ text: 'Sign Up', exact: true });

Action Tools

Method	Description
`click(options)`	Click an element (left/right/middle, double-click, modifiers)
`type(options)`	Type text into inputs (supports React/Vue synthetic events)
`select(options)`	Select options in a `<select>` element
`check(options)`	Check or uncheck a checkbox/radio
`hover(options)`	Hover over an element
`focus(options)`	Focus an element
`scroll(options)`	Scroll the page or a specific element
`pressKey(options)`	Simulate a keyboard key press

// Click with modifiers
dom.click({ selector: '.item', button: 'left', modifiers: { ctrl: true } });

// Type into input (clears first by default)
dom.type({ selector: '#email', text: 'user@example.com' });

// Type and press Enter
dom.type({ selector: '#search', text: 'query', pressEnter: true });

// Select option
dom.select({ selector: '#country', value: 'Spain' });

// Check a checkbox
dom.check({ selector: '#agree-terms', checked: true });

// Scroll down
dom.scroll({ y: 500, behavior: 'smooth' });

// Press Escape
dom.pressKey({ key: 'Escape' });

Modification Tools

Method	Description
`setAttribute(options)`	Set an HTML attribute
`toggleClass(options)`	Toggle a CSS class
`setStyle(options)`	Set an inline CSS style

Form Tools

Method	Description
`submitForm(options)`	Submit a form element
`fillForm(options)`	Auto-fill a form by matching field names/labels to data

dom.fillForm({
  selector: '#checkout',
  data: {
    name: 'Juan Perez',
    email: 'juan@example.com',
    phone: '+34612345678',
    address: 'Calle Principal 123',
    city: 'Madrid',
    country: 'Spain',
    accept_terms: true,
  },
});
// => { success: true, data: { name: true, email: true, ... } }

Async Tools

Method	Description
`waitFor(options)`	Wait for an element to reach a state (`attached`, `detached`, `visible`, `hidden`)

const { data: promise } = dom.waitFor({
  selector: '.modal',
  state: 'visible',
  timeout: 5000,
});
await promise;

Dynamic Execution

// Execute any tool by name
const result = dom.execute('click', { selector: '#btn' });

Tool Descriptors

// Get schema of all tools (for handshake with LLM backends)
const tools = dom.getToolDescriptors();

FormHandler

Advanced form analysis and filling with support for complex field types.

const handler = new FormHandler(document?, window?);

Method	Description
`analyzeForm(selector)`	Analyze form structure, fields, steps, and validation errors
`fillForm(options)`	Fill form with intelligent field matching and optional auto-submit
`getValidationErrors(selector)`	Read all validation errors from a form
`goToStep(selector, index)`	Navigate to a specific step in a multi-step form

Supported field types: text, email, password, number, tel, url, select, multi-select, custom-select (React Select, Headless UI, etc.), checkbox, radio, date, datetime-local, time, custom-date (flatpickr, react-datepicker), file, textarea, rich-text (Quill, ProseMirror, CKEditor, TinyMCE, contentEditable), range, color.

// Analyze a form
const { data } = handler.analyzeForm('#registration');
console.log(data.fields);       // All detected fields
console.log(data.isMultiStep);  // true if wizard-style
console.log(data.steps);        // Step info for multi-step forms

// Fill with auto-submit
handler.fillForm({
  selector: '#checkout-form',
  data: {
    name: 'Jane Doe',
    email: 'jane@example.com',
    card_number: '4242424242424242',
  },
  submitAfterFill: true,
});

SiteDiscovery

Auto-discovers the site's structure and URL patterns.

const discovery = new SiteDiscovery(document?, window?);

Method	Description
`discover(options?)`	Run all discovery strategies and return a site map
`startWatching(callback)`	Watch for SPA navigation and re-discover on route change
`stopWatching()`	Stop watching for navigation changes
`getRoutes()`	Get the currently discovered routes
`getCurrentSiteMap()`	Get the full site map snapshot

Discovery strategies:

nav — parses <nav>, [role="navigation"], header, sidebar
sitemap — fetches /sitemap.xml
links — collects all internal <a href> on the current page
structured-data — parses JSON-LD (BreadcrumbList, SiteNavigationElement, etc.)
meta — reads canonical, alternate, hreflang tags

const { data: siteMap } = await discovery.discover({
  strategies: ['nav', 'links', 'structured-data'],
  maxLinks: 100,
  detectPatterns: true,
});

console.log(siteMap.routes);     // Discovered routes
console.log(siteMap.framework);  // Detected framework ('react', 'vue', etc.)
console.log(siteMap.patterns);   // URL patterns (e.g., /products/:slug)

// Watch for SPA navigation
discovery.startWatching((updatedMap) => {
  console.log('Routes changed:', updatedMap.routes.length);
});

RouterManager

Framework-aware SPA navigation with automatic framework detection.

const router = new RouterManager(config?);

Method	Description
`navigate(url)`	Navigate using the framework's native router
`back()`	Go back in history
`forward()`	Go forward in history
`getCurrentRoute()`	Get current route with params, query, hash
`onRouteChange(callback)`	Subscribe to route changes
`canNavigate(url)`	Check if a URL can be navigated to
`redetect()`	Re-detect framework and swap adapter if needed
`destroy()`	Clean up the router manager

Supported frameworks: React (react-router), Vue (vue-router), Angular, Next.js (App Router & Pages Router), Nuxt, SvelteKit, Astro (View Transitions), and a History API fallback.

const router = new RouterManager({
  onRouteChange: (route) => console.log('Navigated to:', route.path),
  onFrameworkDetected: (info) => console.log(`Detected ${info.framework} v${info.version}`),
});

console.log(router.framework); // 'react', 'vue', 'nextjs', etc.

await router.navigate('/dashboard');

const route = router.getCurrentRoute();
// => { path, fullPath, params, query, hash, meta }

Framework Detection

import { detectFramework, detectAllFrameworks } from 'agent-web-tools';

const detected = detectFramework();
// => { framework: 'nextjs', version: '14.2.0', router: 'app-router', confidence: 0.95 }

const all = detectAllFrameworks();
// => Sorted by confidence, useful when multiple frameworks coexist

Security

SecurityPolicy

Enforces security policies on tool execution.

const policy = new SecurityPolicy(config?);

Method	Description
`validate(tool, params)`	Validate a tool execution against all security checks
`logExecution(tool, params, success, duration, error?)`	Log execution result
`isToolAllowed(tool)`	Check if a tool is in the allowlist
`isSelectorBlocked(selector)`	Check if a selector matches a blocked pattern
`isRateLimited()`	Check if rate limit has been reached
`getRateLimitStatus()`	Get remaining actions and reset time
`getAuditLog()`	Get the full audit log
`clearAuditLog()`	Clear the audit log
`reset()`	Reset all internal state

const policy = new SecurityPolicy({
  allowedTools: ['snapshot', 'readText', 'click', 'type', 'fillForm'],
  blockedSelectors: ['.admin-panel', '#delete-account', '[data-sensitive]'],
  rateLimit: { maxActions: 30, windowMs: 60_000 },
  requireConfirmation: ['submitForm', 'fillForm'],
  confirmationHandler: async (request) => {
    return confirm(`Allow agent to ${request.tool} on ${request.selector}?`);
  },
  auditLog: true,
  onAuditEntry: (entry) => console.log('[Audit]', entry.tool, entry.action),
});

// Validate before executing
const blocked = await policy.validate('click', { selector: '.admin-panel' });
if (blocked) {
  console.error(blocked.error); // "Blocked by security policy: ..."
}

Transports

Three transport options for communicating between the SDK (browser) and your agent backend.

WebSocketTransport

Full-featured WebSocket with automatic reconnection and heartbeat.

import { WebSocketTransport } from 'agent-web-tools';

const ws = new WebSocketTransport({
  url: 'wss://agent-server.example.com/ws',
  token: 'auth-token-123',
  reconnect: true,
  maxRetries: 10,
  reconnectBaseDelay: 1000,
  reconnectMaxDelay: 30000,
  heartbeatInterval: 30000,
  heartbeatTimeout: 10000,
  protocols: ['agent-v1'],
});

ws.on('connected', () => console.log('Connected!'));
ws.on('disconnected', ({ reason }) => console.log('Disconnected:', reason));
ws.on('reconnecting', ({ attempt, maxRetries }) => console.log(`Reconnecting ${attempt}/${maxRetries}`));
ws.on('message', (msg) => console.log('Received:', msg));
ws.on('error', (err) => console.error('Error:', err));

await ws.connect();
ws.send(BaseTransport.createMessage('init', { tools: dom.getToolDescriptors(), pageInfo: { url: location.href, title: document.title } }));

PostMessageTransport

For embedding the SDK in iframes with secure origin verification.

import { PostMessageTransport } from 'agent-web-tools';

const pm = new PostMessageTransport({
  targetWindow: parent,
  targetOrigin: 'https://agent-host.example.com',
  sourceOrigin: 'https://my-app.example.com',
});

await pm.connect();
pm.send(BaseTransport.createMessage('init', { /* ... */ }));

CustomTransport

Integrate with any messaging system (WebRTC, SSE, message queues, etc.).

import { CustomTransport } from 'agent-web-tools';

const transport = new CustomTransport({
  send: (message) => myChannel.postMessage(JSON.stringify(message)),
  connect: () => myChannel.open(),
  disconnect: () => myChannel.close(),
});

await transport.connect();

// Feed incoming messages from your custom channel
myChannel.onMessage((raw) => transport.receive(raw));

Transport Protocol

The SDK uses a structured message protocol:

Message Type	Direction	Description
`init`	Client → Server	Handshake with tool descriptors + page info
`ack`	Server → Client	Session acknowledgment
`execute`	Server → Client	Tool execution request
`result`	Client → Server	Tool execution result
`page_changed`	Client → Server	SPA navigation event
`agent_message`	Server → Client	Agent text message
`agent_thinking`	Server → Client	Agent thinking/reasoning
`agent_partial`	Server → Client	Streaming text chunk (real-time)
`request_tools`	Server → Client	Re-request tool descriptors
`request_snapshot`	Server → Client	Request DOM snapshot
`disconnect`	Either	Graceful disconnect

DataProviderManager

Optional extension that connects the SDK with backend data APIs, dynamically adding data tools to the agent's toolset.

import { DataProviderManager } from 'agent-web-tools';
import type { DataProvider } from 'agent-web-tools';

const catalogProvider: DataProvider = {
  name: 'catalog',
  async search(query, options) {
    const res = await fetch(`/api/search?q=${query}&limit=${options?.limit ?? 10}`);
    const items = await res.json();
    return { success: true, data: items };
  },
  async getByUrl(url) {
    const res = await fetch(`/api/page?url=${encodeURIComponent(url)}`);
    const data = await res.json();
    return { success: true, data };
  },
};

const manager = new DataProviderManager({ providers: [catalogProvider] });
await manager.init();

// Dynamic tool descriptors (merge with AgentDOM tools for handshake)
const dataTools = manager.getToolDescriptors();
// => [{ name: 'data_search', ... }, { name: 'data_get_by_url', ... }]

// Execute data tools
const results = await manager.execute('data_search', { query: 'wireless headphones', limit: 5 });

Server Architecture: Perception, Memory & Intelligence

The server/ directory contains the WebSocket server that bridges the browser SDK with OpenAI. Beyond simple prompt relay, it implements a cognitive layer that gives the agent human-like perception, memory, and decision-making — all without screenshots or extra API calls.

┌─────────────────────────────────────────────────────────────┐
│                     Client (browser, cost = 0)              │
│                                                             │
│  PagePerception ──┬── LayoutAnalyzer                        │
│                   ├── VisualStateDetector                   │
│                   └── ActionInventory                       │
│                          │                                  │
│                     PageContext (compact JSON)               │
└──────────────────────────┬──────────────────────────────────┘
                           │ WebSocket
┌──────────────────────────▼──────────────────────────────────┐
│                        Server                               │
│                                                             │
│  IntentAnalyzer ──── regex fast-path + LLM fallback         │
│  ActionPatterns ──── shortcut execution (skip LLM)          │
│  SessionMemory  ──── per-session context (pages, actions)   │
│  ContextBuilder ──── optimized prompt assembly              │
│  LLMBridge      ──── model selection + streaming            │
└─────────────────────────────────────────────────────────────┘

Page Perception (client-side, zero cost)

The perception subsystem runs entirely in the browser, analyzing the existing DOM snapshot data (including rect coordinates) to produce a structured PageContext — the equivalent of a human "glancing" at a page and instantly understanding its layout, state, and available actions.

Files: src/perception/layout-analyzer.ts, src/perception/visual-state-detector.ts, src/perception/action-inventory.ts, src/perception/page-perception.ts

LayoutAnalyzer

Detects spatial zones using element positions (rect) and semantic landmarks — no LLM required:

Zone	Detection Heuristic
Header	`role="banner"`, `<header>`, or first block with `y < 100` and `width > 80%` viewport
Sidebar	`role="navigation"`, `<aside>`, with `height > 50%` viewport and `width < 30%` viewport
Main	`<main>`, `role="main"`, or largest area block after header/sidebar
Footer	`<footer>`, `role="contentinfo"`, or block near the document end
Modal	`role="dialog"`, `aria-modal="true"`, `class*="modal"` that is visible

interface LayoutAnalysis {
  viewport: { width: number; height: number };
  zones: LayoutZone[];          // detected spatial regions
  scrollPosition: { x: number; y: number };
  hasFixedHeader: boolean;
  hasSidebar: boolean;
  mainContentSelector: string;
}

VisualStateDetector

Instantly identifies visual states a human would notice:

State	Signals
Loading	`[class="spinner"]`, `[class="skeleton"]`, `role="progressbar"` with `aria-busy="true"`
Empty	`[class="empty"]`, `[class="no-results"]`, text "no hay resultados"
Error	`[role="alert"]`, `[class="error"]`, `[class="danger"]`
Notifications	`[role="status"]`, `[class="toast"]`, `[class="snackbar"]` (visible)
Modal	`role="dialog"`, `aria-modal="true"` (visible)
Selected Tab	`[aria-selected="true"]`, `[class*="active"]` within tab containers

ActionInventory

Scans the DOM and classifies every available action, so the LLM doesn't need to "search" for buttons:

interface ActionInventory {
  primary: AvailableAction[];    // CTAs, main buttons
  navigation: AvailableAction[]; // nav links, menu items
  forms: FormSummary[];          // forms with purpose + field count
  tableActions: AvailableAction[]; // per-row actions (edit, delete, view)
  filters: AvailableAction[];   // search, filter, sort controls
  pagination: PaginationInfo | null;
}

Action classification uses visible text, aria-label, and element context:

"crear"/"new"/"add" → create
"editar"/"edit" → edit
"eliminar"/"delete"/"remove" → delete
input[type="search"], [role="search"] → search
Links in <nav> → navigate

PageContext (combined output)

All sub-analyzers feed into a single PageContext with a human-readable summary:

Pagina: Dashboard de usuarios | Layout: header + sidebar + main
Vista: tabla CRUD con 15 filas | Estado: normal (sin loading/error)
Acciones: [Crear usuario] [Buscar] [Filtrar por rol] [Exportar]
Por fila: [Editar] [Eliminar] [Ver detalle]
Paginación: página 1 de 5

This replaces large portions of the raw DOM snapshot in prompts, giving the LLM structured context instead of making it parse HTML.

Smart Intent Analysis

File: server/src/intent-analyzer.ts

A tiered intent classifier that uses regex as a fast-path (~~0ms) and falls back to LLM only when confidence is low (~~10% of cases).

User command ──► Regex patterns (ES + EN) ──► confidence ≥ 0.7? ──► YES ──► return
                                                      │
                                                      NO
                                                      │
                                              LLM classify (gpt-4o-mini)
                                              ~200ms, ~$0.00003/call
                                                      │
                                                      ▼
                                                   return

Supported intents: navigate, fill_form, crud_create, crud_read, crud_update, crud_delete, cart_action, read_content, search, scroll_to, general

Each intent carries:

Confidence score (0–1) based on keyword-to-command length ratio
Target extraction (e.g., "elimina el usuario Juan" → target: "usuario Juan")
Expected mutation (e.g., crud_delete → element_removed, hint: "fila desaparece")

The mutation expectation is used downstream to verify the action succeeded.

const analyzer = new IntentAnalyzer();
analyzer.configure(OPENAI_API_KEY, 'gpt-4o-mini');

const result = await analyzer.analyze('Elimina el usuario Juan');
// => { intent: 'crud_delete', confidence: 0.95, target: 'usuario Juan',
//      expectedMutation: { type: 'element_removed', hint: 'fila/row/item desaparece' } }

Action Patterns (Shortcuts)

File: server/src/action-patterns.ts

Pre-computed patterns that map intent + page context to immediate tool calls, bypassing the LLM entirely for common operations. This saves ~1.5–2s on the first action.

Intent	Shortcut Condition	Action
`navigate`	Target matches a discovered route	`navigate(matched_route_path)`
`scroll_to`	Target text specified	`findByText(target)`
`search`	Search filter found in page context	`type(search_selector, target)`
`crud_create`	Create button found in page context	`click(create_button_selector)`
`crud_delete`	Delete button + target found in context	`findByText(target)` to locate row
`crud_read`	Target text specified	`findByText(target)`

When a shortcut fires, the LLM only enters on the continuation step (interpreting the result), not the first step. When no shortcut applies, the full LLM reasoning flow runs as normal.

import { tryActionShortcut } from './action-patterns.js';

const calls = tryActionShortcut(intentResult, pageContext, routes);
if (calls) {
  // Execute directly — no LLM call needed
  await processAction(session, { type: 'tool_calls', calls });
} else {
  // Fall back to LLM reasoning
  const { action } = await bridge.startConversation(...);
}

Session Memory

File: server/src/session-memory.ts

Lightweight per-session memory that tracks what the agent has seen and done, preventing re-discovery of known elements.

interface SessionMemory {
  pagesVisited: PageVisit[];        // max 10 (FIFO)
  lastActions: ActionRecord[];      // max 5 (FIFO)
  knownElements: Map<string, string>; // max 20 (LRU), label → selector
  discoveredRoutes: string[];       // max 50
}

Injected into prompts as:

═══ MEMORIA DE SESIÓN ═══
Páginas visitadas: /dashboard (table), /usuarios/crear (form)
Acciones recientes: click(#btn-new) [OK] → fill_form(#user-form) [OK] → navigate(/dashboard) [OK]
Elementos conocidos: formulario: #user-form, boton-crear: #btn-new

Lifecycle:

Created when a WebSocket session starts
Fed with every page visit, tool execution, and route discovery
Destroyed when the connection closes
~2KB per session, never shared between sessions

Auto-learning: when the agent clicks an element or fills a form, the memory automatically registers the selector with a human-readable label for future reference.

Context Builder

File: server/src/context-builder.ts

Centralizes all prompt construction, replacing ad-hoc string assembly. Produces compact, structured prompts by combining:

Page info — URL, title, framework
Perception block — PageContext summary, visual state warnings, available actions
Memory block — visited pages, recent actions, known elements
DOM snapshot — pruned, summarized tree (max 200 nodes)
Routes — discovered site routes with titles
Intent analysis — detected intent, target, expected mutation
User context — custom data for form filling

The resulting prompt is more compact because the perception block eliminates redundant information the LLM would otherwise have to extract from raw DOM.

const contextBuilder = new ContextBuilder();

const systemPrompt = contextBuilder.buildSystemMessage(siteType, intentResult);
const userMessage = contextBuilder.buildUserMessage({
  command: 'Elimina el usuario Juan',
  pageInfo: { url: '/dashboard', title: 'Users' },
  snapshotSummary: '...',
  routes: [...],
  intent: intentResult,
  pageContext: pageContext,
  memory: sessionMemory,
});

Optimized Flow

The complete flow for a typical command like "Elimina el usuario Juan":

1. Client: PagePerception analyzes DOM → PageContext          (0ms, browser)
2. Client → Server: command + PageContext via WebSocket
3. Server: IntentAnalyzer.analyze("Elimina el usuario Juan")  (0ms, regex match)
   → intent=crud_delete, confidence=0.95, target="usuario Juan"
4. Server: tryActionShortcut() checks patterns
   → shortcut found: findByText("usuario Juan")
5. Server → Client: execute findByText (no LLM call, saves ~2s)
6. Client → Server: result + updated snapshot
7. Server: SessionMemory.recordAction()
8. Server → LLM: continueWithResults (mini model, 512 tokens)
   → LLM sees the row, clicks delete button
9. ... confirmation modal → click confirm → verify deletion
10. Server → Client: "Listo, eliminé al usuario Juan"

Key optimizations:

Steps 1–5 involve zero LLM calls — perception + regex + shortcut handle the first action
Step 8 uses gpt-4o-mini with 512 max tokens for the continuation
Memory prevents re-discovering selectors on subsequent commands

Server Setup

cd server
cp .env.example .env
# Edit .env with your OpenAI API key
npm install
npm run dev

Environment variables:

Variable	Default	Description
`OPENAI_API_KEY`	—	Required. Your OpenAI API key
`OPENAI_MODEL`	`gpt-4o`	Primary model for complex reasoning
`PORT`	`3100`	WebSocket server port
`AGENT_API_KEY`	—	Optional API key for client authentication
`ALLOWED_ORIGINS`	`*`	Comma-separated allowed CORS origins
`RATE_LIMIT_MAX`	`20`	Max messages per window per IP
`RATE_LIMIT_WINDOW_MS`	`60000`	Rate limit window in milliseconds
`MAX_SESSIONS_PER_IP`	`3`	Max concurrent WebSocket sessions per IP

Scalability Considerations

Concern	Design Decision
Perception cost	Runs in the browser — zero server/API cost, parallelized with command send
Memory footprint	~2KB per session; max 10 pages, 5 actions, 20 elements; destroyed on disconnect
LLM call reduction	Action shortcuts skip LLM for common commands (navigate, search, create, delete)
Token efficiency	ContextBuilder produces ~40% smaller prompts vs raw DOM by using structured PageContext
Model selection	Simple intents use `gpt-4o-mini`; complex intents (forms, general) use the full model
Backward compatibility	If `PageContext` is absent (older client), server falls back to the snapshot-only flow
Rate limiting	Per-IP rate limiting + max sessions per IP prevent abuse
Context growth	`trimHistory()` keeps only the last 2 tool-call round-trips, preventing unbounded growth

Basic Commands (Natural Language)

The agent understands commands in Spanish and English. Type them in the chat widget or pass them to agent.run(). Here are examples for every common interaction:

Command	What it does
`Navega a la página de contacto`	Navigates using discovered routes
`Abre /products`	Navigates to a specific path
`Ve a la sección de precios`	Scrolls to the pricing section
`Regresa a la página anterior`	Goes back in history

Click & Interact

Command	What it does
`Haz click en el botón "Enviar"`	Clicks a button by its visible text
`Haz click en el enlace "Ver más"`	Clicks a link
`Cierra el modal`	Finds and clicks the close/X button
`Acepta las cookies`	Finds and clicks the cookie consent button
`Haz click en el menú hamburguesa`	Opens mobile navigation

Read Content

Command	What it does
`¿Qué dice la página sobre nosotros?`	Scrolls to the section and reads it
`Lee el contenido del artículo`	Reads text from the main article
`¿Cuál es el precio del producto?`	Finds and reads the price
`Muéstrame la información de contacto`	Scrolls to and reads contact info
`¿Cuántos items hay en la lista?`	Counts and reports list items

Search

Command	What it does
`Busca "laptop" en el sitio`	Finds the search field, types, and submits
`Busca productos de menos de $50`	Searches and reports results
`Encuentra el artículo sobre inteligencia artificial`	Locates content by topic

Forms

Command	What it does
`Llena el formulario de contacto`	Analyzes the form and fills it with user context data
`Escribe "juan@email.com" en el campo de email`	Types into a specific field
`Selecciona "España" en el campo de país`	Selects a dropdown option
`Marca la casilla de términos y condiciones`	Checks a checkbox
`Envía el formulario`	Submits the form
`Completa el formulario de registro con nombre Juan y email juan@test.com`	Fills specific fields

E-commerce / Cart

Command	What it does
`Agrega el primer producto al carrito`	Clicks "Add to cart" on the first product
`Agrega "Wireless Headphones" al carrito`	Finds the product and adds it
`Ve al carrito`	Navigates to the cart page
`¿Cuántos productos hay en el carrito?`	Reads the cart counter
`Elimina el segundo producto del carrito`	Removes an item from cart
`Compra el producto más barato`	Finds the cheapest and adds to cart
`Navega a checkout`	Goes to the checkout page

CRUD / Admin Panels

Command	What it does
`Crea un nuevo usuario con nombre "Ana" y email "ana@test.com"`	Opens create form, fills, and submits
`Edita el usuario "Carlos" y cambia su email a "carlos@new.com"`	Finds the row, clicks edit, updates
`Elimina el registro "Producto Demo"`	Finds, clicks delete, confirms in modal
`Muestra todos los usuarios`	Reads and lists table data
`Busca "pendiente" en la tabla`	Filters table by search

Theme & Appearance

Command	What it does
`Cambia a modo oscuro`	Finds and toggles the dark mode switch
`Activa el tema claro`	Switches to light theme
`Pon el modo oscuro / dark mode`	Toggles dark mode on
`Aumenta el tamaño de la fuente`	Finds and clicks the font size control
`Activa el modo de alto contraste`	Toggles high contrast / accessibility mode
`Cambia el color del tema a azul`	Selects a theme color option

Language & i18n

Command	What it does
`Cambia el idioma a inglés`	Finds the language selector and switches to English
`Switch language to Spanish`	Selects Spanish from the language dropdown
`Pon el sitio en francés`	Changes locale to French
`Selecciona "Português" en el selector de idioma`	Picks a specific language option

Authentication

Command	What it does
`Inicia sesión con usuario "admin" y contraseña "1234"`	Fills and submits the login form
`Haz login con email juan@test.com`	Fills email and submits
`Cierra sesión`	Finds and clicks the logout button
`Regístrate con nombre "Ana" y email "ana@test.com"`	Fills and submits signup form
`Haz click en "Olvidé mi contraseña"`	Navigates to password recovery

Tables & Data

Command	What it does
`Ordena la tabla por fecha`	Clicks the date column header to sort
`Filtra por estado "activo"`	Finds filter controls and applies status filter
`Ve a la siguiente página de la tabla`	Clicks the next page button
`Exporta los datos a CSV`	Clicks the export/download button
`Selecciona todas las filas`	Checks the "select all" checkbox
`Muestra 50 resultados por página`	Changes the page size selector

Dashboard & Analytics

Command	What it does
`Cambia el rango de fechas a "Últimos 7 días"`	Selects a date range preset
`Aplica el filtro de fecha desde 01/01/2025 hasta 31/03/2025`	Fills the date range picker
`Cambia a la vista de gráfico`	Switches from table to chart view
`Muéstrame las métricas de ventas`	Navigates to sales dashboard
`Descarga el reporte en PDF`	Clicks the PDF export button
`Compara con el período anterior`	Toggles comparison mode

Media & Video

Command	What it does
`Reproduce el video`	Clicks the play button
`Pausa la reproducción`	Clicks pause
`Pon el video en pantalla completa`	Clicks the fullscreen button
`Sube el volumen`	Adjusts the volume slider up
`Silencia el audio`	Clicks the mute button
`Avanza al minuto 5`	Seeks the video timeline

Command	What it does
`Dale like al post`	Clicks the like/heart button
`Comparte en redes sociales`	Clicks the share button
`Escribe un comentario: "Excelente artículo"`	Fills and submits the comment form
`Sigue a este usuario`	Clicks the follow button
`Envía un mensaje directo`	Opens and fills the DM form

Booking & Reservations

Command	What it does
`Reserva para el 15 de marzo a las 14:00`	Fills the booking form with date and time
`Selecciona 2 adultos y 1 niño`	Adjusts guest count selectors
`Elige la habitación doble`	Selects a room type
`Busca vuelos de Madrid a Barcelona`	Fills origin/destination and searches
`Selecciona el horario de las 10:00 AM`	Picks a time slot

Settings & Profile

Command	What it does
`Ve a configuración`	Navigates to the settings page
`Cambia mi nombre a "Carlos López"`	Updates the profile name field
`Desactiva las notificaciones por email`	Unchecks the notification toggle
`Actualiza mi foto de perfil`	Clicks the avatar upload button
`Guarda los cambios`	Clicks the save/update button
`Activa la autenticación de dos factores`	Toggles 2FA on

Notifications & Modals

Command	What it does
`Cierra la notificación`	Dismisses the notification/toast
`Acepta las cookies`	Clicks accept on the cookie banner
`Cierra el popup`	Finds and clicks close on the modal
`Marca todas las notificaciones como leídas`	Clicks "mark all as read"
`Descarta la alerta`	Dismisses the alert/banner

Content Management

Command	What it does
`Crea un nuevo artículo con título "Mi Post"`	Opens editor, fills title
`Publica el borrador`	Clicks the publish button
`Archiva este contenido`	Clicks archive/move to trash
`Cambia el estado a "Revisión"`	Updates the content status
`Sube una imagen`	Clicks the upload/media button
`Previsualiza el artículo`	Clicks the preview button

Maps & Location

Command	What it does
`Busca "restaurantes cerca de mí"`	Types into the map search field
`Muéstrame las indicaciones a Madrid`	Fills destination and gets directions
`Haz zoom en el mapa`	Clicks the zoom in control
`Cambia a vista satélite`	Switches the map layer

Scroll & Visibility

Command	What it does
`Baja hasta el footer`	Scrolls to the bottom of the page
`Muéstrame la sección de testimonios`	Scrolls to the testimonials section
`Scroll arriba`	Scrolls to the top

Programmatic Usage

All commands work both in the chat widget and via JavaScript:

// E-commerce flow
await agent.run('Agrega "Wireless Headphones" al carrito');
await agent.run('Ve al carrito');
await agent.run('Navega a checkout');
await agent.run('Llena el formulario de checkout');

// CRUD flow
await agent.run('Crea un nuevo producto con nombre "Widget Pro" y precio "29.99"');
await agent.run('Busca "Widget Pro" en la tabla');
await agent.run('Elimina "Widget Pro"');

// Content reading
const response = await agent.run('¿Cuál es el horario de atención?');
console.log(response); // "El horario de atención es de lunes a viernes, 9:00 a 18:00..."

// Theme & preferences
await agent.run('Cambia a modo oscuro');
await agent.run('Cambia el idioma a inglés');

// Authentication
await agent.run('Inicia sesión con email "admin@test.com" y contraseña "secret"');

// Dashboard
await agent.run('Cambia el rango de fechas a "Últimos 30 días"');
await agent.run('Descarga el reporte en PDF');

// Booking
await agent.run('Reserva para el 20 de abril a las 16:00');
await agent.run('Selecciona 2 adultos');

Build Formats

The SDK is distributed in multiple formats:

Format	Path	Use Case
ESM	`dist/esm/index.js`	Modern bundlers (Vite, webpack, Rollup)
CJS	`dist/cjs/index.js`	Node.js / CommonJS environments
UMD	`dist/umd/agent-web-tools.js`	Script tag, RequireJS
UMD (minified)	`dist/umd/agent-web-tools.min.js`	Production script tag

// package.json entry points
{
  "main": "dist/cjs/index.js",
  "module": "dist/esm/index.js",
  "browser": "dist/umd/agent-web-tools.min.js",
  "types": "dist/esm/index.d.ts"
}

Development

Prerequisites

Node.js >= 20.0.0
npm (or pnpm)

Scripts

# Build all formats
npm run build

# Run unit tests
npm test

# Run tests in watch mode
npm run test:watch

# Run tests with coverage
npm run test:coverage

# Run E2E tests (builds first)
npm run test:e2e

# Run E2E tests with UI
npm run test:e2e:ui

# Lint
npm run lint

# Format
npm run format

# Type check
npm run typecheck

# Clean build output
npm run clean

Project Structure

agent-web-tools/
  src/
    index.ts                    # Entry point — all public exports
    core/
      agent-dom.ts              # AgentDOM — 18 DOM interaction tools
      form-handler.ts           # FormHandler — advanced form analysis & filling
      types.ts                  # All TypeScript interfaces and types
    perception/                 # Page perception (runs in browser, zero cost)
      page-perception.ts        # Coordinator — produces PageContext
      layout-analyzer.ts        # Spatial zone detection from rect data
      visual-state-detector.ts  # Loading/empty/error/modal state detection
      action-inventory.ts       # Available actions classifier
    agent/
      agent-brain.ts            # Agent orchestration + perception integration
      agent-controller.ts       # High-level agent control
      console-ui.ts             # Console-based chat UI
    router/
      detector.ts               # Framework auto-detection
      router-manager.ts         # RouterManager orchestrator
      adapters/                 # Per-framework router adapters
        react-router.ts
        vue-router.ts
        angular-router.ts
        nextjs.ts
        nuxt.ts
        sveltekit.ts
        astro.ts
        fallback.ts             # History API fallback
    discovery/
      site-discovery.ts         # SiteDiscovery orchestrator
      strategies/               # Discovery strategies
        nav-elements.ts
        sitemap-xml.ts
        links.ts
        structured-data.ts
        meta-tags.ts
    transport/
      transport.ts              # Transport interface & BaseTransport
      websocket.ts              # WebSocket with reconnection
      post-message.ts           # postMessage for iframes
      custom.ts                 # Custom transport adapter
    security/
      policy.ts                 # SecurityPolicy
      confirmation.ts           # User confirmation system
    providers/
      data-provider.ts          # DataProviderManager
  server/
    src/
      index.ts                  # WebSocket server + message routing
      llm-bridge.ts             # OpenAI integration + streaming
      intent-analyzer.ts        # Tiered intent classification (regex + LLM)
      action-patterns.ts        # Shortcut patterns (skip LLM)
      session-memory.ts         # Per-session memory (pages, actions, elements)
      context-builder.ts        # Optimized prompt assembly
  tests/
    unit/                       # Vitest unit tests
    e2e/                        # Playwright E2E tests
      apps/                     # Test applications (vanilla, react, vue)
  examples/                     # Usage examples
    basic/                      # Minimal script tag example
    react/                      # React integration example
    with-data-provider/         # Data provider example
  dist/                         # Build output (ESM, CJS, UMD)

Publishing

This project uses semantic versioning. The package is published to npm manually, and CDN availability via unpkg and jsdelivr is automatic.

Prerequisites

An npmjs.com account with 2FA enabled
Run npm login before your first publish

Release Commands

# Dry run — preview what would be published without actually publishing
npm run release:dry

# Patch release (0.1.0 → 0.1.1) — bug fixes
npm run release:patch

# Minor release (0.1.0 → 0.2.0) — new features, backwards compatible
npm run release:minor

# Major release (0.1.0 → 1.0.0) — breaking changes
npm run release:major

# Check bundle size
npm run size

Each release:* command runs the full pipeline automatically:

npm run lint + npm test (via prerelease)
npm version (bumps version in package.json, creates git tag)
npm run build (clean + rollup, via prepublishOnly)
npm run typecheck (via prepublishOnly)
npm publish (pushes to npm registry)

After publishing, update the CHANGELOG.md with the new version's changes and push the version commit + tag:

git push && git push --tags

Conventional Commits

Use conventional commit messages for clear changelogs:

Prefix	When to use	Version bump
`feat:`	New feature	minor
`fix:`	Bug fix	patch
`perf:`	Performance improvement	patch
`refactor:`	Code change (no new feature/fix)	—
`docs:`	Documentation only	—
`test:`	Adding/updating tests	—
`chore:`	Maintenance, dependencies	—
`BREAKING CHANGE:`	Breaking change (in body/footer)	major

Browser Support

The SDK targets ES2020 and works in all modern browsers:

Chrome/Edge 80+
Firefox 80+
Safari 14+

License

MIT

agent-web-tools - v0.3.0

agent-web-tools

Features

Quick Start

Script Tag (UMD)

ES Modules / npm

Installation

Integration with external LLM backends (Direct Mode)

API Reference

AgentDOM

Perception Tools

Action Tools

Modification Tools

Form Tools

Async Tools

Dynamic Execution

Tool Descriptors

FormHandler

SiteDiscovery

RouterManager

Framework Detection

Security

SecurityPolicy

Transports

WebSocketTransport

PostMessageTransport

CustomTransport

Transport Protocol

DataProviderManager

Server Architecture: Perception, Memory & Intelligence

Page Perception (client-side, zero cost)

LayoutAnalyzer

VisualStateDetector

ActionInventory

PageContext (combined output)

Smart Intent Analysis

Action Patterns (Shortcuts)

Session Memory

Context Builder

Optimized Flow

Server Setup

Scalability Considerations

Basic Commands (Natural Language)

Navigation

Click & Interact

Read Content

Search

Forms

E-commerce / Cart

CRUD / Admin Panels

Theme & Appearance

Language & i18n

Authentication

Tables & Data

Dashboard & Analytics

Media & Video

Social & Community

Booking & Reservations

Settings & Profile

Notifications & Modals

Content Management

Maps & Location

Scroll & Visibility

Programmatic Usage

Build Formats

Development

Prerequisites

Scripts

Project Structure

Publishing

Prerequisites

Release Commands

Conventional Commits

Browser Support

License

Settings

On This Page