Desktop Automation

Control full desktop environments - click, type, read the screen, and automate native applications. Whether you need to automate a Windows app, interact with a Linux GUI, or control your own machine remotely, BCTRL provides a unified API.

Choose Your Environment

Environment	Best For	AI Control	Browser Integration
Cloud Windows	Windows apps, Office automation	Yes	Yes
Cloud Linux	Linux GUIs, headless with display	Yes	Yes
Local Desktop	Your own machines, internal networks	Yes	Yes

I need a Windows desktop

Use Cloud Windows. Get a full Windows VM with pre-installed software, perfect for Office automation, Windows-only apps, and testing.

I need a Linux desktop

Use Cloud Linux. Get a Linux VM with a display server, ideal for Linux GUI apps and cross-platform testing.

I want to control my own machine

Use Local Desktop. Install the BCTRL agent on your machine and control it remotely - perfect for internal networks and air-gapped systems.

Connect to a Desktop

Cloud Desktop

Windows
Linux

import { bctrl } from '@bctrl/sdk';

const desktop = await bctrl.desktop.connect({
  apiKey: process.env.BCTRL_API_KEY,
  os: 'windows',
  options: {
    screen: { width: 1920, height: 1080 }
  }
});

import { bctrl } from '@bctrl/sdk';

const desktop = await bctrl.desktop.connect({
  apiKey: process.env.BCTRL_API_KEY,
  os: 'linux',
  options: {
    screen: { width: 1920, height: 1080 }
  }
});

Local Desktop

Connect to your own machine with the BCTRL agent installed:

import { bctrl } from '@bctrl/sdk';

const desktop = await bctrl.desktop.connectLocal({
  apiKey: process.env.BCTRL_API_KEY,
  agentId: 'your-agent-id'  // From bctrl agent register
});

Local Agent Setup

Learn how to install and configure the BCTRL agent on your machines

Mouse Control

Control the mouse cursor with pixel-perfect precision:

// Move cursor
await desktop.mouse.move(500, 300);
await desktop.mouse.move(500, 300, { duration: 500 });  // Smooth movement over 500ms

// Click actions
await desktop.mouse.click();                      // Left click at current position
await desktop.mouse.click(500, 300);              // Left click at coordinates
await desktop.mouse.doubleClick();                // Double click
await desktop.mouse.doubleClick(500, 300);        // Double click at coordinates
await desktop.mouse.rightClick();                 // Right click
await desktop.mouse.rightClick(500, 300);         // Right click at coordinates

// Drag and drop
await desktop.mouse.drag(100, 100, 400, 400);     // Drag from (100,100) to (400,400)

// Scrolling
await desktop.mouse.scroll(3);                    // Scroll down 3 units
await desktop.mouse.scroll(-3);                   // Scroll up 3 units
await desktop.mouse.scroll(3, { x: 500, y: 300 }); // Scroll at specific position

Mouse Button Options

// Specify button
await desktop.mouse.click(500, 300, { button: 'left' });
await desktop.mouse.click(500, 300, { button: 'right' });
await desktop.mouse.click(500, 300, { button: 'middle' });

// Click and hold
await desktop.mouse.down();
await desktop.mouse.move(600, 400);
await desktop.mouse.up();

Keyboard Control

Type text and send key combinations:

// Type text
await desktop.keyboard.type('Hello World');
await desktop.keyboard.type('Hello World', { delay: 50 });  // 50ms between keystrokes

// Press single keys
await desktop.keyboard.press('Enter');
await desktop.keyboard.press('Tab');
await desktop.keyboard.press('Escape');
await desktop.keyboard.press('Backspace');
await desktop.keyboard.press('Delete');

// Arrow keys
await desktop.keyboard.press('ArrowUp');
await desktop.keyboard.press('ArrowDown');
await desktop.keyboard.press('ArrowLeft');
await desktop.keyboard.press('ArrowRight');

// Function keys
await desktop.keyboard.press('F1');
await desktop.keyboard.press('F5');
await desktop.keyboard.press('F11');

Keyboard Shortcuts

Use hotkey combinations for common operations:

// Common shortcuts
await desktop.keyboard.hotkey('ctrl', 'c');        // Copy
await desktop.keyboard.hotkey('ctrl', 'v');        // Paste
await desktop.keyboard.hotkey('ctrl', 'x');        // Cut
await desktop.keyboard.hotkey('ctrl', 'z');        // Undo
await desktop.keyboard.hotkey('ctrl', 'a');        // Select all
await desktop.keyboard.hotkey('ctrl', 's');        // Save

// Multi-key combinations
await desktop.keyboard.hotkey('ctrl', 'shift', 's');  // Save as
await desktop.keyboard.hotkey('alt', 'Tab');          // Switch windows
await desktop.keyboard.hotkey('ctrl', 'alt', 'Delete'); // Task manager (Windows)
await desktop.keyboard.hotkey('super', 'd');          // Show desktop (Windows)

// Windows-specific
await desktop.keyboard.hotkey('win', 'r');         // Run dialog
await desktop.keyboard.hotkey('win', 'e');         // Open Explorer

// macOS-style (when applicable)
await desktop.keyboard.hotkey('cmd', 'space');     // Spotlight

Key Hold and Release

For complex interactions:

// Hold modifier keys
await desktop.keyboard.down('Shift');
await desktop.keyboard.press('ArrowDown');
await desktop.keyboard.press('ArrowDown');
await desktop.keyboard.press('ArrowDown');
await desktop.keyboard.up('Shift');
// Selected 3 lines of text

// Or use modifiers with type
await desktop.keyboard.down('Ctrl');
await desktop.keyboard.type('test');  // Types while Ctrl is held
await desktop.keyboard.up('Ctrl');

Screen Access

Capture screenshots of the entire screen or specific regions:

// Full screenshot
const screenshot = await desktop.screenshot();
// Returns: { base64: string, width: number, height: number }

// Save to file
await desktop.screenshot({ path: 'desktop.png' });

// Screenshot specific region
const region = await desktop.screenshot({
  region: { x: 100, y: 100, width: 500, height: 400 }
});

// Get as buffer
const buffer = await desktop.screenshot({ encoding: 'buffer' });

// Get dimensions
const { width, height } = await desktop.getScreenSize();
console.log(`Screen: ${width}x${height}`);

Using Screenshots for Verification

// Take screenshot after action
await desktop.mouse.click(500, 300);
await new Promise(r => setTimeout(r, 1000));  // Wait for UI to update

const screenshot = await desktop.screenshot();
// Send to AI for verification or save for debugging

AI Desktop Control (CUA)

Use AI-powered Computer Use Agents (CUA) to control the desktop with natural language:

Simple Commands

// Open applications
await desktop.cua.run('Open Chrome');
await desktop.cua.run('Open Notepad');
await desktop.cua.run('Open File Explorer');

// Navigate
await desktop.cua.run('Go to google.com in the browser');
await desktop.cua.run('Find the Downloads folder and open it');

// Interact
await desktop.cua.run('Click the Start menu');
await desktop.cua.run('Type "calculator" in the search box and press Enter');

Multi-Step Tasks

await desktop.cua.run(`
  Open Microsoft Word,
  Create a new document,
  Type "Hello World" as the title,
  Save the document as "test.docx" on the Desktop
`);

Complex Workflows

const result = await desktop.cua.run(`
  1. Open Chrome and go to gmail.com
  2. Click Compose
  3. Enter recipient: [email protected]
  4. Subject: Weekly Report
  5. Type a brief message about project status
  6. Click Send
`, {
  maxSteps: 20,
  timeout: 120000
});

if (result.success) {
  console.log('Email sent successfully!');
} else {
  console.log('Failed:', result.message);
}

CUA Options

const result = await desktop.cua.run('Complete task description', {
  maxSteps: 15,          // Maximum actions to take
  timeout: 60000,        // Timeout in milliseconds
  modelName: 'gpt-4o',   // AI model to use
  screenshot: true       // Include screenshots in reasoning
});

console.log(result.success);   // boolean
console.log(result.message);   // Summary of what happened
console.log(result.steps);     // Array of actions taken

Combining Browser + Desktop

Use browser automation inside a desktop environment for hybrid workflows:

Browser Inside Desktop

import { bctrl, playwright } from '@bctrl/sdk';

async function hybridAutomation() {
  // Connect to desktop
  const desktop = await bctrl.desktop.connect({
    apiKey: process.env.BCTRL_API_KEY,
    os: 'windows'
  });

  // Use AI to open browser
  await desktop.cua.run('Open Chrome and navigate to google.com');

  // Wait for browser to be ready
  await new Promise(r => setTimeout(r, 2000));

  // Now use direct mouse/keyboard for precise control
  await desktop.keyboard.type('BCTRL automation');
  await desktop.keyboard.press('Enter');

  // Take screenshot of results
  await desktop.screenshot({ path: 'search-results.png' });

  await desktop.close();
}

Desktop + Web Session

For complex scenarios, use both a desktop and a separate browser session:

import { bctrl, playwright } from '@bctrl/sdk';

async function multiSessionWorkflow() {
  // Desktop for native app
  const desktop = await bctrl.desktop.connect({
    apiKey: process.env.BCTRL_API_KEY,
    os: 'windows'
  });

  // Separate browser session for web
  const browser = await playwright.connect({
    apiKey: process.env.BCTRL_API_KEY
  });

  // Work with native app
  await desktop.cua.run('Open Excel and create a new spreadsheet');

  // Get data from web
  await browser.page.goto('https://api.example.com/data');
  const data = await browser.stagehand.extract(
    'Get all data entries',
    z.array(z.object({ name: z.string(), value: z.number() }))
  );

  // Enter data into Excel using AI
  for (const entry of data) {
    await desktop.cua.run(`Enter "${entry.name}" in the next row, column A`);
    await desktop.cua.run(`Enter "${entry.value}" in column B`);
    await desktop.keyboard.press('Enter');
  }

  await desktop.cua.run('Save the spreadsheet as data-export.xlsx');

  await desktop.close();
  await browser.close();
}

Application Automation Examples

Notepad (Windows)

// Open and use Notepad
await desktop.cua.run('Open Notepad');
await desktop.keyboard.type('This is automated text!\n');
await desktop.keyboard.type('Line 2 of the document.');
await desktop.keyboard.hotkey('ctrl', 's');
await desktop.cua.run('Save as "notes.txt" on Desktop');

File Explorer

// Navigate file system
await desktop.keyboard.hotkey('win', 'e');  // Open Explorer
await new Promise(r => setTimeout(r, 1000));

await desktop.cua.run('Navigate to the Documents folder');
await desktop.cua.run('Create a new folder called "Automation"');
await desktop.cua.run('Open the Automation folder');

Office Automation

// Excel automation
await desktop.cua.run(`
  Open Microsoft Excel,
  Create a new workbook,
  In cell A1 type "Name",
  In cell B1 type "Value",
  In cell A2 type "Product A",
  In cell B2 type "100",
  Save as "data.xlsx" on Desktop
`);

// Word automation
await desktop.cua.run(`
  Open Microsoft Word,
  Type "Meeting Notes" and press Enter twice,
  Type today's date and press Enter,
  Type "Attendees:" and make it bold
`);

Terminal/Command Line

// Windows Command Prompt
await desktop.keyboard.hotkey('win', 'r');
await desktop.keyboard.type('cmd');
await desktop.keyboard.press('Enter');
await new Promise(r => setTimeout(r, 500));
await desktop.keyboard.type('dir');
await desktop.keyboard.press('Enter');

// Or use AI
await desktop.cua.run('Open Command Prompt and run "ipconfig"');

Best Practices

Use AI for complex, multi-step tasks

AI excels at tasks that require visual understanding and adaptation:

// Good use of AI
await desktop.cua.run('Find the Settings app and enable Dark Mode');
await desktop.cua.run('Open Chrome, go to gmail.com, and compose a new email');

// AI handles the details - finding buttons, handling popups, etc.

Use direct control for precise actions

When you know exact coordinates or need precise timing:

// Precise mouse control
await desktop.mouse.move(100, 200);
await desktop.mouse.click();

// Exact keyboard input
await desktop.keyboard.type('precise-value-123');
await desktop.keyboard.hotkey('ctrl', 'Enter');

Add delays for UI responsiveness

Desktop UIs need time to respond - add appropriate waits:

await desktop.mouse.click(500, 300);
await new Promise(r => setTimeout(r, 500));  // Wait for menu to open

await desktop.keyboard.hotkey('alt', 'Tab');
await new Promise(r => setTimeout(r, 300));  // Wait for window switch

Use screenshots for verification

Take screenshots to verify state and debug issues:

await desktop.cua.run('Open the Settings app');
await desktop.screenshot({ path: 'step-1-settings.png' });

await desktop.cua.run('Click on System');
await desktop.screenshot({ path: 'step-2-system.png' });

Combine both approaches

Use AI for discovery and navigation, direct control for data entry:

// AI to navigate to the right place
await desktop.cua.run('Open Excel and create a new spreadsheet');

// Direct control for fast data entry
for (const row of data) {
  await desktop.keyboard.type(row.name);
  await desktop.keyboard.press('Tab');
  await desktop.keyboard.type(row.value.toString());
  await desktop.keyboard.press('Enter');
}

// AI to save
await desktop.cua.run('Save the spreadsheet as data.xlsx');

Set appropriate timeouts for AI tasks

Complex AI tasks may need longer timeouts:

// Simple task - default timeout is fine
await desktop.cua.run('Click the Start button');

// Complex task - increase timeout
await desktop.cua.run('Open Photoshop and create a new 1920x1080 canvas', {
  timeout: 60000,  // 60 seconds
  maxSteps: 15
});

Troubleshooting

Clicks not registering

Add a small delay after moving the mouse:

await desktop.mouse.move(500, 300);
await new Promise(r => setTimeout(r, 100));
await desktop.mouse.click();

Keyboard shortcuts not working

Some applications may require focus first:

await desktop.mouse.click(500, 300);  // Focus the window
await new Promise(r => setTimeout(r, 200));
await desktop.keyboard.hotkey('ctrl', 's');

AI not finding elements

Be more specific in your instructions:

// Too vague
await desktop.cua.run('Click the button');

// More specific
await desktop.cua.run('Click the blue "Save" button in the bottom right corner');

Next Steps

Cloud Computers

Set up Windows and Linux VMs

Local Desktop

Connect your own machines

Web Automation

Browser automation with Playwright, Puppeteer, and Selenium

AI Agents

Deep dive into Stagehand and Browser-use

Get Started

Connect Resources

Automation & AI

Advanced

Desktop Automation

Choose Your Environment

Connect to a Desktop

Cloud Desktop

Local Desktop

Local Agent Setup

Mouse Control

Mouse Button Options

Keyboard Control

Keyboard Shortcuts

Key Hold and Release

Screen Access

Using Screenshots for Verification

AI Desktop Control (CUA)

Simple Commands

Multi-Step Tasks

Complex Workflows

CUA Options

Combining Browser + Desktop

Browser Inside Desktop

Desktop + Web Session

Application Automation Examples

Notepad (Windows)

File Explorer

Office Automation

Terminal/Command Line

Best Practices

Troubleshooting

Next Steps

Cloud Computers

Local Desktop

Web Automation

AI Agents

Get Started

Connect Resources

Automation & AI

Advanced

​Choose Your Environment

​Connect to a Desktop

​Cloud Desktop

​Local Desktop

Local Agent Setup

​Mouse Control

​Mouse Button Options

​Keyboard Control

​Keyboard Shortcuts

​Key Hold and Release

​Screen Access

​Using Screenshots for Verification

​AI Desktop Control (CUA)

​Simple Commands

​Multi-Step Tasks

​Complex Workflows

​CUA Options

​Combining Browser + Desktop

​Browser Inside Desktop

​Desktop + Web Session

​Application Automation Examples

​Notepad (Windows)

​File Explorer

​Office Automation

​Terminal/Command Line

​Best Practices

​Troubleshooting

​Next Steps

Cloud Computers

Local Desktop

Web Automation

AI Agents

Choose Your Environment

Connect to a Desktop

Cloud Desktop

Local Desktop

Mouse Control

Mouse Button Options

Keyboard Control

Keyboard Shortcuts

Key Hold and Release

Screen Access

Using Screenshots for Verification

AI Desktop Control (CUA)

Simple Commands

Multi-Step Tasks

Complex Workflows

CUA Options

Combining Browser + Desktop

Browser Inside Desktop

Desktop + Web Session

Application Automation Examples

Notepad (Windows)

File Explorer

Office Automation

Terminal/Command Line

Best Practices

Troubleshooting

Next Steps