How to Run an AI OCR Document Reader Agent with OE Runtime

Extract text from scanned documents, invoices, receipts, and forms automatically. OE Runtime uses Azure Vision to run OCR, classifies the document type, and structures the extracted data into ready-to-use fields — no manual data entry required.

The Agent YAML

Create a file called agent.yaml inside an ocr-vision/ directory:

name: Document Reader
description: Extract and analyze text from images using OCR
instructions: |
  You are a document analysis agent. Extract text from images using OCR,
  then analyze, classify, and summarize the extracted content.
  Complete all steps fully before writing your report.
steps:
  - name: Extract Text
    content: |
      Call the Azure Vision connector to run OCR on the provided image.
      POST /read/analyze with body:
      {
        "url": "<image URL to analyze>"
      }
      Save the Operation-Location URL from the response headers.
      Then poll GET <operation-path> (the path portion of the Operation-Location URL)
      every 2 seconds until status is "succeeded".
      Extract all text lines from the analyzeResult.readResults.
  - name: Classify and Structure
    content: |
      Analyze the extracted text to determine the document type
      (invoice, receipt, contract, ID, form, letter, or other).
      Extract key structured fields based on type — e.g. for an invoice: vendor, date, line items, total.
  - name: Report
    content: |
      Produce a document analysis result:
      - Document type identified
      - Structured fields extracted with values
      - Full raw text extracted
      - Any low-confidence regions or unreadable sections
      - Suggested next action (e.g. file, process payment, follow up)
connectors:
  - connection_name: Azure Vision
    connection_type: azure-vision

The Config File

Create oe-config.json in the same directory:

{
  "llm": {
    "provider": "openai",
    "model": "gpt-4o",
    "apiKey": "YOUR_OPENAI_API_KEY"
  },
  "server": {
    "enabled": false,
    "port": 3333,
    "apiKey": "your-secret-api-key"
  },
  "connectors": [
    {
      "connection_name": "Azure Vision",
      "connection_type": "azure-vision",
      "apiKey": "YOUR_AZURE_VISION_KEY",
      "headerName": "Ocp-Apim-Subscription-Key",
      "baseUrl": "https://YOUR_RESOURCE.cognitiveservices.azure.com/vision/v3.2"
    }
  ]
}

Create an Azure Computer Vision resource at portal.azure.com and copy the endpoint and API key from the resource's Keys and Endpoint page. The free tier (F0) allows up to 5,000 OCR transactions per month.

Download OE Runtime

Run the Agent

Windows

oe-runtime-win.exe ocr-vision/agent.yaml --config ocr-vision/oe-config.json

macOS

chmod +x oe-runtime-macos

oe-runtime-macos ocr-vision/agent.yaml --config ocr-vision/oe-config.json

Linux

chmod +x oe-runtime-linux

oe-runtime-linux ocr-vision/agent.yaml --config ocr-vision/oe-config.json

Running as an API Server

Add --serve to start OE Runtime as an HTTP server on port 3333:

oe-runtime-win.exe --serve --config ocr-vision/oe-config.json

Then send a request:

curl -X POST http://localhost:3333/run \
  -H "Content-Type: application/json" \
  -d '{"yaml": "ocr-vision/agent.yaml", "params": {}}'

Or import the Postman collection (download above) to test all endpoints visually.

Use Cases

• Invoice processing automation — Extract vendor name, invoice number, line items, amounts, and due dates from scanned or photographed invoices automatically. The structured output can be fed directly into your accounts payable system, eliminating manual data entry and reducing processing time from days to seconds.
• Receipt digitisation — Convert paper receipts into structured expense records by photographing them and running the OCR agent. The agent extracts merchant name, date, amount, and category, ready to import into your expense management system or spreadsheet.
• Contract data extraction — Parse legal contracts and agreements to extract key clauses, parties, dates, payment terms, and renewal conditions. The agent surfaces the critical information from long documents so your legal or procurement team can review what matters without reading every page.
• ID and form verification — Extract data from identity documents, application forms, and registration papers to populate database records automatically. This reduces manual transcription errors in KYC, onboarding, and registration workflows where accuracy is critical.
• Medical record processing — Extract structured data from handwritten or printed medical forms, referral letters, and lab reports. The agent can identify patient details, diagnosis codes, medication names, and test results to feed into electronic health record systems.
• Logistics and shipping document processing — Process bills of lading, packing lists, and customs declaration forms to extract shipment details, quantities, and consignee information. Automating this step in import/export workflows significantly reduces manual processing time and customs clearance delays.
• Historical document digitisation — Convert archives of scanned historical documents, letters, or records into searchable digital text. The agent processes each document, extracts the readable content, and flags sections with low OCR confidence for manual review.
• Quality control and compliance — Read product labels, certificates of conformity, and inspection reports from photographed documents to verify that required fields and values are present. The agent can flag documents where mandatory information is missing or illegible before they are filed.