Local Model, API, or Agent: Six Decisions Before Building an AI Workflow

Begin with the task

“Deploy AI” is not a usable goal. Describe the input, output and acceptance test. Is the task summarizing documents, writing code, searching local material—or reading files, calling APIs and executing commands?

Content generation may need only a model call. A workflow that observes an environment, chooses tools and keeps acting enters agent territory. An agent adds permissions, state, logs and recovery—not only capability.

One: can the data leave the environment?

Classify inputs as public, internal, sensitive or regulated. An API design must define where data goes, how long it is retained and who controls credentials. A local model reduces outbound transfer, but host logs, caches, vector stores and backups may still preserve content.

Local does not automatically mean secure. Service accounts, directory permissions, remote access and model sources still matter.

Two: what does the real task require?

Do not compare parameter counts alone. Build a small set of real inputs and evaluate:

• instruction following and output format;
• language and domain terminology;
• required context length;
• reliability of tool calls;
• whether failures can be detected instead of confidently continued.

A model that looks strong in a chat demo may still be unsuitable for structured output or autonomous actions.

Three: hardware, latency and concurrency

For local inference, consider memory, quantization, context overhead, time to first token and concurrency together. Start with a model that runs and complete a minimal experiment before deciding whether the bottleneck is capability or hardware.

An API removes local inference maintenance, but introduces network dependency, quotas, cost and provider change. A hybrid can keep routine tasks local and use an API only when a task needs more capability.

Four: what may the agent access?

Design permissions before features:

• separate read and write tools;
• restrict file access to explicit directories;
• allow-list commands or use an isolated runtime;
• require human approval for deletion, publishing, payment and external messaging;
• inject secrets through the runtime, never prompts or logs;
• record each tool call, its parameters, result and initiator.

Giving an agent unrestricted SSH and filesystem access to production turns one bad decision into system-level risk.

Five: what happens when it fails?

Define timeouts, retry limits, idempotency and fallback for each step. If the model is unavailable, does the workflow switch models, return to a human process or stop? If a tool stops halfway, can the system determine what already happened?

A recoverable workflow needs explicit durable state, not only a transcript.

Six: who maintains it?

Record model and quantization versions, system prompts, tool schemas, dependencies and test cases. Before upgrading any layer, rerun the same real tasks. A newer model does not guarantee a more stable workflow.

Three practical starting points

Local-first: useful when data cannot leave, tasks are stable and hardware is controlled; it carries model and driver maintenance.

API-first: useful for rapid validation, demanding capability or irregular load; manage credentials, cost, limits and fallback.

Hybrid agent: useful when a workflow connects multiple tools; begin read-only, low-risk and human-confirmed with the smallest permissions possible.

The final choice should come from a measurable experiment: complete one real task with real data, hardware and permission boundaries, then decide what deserves to expand.