Remote Inference
Remote inference refers to the execution of inference tasks on external servers rather than on local devices or machines. In this architecture, computational requests are sent to remote systems where models process inputs and return results to the client. This approach contrasts with local inference, where models run directly on the user’s device or edge hardware. Remote inference is particularly common in cloud-based AI services and server-based deployments of large language models.
Architecture and Workflow
In a remote inference setup, a client application sends input data to a remote server or service endpoint over a network connection. The server hosts the machine learning model and performs the actual computation, then returns the output back to the client. This separation of inference computation from the client allows for scalable resource management but introduces latency and privacy considerations.
Local-First Alternatives
While remote inference dominates cloud-based AI, there is a growing movement toward local-first architectures that prioritize privacy, control, and offline capability.
- OpenJarvis Framework: A local-first, open-source personal AI framework developed by Stanford University’s Hazy Research and Scaling Intelligence Labs. It enables users to run powerful AI models directly on personal devices, reducing reliance on cloud services.
- Integration with Ollama: OpenJarvis leverages Ollama to manage local model execution, allowing for efficient tracking of computational resources (e.g., wattage) and ensuring data remains on-device.
- Privacy and Control: By shifting inference from remote servers to local hardware, frameworks like OpenJarvis address concerns regarding data sovereignty and network dependency.
For detailed implementation notes and context on this framework, see OpenJarvis: Stanford’s Local-First Personal AI Framework with Ollama.