Many small businesses require AI tools for tasks such as drafting emails, summarizing calls, reading invoices, and searching documents. However, data privacy and sovereignty concerns often prevent them from sending sensitive customer data to cloud APIs. Running AI workflows locally on hardware like the NVIDIA DGX Spark allows these businesses to maintain control over their data, ensuring it remains on-premises and compliant with privacy obligations. This approach also enables clear auditing of data usage and storage, which is critical for clinics, law firms, and accountants among others (source).
The NVIDIA DGX Spark is equipped with 128GB of unified memory shared between CPU and GPU, providing sufficient resources to run multiple AI models simultaneously. This setup supports a variety of AI jobs by loading specialized models for tasks such as chat, coding, vision, extraction, embeddings, reranking, speech-to-text, OCR, image generation, and text-to-speech. Each model used in this stack allows commercial use under appropriate licenses, which is a crucial consideration for deployment (source).
To ensure offline functionality and data privacy, all models are downloaded once and cached locally, with exact commit versions locked to prevent unauthorized updates or license changes. The main AI model, Qwen3.6-35B-A3B, runs efficiently by activating only a fraction of its parameters per token, enabling fast text generation and multi-user support. This model serves a wide range of text-based tasks such as summarization, classification, extraction, drafting, coding, and reading screenshots, reducing the need for multiple specialized models (source).
The DGX Spark’s shared memory architecture requires careful management of GPU memory utilization to accommodate all models. For example, the main model is capped at about one-third of the memory, while other models share the remainder. Loading all seven models takes approximately 12 minutes and uses around 108GB of the available 128GB, leaving limited free memory. Proper startup sequencing and cache management are necessary to avoid memory conflicts during model initialization (source).
To provide a unified interface for clients, LiteLLM acts as a single OpenAI-compatible endpoint routing requests to the appropriate backend models. It supports virtual keys, team-based access control, and budget tracking, making it suitable for multi-user environments. LiteLLM handles specialized tasks such as audio transcription and image generation by routing requests to the corresponding models and managing parameters like response formats (source).
Integration with workflow automation tools like n8n is straightforward. By creating LiteLLM virtual keys and configuring n8n OpenAI credentials with custom base URLs, teams can securely connect to the DGX Spark AI stack. This setup enables workflows such as invoice processing that perform OCR and data extraction locally without any cloud calls, preserving data privacy. For example, an n8n workflow can accept an invoice image, convert it to base64, run OCR and field extraction using the local AI models, and display the extracted data, all within seconds (source).
Overall, running AI workflows on NVIDIA DGX Spark provides small businesses a powerful, secure, and efficient way to leverage AI capabilities on-premises. This approach addresses data privacy concerns by eliminating the need to send sensitive information to external cloud services, while supporting a broad range of AI tasks through a carefully managed model stack and unified API endpoint (source).
Leave a Reply