Project
AIDP: distributed LLM inference on GPUs
Serving LLMs on decentralized GPUs for less.
AIDP Neural Cloud serves LLMs on distributed GPU capacity. It uses vLLM and llama.cpp for inference, with a guardrails gateway and evaluation datasets that score accuracy, safety and latency.
The repo’s results table compares it with GPT-4o-mini: about 47% lower cost per million tokens ($0.08 against $0.15), and a 180 ms p50 latency against 250 ms. The table also lists 50 requests per second.
Those are the repo’s numbers from its own benchmark. The comparison is with one model, not with the OpenAI API in general.
The code is in the aidp-neural-cloud repo. The work page has the case study, and the GPU video side is in the next post.