Deploy a model to RunPod Serverless

This workshop packages a sentiment model as a RunPod Serverless worker. The model is DistilBERT, fine-tuned on SST-2. It is small enough to run on a CPU for local testing and fast on a GPU in production.

The handler

• The model loads at module level, so it loads once per cold start and is reused across jobs.
• Input is validated, and text is capped at 512 tokens, DistilBERT's maximum context.
• top_k controls how many labels come back. The default is 2.
• Output is structured JSON with a label and a score for each result.

Test locally

Install the requirements, then run python handler.py. The SDK detects test_input.json and runs one local job. You can also pass input inline with --test_input.

Build and deploy

Build the image for x86_64 with docker build --platform=linux/amd64. RunPod hosts are x86_64, even when you build from an ARM Mac. Then create a template and an endpoint with runpodctl.

Cost

The endpoint scales to zero, so it costs nothing while idle. The repo's notes put it at about $0.69 per hour on AMPERE_24 while serving.

The code is in the runpod-demo repo.

More labs

• 
• 
• 

← All posts