Nicolas Smits

Movie recommender with LLM cold start

Team project, CMUSep 2026 – present

Context

A team project for CMU's Machine Learning in Production course: run a recommendation service for a movie-streaming platform with live users for a whole semester. Every request must get 20 films inside 600 ms, and the service has to stay up while it is monitored, retrained and A/B tested. My parts were the cold start and the deployment.

Cold start

A ranking model only knows users who were in its training data, and new users arrive all the time. At signup each user writes a sentence or two about the films they like and dislike. A background worker sends that text to Jev once, stores the answers as a taste profile, and a ranker scores the catalogue from it in code. A new user's first request comes about five minutes after signup, so the profile is almost always ready, and no request ever waits on the model.

Grafana panel: profiles waiting for the cold-start worker fall from about 80,000 to zero overnight
The cold-start worker clearing a backlog of about 80,000 profiles overnight, 10 to 11 November (UTC).Team Grafana dashboard, Platform Health.

Jev by TypeSafe

Jev is TypeSafe's first System One model, released in September 2026. It doesn't write text. It answers typed questions with probabilities, such as whether this person wants or avoids horror films, with "not stated" as an explicit option so silence never counts as dislike. The answer always fits the schema, so the ranker never meets a malformed profile, and the probabilities go straight into a scoring formula. Twenty-four questions cost about 2,400 input tokens and return in a median 169 ms, which comes to about $1 a day for every new user. A usability question screens out descriptions like "idk".

Operations

LightGBM LambdaMART ranks the users the model knows. A new model is retrained each day, served first as a canary candidate beside the champion, and promoted without downtime. Grafana dashboards track availability, latency, model versions, drift and the cold-start backlog.

Grafana state timeline: one control model per day, each preceded by a short candidate period
One week of serving models. Each day’s LightGBM model runs first as a candidate beside the champion, then takes over.Team Grafana dashboard, Model Quality.
Grafana time series: p50, p95 and p99 HTTP latency over one week, p99 near 500 ms
HTTP latency over the same week. p99 stays under 500 ms apart from two brief spikes; percentiles come from histogram buckets, so read them as bounds.Team Grafana dashboard, Service Availability.