Nicolas Smits

DQN variants for ICU sepsis treatment

McGill, COMP 579with James Randolph and Abdullah ParachaJan – Apr 2025

Mean return over the last 10,000 episodes. The three dim variants stayed near a random policy.

Formulation

ICU-Sepsis-v2 models a patient's stay in intensive care as a Markov decision process built from MIMIC-III records: 716 discrete health states and 25 treatments, each a combination of intravenous-fluid and vasopressor doses. The reward is 1 if the patient survives and 0 otherwise, with no discounting, so a policy's expected return is its survival rate.

Deep Q-learning

DQN learns Q(s,a), the expected return of giving treatment a in state s and acting well afterwards, with a neural network. Transitions go into a replay buffer 𝒟, and the network regresses on targets computed by a slowly updated copy of itself, θ‾:

ℒ(θ)=𝔼(s,a,r,s′)∼𝒟[(r+γmaxa′⁡Qθ‾(s′,a′)−Qθ(s,a))2]

Each extension below changes one part of this loop.

Store transition

Extensions

  • Double: the online network picks the next action and the target network scores it, r+γQθ‾(s′,arg⁡maxa′⁡Qθ(s′,a′)), which curbs overestimation.
  • Prioritized replay: transitions with large TD error δ are replayed more often, P(i)∝|δi|α, with importance weights wi=(NP(i))−β removing the bias.
  • Dueling: Q(s,a)=V(s)+A(s,a)−1|𝒜|∑a′A(s,a′) separates how good a state is from how much the treatment matters.
  • n-step: the target sums n rewards before bootstrapping, ∑k<nγkrt+k+γnmaxa′⁡Q(st+n,a′), so the terminal reward travels back faster.
  • Noisy layers: weights μ+σ⊙ε with learned σ replace ε-greedy with exploration that depends on the state.
  • Distributional: the network predicts the whole return distribution Z(s,a) on fixed atoms, with Q=𝔼[Z], trained against the projected Bellman target.

Rainbow combines all six.

Comparison

Starting from a two-layer DQN with 128 hidden units, the team added one extension at a time and then all six together as Rainbow. Each variant was tuned on a grid of about four settings and run for 10 seeds of 300,000 episodes.

Results

The dueling head reached the best return, at the cost of noisier learning; prioritized replay came second and converged about 100,000 episodes sooner. The combined methods failed to learn on grids of about four settings each, which says more about tuning budgets than about the methods.

Return over training for the eight DQN variants
Survival rate over training for the best settings of each variant.Figure 1a of the team’s report.

Further reading