In my first semester of the double MSc at KTH (Systems, Control and Robotics), I took the Reinforcement Learning course — and it remains one of my favourites of the whole degree. I solved the two lab assignments entirely from scratch: a classic grid-world problem with full value iteration, and a Deep Q-Network trained on Lunar Lander. Both labs later grew playable browser editions, so you can try the exact algorithms live.
🤖 Lab 1 — Minotaur Maze & Mountain Car
The first lab covered the fundamentals: MDPs, Bellman equations, value iteration, Q-learning and SARSA. Our first problem was the Minotaur Maze — Thomas must reach the exit while a minotaur wanders the maze. Because both agent and minotaur move, I modelled it as a joint MDP of 2,240 states and solved it with full value iteration, comparing it against Q-learning and SARSA. The web edition lets you drive Thomas yourself while an AI arrow shows what the optimal policy would do at every step, and a live heatmap visualises the V-values converging sweep by sweep:
The big idea, in plain words: imagine every maze position glowing with a number — how good it is to be there — with the exit glowing +500 and being caught −1000. Value iteration spreads that glow through the maze like water finding its level, until each position knows its best move’s promise minus the cost of moving; then the arrows simply point to the brightest neighbour. Formally, the maze is a stochastic shortest-path MDP whose state is the joint configuration of both agents, \(\mathcal{S} = \{(t_i, t_j, m_i, m_j, k)\}\), and the transition model over the minotaur’s uniformly random moves is \[ P(s' \mid s, a) = \frac{1}{|A_{\text{mino}}(s)|} \] with rewards \(+500\) for escaping and \(-1000\) for being caught.
The second problem was the classic Mountain Car: a weak car that must climb a steep hill by building momentum. Here we implemented SARSA(λ) with a Fourier linear basis, eligibility traces and Nesterov momentum — a great lesson in how function approximation turns tabular algorithms into continuous-state learners:
🤖️ Lab 2 — Deep Q-Network on Lunar Lander
The second lab jumped to deep reinforcement learning: a DQN trained from scratch (PyTorch) on LunarLander-v2 from OpenAI Gym — an 8 → 64 → 64 → 4 network with experience replay (L = 16,384 transitions), a target network updated every C = L/N = 256 steps, mini-batches of 64 and ε-greedy exploration decaying from 0.99. I ran a proper ablation study — discount factor (γ = 0.5 / 0.99 / 1.0), training length (500 / 1000 / 2000 episodes) and DQN vs. random baselines — and saved the trained models.
The big idea, in plain words: the lander learns by replaying memories — it stores what it experienced (situation → action → outcome) and practices on random memories between flights, like a pilot studying past landings. The optimal action-value function satisfies the Bellman equation \[ Q^*(s,a) = \mathbb{E}\!\left[\, r + \gamma \max_{a'} Q^*(s',a')\, \right] \] and the network \(Q(s,a;\theta)\) approximates it by minimising the TD loss \[ \mathcal{L}(\theta) = \mathbb{E}\!\left[\left( r + \gamma \max_{a'} Q(s',a';\theta^-) - Q(s,a;\theta) \right)^2\right] \] against a frozen copy \(\theta^-\) of the network — the target network keeps the scoreboard stable enough to learn from, while \(\varepsilon\)-greedy exploration occasionally presses a random button so the lander discovers moves it would never try otherwise.
The web edition is the fun part: Learn mode trains the DQN live in your browser with adjustable hyperparameters (γ, α, ε, decay, replay buffer, batch size and speed), plotting the running reward and logging every landing; Fly mode hands you the thrusters:
🎮 Play both labs in the browser
Both labs are deployed on GitHub Pages — the JavaScript cores are 1:1 ports of the Python code (same state
space, same Bellman backups, same replay buffer) with matching node:test suites:
- RL Lab 1 — Minotaur Maze + Value Iteration + Mountain Car
- RL Lab 2 — train the DQN or fly the lander yourself
🎓 What the course gave me
- MDP thinking: modelling sequential decisions — states, actions, rewards, and the Bellman backup — is now second nature.
- The tabular → deep bridge: SARSA(λ) with Fourier features in Lab 1, DQN with replay and target nets in Lab 2 — the same ideas, scaled up.
- Deep-RL stability toolkit: experience replay, target networks, ε decay — and the discipline of ablating each one to see it matter.
- Experiment hygiene: every run logged, every model saved, every figure labelled — the habit that made the labs reproducible (and later, ported to the browser).
GitHub · rl_lab1 GitHub · rl_lab2 Play Lab 1 Play Lab 2 KTH · Systems, Control and Robotics PyTorch · OpenAI Gym · SARSA(λ) · DQN