minimalRL: reinforcement learning in one file each

Eleven reinforcement learning algorithms, each in a single PyTorch file of about 70–190 lines, all aimed at the same pole-balancing game.

Tested at commit c8efed8 on a MacBook with an Apple M4, Python 3.14.5, torch 2.15.0, gymnasium 1.4.0. Short code excerpts are quoted with credit to seungeunrho (MIT).

Going deeper: everything below is the full version. On a first read, look at the real scores and the Robo-Helper experiment, then come back for the code.

1. Summary

• One file each: REINFORCE, Actor-Critic, DQN, PPO, DDPG, A3C, ACER, A2C, SAC, PPO-Continuous, V-trace, plus a PPO-LSTM file not listed in the README.
• Install snag: "gym > 0.26.2" doesn't exist. 0.26.2 was the last Gym. It installs on Python 3.14 but crashes with numpy 2. We used Gymnasium, Gym's maintained successor, by changing one import.
• Real results: every run used the full 180-second cap, because each script loops for 10,000 episodes. All three scored 200+ within the cap.
• Robo-Helper: a DQN learned the 9-step route around the cat in about 200 episodes, and was optimal from all 40 start squares. A* needs no training and plans in microseconds.
• Where each wins: learned control coped with wheel slip as well as A* replanning every step, and far better than a plan followed blindly. Planning won when the world changed (the cat moved).

2. How the code works: the algorithm ladder

Every file has the same shape: a small network class, a train_net (or train) function, and a main() loop that plays episodes and prints the average score every 20 episodes. The algorithms differ only in what the network predicts and what loss it learns from.

REINFORCE in three lines

# REINFORCE.py (MIT, seungeunrho)
for r, prob in self.data[::-1]:
    R = r + gamma * R
    loss = -torch.log(prob) * R

Walk the episode backwards, adding up discounted reward R. Then push up the log-probability of each action, scaled by R. Good episodes make their actions more likely. That's the whole algorithm.

DQN: the one idea that makes it click

# dqn.py (MIT, seungeunrho)
q_a = q_out.gather(1,a)
max_q_prime = q_target(s_prime).max(1)[0].unsqueeze(1)
target = r + gamma * max_q_prime * done_mask

This is exactly the Q-learning target from Lesson 7: reward plus γ times the best Q at the next state, and just the reward if the episode ended (done_mask = 0). Three things are new:

• A neural network instead of a table. CartPole's four numbers are continuous, so you can't list every state as a dictionary key. The network fills in Q for states it has never seen.
• Replay memory (ReplayBuffer, up to 50,000 moves). It trains on random old moves, not only the latest ones, so the examples aren't all near-copies of each other.
• A target network (q_target), copied from q only every 20 episodes. The target stays still for a while, so the network isn't chasing its own moving guess.

PPO's clip

# ppo.py (MIT, seungeunrho)
surr1 = ratio * advantage
surr2 = torch.clamp(ratio, 1-eps_clip, 1+eps_clip) * advantage
loss = -torch.min(surr1, surr2) + F.smooth_l1_loss(self.v(s) , td_target.detach())

ratio is "how much more likely is this action now than when we collected the data". Clamping it to 0.9–1.1 (eps_clip = 0.1) stops one update from changing the policy too much. That is what lets PPO safely reuse each batch 3 times (K_epoch = 3).

3. Run it, tested (CartPole)

All runs on 7 Nov 2026, MacBook (Apple M4), CPU, Python 3.14.5, torch 2.15.0, in a private venv in a scratch folder. Other jobs were running on the laptop at the same time, so on a quiet machine more episodes would fit in 3 minutes.

The Gym problem

$ pip install "gym>0.26.2"
ERROR: Could not find a version that satisfies the requirement gym>0.26.2 (from versions: 0.0.2, ... 0.26.1, 0.26.2)

$ pip install "gym==0.26.2"        # installs, then:
AttributeError: module 'numpy' has no attribute 'bool8'. Did you mean: 'bool'?

Gym stopped at 0.26.2 and uses np.bool8, which numpy 2 removed. Gym's own install message points to Gymnasium: "replace 'import gym' with 'import gymnasium as gym'". So we made modified copies of three files in the scratch folder only. The clone stays untouched. Two changes per file:

$ diff minimalRL/dqn.py mrl-mod/dqn.py
1c1
< import gym
---
> import gymnasium as gym
95a96
>             done = done or truncated

The second change is a bug fix. Since Gym 0.26, hitting the 500-step time limit sets truncated, not done. The original loops only stop on done. Once the pole is balanced well, they would keep calling step() past the limit. The comment in REINFORCE.py, "CartPole-v1 forced to terminates at 500 step", is from before that API change. We found this by reading the code. We didn't run the unfixed version.

pip install torch gymnasium
python runcap.py mrl-mod/REINFORCE.py 180     # runs a script, kills it after 180 s, prints wall time
python runcap.py mrl-mod/dqn.py 180
python runcap.py mrl-mod/ppo.py 180

REINFORCE

# 7 Nov 2026, MacBook (Apple M4), CPU. Selected lines (one every 200 episodes) + the end.
# of episode :20, avg score : 21.65
# of episode :200, avg score : 31.65
# of episode :400, avg score : 53.1
# of episode :600, avg score : 59.35
# of episode :800, avg score : 104.45
# of episode :1000, avg score : 136.5
# of episode :1200, avg score : 177.0
# of episode :1400, avg score : 200.05
# of episode :1600, avg score : 194.65
# of episode :1800, avg score : 241.1
# of episode :1900, avg score : 286.45
# of episode :1920, avg score : 292.2
[wall time 180.1s, cap 180s]

Slow and steady. First 20-episode average above 200 at episode 1,140 (221.7). Best: 317.05 at episode 1,700. It never reached 500 in 3 minutes.

DQN

# 7 Nov 2026, MacBook (Apple M4), CPU. Selected lines + the end.
n_episode :20, score : 10.2, n_buffer : 203, eps : 7.9%
n_episode :200, score : 9.6, n_buffer : 1976, eps : 7.0%
n_episode :300, score : 13.9, n_buffer : 3110, eps : 6.5%
n_episode :320, score : 30.8, n_buffer : 3726, eps : 6.4%
n_episode :340, score : 101.4, n_buffer : 5754, eps : 6.3%
n_episode :360, score : 207.9, n_buffer : 9913, eps : 6.2%
n_episode :560, score : 160.9, n_buffer : 45130, eps : 5.2%
n_episode :960, score : 314.1, n_buffer : 50000, eps : 3.2%
n_episode :1760, score : 478.8, n_buffer : 50000, eps : 1.0%
n_episode :1960, score : 485.1, n_buffer : 50000, eps : 1.0%
n_episode :2560, score : 286.1, n_buffer : 50000, eps : 1.0%
n_episode :2840, score : 475.4, n_buffer : 50000, eps : 1.0%
[wall time 180.1s, cap 180s]

Nothing happens for 200 episodes. That's on purpose: training only starts once the replay memory holds 2,000 moves. Then it jumps from about 10 to 208 in 60 episodes. It hit a perfect 500.0 average at episode 1,900, and wobbled between about 290 and 485 after that. (DQN prints "score", not "avg score", but it is the same 20-episode average.)

PPO

# 7 Nov 2026, MacBook (Apple M4), CPU. Every line to episode 340, then selected lines.
# of episode :20, avg score : 33.0
# of episode :40, avg score : 35.5
# of episode :60, avg score : 45.4
# of episode :80, avg score : 68.2
# of episode :100, avg score : 81.2
# of episode :120, avg score : 137.3
# of episode :140, avg score : 212.9
# of episode :160, avg score : 275.7
# of episode :180, avg score : 371.2
# of episode :200, avg score : 279.7
# of episode :220, avg score : 338.6
# of episode :240, avg score : 196.4
# of episode :260, avg score : 269.9
# of episode :280, avg score : 448.9
# of episode :300, avg score : 482.6
# of episode :320, avg score : 500.0
# of episode :340, avg score : 497.2
# of episode :400, avg score : 500.0
# of episode :560, avg score : 334.7
# of episode :640, avg score : 499.1
# of episode :720, avg score : 152.3
# of episode :800, avg score : 149.2
# of episode :820, avg score : 170.2
[wall time 180.1s, cap 180s]

Fastest learner by episodes: above 200 at episode 140, a perfect 500.0 at episode 320. Fewer episodes fit in 3 minutes (820) because good episodes are long, up to 500 steps each. Then it partly collapsed, to about 150 by episode 720. This is a known RL failure. Nothing in the file saves the best policy or lowers the learning rate, so it can unlearn. Real projects save checkpoints and evaluate them separately.

Both DQN and PPO also printed a PyTorch warning that building a tensor from a list of numpy arrays "is extremely slow". It's harmless here.

4. Connect to Robo-Helper: DQN vs A* in our room

Lesson 7 did tabular Q-learning in a maze. dqn.py is the next rung: the same target rule, plus a neural network, replay memory and a target network. So we tested it on Robo-Helper's real room, against the A* planner from Lesson 6.

The setup

• Room: world.ROOM (10×8, 42 free squares). Start ROBOT_START = (1, 1). Goal: the blue block at (6, 3). The cat sits at sim.CAT = (4, 1), right on the shortest route.
• Environment (RoomEnv in grid_dqn.py, gym-like reset/step): state = the robot's square as a one-hot list of 80 numbers (plus the cat's square in one version). Actions = N, S, E, W using Lesson 1's move. Reward: −1 per step, −5 for bumping a wall, −10 for bumping the cat (you stay put), +20 for reaching the block. Maximum 60 steps.
• DQN: the same network shape, replay size (50,000), batch (32), γ (0.98), learning rate, smooth L1 loss and target copy every 20 episodes as dqn.py. Two changes, both honest: exploration starts at 100% and falls to 5% (dqn.py uses 8%→1%, too little to find a block in a room), and each training episode starts on a random free square ("exploring starts").
• Baselines: lesson06_paths.astar_path told where the cat is, and Lesson 11's real RoboHelper.drive_to, which only discovers the cat when its sensor sees it, then replans.

Results

$ SOLUTIONS=…/18-robot-brain/project/solutions python -I grid_dqn.py
# 7 Nov 2026, MacBook (Apple M4), CPU, torch 2.15.0
room 10x8, 42 free squares, start (1, 1), goal blue block (6, 3), cat (4, 1)
A* knowing the cat: 9 steps, 14 us per plan, 0 training
Lesson 11 drive_to (finds the cat on the way, replans): ok=True, 11 moves, 0.10 ms
DQN (position only): trained 1500 episodes in 9.3 s; first optimal greedy run at episode 200
  from start: ok=True, 9 steps, 0 bumps | from every free square: reaches block 40/40, shortest path 40/40
  twist, cat moves to (5, 3): DQN ok=False, 60 steps, 53 bumps | A* 7 steps
  twist, cat leaves the room: DQN ok=True, 9 steps, 0 bumps | A* 7 steps
  twist, 10% wheel slip (200 runs): DQN success 100% (10.2 steps) | A* open-loop 50% (9.0) | A* replanning every step 100% (10.0)
  twist, 20% wheel slip (200 runs): DQN success 100% (11.8 steps) | A* open-loop 22% (9.0) | A* replanning every step 100% (11.6)
DQN (position + cat), 4000 episodes in 78.8 s: reaches the block for 40/40 cat positions, shortest path on 40/40 | A*: 40/40

A second run with the same seeds gave identical results, but different times (DQN training 39.7 s and 165.7 s, A* 54 µs per plan), because the laptop was busy with other jobs (load average about 20). Times here are rough. The step counts and success rates are exact.

What it shows:

• In a room you have a map of, planning wins. A* gives the same 9-step route with zero training, in microseconds. DQN needed hundreds of episodes to learn what A* computes instantly. That's the same honest note as Lesson 7.
• A learned policy reacts. With 20% wheel slip, a plan followed blindly reached the block only 22% of the time. The DQN reached it 100% of the time, because it picks a move from wherever it actually is. A* matched it only when we made it replan every step. A policy is a closed-loop controller for free.
• A learned policy only knows what it was shown. When the cat moved onto its route, the position-only DQN drove into the cat 53 times. It cannot see the cat, so it has no reason to change. A* just replanned. When the cat was put into the state and the cat moved around during training, the DQN handled all 40 cat positions with the shortest path. That took 2.7× more episodes and a 160-number input. In a bigger world, "train on every situation" stops being possible. That's where planning, or a learned model plus planning, comes back.

And PPO? In Lesson 12, RLHF by hand, Robo-Helper learned a reward model from a person's route choices, then used best-of-n to pick a route. The lesson explains that real RLHF goes one step further: it trains the policy itself with PPO (or GRPO) on the learned reward, with a KL penalty as a leash. ppo.py is that update in 118 lines. The clip in the loss above is a second leash of the same kind: "improve, but don't move too far from the policy that collected the data".

5. Why it matters for robotics

6. Reality check

• Gym is dead; use Gymnasium. The README's "gym > 0.26.2" can't be installed. Swap the import, and handle truncated as well as terminated.
• "Every algorithm can be trained within 30 seconds." We didn't timestamp each line, so we can't say exactly when each passed a given score. All three runs used their full 3-minute cap, because each script loops for 10,000 episodes and never stops on its own.
• Not every file is CartPole. The README says environments are fixed to CartPole-v1, but ddpg.py, sac.py and ppo-continuous.py use Pendulum-v1.
• No seeds, so results vary. The files don't set random seeds. Your numbers will differ from ours.
• Learning can collapse. Our PPO hit 500 then fell to about 150. Save the best checkpoint.
• For real projects: use a maintained library such as Stable-Baselines3 or CleanRL (single-file implementations with logging). minimalRL is for reading.

7. Self-check

8. Further reading

• minimalRL on GitHub: start with REINFORCE.py, dqn.py and ppo.py
• Gymnasium migration guide (linked from Gym's own install warning)
• Robo-Helper: Lesson 7, Q-learning (the table version of DQN) and Lesson 12, RLHF by hand. Real RLHF trains the language model with PPO (or GRPO) on a learned reward, with a KL leash. That is ppo.py's clipped update, at a much larger scale.
• Page 5, tinygrad: the previous page in this module