1. The Mental Model: What RL Is Actually Solving
Reinforcement learning studies an agent that repeatedly observes a situation, chooses an action, receives a reward, and changes the future situation. Unlike supervised learning, the correct action is not given directly. The agent must learn from consequences, and an action that looks good immediately may be bad in the long run. The central difficulty is therefore sequential credit assignment under uncertainty.
Key points
- State tells the agent what situation it is in; action is what it can choose; reward is an immediate scalar signal; return is the cumulative future reward.
- A policy is the decision rule. Learning a policy means learning how actions should change with state.
- The environment is not just a dataset: the action changes what data the agent will see next.
- Exploration asks whether to try uncertain actions; exploitation uses actions currently believed to be best.
What you should remember
If you remember only one sentence: RL learns a feedback decision rule from delayed consequences, not from labeled correct actions.
2. Markov Decision Process: The Mathematical Language
A Markov decision process (MDP) is the standard mathematical model for RL. At time t the agent observes state s_t, samples an action a_t from policy π(a|s), receives reward r_t, and the environment samples the next state according to P(s_{t+1}|s_t,a_t). The Markov property does not mean the world has no history; it means the chosen state representation already summarizes all history needed to predict the future under an action.
Key points
- S is the state space and A is the action space. Either can be discrete or continuous.
- P is the transition model. Model-free RL does not require P to be known explicitly; model-based RL learns or uses it.
- If the observation does not make the process Markov, the problem is better described as a POMDP; history or belief state may then be needed.
Mini example
For a mobile robot, state can be position and velocity, action can be acceleration, reward can penalize distance to the goal and energy use, and transition dynamics describe how acceleration changes motion.
3. Reward, Return, Discounting, and Episodes
The reward r_t is local, while the return G_t evaluates the future consequence of a decision. Discounting with γ∈[0,1) gives near-term rewards more weight and keeps infinite-horizon sums finite. In episodic tasks the process terminates; in continuing tasks it may run indefinitely. Choosing reward and γ changes the task the agent is actually optimizing, so reward design is part of system design rather than a cosmetic detail.
Key points
- γ near 0 makes the agent myopic; γ near 1 emphasizes long-term consequences.
- Reward shaping can accelerate learning, but a poorly shaped reward can create unintended behavior.
- For control problems, it is common to use negative cost as reward: r_t = −(state error + control effort + penalties).
Numerical intuition
If rewards are 1,1,1 and γ=0.9, then G_0=1+0.9+0.81=2.71. The same future reward contributes less when it is farther away.
4. Value Functions: V, Q, and Advantage
A value function converts delayed future rewards into a quantity that can be evaluated at the current state. V^π(s) asks how good state s is when policy π is followed. Q^π(s,a) asks how good it is to take action a now and then follow π. The advantage A^π(s,a)=Q^π(s,a)−V^π(s) measures whether an action is better or worse than the policy's typical action at that state.
Key points
- Value-based methods learn V or Q and derive a policy by choosing high-value actions.
- Policy-based methods parameterize the policy directly; a critic may still estimate V or Q to reduce variance.
- Advantage is especially important in actor–critic and PPO because it tells the actor which sampled actions deserve increased probability.
5. Bellman Equations: Turning Long Horizons into One-Step Recursions
The Bellman idea is the algebraic core of RL: the value of the current state equals the immediate reward plus the discounted value of the next state. This decomposition makes a long-horizon optimization problem recursively solvable. Bellman expectation equations evaluate a fixed policy, while Bellman optimality equations characterize the best possible value function.
Key points
- Policy evaluation solves the expectation equation for a fixed policy.
- Policy improvement makes the policy greedier with respect to the current value estimate.
- Dynamic programming alternates these operations when the model is known; TD methods approximate the same recursion from samples when the model is unknown.
What you should remember
Bellman recursion is the bridge from 'future cumulative reward' to a target that can be learned one transition at a time.
6. Dynamic Programming, Monte Carlo, and TD Learning
Three families differ mainly in what they use as a learning target. Dynamic programming uses the known transition model and expected next-state values. Monte Carlo waits until an episode finishes and uses the realized return. Temporal-difference (TD) learning updates before termination by bootstrapping from the current estimate of the next state. TD is usually the practical bridge to modern RL.
Key points
- Monte Carlo has no bootstrap bias but often high variance and requires complete returns.
- TD has lower variance and learns online, but bootstrapping can propagate estimation error.
- n-step returns and TD(λ) interpolate between one-step TD and Monte Carlo.
7. SARSA and Q-Learning: On-Policy vs Off-Policy
Both SARSA and Q-learning estimate action values, but their targets differ. SARSA uses the next action actually selected by the behavior policy, so it learns the value of that policy. Q-learning replaces the next action by the greedy maximum, so it learns toward an optimal greedy target even while behavior explores. This is the classic distinction between on-policy and off-policy learning.
Key points
- ε-greedy is a simple exploration rule: with probability ε take a random action, otherwise take argmax Q.
- Q-learning can learn from data generated by another behavior policy, which is useful for replay buffers and offline data.
- With function approximation, off-policy + bootstrapping + approximation can become unstable; this is one reason deep Q-learning needs stabilizing tricks.
8. DQN: Why Deep Q-Learning Needs Replay and Target Networks
DQN replaces the Q-table with a neural network Q_θ(s,a). The naive idea is simple, but training becomes unstable because consecutive samples are strongly correlated and the target itself changes whenever θ changes. Experience replay breaks short-range correlation by sampling old transitions, while a separate target network θ⁻ changes slowly and stabilizes the bootstrap target.
Key points
- Replay buffer stores (s,a,r,s′,done) and samples mini-batches approximately i.i.d.
- Target network can be hard-updated every C steps or soft-updated with Polyak averaging.
- Double DQN reduces maximization bias by decoupling action selection and action evaluation.
- DQN is naturally suited to discrete actions; continuous control usually uses policy-gradient or actor–critic methods.
Training loop
Interact with the environment → append transition to replay → sample a batch → build y with the target network → minimize squared TD error → occasionally update the target network.
9. Policy Gradient: Optimizing the Policy Directly
Instead of learning Q and then taking argmax, policy-gradient methods parameterize π_θ(a|s) directly and maximize expected return. The policy-gradient theorem turns this objective into an expectation that can be estimated from sampled trajectories. Multiplying log-probability gradients by return or advantage increases the probability of better-than-expected actions and decreases the probability of worse actions.
Key points
- For discrete actions, π may be a categorical distribution; for continuous actions, a Gaussian policy is common.
- REINFORCE is unbiased in its basic form but can have very high variance.
- Subtracting a state-dependent baseline such as V(s) does not change the expected gradient but reduces variance; this leads naturally to advantage methods.
10. Actor–Critic and GAE
Actor–critic methods split the job into two learned components. The actor π_θ chooses actions; the critic V_φ or Q_φ evaluates them. The critic supplies a lower-variance learning signal to the actor. Generalized Advantage Estimation (GAE) combines multi-step TD errors with a parameter λ, creating a practical bias–variance trade-off used by PPO and many modern on-policy algorithms.
Key points
- λ≈0 behaves like one-step TD: lower variance, more bootstrap bias.
- λ≈1 approaches Monte Carlo-style advantages: lower bias but higher variance.
- The critic is not merely auxiliary: a poor value estimate can make the actor update noisy or systematically wrong.
11. PPO: Stable Policy Updates in Practice
PPO is popular because it keeps the basic policy-gradient workflow while limiting destructive policy updates. It compares the probability of each sampled action under the new and old policies through the ratio r_t(θ). The clipped objective prevents this ratio from moving too far in a direction that would over-amplify the advantage signal. PPO is still an on-policy method: data should not be reused indefinitely after the policy has changed substantially.
Key points
- Typical PPO training alternates rollout collection and several epochs of mini-batch optimization on that rollout.
- The total loss usually contains policy loss, value loss, and an entropy bonus that encourages exploration.
- Advantage normalization, observation normalization, reward scaling, gradient clipping, and correct terminal handling often matter as much as the headline formula.
Common pitfalls
- Treating PPO as off-policy and training too many epochs on stale data.
- Using reward magnitudes that differ by several orders, causing value loss or policy gradients to dominate.
- Ignoring action bounds for continuous control; a Gaussian sample may need squashing or clipping with the corresponding log-probability correction.
12. Constrained and Safe RL: Connecting RL to Control
In engineering systems, reward maximization alone is rarely enough. Energy budgets, collision avoidance, latency limits, queue stability, and actuator bounds are constraints, not preferences. A constrained MDP separates reward from one or more cost signals and optimizes return subject to cost limits. In safety-critical applications, RL is often combined with model predictive control, control barrier functions, shielding, or optimization-based repair so that learned actions are filtered before execution.
Key points
- Lagrangian methods turn constraints into adaptive penalties, but feasibility is usually not guaranteed at every time step.
- Safety layers solve a small optimization problem that minimally modifies the RL action to satisfy known constraints.
- Use model-free RL when a reliable model is unavailable and interaction is affordable; use MPC when a useful model and hard constraints dominate; hybrid methods are often attractive in networked control.
What you should remember
After this note, you should be able to read a modern RL paper and identify its state/action/reward, value target, policy update, on/off-policy nature, exploration mechanism, and constraint-handling strategy.