{"id":513,"date":"2026-06-20T21:29:12","date_gmt":"2026-06-20T13:29:12","guid":{"rendered":"https:\/\/arkdoc.cc\/?p=513"},"modified":"2026-06-20T21:39:45","modified_gmt":"2026-06-20T13:39:45","slug":"lecture-policy-gradient","status":"publish","type":"post","link":"https:\/\/arkdoc.cc\/index.php\/2026\/06\/20\/lecture-policy-gradient\/","title":{"rendered":""},"content":{"rendered":"\n
\n\n

Lecture: Policy Gradient<\/h1>\n

For simplicity, let us first consider a one-step decision-making\nprocess (also known as a contextual bandit).<\/p>\n

Problem Setup<\/h4>\n

Let \\(s \\in \\set S\\)<\/span> be the state\nvector of the environment, and let \\(a \\in\n\\set A\\)<\/span> be the action taken by the agent. The reward function is\ndenoted by \\(r(a,s)\\)<\/span>. The agent\u2019s\nbehavior is characterized by a policy, which is conveniently represented\nas a conditional probability distribution, \\(\\pi(a \\mid s)\\)<\/span>, specifying the probability\nof selecting each action \\(a\\)<\/span> given a\nstate \\(s\\)<\/span>. The objective is to find\nthe optimal policy \\(\\pi^*\\)<\/span> that\nmaximizes the expected reward: \\[%R(\\pi \\mid\ns)\nR(\\pi) = \\mathbb{E}_{s\\sim p_0}\\mathbb{E}_{a\\sim \\pi(\\cdot | s)} [r(a,s)] = \\sum_{s,a}\np_\\pi(a, s) r(a,s),\\]<\/span> where \\(p_0\\)<\/span> denotes the distribution of states,\nand we have \\(p_{\\pi}(a,s)\\)<\/span> denotes\nthe joint distribution of \\((a,s)\\)<\/span>\nwhen policy \\(\\pi\\)<\/span> is used: \\[p_\\pi(a,s) = \\pi(a\\mid s) p_0(s).\\]<\/span>\nTypically, \\(p_0\\)<\/span> is unknown to us,\nbut observed through a collection of data \\(\\{s^{(i)}\\}\\)<\/span> drawn from \\(p_0\\)<\/span>.<\/p>\n

\n\n<\/div>\n

Policy Gradient<\/h4>\n

In practice, the policy \\(\\pi(a \\mid s) =\n\\pi_\\theta(a \\mid s)\\)<\/span> is parameterized by a function with\nparameter \\(\\theta\\)<\/span>. Assume the\nstate-action space \\(\\mathcal S \\times\n\\mathcal A\\)<\/span> is finite or accountable, the gradient of the\nexpected reward with respect to \\(\\theta\\)<\/span> is \\[\\begin{align}\n\\nabla_\\theta R(\\textcolor{blue}{\\pi_\\theta})\n& = \\nabla_\\theta \\left ( \\sum_{s,a} p_0(s) \\textcolor{blue}{\\pi_\\theta(a \\mid\ns)} r(a, s) \\right ) \\notag \\\\\n& = \\sum_{s,a} p_0(s) \\textcolor{blue}{\\nabla_\\theta \\pi_\\theta(a \\mid s)} r(a,\ns) \\notag \\\\\n& = \\sum_{s,a} p_0(s) \\textcolor{blue}{\\pi_\\theta(a \\mid s)}\\textcolor{blue}{ \\frac{\n\\nabla_\\theta \\pi_\\theta(a \\mid s) }{\\pi_\\theta(a \\mid s)} } r(a, s)\n\\notag \\\\\n& = \\mathbb{E}_{\\textcolor{blue}{p_{\\pi_\\theta}}} \\left[ r(a, s) \\textcolor{blue}{\\nabla_\\theta\n\\log \\pi_\\theta(a \\mid s)} \\right], \\notag\n\\end{align}\\]<\/span> where the parts depending on \\(\\theta\\)<\/span> are highlighted, and we use the\nlog-derivative trick \\(\\nabla_\\theta \\log \\pi =\n\\nabla_\\theta \\pi_\\theta \/ \\pi_\\theta\\)<\/span>. This allows us to\nexpress the derivative of an expectation with respect to a distribution\nas the expectation of the reward weighted by \\(\\nabla_\\theta \\log \\pi_\\theta(a \\mid s)\\)<\/span>.<\/p>\n

Reward Baseline<\/h4>\n

For any policy \\(\\pi^\\theta\\)<\/span>, it is\neasy to prove the following identity: \\[\\mathbb{E}_{a\\sim \\pi_\\theta(\\cdot|s)}[\\nabla_\\theta \\log\n\\pi_\\theta(a\\mid s)] =0,~~~~~ \\forall s,~\\theta.\\]<\/span><\/p>\n

Hence, we can generalize the gradient formula above to \\[\\begin{align}\n\\label{equ:pcgrad2}\n\\nabla_\\theta R(\\pi_\\theta )\n& = \\mathbb{E}_{p_{\\pi_\\theta}} \\left [ (r(a,s) –\n\\textcolor{magenta}{v(s)}) \\textcolor{blue}{\\nabla_\\theta \\log \\pi_\\theta (a\\mid s)} \\right ],\n\\end{align}\\]<\/span> where \\(v(s)\\)<\/span> is\nany<\/em> function that does not depend on the action \\(a\\)<\/span>. The choice of \\(v\\)<\/span> does not alter the expectation of the\nformula, but a proper choice of \\(v\\)<\/span>\ncan reduce the variance of the mean estimation given empirical\nobservations.<\/p>\n

One common choice of baseline is \\(v(s) =\n\\mathbb{E}_{a\\sim \\pi_\\theta(\\cdot|s)}[r(a,s)]\\)<\/span>, in which case the\ndifference \\(r(a,s) – v(s)\\)<\/span> is known\nas the advantage function<\/em>: \\[A(a, s)\n= r(a,s) – v(s).\\]<\/span> A positive value \\(A(a,s) > 0\\)<\/span> (respectively, negative\n\\(<0\\)<\/span>) indicates that the action\n\\(a\\)<\/span> performs above (respectively,\nbelow) the average.<\/p>\n

On-Policy Estimation<\/h4>\n

In practice, given observations \\(\\{s^{(i)}\\}\\)<\/span> drawn from \\(p_0\\)<\/span> and \\(a^{(i)} \\sim \\pi^\\theta(\\cdot|s^{(i)})\\)<\/span>\ndrawn from the ongoing policy \\(\\pi^\\theta\\)<\/span>, the gradient can be estimated\nby \\[\\begin{align}\n\\label{equ:onpolicyestimate}\n\\nabla_\\theta R(\\pi_\\theta )\n\\approx \\frac{1}{n}\\sum_{i=1}^n \nA^{(i)} \\nabla_\\theta \\log \\pi_\\theta (a^{(i)} \\mid s^{(i)}),~~~~~~\nA^{(i)} = A(a^{(i)}, s^{(i)}).\n\\end{align}\\]<\/span> The gradient ascent update, also known as\nREINFORCE, \\[\\theta\\gets \\theta + \\epsilon\n\\nabla_\\theta R(\\pi_\\theta ).\\]<\/span> Intuitively, the gradient \\(\\nabla_\\theta \\log \\pi_\\theta (a^{(i)} \\mid\ns^{(i)})\\)<\/span> for data points \\((a^{(i)},\ns^{(i)})\\)<\/span> with positive advantages \\(A^{(i)} > 0\\)<\/span> is positively weighted,\nincreasing the probability \\(\\pi_\\theta(a \\mid\ns)\\)<\/span> during gradient ascent. Conversely, for data points with\nnegative advantages, the gradient is negatively weighted, decreasing the\nprobability.<\/p>\n

Connection to Weighted MLE<\/h4>\n

Assuming the data \\(\\{a^{(i)},\ns^{(i)}\\}\\)<\/span> are fixed, the policy gradient \\(\\nabla_\\theta R(\\pi_\\theta)\\)<\/span> coincides with\nthe gradient of the \\(A^{(i)}\\)<\/span>-weighted log-likelihood function:\n\\[\\ell(\\pi_\\theta) = \\frac{1}{n} \\sum_{i=1}^n\nA^{(i)} \\log \\pi_\\theta(a^{(i)} \\mid s^{(i)}).\\]<\/span> This aligns with\nthe intuition of maximizing \\(\\log \\pi(a \\mid\ns)\\)<\/span> for data points with large (and positive) \\(A^{(i)}\\)<\/span>, while minimizing the\nlog-likelihood for negative \\(A^{(i)}\\)<\/span>.<\/p>\n

The key difference, however, is that the data \\(\\{a^{(i)}, s^{(i)}\\}\\)<\/span> are drawn from the\npolicy \\(\\pi_\\theta\\)<\/span> and thus depend\non the current parameter \\(\\theta\\)<\/span>. As\n\\(\\theta\\)<\/span> is updated during policy\noptimization, the data must either be regenerated or reweighted to\nreflect changes in \\(\\pi_\\theta\\)<\/span>. In\ncontrast, maximum likelihood estimation assumes the data are fixed.<\/p>\n

Hence, policy optimization can be viewed as an adaptively weighted\nmaximum likelihood estimation (MLE), where the weights (\\(A^{(i)}\\)<\/span>) are adjusted across iterations\nusing newly generated data.<\/p>\n

Off-Policy\nEstimation via Importance Sampling<\/h4>\n

The gradient estimation in [equ:onpolicyestimate]<\/a> is\non-policy<\/em>, because the actions \\(a\\)<\/span> must be drawn from the current policy\n\\(\\pi_\\theta(\\cdot | s)\\)<\/span>. This does\nnot yield efficient use of data as new actions must be rolled out once\nthe policy is updated, and data from different policies cannot be used.\nIn comparison, off-policy<\/em> methods allow us to leverage actions\nfrom different policies.<\/p>\n

One common approach to off-policy estimation is importance sampling.\nAssume that the data \\(\\{a^{(i)},\ns^{(i)}\\}\\)<\/span> are drawn from a behavior policy<\/em> \\(\\pi^{\\mathtt{data}}\\)<\/span>. Using importance\nsampling, we have<\/p>\n

\\[\\begin{align*}\n\\nabla_\\theta R(\\pi_\\theta) \n& = \\mathbb{E}_{p_{\\pi^{\\texttt{data}}}}\n\\left[\\frac{\\pi_\\theta(a|s)}{\\pi^{\\mathtt{data}}(a|s)} A(a,s)\n\\nabla_\\theta \\log \\pi_\\theta (a \\mid s) \\right] \\\\\n& \\approx \\frac{1}{z} \\sum_{i=1}^n\nw^{(i)} A^{(i)} \\nabla_\\theta \\log \\pi_\\theta (a^{(i)} \\mid s^{(i)}),\n~~~~~\n\\end{align*}\\]<\/span> where \\(w^{(i)}\\)<\/span>\nis the importance weight: \\[w^{(i)} =\n\\frac{\\pi_\\theta(a^{(i)}|s^{(i)})}{\\pi^{\\mathtt{data}}(a^{(i)}|s^{(i)})},\\]<\/span>\nand the normalization constant \\(z\\)<\/span> is\noften taken as \\(z = \\sum_i\nw^{(i)}\\)<\/span>.<\/p>\n

\n

In RL, on-policy methods<\/em> update the policy using data\ncollected from the current policy itself. Off-policy methods<\/em>\nlearn from data collected under a different policy, allowing reuse of\npast experiences and improving sample efficiency, but at the cost of a\ndistribution mismatch between the behavior policy \\(\\pi^{\\text{data}}\\)<\/span> and the target policy\n\\(\\pi_\\theta\\)<\/span>. Offline RL<\/em> is\nthe case of off-policy learning where training relies entirely on a\nfixed dataset with no further interaction.<\/p>\n<\/div>\n\n\n <\/div>\n","protected":false},"excerpt":{"rendered":"

Lecture: Policy Gradient For simplicity, let us first c … \u7ee7\u7eed\u9605\u8bfb<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-513","post","type-post","status-publish","format-standard","hentry","category-news"],"_links":{"self":[{"href":"https:\/\/arkdoc.cc\/index.php\/wp-json\/wp\/v2\/posts\/513","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/arkdoc.cc\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/arkdoc.cc\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/arkdoc.cc\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/arkdoc.cc\/index.php\/wp-json\/wp\/v2\/comments?post=513"}],"version-history":[{"count":3,"href":"https:\/\/arkdoc.cc\/index.php\/wp-json\/wp\/v2\/posts\/513\/revisions"}],"predecessor-version":[{"id":516,"href":"https:\/\/arkdoc.cc\/index.php\/wp-json\/wp\/v2\/posts\/513\/revisions\/516"}],"wp:attachment":[{"href":"https:\/\/arkdoc.cc\/index.php\/wp-json\/wp\/v2\/media?parent=513"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/arkdoc.cc\/index.php\/wp-json\/wp\/v2\/categories?post=513"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/arkdoc.cc\/index.php\/wp-json\/wp\/v2\/tags?post=513"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}