sprt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wald's Sequential Probability Ratio Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequential test: make a decision after each draw from the population\n",
    "\n",
    "Sequence of data $X_1, X_2, \\ldots$. \n",
    "\n",
    "Two hypotheses, $H_0$ and $H_1$, each of which completely specifies the joint distribution of the data.\n",
    "Either $H_0$ or $H_1$ is true.\n",
    "\n",
    "Assume that the joint distributions of $(X_j)$ under $H_0$ and $H_1$ have densities with respect to the same dominating measure.\n",
    "\n",
    "Let $f_{0m}$ be the likelihood of $H_0$ for data $(X_j)_{j=1}^m$ and let $f_{1m}$ be the likelihood of $H_1$ for data $(X_j)_{j=1}^m$.\n",
    "\n",
    "The likelihood ratio of $H_1$ to $H_0$ is $f_{1m}/f_{0m}$; this is (speaking informally) the probability of observing $X_1, \\ldots, X_m$ if $H_1$ is true, divided by the probability of observing $X_1, \\ldots, X_m$ if $H_0$ is true.\n",
    "\n",
    "The probability of observing the data actually observed will tend to be higher for whichever\n",
    "hypothesis is in fact true, so this likelihood ratio will tend to be greater than $1$ if $H_1$ is true,\n",
    "and will tend to be less than $1$ if $H_0$ is true.\n",
    "The more observations we make, the more probable it is that the\n",
    "resulting likelihood ratio will be small if $H_0$ is true.\n",
    "Wald (1945) showed that if $H_0$ is true, then the probability is at most $\\alpha$\n",
    "that the likelihood ratio\n",
    "is ever greater than $1/\\alpha$, no matter how many observations are made.\n",
    "More generally, we have:\n",
    "\n",
    "<div class=\"admonition\"> Wald's SPRT\n",
    "    <p class=\"title\">Theorem</p>\n",
    "\n",
    "For any $\\alpha \\in (0, 1)$ and $\\beta \\in [0, 1)$, the following sequential algorithm tests the hypothesis $H_0$ at level no larger than $\\alpha$ and with power at least $1-\\beta$ against the alternative $H_1$.\n",
    "\n",
    "Set $m=0$.\n",
    "\n",
    "1. Increment $m$\n",
    "1. If $\\frac{f_{1m}}{f_{0m}} \\ge \\frac{1-\\beta}{\\alpha}$, stop and reject $H_0$.\n",
    "1. If $\\frac{f_{1m}}{f_{0m}} \\le \\frac{\\beta}{1-\\alpha}$, stop and do not reject $H_0$.\n",
    "1. If $\\frac{\\beta}{1-\\alpha} < \\frac{f_{1m}}{f_{0m}} < \\frac{1-\\beta}{\\alpha}$, go to step 1.\n",
    "  \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SPRT miracle\n",
    "\n",
    "Don't need to know the distribution of the likelihood ratio $\\mbox{LR}=\\frac{f_{1m}}{f_{0m}}$ under the null hypothesis\n",
    "to find the critical values for the test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connection to Gambler's ruin\n",
    "\n",
    "For IID data, the likelihood ratio is a product of terms. On a log scale, it's a sum.  Each \"trial\" produces another term in the sum, positive or negative. But&mdash;unlike the classical Gambler's Ruin problem in which the game is fair&mdash;the terms are not equal in magnitude: the steps are not all the same size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IID Bernoulli trials: sampling with replacement from binary populations\n",
    "\n",
    "Suppose $X_1, X_2, \\ldots,$ are IID $\\mbox{Bernoulli}(p)$ random variables\n",
    "and let $p_1 > p_0$.\n",
    "\n",
    "Set $\\mbox{LR} \\leftarrow 1$ and $j \\leftarrow 0$.\n",
    "\n",
    " + Increment $j$\n",
    " + If $X_j = 1$, $\\mbox{LR} \\leftarrow \\mbox{LR} \\times p_1/p_0$. \n",
    " + If $X_j = 0$, $\\mbox{LR} \\leftarrow \\mbox{LR} \\times (1-p_1)/(1- p_0)$.\n",
    "\n",
    "What's $\\mbox{LR}$ at stage $m$?\n",
    "Let $T_m := \\sum_{j=1}^m X_j$.\n",
    "\\begin{equation*} \n",
    "     \\frac{p_{1m}}{p_{0m}} := \n",
    "         \\frac{p_1^{T_m}(1-p_1)^{m-T_m}}{p_0^{T_m}(1-p_0)^{m-T_m}}.\n",
    "\\end{equation*}\n",
    "This is the ratio of binomial probability when $p = p_1$ to binomial probability when\n",
    "$p = p_0$ (the binomial coefficients in the numerator and denominator cancel).\n",
    "It simplifies further to\n",
    "\\begin{equation*} \n",
    "  \\frac{p_{1m}}{p_{0m}} = (p_0/p_1)^{T_m} ((1-p_0)/(1-p_1))^{m-T_m}.\n",
    "\\end{equation*}\n",
    "\n",
    "## Wald's SPRT for $p$ in iid Bernoulli trials\n",
    "\n",
    "Conclude $p > p_0$ if \n",
    "\\begin{equation*} \n",
    "     \\frac{p_{1m}}{p_{0m}} \\ge \\frac{1-\\beta}{\\alpha}.\n",
    "\\end{equation*}\n",
    "Conclude $p \\le p_0$ if\n",
    "\\begin{equation*} \n",
    "     \\frac{p_{1m}}{p_{0m}} \\le \\frac{\\beta}{1-\\alpha}.\n",
    "\\end{equation*}\n",
    "Otherwise, draw again.\n",
    "\n",
    "The SPRT approximately minimizes the \n",
    "expected sample size\n",
    "when $p \\le p_0$ or $p > p_1$.\n",
    "For values in $(p_1, p_0)$, it can have larger sample sizes than fixed-sample-size tests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simulation of SPRT for Binomial\n",
    "\n",
    "Let's watch the SPRT in action for iid Bernoulli trials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is the first cell with code: set up the Python environment\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "import numpy as np\n",
    "import scipy as sp\n",
    "import scipy.stats\n",
    "from scipy.stats import binom\n",
    "import pandas as pd\n",
    "# For interactive widgets\n",
    "from ipywidgets import interact, interactive, fixed\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import clear_output, display, HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "28f6e62ba90642a0836a69dbe4f391d5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=100, description='n', max=300, min=5, step=5), FloatSlider(value=0.45, d…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.plotBinomialSPRT(n, p, p0, p1, alpha, beta)>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def plotBinomialSPRT(n, p, p0, p1, alpha, beta):\n",
    "    '''\n",
    "       Plots the progress of the SPR for n iid Bernoulli trials with probabiity p\n",
    "       of success, for testing the hypothesis that p=p0 against the hypothesis p=p1\n",
    "       with significance level alpha and power 1-beta\n",
    "    '''\n",
    "    fig, ax = plt.subplots(nrows=2, ncols=1, sharex=True)\n",
    "    trials = sp.stats.binom.rvs(1, p, size=n+1)  # leave room to start at 1\n",
    "    terms = np.ones(n+1)\n",
    "    sfac = p1/p0\n",
    "    ffac = (1.0-p1)/(1.0-p0)\n",
    "    terms[trials == 1.0] = sfac\n",
    "    terms[trials == 0.0] = ffac\n",
    "    terms[0] = 1.0\n",
    "    logterms = np.log(terms)\n",
    "    #\n",
    "    ax[0].plot(range(n+1),np.cumprod(terms), color='b')\n",
    "    ax[0].axhline(y=(1-beta)/alpha, xmin=0, xmax=n, color='g', label=r'$(1-\\beta)/\\alpha$')\n",
    "    ax[0].axhline(y=beta/(1-alpha), xmin=0, xmax=n, color='r', label=r'$\\beta/(1-\\alpha)$')\n",
    "    ax[0].set_title('Simulation of Wald\\'s SPRT')\n",
    "    ax[0].set_ylabel('LR')\n",
    "    ax[0].legend(loc='best')\n",
    "    #\n",
    "    ax[1].plot(range(n+1),np.cumsum(logterms), color='b', linestyle='--')\n",
    "    ax[1].axhline(y=math.log((1-beta)/alpha), xmin=0, xmax=n, color='g', label=r'$log((1-\\beta)/\\alpha)$')\n",
    "    ax[1].axhline(y=math.log(beta/(1-alpha)), xmin=0, xmax=n, color='r', label=r'$log(\\beta/(1-\\alpha))$')\n",
    "    ax[1].set_ylabel('log(LR)')\n",
    "    ax[1].set_xlabel('trials')\n",
    "    ax[1].legend(loc='best')\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "interact(plotBinomialSPRT, n=widgets.IntSlider(min=5, max=300, step=5, value=100),\\\n",
    "         p=widgets.FloatSlider(min=0.001, max=1, step=0.01, value=.45),\\\n",
    "         p0=widgets.FloatSlider(min=0.001, max=1, step=0.01, value=.5),\\\n",
    "         p1=widgets.FloatSlider(min=0.001, max=1, step=0.01, value=.6),\\\n",
    "         alpha=widgets.FloatSlider(min=0.001, max=1, step=0.01, value=.05),\\\n",
    "         beta=widgets.FloatSlider(min=0.001, max=1, step=0.01, value=.05)\n",
    "         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Behavior\n",
    "For $p_0 < p_1$,\n",
    "+ when $p \\ge p_1$, the likelihood ratio is likely to cross the upper (green) line eventually and unlikely to cross the lower (red) line.\n",
    "+ when $p \\le p_0$, the likelihood ratio is likely to cross the lower (red) line eventually and unlikely to cross the upper (green) line.\n",
    "+ the SPRT approximately minimizes the expected number of trials before rejecting one or the other hypothesis, provided $p \\notin (p_0, p_1)$.\n",
    "\n",
    "For $p_1 < p_0$, the directions are reversed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## It works for $\\beta = 0$ too: $P$-values\n",
    "The inequalities hold when $\\beta = 0$ also. Then the likelihood ratio has chance at most $\\alpha$ of ever being greater than $1/\\alpha$, if in fact $p > p_0$.\n",
    "\n",
    "Hence, $1/\\mbox{LR}$ is the $P$-value of the hypothesis $p > p_0$. This can be used to construct one-sided confidence bounds for $p$ and other parameters. The next chapter does exactly that, to find a lower confidence bound for the mean of a nonnegative population."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependent observations\n",
    "As an illustration of SPRT in for obervations that are not iid, consider the following problem.\n",
    "\n",
    "There is a population of $B$ items. Item $b$ has \"weight\" $N_b$ and \"value\" $a_b \\in \\{0, 1\\}$. Let $N = \\sum_{b=1}^B N_b$. (You might think of items as manufacturing lots, where $a_b = 1$ means the lot is acceptable and $a_b=0$ means the lot is defective.)\n",
    "\n",
    "We want to test the hypothesis $H_0$ that $\\frac{1}{N}\\sum_b N_b a_b = 1/2$ against the\n",
    "alternative hypothesis $H_1$ that $\\frac{1}{N}\\sum_b N_b a_b = \\gamma $, for some\n",
    "fixed $\\gamma > 1/2$.\n",
    "\n",
    "We will draw items sequentially, without replacement, such that the chance that item $b$ is selected in draw $i$, assuming it has not been selected already, is $N_b/\\sum_{j \\notin {\\mathcal B_{i-1}}} N_j$,\n",
    "where ${\\mathcal B_{i-1}}$ is the sample up to and including the $i-1$st draw,\n",
    "and ${\\mathcal B_0} := \\emptyset$. \n",
    "That is, the chance of selecting an item is in proportion to its weight among the items\n",
    "that have not yet been selected.\n",
    "\n",
    "Let $\\mathbb B_i$ denote the item selected at random in such a manner in the $i$th draw.\n",
    "\n",
    "The chance that the first draw ${\\mathbb B_1}$ gives an item with value 1, i.e., \n",
    "$\\Pr \\{a_{\\mathbb B_1} = 1\\}$, is $\\frac{1}{N}\\sum_b N_b a_b$.\n",
    "Under $H_0$, this chance is $p_{01} = 1/2$; under $H_1$, this chance is \n",
    "$p_{11} = \\gamma$.\n",
    "\n",
    "Given the values of $\\{a_{\\mathbb B_k}\\}_{k=1}^i$, the conditional\n",
    "probability that the $i$th draw gives an item with value 1 is\n",
    "\n",
    "\\begin{equation*} \n",
    "   \\Pr \\{a_{\\mathbb B_i} = 1 | {\\mathcal B_{i-1}} \\} = \\frac{ \\sum_{b \\notin {\\mathcal B_{i-1}}} N_b a_b}{\\sum_{b \\notin {\\mathcal B_{i-1}}} N_b}.\n",
    "\\end{equation*}\n",
    "\n",
    "Under $H_0$, this chance is\n",
    "\n",
    "\\begin{equation*} \n",
    "   p_{0i} =  \\frac{N/2 - \\sum_{b \\in {\\mathcal B_{i-1}}} N_b a_b}{N - \\sum_{b \\in {\\mathcal B_{i-1}}} N_b}.\n",
    "\\end{equation*}\n",
    "\n",
    "Under $H_1$, this chance is\n",
    "\n",
    "\\begin{equation*} \n",
    "   p_{1i} = \\frac{N \\gamma - \\sum_{b \\in {\\mathcal B_{i-1}}} N_b a_b}{N - \\sum_{b \\in {\\mathcal B_{i-1}}} N_b}.\n",
    "\\end{equation*}\n",
    "\n",
    "Let $X_k$ be the indicator of the event that the $k$th draw gives an item with\n",
    "value $1$, i.e., the indicator of the event $a_{\\mathbb B_k} = 1$.\n",
    "The likelihood ratio for a given sequence $\\{X_k\\}_{k=1}^i$ is\n",
    "\n",
    "\\begin{equation*} \n",
    "    \\mbox{LR} = \\frac{\\prod_{k=1}^i p_{1k}^{X_k}(1-p_{1k})^{1-X_k}}\n",
    "         {\\prod_{k=1}^i p_{0k}^{X_k}(1-p_{0k})^{1-X_k}}.\n",
    "\\end{equation*}\n",
    "\n",
    "This can be simplified: $p_{0k}$ and $p_{1k}$ have the same denominator,\n",
    "$N - \\sum_{b \\in {\\mathcal B_{i-1}}} N_b$,\n",
    "and their numerators share a term.\n",
    "Define $N(k) := \\sum_{j = 1}^{k-1}N_b$ and\n",
    "$A(k) := \\sum_{j = 1}^{k-1}N_b a_b$.\n",
    "Then\n",
    "\n",
    "\\begin{equation*} \n",
    "    \\mbox{LR} = \\prod_{k=1}^i \n",
    "    \\left ( \\frac{N\\gamma - A(k)}{N/2 - A(k)} \\right )^{X_k}\n",
    "    \\left ( \\frac{N-N\\gamma - (N(k) - A(k))}{N-N/2 - (N(k)-A(k))} \\right )^{1-X_k}\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*} \n",
    "   = \\prod_{k=1}^i\n",
    "\\left ( \\frac{N\\gamma - A(k)}{N/2 - A(k)} \\right )^{X_k}\n",
    "\\left ( \\frac{N(1-\\gamma) - N(k) + A(k)}{N/2 - N(k) + A(k)} \\right )^{1-X_k},\n",
    "\\end{equation*}\n",
    "where the products are defined to be infinite if the denominator vanishes anywhere.\n",
    "\n",
    "If $H_0$ is true, the chance that $\\mbox{LR}$ is ever greater than $1/\\alpha$\n",
    "is at most $\\alpha$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nuisance parameters\n",
    "\n",
    "This example arises in constructing risk-limiting audits using \"ballot polling,\" which involves selecting\n",
    "individual cast ballots and reading voter intent from them manually.\n",
    "\n",
    "\n",
    "Consider a population of $N$ items, each of which has one of three possible labels, $\\{w, \\ell, u\\}$.\n",
    "Let $N_w$ denote the number of items labeled $w$, $N_\\ell$ denote the number labeled $\\ell$, and $N_u$ \n",
    "denote the number labeled $u$.\n",
    "We want to test the compound hypothesis that $N_\\ell \\ge N_w$ against the alternative that $N_w = V_w$, $N_\\ell = V_\\ell$, and $N_u = V_u$, with $V_w > V_\\ell$.\n",
    "\n",
    "In election auditing, the items are ballots and the types are votes: $w$ means the ballot shows a vote \n",
    "for reported a particular reported winner $w$ and not for particular reported loser $\\ell$, $\\ell$ is a ballot with vote for $\\ell$ but not for $w$, and $u$ means the ballot shows an undervote or invalid vote or a vote for both $w$ and $\\ell$.\n",
    "The values $V_w$, $V_\\ell$, and $V_u$ are the reported results (or related values); the values $N_w$, $N_\\ell$, and $N_u$ are what a full manual tally of the paper ballots would show.\n",
    "Reported winner $w$ really beat reported loser $\\ell$ if $N_w > N_\\ell$. \n",
    "The null hypothesis $N_\\ell \\ge N_w$ is that the reported out come is wrong: the outcome was a tie or $\\ell$ beat $w$.)\n",
    "\n",
    "In this problem, $N_u$ (equivalently, $N_w + N_\\ell$) is a nuisance parameter: we care about $N_\\ell - N_w$.\n",
    "\n",
    "Suppose we sample sequentially without replacement from the $N$ items in such a way that in every draw, each of the unselected items is equally likely to be selected.\n",
    "Let $X_k$ be the label of the item selected on the $k$th draw; let $W_n := \\sum_{k=1}^n 1_{X_k = w}$;\n",
    "and define $L_n$ and $U_n$ analogously.\n",
    "\n",
    "The probability of a given data sequence $X_1, \\ldots, X_n$ under the alternative hypothesis is\n",
    "\n",
    "\\begin{equation*} \n",
    "    \\frac{V_w(V_w - 1) \\cdots (V_w - W_n+1) \\times \n",
    "    V_\\ell(V_\\ell - 1) \\cdots (V_\\ell - L_n+1) \\times\n",
    "    V_u(V_u - 1) \\cdots (V_u - U_n+1)}{N(N-1) \\cdots (N-n+1)}.\n",
    "\\end{equation*}\n",
    "\n",
    "If $L_n \\ge W_n$, the data obviously do not provide evidence against the null, so we suppose that\n",
    "$L_n < W_n$, in which case, the element of the null that will maximize the probability of the observed data\n",
    "has a tie: $N_w = N_\\ell$.\n",
    "Under the null hypothesis, the probability of $X_1, \\ldots, X_n$ is\n",
    "\n",
    "\\begin{equation*} \n",
    "    \\frac{N_w(N_w - 1) \\cdots (N_w - W_n+1) \\times \n",
    "    N_w(N_w - 1) \\cdots (N_w - L_n+1) \\times\n",
    "    N_u(N_u - 1) \\cdots (N_u - U_n+1)}{N(N-1) \\cdots (N-n+1)},\n",
    "\\end{equation*}\n",
    "\n",
    "for some value $N_w$ and the corresponding $N_u = N - 2N_w$.\n",
    "\n",
    "How large can that probability be under the null? \n",
    "Certainly no larger than the likelihood of the maximum likelihood estimate of $N_w$ when $N_w$ is constrained to\n",
    "equal $N_\\ell$.\n",
    "We thus have a one-dimensional optimization problem. \n",
    "The probability under the null is maximized by any integer value $x \\in \\{ \\max(W_n, L_n), \\ldots, (N-U_n)/2 \\}$ that maximizes \n",
    "\n",
    "\\begin{equation*} x(x - 1) \\cdots (x - W_n+1) \\times x(x - 1) \\cdots (x - L_n+1) \\times (N-2x)(N-2x - 1) \\cdots (N-2x - U_n+1).\\end{equation*}\n",
    "\n",
    "The logarithm is monotonic, so any maximizer, $x^*$, also maximizes\n",
    "\n",
    "\\begin{equation*}  f(x) = \\sum_{i=0}^{W_n-1} \\ln (x-i) + \\sum_{i=0}^{L_n-1} \\ln (x-i) + \\sum_{i=0}^{U_n-1} \\ln (N-2x-i).\\end{equation*}\n",
    "\n",
    "If we treat $x$ as a continuous variable, we can differentiate $f$ with respect to $x$.\n",
    "The second derivative is negative, so the function is convex, and a local maximum is\n",
    "a global maximum.\n",
    "The optimal value will be the integer value of $x$ closest to the real value of $x$ that is\n",
    "a stationary point.\n",
    "\n",
    "A conservative $p$ value for the null hypothesis after $n$ items have been drawn is thus\n",
    "\n",
    "\\begin{equation*} \n",
    "    \\frac{\\prod_{i=0}^{W_n-1} (x^*-i) \\; \\prod_{i=0}^{L_n-1}  (x^*-i) \\; \\prod_{i=0}^{U_n-1} (N-2x^*-i)}{\\prod_{i=0}^{W_n-1}(V_w-i) \\; \\prod_{i=0}^{L_n-1} (V_\\ell-i) \\; \\prod_{i=0}^{U_n-1} (V_u-i)}.\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative approach that avoids the need to maximize over a nuisance parameter is given in SHANGRLA:\n",
    "Sets of Half-Average Nulls Generate Risk-Limiting Audits\n",
    "[Stark, 2020](https://dl.acm.org/doi/abs/10.1007/978-3-030-54455-3_23): in this example, it would encode a vote for \n",
    "$w$ as $1$, a vote for $u$ as $1/2$, and a vote for $\\ell$ as zero.\n",
    "That yields a finite list of values between $0$ and $1$.\n",
    "If the mean of that list is greater than $1/2$, $w$ really got more votes than $\\ell$.\n",
    "This transformation allows any method for testing whether the mean of a finite, bounded population is less\n",
    "than or equal to $1/2$ to be used.\n",
    "See also [the martingales notebook](./martingales.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TO DO: implement stationary point calculation\n",
    "\n",
    "def f_plus(x, Wn, Ln):\n",
    "    return sum([math.log(x-i) for i in range(0,Wn)]) + sum([math.log(x-i) for i in range(0,Ln)])\n",
    "\n",
    "def f_minus(x, Un, N):\n",
    "    return sum([math.log(N-2*x-i) for i in range(0, Un)])\n",
    "\n",
    "def f_all(x, Wn, Ln, Un, N):\n",
    "    return f_plus(x, Wn, Ln)+f_minus(x, Un, N)\n",
    "\n",
    "def feasible_range(Wn, Ln, Un, N):\n",
    "    '''\n",
    "    Returns the range of values of Nw consistent with the data\n",
    "    '''\n",
    "    return max(Wn,Ln), (N-Un)/2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## What's next?\n",
    "+ [Optional next: The SPRT for the population percentage, sampling without replacement](pSPRTnoReplacement.ipynb)\n",
    "+ [Next: The Kaplan-Wald Confidence Bound for a Nonnegative Mean](kaplanWald.ipynb)\n",
    "+ [Lower confidence bounds for the mean of a nonnegative population: Markov's Inequality & methods based on the empirical distribution](markov.ipynb)\n",
    "+ [Index](index.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}