This homework is due Tuesday March 8, 2016 at 8PM EST. When complete, submit your code in an R Markdown file and the knitted HTML via GitHub.
In 2012 Nate Silver, and other data scientists, predicted the outcome of each state correctly. They did this by aggregating data from many polls to create more precise estimates than what one single poll can provide.
In this homework, we will try to predict the results of the democratic and republican primaries by studying the performance of polls in elections that already occurred and then aggregating results.
The first step in our analysis will be to wrangle the data in a way that will simplify the analysis. Ultimately, we want a table of results with each poll represented by a row and including results for each candidate as well as information about the poll such as name and date.
Install and load the pollstR package. This package provides functions to access data in the Huffington Post’s database. Read the help file for the pollstr_polls() function and write a function that reads all the polls related to the republican primaries. Name the object race2016. Hint: Visit this webpage to select the right topic and make sure to change the max_pages argument.
Examine and familiarize yourself with the race2016 object. Note that the questions component has a table with election results. Look at the topic component of the questions component. Create a new table with only the results from the 2016-president-gop-primary and only state (or territory) polls, no national polls. Hint: create a new object called results with the table of results and use dplyr. How many rows are we left with?
##Your code hereIn Problem 1B, we created a table called results with over 4000 rows. Does this mean that we have data for 4000 polls? How many polls did we actually have? Hint: look at the id column and use the group_by command.
##Your code hereLook at the first row of your results table. What date was this poll conducted? Hint: Use the polls component of the race2016 object to find the date.
##Your code hereNow examine the candidates in the “choices” column included in results table. Hint: use the table() function. Note that there are several choices that not going to be informative. For example, we have candidates that have dropped out. We also have entries such as No one, No One and No Preference. Filter the results table to include only Rubio and Trump.
##Your code hereIn our results table, we have one row for each candidate in each poll. Transform the results table to have one row for each poll and columns for each Rubio and Trump. Next, create a column called diff with the difference between Trump and Rubio. Hint: Remove the first_name and last_name columns then use the tidyr function spread().
##Your code hereFor each poll in the results table, we want to know the start date and the end date of the poll along with the pollster name and the type of poll it was. Hint: This information is in the polls component of race2016. You can select the relevant columns then use the id column to join the tables. One of the join functions in tidyr will do the trick.
##Your code hereStudy the type of values in the pollster column. Notice that you have many different values but that certain names commonly appear in these values. For example, consider the name “NBC” in the pollster column. NBC here is the Survey House. Use a join function again to add the survey house to the results table. Rename the column house. Hint: race2016$survey_house has the information you need.
##Your code hereWe now have a table with all the information we need. We will now use the results from Iowa, New Hampshire, Nevada and South Carolina to determine how to create a prediction for upcoming primaries.
Use an internet search to determine the results for the Iowa, New Hampshire, Nevada and South Carolina primaries for the top three candidates. Create a table called actual with this information. Also, create a column with the actual election difference. Use a join function to add this information to our results table.
##Your code hereCreate boxplots of the poll results for Trump in Iowa stratified by the pollster survey house for polls having more than 4 total results. Add a horizontal line with the actual results. Hint: Use the group_by, mutate, filter and ungroup functions in dplyr for the filtering step.
##Your code hereUsing the poll results for Trump in Iowa, compute the standard deviation for the results from each pollster house for polls having more than 4 total results. Then, study the typical standard deviation sizes used in these polls. Create a new table with two columns: the observed standard deviation and the standard deviations that theory predicts. For the prediction you have several observations. Pick the smallest one. Which is larger, the observed or the theoretical?
##Your code hereNow using the data from Problem 2C, plot the individual values against the time the poll was taken (use the end_date). Repeat this for each of the four states. Use color to denote pollster house. Using this plot, explain why the theory does not match the observed results?
##Your code hereConsider the Trump - Rubio difference. For each poll in IA, NH, SC and NV, compute the error between the prediction and actual election results. Use exploratory data analysis to get an idea of how time and pollster impacts accuracy.
##Your code hereFor polls from IA, NH, and SC, aggregate all polls from within 1 week of the election (use the start_date to determine cutoff) to provide a 95% confidence interval for the difference between Trump and Rubio. Compare the following two approaches: (1) the method that assumes that all variance comes from sampling error and (2) the approach that estimates variance empirically.
##Your code hereBefore seeing any polls my prior belief is that Rubio will beat Trump in Florida. If I were to quantify this belief I would say that the distribution of the Trump - Rubio was normal with mean \(\mu=-20\) percent and standard deviation \(\tau=10\). Let’s call the difference \(\theta\). Then
\[ \theta \sim N( \mu, \tau) \]
Under my prior belief, what is the chance that Trump would beat Rubio in Florida.
##Your code hereConsider the latest 25 Florida polls. Assume the poll results for the difference are normal distributed with mean \(\theta\) and standard deviation \(\sigma\). Provide an estimate for \(\theta\) and an estimate of the standard deviation \(\sigma\).
##Your code here\[ \hat{\theta} \sim N( \theta, \sigma/ \sqrt{25})\]
Now use the Central Limit Theorem to construct a confidence interval.
##Your code hereCombine these two results to provide the mean and standard deviation of a posterior distribution for \(\theta\).
##Your code hereUse the result form Problem 3C to provide your estimate of Trump beating Rubio in Florida.
##Your code hereUse the poll data as well as the results from Super Tuesday (March 1st) and other election results that happen before the deadline to make predictions for each remaining primary. Then use these results to estimate the probability of Trump winning the republican nomination. Justify your answer with figures, statistical arguments, and Monte Carlo simulations.
It will help to learn about how delegates are assigned. Here is the manual