laser-lab-slides.html

<!DOCTYPE html>
<html lang="" xml:lang="">
  <head>
    <title>LASER Learning Lab Orientation</title>
    <meta charset="utf-8" />
    <meta name="author" content="The LASER Team" />
    <meta name="date" content="2023-07-17" />
    <script src="libs/header-attrs-2.26/header-attrs.js"></script>
    <link href="libs/remark-css-0.0.1/default.css" rel="stylesheet" />
    <link href="libs/panelset-0.2.6/panelset.css" rel="stylesheet" />
    <script src="libs/panelset-0.2.6/panelset.js"></script>
    <script src="libs/clipboard-2.0.6/clipboard.min.js"></script>
    <link href="libs/xaringanExtra-clipboard-0.2.6/xaringanExtra-clipboard.css" rel="stylesheet" />
    <script src="libs/xaringanExtra-clipboard-0.2.6/xaringanExtra-clipboard.js"></script>
    <script>window.xaringanExtraClipboard(null, {"button":"<i class=\"fa fa-clipboard\"><\/i>","success":"<i class=\"fa fa-check\" style=\"color: #90BE6D\"><\/i>","error":"Press Ctrl+C to Copy"})</script>
    <link href="libs/font-awesome-6.4.2/css/all.min.css" rel="stylesheet" />
    <link href="libs/font-awesome-6.4.2/css/v4-shims.min.css" rel="stylesheet" />
    <link href="libs/tile-view-0.2.6/tile-view.css" rel="stylesheet" />
    <script src="libs/tile-view-0.2.6/tile-view.js"></script>
    <link rel="stylesheet" href="css/laser.css" type="text/css" />
    <link rel="stylesheet" href="css/laser-fonts.css" type="text/css" />
  </head>
  <body>
    <textarea id="source">

class: clear, title-slide, inverse, center, top, middle


# LASER Learning Lab Orientation
## Reproducible Research &amp; The Data-Intensive Research Workflow
### The LASER Team
### July 17, 2023


---
# Agenda

.pull-left[
## Part 1: Core Concepts
### Reproducible Research with RStudio
- What is reproducible research? 
- The benefits of reproducibility 
- The tools of reproducible research 

]

.pull-right[

## Part 2: R Code-Along
### Data-Intensive Research Workflow
- Prepare 
- Wrangle 
- Explore 
- Model
- Communicate
]


???

Each Learning Lab begins with a 60-minute interactive presentation consisting of two parts. The first focuses on a conceptual overview of key terminology, techniques, and applications. The second part provides a short but highly structured code-along activity that demonstrates key packages and functions required for specific data analysis techniques highlighted in each unit and an exemplary research study.

In this learning lab we'll focus on the importance of reproducible research and tools for reproducible research; and then gain some hands on experience with the data-intensive research workflow to conduct a simple analysis with R. 

Everything we introduce in this presentation you will have an opportunity to explore in much greater depth through the essential readings and a coding case study after the break. 

---

class: clear, inverse, middle, center

# Part 1: Core Concepts

----
Reproducible Research with RStudio 

---
class: clear, laser

.quote[“...future you and anyone else will be able to **pick up your analysis and reproduce the same results**, including figures and tables.”]

.author[~ Harrison, 2020]


---
# What is Reproducible Research? 

.pull-left[
In computational sciences like LA:
- Ideally... researchers can ***replicate*** your findings by following the procedures used to gather the data and run the computer code.
- Realistically... the data and code used to make a finding are available and sufficient to ***recreate*** the finding. 
]
.pull-right[
.center[&lt;img src="img/reproducible-research.png" height="400px"/&gt;
]
]

.footnote[
Gandrud, C. (2013). [Reproducible research with R and R studio (3rd Edition).](https://github.com/christophergandrud/Rep-Res-Book). CRC Press.
]

???
The information from this conceptual overview on reproducible research is drawn largely from the excellent book by Christopher Gandrud,Reproducible research with R and R studio. 

The distinction between these two definitions is that we recognize  it is sometimes difficult to replicate the original data set in a study because of issues such as limited resources to gather new data, or because the original study may have already sampled the full universe of cases.

---
# The Benefits of Reproducible Research

.pull-left[
**For Science:**
- Standard to judge scientific claims 
- Enhances *replicability*
- Avoiding effort duplication
- Cumulative knowledge development
]
.pull-right[
**For Yourself:**
- Better work habits 
- Better teamwork
- Changes are easier
- Higher impact research
]


???
##For Science

###Standard to judge scientific claims 
Reproducibility opens up your findings and work to scrutiny, and is the **minimum** standard for judging scientific claims. However, “a study can be reproducible but still be wrong” (Peng, 2014).

###Reproducibility enhances replicability. 
If other researchers are able to clearly understand how a finding was originally made, then they will be better able to conduct comparable research, in meaningful attempts to replicate the original findings. However, a finding in one study may be statistically significant when **reproduced** using the original data/code, but when researchers try to **replicate** it using new data and even methods, they are unable to find a similar result. The original finding could have been noise, even though it is fully reproducible.

### Avoiding effort duplication
Reproducible research, in theory, should cut down on the amount of time others have to spend gathering data or developing procedures that have already been collected or figured out. 

### Cumulative knowledge development
They can instead use that time to more quickly build on established findings and develop new knowledge.


## For Yourself

### Better work habits 
It can spur you to more effectively plan and organize your research. It should push you to bring your data and source code up to a higher level of quality than you might if you thought no one was looking.

### Better teamwork
RR makes it easier for your collaborators figure out what you have done also to understand your work and up to speed on a research project. 

### Changes are easier
No actual research process is completely linear. You almost never gather data, run analyses, and present your results without going backwards to add variables, make changes to your statistical models, create new graphs, alter results tables in light of new findings. . RR makes it easier to reproduce parts or even all of your own research. 

### Higher impact research
Reproducible research is more likely to be useful for other researchers than nonreproducible research. Useful research is cited more frequently! 

---
class: middle, no-logo
# Reproducible Research?

&lt;img src="img/phd-comic.jpeg" height="500px"/&gt;

???

In practice, your research should be easy for independent researchers to reproduce! If a study is difficult to reproduce, it’s more likely that no one will reproduce it. If someone does attempt to reproduce it, it will be difficult for them to tell if any errors they find were in the original research or problems introduced during the reproduction.

So next we're going to talk about some general best practices for reproducible research, and then give an overview of the research tools we'll be working with throughout the LASER institute to help ensure it's relatively easy to reproduce our research. 

---
# Discussion

Think about the following questions and then discuss at your table: 

- What have your experiences been with reproducible research?

- What tools have you used to ensure the reproducibility of your work? 

- What questions do you have about reproducible research? 

---
# Best Practices for Reproducible Research 

1. Document everything! 

2. Everything is a (text) file.

3. All files should be human readable.

4. Explicitly tie your files together.

5. Have a plan to organize, store, and make your files available.

To learn more: [Reproducible research with R and R studio (3rd Edition)](https://github.com/christophergandrud/Rep-Res-Book). CRC Press.


???
## 1. Document everything! 
Obviously, in order to reproduce your research, others must be able to know what you did. Documentation of includes, but is not limited to the data itself, analysis code, the write-up of results, and explanations of these files (e.g. data set codebooks, software information, and so on). 

## 2. Everything is a (text) file. 
Ideally, your documentation will be stored in the simplest file format possible to store this information. Usually the simplest file format is the humble, but versatile, text file. Text files are extremely nimble and can used your data in, like in comma-separated values (CSV) format for example, can contain your analysis code in files, and can be the basis for publishable formats like articles, webpages, books, etc. using markup languages such as Markdown.
One reason reproducible research is best stored in text fles is that this helps future-proof your research

## 3. All files should be human readable. 
Treat all of your research fles as if someone who has not worked on the project will, in the future, try to understand them. Computer code is a way of communicating with the computer. It is ‘machine readable’ in that the computer is able to use it to understand what you want to do.2 However, there is a very good chance that other people (or you six months in the future) will not understand what you were telling the computer. So, you need to make all of your files ‘human readable’. To make them human readable, you should comment on your code with the goal of communicating its design and purpose.

## 4. Explicitly tie your files together. 
Researchers often do not explicitly document the relationships between fles that they used in their research. For example, the results of an analysis–a table or figure–may be copied and pasted into a presentation document. It can be very difficult for future researchers to trace the table or fgure back to a particular statistical model and a particular data set without clear documentation. Therefore, it is important to make the links between your files explicit.

## 5. Have a plan to organize, store, and make your files available.
Finally, in order for independent researchers to reproduce your work, they need to be able access the fles that instruct them how to do this. Files also need to be organized so that independent researchers can fgure out how they ft together. So, from the beginning of your research process, you should have a plan for organizing your fles and a way to make them accessible.

## Learn More
We've only scratched the surface Reproducible Research practices and highlight recommend the book Reproducible research with R and R studio to learn more. 

---
class: no-logo

# The Tools of Reproducible Research

.panelset[

.panel[.panel-name[Tool Types]
Reproducible research involves two broad sets of tools: 
1. *A Reproducible Research Environment*, that includes the statistical tools you need to run your analyses; automatically track the provenance of data, analyses, and results; and to package them for redistribution”. 

2. *A Reproducible Research Publisher* the prepares dynamic documents for presenting results and is easily linked to the reproducible research environment.  
]

.panel[.panel-name[Posit Cloud]

.center[
&lt;img src="img/posit-cloud.png" height="350px"/&gt;
]

.center[[go.ncsu.edu/laser-labs](https://go.ncsu.edu/laser-labs)
]

]

.panel[.panel-name[RStudio]
.center[
&lt;img src="img/rstudio.png" height="425px"/&gt;
]
]

.panel[.panel-name[R Markdown]
.center[
&lt;img src="img/r-markdown.png" height="425px"/&gt;
]
]


.panel[.panel-name[GitHub]

.center[
&lt;img src="img/github.png" height="350px"/&gt;
]

.center[[go.ncsu.edu/laser-github](https://go.ncsu.edu/laser-github)
]
]


.panel[.panel-name[R]
.center[
&lt;img src="img/r-script.png" height="425px"/&gt;
]
]

]

???

## Tool Types
N/A

## Posit Cloud
Posit Cloud (formerly RStudio Cloud) provides a powerful set of data science tools right in your browser with no installation or complex configuration required. 

All of our Learning Lab materials will be access through Posit Cloud this year so we encourage you bookmark posit.cloud in your browser. 

&lt; Have scholars navigate to our LASER Learning Labs 2023 workspace using the link provide and provide a breif overview of the the Posit Cloud features like workspaces, primers, cheatsheets, etc. &gt; 

## RStudio
&lt; Next have scholars navigate to our laser-orientation RStudio Project "assignment" and click the link to make a copy for themselves and open up the project. &gt; 

RStudio is an integrated development environment (IDE) for R and Python and includes a console for running R code directly, syntax-highlighting editor that supports direct code execution in the source pane, and tools for plotting, history, debugging, and management of research projects in the Environment and Files panes. RStudio is basically your home for all of the files, images, reports, and code that are used in a research project given project.

When you create a "project" with RStudio, it's straightforward to divide your work into multiple contexts, each with their own working directory, workspace, history, and source documents.

&lt; Provide a brief walkthrough of the different panes and discuss what they are used for including how they might make their own R Project is they so desired. &gt;

## R Markdown
&lt; Show scholars how to navigate to the files tab and open up the file named laser-lab-slides.Rmd&gt;

As we noted earlier, plain text files written in markup languages like Markdown can be the basis for documents such as presentations, webpages, dashboards, and books. 

This R Markdown file can be opened not just with RStudio, but a simple text editor built into most operating systems. This specific text file is what was used to make the slides you're looking at now. 

Click the "knit" button located at the top of the toolbar to convert this markdown file into an HTML slide show. I should appear in the Viewer table or in a separate browser window and you will need it for our Part 2 code along. 

## GitHub
GitHub is an interface and cloud hosting service built on top of the Git version control system. GitHub was not explicitly designed to host research projects or even data, but for programmers to work together to build computer programs. This seems very far from reproducible research but remember that as reproducible researchers, we are building projects out of interconnected text fles. In important ways, this is exactly the same as building a computer program and offers many highly developed and robust features for reproducible researchers.

We won't be using GitHub in our Learning Labs, but I did want to note that if you click the link below, it will take you to our GitHub organization site where you will find our RStudio Projects stored as Repositories, or repos, including the laser-orientation repository you just opened. 

## R 
Underlying all of this is R, a programming language originally for statistics and graphics but has evolved into rich platform for data science and learning analytics that can be used to create a wide range of static and dynamic outputs for communicating your research.

Let's create a new R script so we can begin working with R to help illustrate a simple research workflow and analyze some data. 

&lt; Direct scholars to create a new R script as illustrated in the screenshot. Show them how to name the file and save to their main directory. &gt; 

---
class: clear, inverse, middle, center

# part_2(R, code_along)
----
The Data-Intensive Research Workflow

---
# The Data-Intensive Research Workflow
.center[
&lt;img src="img/laser-cycle.png" height="400px"/&gt;
]

.footnote[
Krumm, A., Means, B., &amp; Bienkowski, M. (2018). [Learning Analytics Goes to School.](https://www.routledge.com/Learning-Analytics-Goes-to-School-A-Collaborative-Approach-to-Improving/Krumm-Means-Bienkowski/p/book/9781138121836). Routledge.
]

???


---
class: no-logo
# Prepare

.panelset[

.panel[.panel-name[Research Context]
.pull-left[
## The Study 
  
The setting of this study was a public provider of individual online courses in a Midwestern state. Data was collected from two semesters of five online science courses and aimed to understand students motivation. 
]

.pull-right[
## Research Question 

Is there a relationship between the time students spend in a learning management system and their final course grade?

]

]


.panel[.panel-name[Packages]

Sometimes referred to as "libraries", packages are shareable collections of R code that can contain functions, data, and/or documentation.


```r
# Load the {tidyverse} package

library(tidyverse)
```

```
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ readr     2.1.5
## ✔ forcats   1.0.0     ✔ stringr   1.5.1
## ✔ ggplot2   3.5.0     ✔ tibble    3.2.1
## ✔ lubridate 1.9.3     ✔ tidyr     1.3.1
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (&lt;http://conflicted.r-lib.org/&gt;) to force all conflicts to become errors
```

```r
# How to install the package if not already installed
# install.packages("tidyverse")
# Note this code is commented out so it will not run
```

]

.panel[.panel-name["Read" Data]

The {readr} tidyverse package has several functions like `read_csv()` for importing rectangular data from delimited text files such as comma-separated values (CSV), a preferred file format for reproducible research. 


```r
# "read" into your R environment the sci-online-classes.csv file 
# stored in the data folder of your R project and
# save as a new data "object" named sci_data 

sci_data &lt;- read_csv("data/sci-online-classes.csv", 
                     col_names = TRUE)
```

R has also packages for proprietary data formats like .xls, .sas, and .dat. 
]

.panel[.panel-name[Inspect Data]
You should now see a new data "object" named `sci_data` saved in your Environment pane. Try clicking on it and see what happens!

Now let's "print" the output to our Console to view another way:


```r
# print data to console to view

sci_data
```

What variables do you think might help us answer our research question? 
]

.panel[.panel-name[R Syntax]
The R language can express complex ideas like in this simple "sentence": 

`sci_data &lt;- read_csv("data/sci-online-classes.csv", col_names = TRUE)`

- *Functions* are like verbs: `read_csv()`
- *Objects* are the nouns: `read_csv("data/sci-online-classes.csv")`
- *Arguments* are like adverbs: `read_csv("data/sci-online-classes.csv", col_names = TRUE)`
-	*Operators* are like “punctuation” `sci_data &lt;- read_csv("data/sci-online-classes.csv", col_names = TRUE)`
]
]

---
class: 
# Wrangle

Data wrangling is the process of cleaning, ["tidying"](https://r4ds.had.co.nz/tidy-data.html?q=tidy%20data#tidy-data-1), and transforming data. In Learning Analytics, it often involves merging (or joining) data from multiple sources. 

Since we are interested the relationship between time spent in an online course and final grades, let's `select()` the `FinalGradeCEMS` and `TimeSpent` from `sci_data`.


```r
# select student_id and course_id and save as 

sci_data |&gt; 
  select(FinalGradeCEMS, TimeSpent)
```

Note the use of a the powerful `|&gt;` operator called a **pipe**, which are used for combining a sequence of functions or processes. 

---
class: no-logo
# Explore

.panelset[

.panel[.panel-name[EDA]
Exploratory data analysis involves processes of **describing** your data numerically or graphically and often involves: 
- **calculating** summary statistics like frequency, means and standard deviations
- **visualizing** your data through charts and graphs 

EDA can be used to help answer research questions or generate new questions about your data, discover relationships between and among variables, and create new variables (i.e., feature engineering) for data modeling.

]

.panel[.panel-name[Graph Template]
The [{ggplot2}](https://ggplot2.tidyverse.org) package follows a common [graphing workflow](https://r4ds.had.co.nz/data-visualisation.html?q=template#a-graphing-template) for making graphs. To make a graph, you simply:

1. Start the graph with `ggplot()` function and include your data as an argument;
2. "Add" elements to the graph using the `+` operator `a geom_()` function;
3. Select variables to graph on each axis with the `aes()` argument.

A common graphing template is as simple as two lines of code: 

```r
ggplot(&lt;DATA&gt;) + 
  &lt;GEOM_FUNCTION&gt;(aes(x = &lt;VARIABLE1&gt;, y = &lt;VARIABLE2&gt;))
```
]


.panel[.panel-name[Our First Graph!]

Scatterplots use the point **geom**, i.e., the `geom_point()` function, and are most useful for displaying the relationship between two continuous variables.


```r
# make a scatter plot showing time spent and final grades
ggplot(sci_data) +
  geom_point(aes(x = TimeSpent, y = FinalGradeCEMS))
```

]

.panel[.panel-name[Interpret Graph]
.pull-left[Hopefully your scatterplot looks like something like the one to the right. 

*How would you interpret this graph?*
]

.pull-right[
![](laser-lab-slides_files/figure-html/unnamed-chunk-8-1.png)&lt;!-- --&gt;

]
]
]


---
class: no-logo
# Model

.panelset[

.panel[.panel-name[Goal]
Modeling involves a simple low-dimensional summary of a dataset. According to Krumm et al. (2018), there are two general types of modeling approaches: 
- **Unsupervised** learning algorithms can be used to understand the structure of one’s dataset. 
- **Supervised** models, on the other hand, help to quantify relationships between features and a known outcome.

Ideally, a good model will separate true “signals” (i.e. patterns generated by the phenomenon of interest) from the “noise” (i.e. random variation that you’re not interested in).
]


.panel[.panel-name[A Simple Model]

We'll dive much deeper into modeling in subsequent learning labs, but for now let's see if there is a statistically significant relationship between students' final grades, `FinaGradeCEMS`, and the `TimeSpent` in the LMS:


```r
# use a simple linear regression model to predict final course grades
m1 &lt;- lm(FinalGradeCEMS ~ TimeSpent, data = sci_data)

summary(m1)
```

]

.panel[.panel-name[Interpret]

.code70.remark-code[ 

```
## 
## Call:
## lm(formula = FinalGradeCEMS ~ TimeSpent, data = sci_data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -67.136  -7.805   4.723  14.471  30.317 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(&gt;|t|)    
## (Intercept) 6.581e+01  1.491e+00   44.13   &lt;2e-16 ***
## TimeSpent   6.081e-03  6.482e-04    9.38   &lt;2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 20.71 on 571 degrees of freedom
##   (30 observations deleted due to missingness)
## Multiple R-squared:  0.1335,	Adjusted R-squared:  0.132 
## F-statistic: 87.99 on 1 and 571 DF,  p-value: &lt; 2.2e-16
```
]
]

]


---
class: no-logo
# Communicate

.panelset[

.panel[.panel-name[Data Products]
Krumm et al. (2018) have outlined the following 3-step process for communicating finding with education stakeholders: 

1. **Select.** Selecting analyses that are most important and useful to an intended audience, as well as selecting a format for displaying that info (e.g. chart, table).

2. **Polish.** Refining or polishing data products, by adding or editing titles, labels, and notations and by working with colors and shapes to highlight key points.

3. **Narrate.**  Writing a narrative pairing a data product with its related research question and describing how best to interpret and use the data product. 
]


.panel[.panel-name[Dashboards]

.center[
&lt;img src="img/dashboards.png" height="350px"/&gt;
]

.center[[https://pkgs.rstudio.com/flexdashboard](https://pkgs.rstudio.com/flexdashboard/)]
]

.panel[.panel-name[Websites]
.center[
&lt;img src="img/websites.png" height="350px"/&gt;
]

.center[[https://pkgs.rstudio.com/blogdown/](https://pkgs.rstudio.com/blogdown/)]
]

.panel[.panel-name[Books]
.center[
&lt;img src="img/books.png" height="350px"/&gt;
]

.center[[https://pkgs.rstudio.com/bookdown/](https://pkgs.rstudio.com/bookdown/)]
]

.panel[.panel-name[And More]

R Markdown supports dozens of static and dynamic output formats including [HTML](https://bookdown.org/yihui/rmarkdown/html-document.html), [PDF](https://bookdown.org/yihui/rmarkdown/pdf-document.html), [MS Word](https://bookdown.org/yihui/rmarkdown/word-document.html), [Beamer](https://bookdown.org/yihui/rmarkdown/beamer-presentation.html), [HTML5 slides](https://bookdown.org/yihui/rmarkdown/ioslides-presentation.html), [Tufte-style handouts](https://bookdown.org/yihui/rmarkdown/tufte-handouts.html), [books](https://bookdown.org), [dashboards](flexdashboard/), [shiny applications](https://bookdown.org/yihui/rmarkdown/shiny-documents.html), [scientific articles](https://github.com/rstudio/rticles), [websites](https://bookdown.org/yihui/rmarkdown/rmarkdown-site.html), and more.

]
]


???


---
# What's Next?


.pull-left[
## Your First LASER Badge!
Next you will complete an interactive "case study" which is a key component to each learning lab.  

Navigate to the Files tab and open the following file:

`laser-lab-case-study.RMD`

]

.pull-right[
## Essential Readings

- [Reproducible Research with R and RStudio](https://github.com/christophergandrud/Rep-Res-Book) (chapters 1 &amp; 2)

- [Learning Analytics Goes to School](https://www.routledge.com/Learning-Analytics-Goes-to-School-A-Collaborative-Approach-to-Improving/Krumm-Means-Bienkowski/p/book/9781138121836) (pages 28 - 58)

- [Data Science in Education Using R](https://datascienceineducation.com)

- [R for Data Science](https://r4ds.had.co.nz/index.html)  
]

---
class: clear, center

## .font130[.center[**Thank you!**]]
&lt;br&gt;
&lt;img src="img/laser-instructors.png" height="350px"/&gt;

    </textarea>
<style data-target="print-only">@media screen {.remark-slide-container{display:block;}.remark-slide-scaler{box-shadow:none;}}</style>
<script src="https://remarkjs.com/downloads/remark-latest.min.js"></script>
<script>var slideshow = remark.create({
"highlightStyle": "default",
"highlightLines": true,
"highlightLanguage": "r",
"countIncrementalSlides": false,
"ratio": "16:9",
"slideNumberFormat": "<div class=\"progress-bar-container\">\n <div class=\"progress-bar\" style=\"width: calc(%current% / %total% * 100%);\">\n </div>\n</div>"
});
if (window.HTMLWidgets) slideshow.on('afterShowSlide', function (slide) {
  window.dispatchEvent(new Event('resize'));
});
(function(d) {
  var s = d.createElement("style"), r = d.querySelector(".remark-slide-scaler");
  if (!r) return;
  s.type = "text/css"; s.innerHTML = "@page {size: " + r.style.width + " " + r.style.height +"; }";
  d.head.appendChild(s);
})(document);

(function(d) {
  var el = d.getElementsByClassName("remark-slides-area");
  if (!el) return;
  var slide, slides = slideshow.getSlides(), els = el[0].children;
  for (var i = 1; i < slides.length; i++) {
    slide = slides[i];
    if (slide.properties.continued === "true" || slide.properties.count === "false") {
      els[i - 1].className += ' has-continuation';
    }
  }
  var s = d.createElement("style");
  s.type = "text/css"; s.innerHTML = "@media print { .has-continuation { display: none; } }";
  d.head.appendChild(s);
})(document);
// delete the temporary CSS (for displaying all slides initially) when the user
// starts to view slides
(function() {
  var deleted = false;
  slideshow.on('beforeShowSlide', function(slide) {
    if (deleted) return;
    var sheets = document.styleSheets, node;
    for (var i = 0; i < sheets.length; i++) {
      node = sheets[i].ownerNode;
      if (node.dataset["target"] !== "print-only") continue;
      node.parentNode.removeChild(node);
    }
    deleted = true;
  });
})();
// add `data-at-shortcutkeys` attribute to <body> to resolve conflicts with JAWS
// screen reader (see PR #262)
(function(d) {
  let res = {};
  d.querySelectorAll('.remark-help-content table tr').forEach(tr => {
    const t = tr.querySelector('td:nth-child(2)').innerText;
    tr.querySelectorAll('td:first-child .key').forEach(key => {
      const k = key.innerText;
      if (/^[a-z]$/.test(k)) res[k] = t;  // must be a single letter (key)
    });
  });
  d.body.setAttribute('data-at-shortcutkeys', JSON.stringify(res));
})(document);
(function() {
  "use strict"
  // Replace <script> tags in slides area to make them executable
  var scripts = document.querySelectorAll(
    '.remark-slides-area .remark-slide-container script'
  );
  if (!scripts.length) return;
  for (var i = 0; i < scripts.length; i++) {
    var s = document.createElement('script');
    var code = document.createTextNode(scripts[i].textContent);
    s.appendChild(code);
    var scriptAttrs = scripts[i].attributes;
    for (var j = 0; j < scriptAttrs.length; j++) {
      s.setAttribute(scriptAttrs[j].name, scriptAttrs[j].value);
    }
    scripts[i].parentElement.replaceChild(s, scripts[i]);
  }
})();
(function() {
  var links = document.getElementsByTagName('a');
  for (var i = 0; i < links.length; i++) {
    if (/^(https?:)?\/\//.test(links[i].getAttribute('href'))) {
      links[i].target = '_blank';
    }
  }
})();
// adds .remark-code-has-line-highlighted class to <pre> parent elements
// of code chunks containing highlighted lines with class .remark-code-line-highlighted
(function(d) {
  const hlines = d.querySelectorAll('.remark-code-line-highlighted');
  const preParents = [];
  const findPreParent = function(line, p = 0) {
    if (p > 1) return null; // traverse up no further than grandparent
    const el = line.parentElement;
    return el.tagName === "PRE" ? el : findPreParent(el, ++p);
  };

  for (let line of hlines) {
    let pre = findPreParent(line);
    if (pre && !preParents.includes(pre)) preParents.push(pre);
  }
  preParents.forEach(p => p.classList.add("remark-code-has-line-highlighted"));
})(document);</script>

<script>
slideshow._releaseMath = function(el) {
  var i, text, code, codes = el.getElementsByTagName('code');
  for (i = 0; i < codes.length;) {
    code = codes[i];
    if (code.parentNode.tagName !== 'PRE' && code.childElementCount === 0) {
      text = code.textContent;
      if (/^\\\((.|\s)+\\\)$/.test(text) || /^\\\[(.|\s)+\\\]$/.test(text) ||
          /^\$\$(.|\s)+\$\$$/.test(text) ||
          /^\\begin\{([^}]+)\}(.|\s)+\\end\{[^}]+\}$/.test(text)) {
        code.outerHTML = code.innerHTML;  // remove <code></code>
        continue;
      }
    }
    i++;
  }
};
slideshow._releaseMath(document);
</script>
<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
  var script = document.createElement('script');
  script.type = 'text/javascript';
  script.src  = 'https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-MML-AM_CHTML';
  if (location.protocol !== 'file:' && /^https?:/.test(script.src))
    script.src  = script.src.replace(/^https?:/, '');
  document.getElementsByTagName('head')[0].appendChild(script);
})();
</script>
  </body>
</html>