---
title: "highlightr"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{highlightr}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

This package is designed to map a group of derivative texts to the corresponding parent text, based on the frequency with which phrases occur in the derivative texts.
The parent text is highlighted corresponding to this frequency, in order to create a 'heatmap' of popular phrases found in the derivative texts.

This example is taken from the initial description of a crime used in a study of jury perception of algorithm use and demonstrative evidence.
The `notepad_example` data frame contains an 'ID' number corresponding to a study participant, as well as their notes, labelled as 'Text'.
The first six observations are shown below.

```{r}

# load the library
library(highlightr)
library(knitr)

# View first 6 observations
knitr::kable(head(notepad_example))

```

Additionally, the source document (or study transcript) is included in `notepad_example` with an ID of 'source'.
The original transcript is shown here:

```{r}

study_transcript <- notepad_example[notepad_example$ID == "source",]$Text

knitr::kable(study_transcript)

```

<br>
Fuzzy collocations are used to match the tokenized derivative texts to the phrases in the tokenized source text.
This function first determines the number of times a collocation of length 5 occurs in derivative texts, or participant notes on the case.
Fuzzy (or indirect) matches are then added to the frequency count of the source collocation that is the closest match.
These fuzzy matches are weighted based on the edit distance between the source collocation and the indirect phrase:
$$
\frac{n*d}{m}
$$

Here, $n$ is the frequency of the fuzzy collocation, $d$ is the Jaccard similarity between the fuzzy collocation and the source collocation (ranging from 0 to 1, where 1 indicates identical strings), and $m$ is the number of closest matches for the fuzzy collocation.
The total count is divided by the number of times a collocation occurs in the source document.


The `collocation_frequency()` function attaches the collocation counts to the full text of the transcript.
The collocation frequencies are averaged per word.

```{r}

# connect collocation frequencies to source document

merged_frequency <- collocation_frequency(notepad_example, source_row=which(notepad_example$ID=="source"), text_column = "Text", fuzzy=TRUE)

knitr::kable(head(merged_frequency), digits=2)
```
The warning regarding the chance of comparisons for a threshold of 0.7 is generated from `zoomerjoin::jaccard_right_join()`.
If desired, `threshold`, `n_bands`, and `band_width` can be adjusted via corresponding values in `collocation_frequency()`.
The output assigns the frequency of each collocation to each word that occurs in that collocation. 
For example, the first collocation in the description is “in this case the defendant”, which occurs with a frequency of 6.96. 
This is the only collocation in which the first word will appear, so this is the only collocation value provided for the first word. 
The second word, “this” appears in the next collocation as well: "this case the defendant richard", whose frequency is 7, and so on for all words in the description. Collocations are weighted by the number of times they appear in the transcript text.

The combined document is then fed through ggplot to assign gradient colors based on frequency, and the minimum and maximum values are recorded.

```{r}

# create `ggplot` object of the transcript

freq_plot <- collocation_plot(merged_frequency)
```

```{r}

# add html tags to source document

page_highlight <- highlighted_text(freq_plot)

```

After colors have been assigned, HTML output is created for highlighted text is created based on frequency, as well as a gradient bar indicating the high and low values.
The left side of each word gradient indicates the frequency of the previous word's averaged collocation frequency, while the right side indicates the current word's averaged collocation frequency.
This HTML output can be rendered into highlighted text by specifying `` `r '\x60r page_highlight\x60'` `` in an R Markdown document outside of a code chunk and knitting to HTML:

`r page_highlight`  

<br>

Alternatively, the `xml2` package can be used to save the output as an html file, as shown in the following code:

```{r, eval=FALSE}

# load `xml2` library

library(xml2)

# save html output to desired location

xml2::write_html(xml2::read_html(page_highlight), "filename.html")

```

Non-fuzzy matching can also be used by removing the `fuzzy=TRUE` argument.
This will only include direct matches betweeen derivative documents and the parent document.
In this case, the highlighting pattern resembles that when the fuzzy matches are included, but the maximum value reached is smaller.
Note also that the colors used in highlighting can be changed in the "colors" argument of the `collocation_plot` function.

```{r}

# connect collocation frequencies to source document

merged_frequency_nonfuzzy <- collocation_frequency(notepad_example, source_row=which(notepad_example$ID=="source"), text_column = "Text")

# create a `ggplot` object of the transcript, and change colors of the gradient

freq_plot_nonfuzzy <- collocation_plot(merged_frequency_nonfuzzy, colors=c("#15bf7e", "#fcc7ed"))

# add html tags to source document

page_highlight_nonfuzzy <- highlighted_text(freq_plot_nonfuzzy)
```

`r page_highlight_nonfuzzy`

<br>

Additionally, the length of the collocation can be changed.
The default collocation length (shown above) is 5 words.
Below, this collocation length has been changed to 2 words.

In these shorter collocations, we can see that the collocation containing the name "Richard Cole" is popular, with a frequency of 89.

```{r}

# connect collocation frequencies to source document

merged_frequency_2col <- collocation_frequency(notepad_example, source_row=which(notepad_example$ID=="source"), text_column = "Text", collocate_length = 2)

# create a `ggplot` object of the transcript

freq_plot_2col <- collocation_plot(merged_frequency_2col)

# add html tags to source document

page_highlight_2col <- highlighted_text(freq_plot_2col)
```

`r page_highlight_2col`