7 Adv Data wrangling P2

🧩 Learning Goals

By the end of this lesson, you should be able to:

Manipulate and explore strings using the stringr package
Construct regular expressions to find patterns in strings

Helpful Cheatsheets

The stringr cheatsheet (HTML, PDF) will be useful to have open and reference.

Motivation: 30 Years of American Anxieties

In 2018 the data journalism organization The Pudding featured a story called 30 Years of American Anxieties about themes in 30 years of posts to the Dear Abby column (an American advice column).

One way to understand themes in text data is to conduct a qualitative analysis, a methodology in which multiple readers read through instances of text several times to reach a consensus about themes.

Another way to understand themes in text data is computational text analysis.

This is what we will explore today.

Both qualitative analysis and computational tools can be used in tandem. Often, using computational tools can help focus a close reading of select texts, which parallels the spirit of a qualitative analysis.

To prepare ourselves for a computational analysis, let’s learn about strings.

Strings

Strings are objects of the character class (abbreviated as <chr> in tibbles).

When you print out strings, they display with double quotes:

Code

some_string <- "banana"
some_string

[1] "banana"

Working with strings generally will involve the use of regular expressions, a tool for finding patterns in strings.

Regular expressions (regex, for short) look like the following:

"^the" (Strings that start with "the")
"end$" (Strings that end with "end")

Before getting to regular expressions, let’s go over some fundamentals about working with strings. The stringr package (available within tidyverse) is great for working with strings.

Creating strings

Creating strings by hand is useful for testing out regular expressions.

To create a string, type any text in either double quotes (") or single quotes '. Using double or single quotes doesn’t matter unless your string itself has single or double quotes.

Code

string1 <- "This is a string"
string2 <- 'If I want to include a "quote" inside a string, I use single quotes'
string3 <- c(string1, string2) # string / character vector (of greater than length 1)


class(string1)

[1] "character"

Code

class(string2)

[1] "character"

Code

class(string3)

[1] "character"

Code

length(string1)

[1] 1

Code

length(string2)

[1] 1

Code

length(string3)

[1] 2

We can view these strings “naturally” (without the opening and closing quotes) with str_view():

Code

str_view(string1)

[1] │ This is a string

Code

str_view(string2)

[1] │ If I want to include a "quote" inside a string, I use single quotes

Code

str_view(string3)

[1] │ This is a string
[2] │ If I want to include a "quote" inside a string, I use single quotes

Exercise: Create the string It's Thursday. What happens if you put the string inside single quotes? Double quotes?

Code

# Your code
strThurs <- "It's Thursday"

# strThurs2 <- 'It's Thursday'

str_view(strThurs)

[1] │ It's Thursday

Single quotes results in error.

Because " and ' are special characters in the creation of strings, R offers another way to put them inside a string. We can escape these special characters by putting a \ in front of them:

Code

string1 <- "This is a string with \"double quotes\""
string2 <- "This is a string with \'single quotes\'"
str_view(string1)

[1] │ This is a string with "double quotes"

Code

str_view(string2)

[1] │ This is a string with 'single quotes'

Given that \ is a special character, how can we put the \ character in strings? We have to escape it with \\.

Exercise: Create the string C:\Users. What happens when you don’t escape the \?

Code

# Your code
str_slash <- 'C:\\Users'
str_view(str_slash)

[1] │ C:\Users

Not escaping it results in an error.

Other special characters include:

\t (Creates a tab)
\n (Creates a newline)

Both can be useful in plots to more neatly arrange text.

Code

string1 <- "Record temp:\t102"
string2 <- "Record temp:\n102"

str_view(string1)

[1] │ Record temp:{\t}102

Code

str_view(string2)

[1] │ Record temp:
    │ 102

Can we get str_view() to show the tab instead of {\t}? We can use the html argument to have the string displayed as if on a webpage:

Code

str_view(string1, html = TRUE)

Often we will want to create new strings within data frames. We can use str_c() or str_glue(), both of which are vectorized functions (meaning they take vectors as inputs and provide vectors as outputs - can be used within mutate()):

With str_c() the strings to be combined are all separate arguments separated by commas.
With str_glue() the desired string is written as a template with variable names inside curly braces {}.

Code

df <- tibble(
    first_name = c("Arya", "Olenna", "Tyrion", "Melisandre"),
    last_name = c("Stark", "Tyrell", "Lannister", NA)
)
df

# A tibble: 4 × 2
  first_name last_name
  <chr>      <chr>    
1 Arya       Stark    
2 Olenna     Tyrell   
3 Tyrion     Lannister
4 Melisandre <NA>

Code

df |>
    mutate(
        full_name1 = str_c(first_name, " ", last_name),
        full_name2 = str_glue("{first_name} {last_name}")
    )

# A tibble: 4 × 4
  first_name last_name full_name1       full_name2      
  <chr>      <chr>     <chr>            <glue>          
1 Arya       Stark     Arya Stark       Arya Stark      
2 Olenna     Tyrell    Olenna Tyrell    Olenna Tyrell   
3 Tyrion     Lannister Tyrion Lannister Tyrion Lannister
4 Melisandre <NA>      <NA>             Melisandre NA

Exercise: In the following data frame, create a full date string in month-day-year format using both str_c() and str_glue().

Code

df_dates <- tibble(
    year = c(2000, 2001, 2002),
    month = c("Jan", "Feb", "Mar"),
    day = c(3, 4, 5)
)

df_dates <- df_dates %>%
  mutate(
    month_num = match(month, month.abb),
    full_date1 = str_c(month_num, "-", day, "-", year), full_date2 = str_glue("{month_num}-{day}-{year}")
  )

df_dates

# A tibble: 3 × 6
   year month   day month_num full_date1 full_date2
  <dbl> <chr> <dbl>     <int> <chr>      <glue>    
1  2000 Jan       3         1 1-3-2000   1-3-2000  
2  2001 Feb       4         2 2-4-2001   2-4-2001  
3  2002 Mar       5         3 3-5-2002   3-5-2002

Extracting information from strings

The str_length() counts the number of characters in a string.

Code

comments <- tibble(
    name = c("Alice", "Bob"),
    comment = c("The essay was well organized around the core message and had good transitions.", "Good job!")
)

comments |>
    mutate(
        comment_length = str_length(comment)
    )

# A tibble: 2 × 3
  name  comment                                                   comment_length
  <chr> <chr>                                                              <int>
1 Alice The essay was well organized around the core message and…             78
2 Bob   Good job!                                                              9

The str_sub() function gets a substring of a string. The 2nd and 3rd arguments indicate the beginning and ending position to extract.

Negative positions indicate the position from the end of the word. (e.g., -3 indicates “3rd letter from the end”)
Specifying a position that goes beyond the word won’t result in an error. str_sub() will just go as far as possible.

Code

x <- c("Apple", "Banana", "Pear")

str_sub(x, start = 1, end = 3)

[1] "App" "Ban" "Pea"

Code

str_sub(x, start = -3, end = -1)

[1] "ple" "ana" "ear"

Code

str_sub(x, start = 2, end = -1)

[1] "pple"  "anana" "ear"

Code

str_sub("a", start = 1, end = 15)

[1] "a"

Exercise: Using str_sub(), create a new variable with only the middle letter of each word in the data frame below. (Challenge: How would you handle words with an even number of letters?)

Code

df <- tibble(
    word_id = 1:3,
    word = c("replace", "match", "pattern")
)

df <- df %>%
  mutate(mid_letter = str_sub(word, start = floor(str_length(word)/2) + 1, end = floor(str_length(word)/2) + 1))

Finding patterns in strings with regular expressions

Suppose that you’re exploring text data looking for places where people describe happiness. There are many ways to search. We could search for the word “happy” but that excludes “happiness” so we might search for “happi”.

Regular expressions (regex) are a powerful language for describing patterns within strings.

. . .

Code

data(fruit)
data(words)
data(sentences)

We can use str_view() with the pattern argument to see what parts of a string match the regex supplied in the pattern argument. (Matches are enclosed in <>.)

Code

str_view(fruit, "berry")

 [6] │ bil<berry>
 [7] │ black<berry>
[10] │ blue<berry>
[11] │ boysen<berry>
[19] │ cloud<berry>
[21] │ cran<berry>
[29] │ elder<berry>
[32] │ goji <berry>
[33] │ goose<berry>
[38] │ huckle<berry>
[50] │ mul<berry>
[70] │ rasp<berry>
[73] │ salal <berry>
[76] │ straw<berry>

Essentials of forming a regex

Letters and numbers in a regex are matched exactly and are called literal characters.
Most punctuation characters, like ., +, *, [, ], and ?, have special meanings and are called metacharacters.
Quantifiers come after a regex and control how many times a pattern can match:
- ?: match the preceding pattern 0 or 1 times
- +: match the preceding pattern at least once
- *: match the preceding pattern at least 0 times (any number of times)

. . .

Exercise: Before running the code below, predict what matches will be made. Run the code to check your guesses. Note that in all regex’s below the ?, +, * applies to the b only (not the a).

Code

str_view(c("a", "ab", "abb"), "ab?")
str_view(c("a", "ab", "abb"), "ab+")
str_view(c("a", "ab", "abb"), "ab*")

We can match any of a set of characters with [] (called a character class), e.g., [abcd] matches “a”, “b”, “c”, or “d”.
- We can invert the match by starting with ^: [^abcd] matches anything except “a”, “b”, “c”, or “d”.

Code

# Match words that have vowel-x-vowel
str_view(words, "[aeiou]x[aeiou]")

[284] │ <exa>ct
[285] │ <exa>mple
[288] │ <exe>rcise
[289] │ <exi>st

Code

# Match words that have not_vowel-y-not_vowel
str_view(words, "[^aeiou]y[^aeiou]")

[836] │ <sys>tem
[901] │ <typ>e

Exercise Using the words data, find words that have two vowels in a row followed by an “m”.

Code

# Your code
str_view(words, "[aeiou][aeiou]m")

[154] │ cl<aim>
[714] │ r<oom>
[735] │ s<eem>
[844] │ t<eam>

The alternation operator | can be read just like the logical operator | (“OR”) to pick between one or more alternative patterns. e.g., apple|banana searches for “apple” or “banana”.

Code

str_view(fruit, "apple|melon|nut")

 [1] │ <apple>
[13] │ canary <melon>
[20] │ coco<nut>
[52] │ <nut>
[62] │ pine<apple>
[72] │ rock <melon>
[80] │ water<melon>

Exercise: Using the fruit data, find fruits that have a repeated vowel (“aa”, “ee”, “ii”, “oo”, or “uu”.)

Code

# Your code
str_view(fruit, "aa|ee|ii|oo|uu")

 [9] │ bl<oo>d orange
[33] │ g<oo>seberry
[47] │ lych<ee>
[66] │ purple mangost<ee>n

The ^ operator indicates the beginning of a string, and the $ operator indicates the end of a string. e.g., ^a matches strings that start with “a”, and a$ matches words that end with “a”.
Parentheses group together parts of a regular expression that should be taken as a bundle. (Much like parentheses in arithmetic statements.)
- e.g., ab+ is a little confusing. Does it match “ab” one or more times? Or does it match “a” first, then just “b” one or more times? (The latter, as we saw in an earlier example.) We can be very explicit and use a(b)+.

Exercise: Using the words data, find (1) words that start with “y” and (2) words that don’t start with “y”.

Code

# Your code
str_view(words, "^y")

[975] │ <y>ear
[976] │ <y>es
[977] │ <y>esterday
[978] │ <y>et
[979] │ <y>ou
[980] │ <y>oung

Code

str_view(words, "^[^y]")

 [1] │ <a>
 [2] │ <a>ble
 [3] │ <a>bout
 [4] │ <a>bsolute
 [5] │ <a>ccept
 [6] │ <a>ccount
 [7] │ <a>chieve
 [8] │ <a>cross
 [9] │ <a>ct
[10] │ <a>ctive
[11] │ <a>ctual
[12] │ <a>dd
[13] │ <a>ddress
[14] │ <a>dmit
[15] │ <a>dvertise
[16] │ <a>ffect
[17] │ <a>fford
[18] │ <a>fter
[19] │ <a>fternoon
[20] │ <a>gain
... and 954 more

Exploring `stringr` functions

Read in the “Dear Abby” data underlying The Pudding’s 30 Years of American Anxieties article.

Code

posts <- read_csv("https://raw.githubusercontent.com/the-pudding/data/master/dearabby/raw_da_qs.csv")

Take a couple minutes to scroll through the 30 Years of American Anxieties article to get ideas for themes that you might want to search for using regular expressions.

The following are core stringr functions that use regular expressions:

str_view() - View the first occurrence in a string that matches the regex
str_count() - Count the number of times a regex matches within a string
str_detect() - Determine if (TRUE/FALSE) the regex is found within string
str_subset() - Return subset of strings that match the regex
str_extract(), str_extract_all() - Return portion of each string that matches the regex. str_extract() extracts the first instance of the match. str_extract_all() extracts all matches.
str_replace(), str_replace_all() - Replace portion of string that matches the regex with something else. str_replace() replaces the first instance of the match. str_replace_all() replaces all instances of the match.
str_remove(), str_remove_all() - Removes the portion of the string that matches the pattern. Equivalent to str_replace(x, "THE REGEX PATTERN", "")

Exercise: Starting from str_count(), explore each of these functions by pulling up the function documentation page and reading through the arguments. Try out each function using the posts data.

Code

# str_view(posts$title, "help")
# str_count(posts$title, "love")
# str_detect(posts$question_only, "cat")
# str_subset(posts$title, "husband")
# str_extract(posts$title, "husband")
# str_replace(posts$title, "dog", "chicken")
# str_remove(posts$title, "hate")

Done!

Check the ICA Instructions for how to (a) push your code to GitHub and (b) update your portfolio website

🧩 Learning Goals

Helpful Cheatsheets

Motivation: 30 Years of American Anxieties

Strings

Creating strings

Extracting information from strings

Finding patterns in strings with regular expressions

Exploring stringr functions

Done!

Exploring `stringr` functions