Exploring and auditing new data with RMarkdown (in-class tutorial)

• Group work will get started - ensure your teams are properly registered
• Any pending installation issues? Let me know!

• Data preparation theory (“the bigger picture”)

• In-class part of this week's tutorial: we will use RMarkdown to start exploring new data

• After class

  • Review today's tutorial.
  • Work through self-study material
    • Intermediate R on Datacamp
    • Data carpentry's Getting started with R Markdown

1) Initializing the script/setup

• Loading packages
• Making connections to databases
• Knowing whether to prototype or not 2) Input
• Read data (e.g., unstructured/unstructured data, remote/local locations, files or databases) 3) Transformation
• E.g., filtering, aggregation, merging, transformation, deduplication (more later) 4) Output
• Store (intermediate) data, figures (e.g., auditing, final)

• Recall, there are multiple ways to code in R
  • RStudio
  • Your favorite script editor (e.g., VS Code) + running from the terminal
• But – in all of these cases, the output will be “ugly” (i.e., just some text)
• RMarkdown is a way to format output nicely – and ideally suited for exploring data
  • mix of code and text (formatted in markdown) - you already know the concept from Jupyter Notebook
  • possibility to compile into HTML, PDF, Word, etc.
• great for prototyping code, but bad for “production” (this is also the case for Python's Jupter Notebook)

• We already know markdown syntax, right?
  • If not: find a cheat sheet online right now!
• All compiled the document? Try this first always!
• Code cells & running code (click, ctrl/cmd+enter)
• Using inline code
• Options to run (or not run) code cells when compiling documents
  • that 'little wheel'
  • echo, include

• Datacamp's Intermediate covers “base R” - basic programming concepts that are at the core of R
• We will first review some of these concepts.
• It's good to have a cheat sheet available - use this

Let's create some data!

x <- c(10, 20, NA, 5, 3, 100)

Write code that displays…

1. all values equal to 10
2. all values NOT equal to 10
3. values larger than 20
4. counts the number of missing values
5. all values larger than 20 OR missing values
6. all values larger than 5 and smaller than 20.

Remeber our research workflow? Before we can start with exactly the same data, we all need to download this data set first.

Question: Why not do it manually? What could go wrong?

# your code here
# urls = # Assemble list of URLs here
# then, copy paste code snippet for downloading and renaming data

• Loops don't “return” anything - they just execute stuff over and over again
• Functions from the apply family, though, DO return information.
• Apply comes in multiple flavors:
  • lapply (loops over a vector or list, returns a list)
  • sapply (loops over a vector, returns as a vector)
  • apply (loops over rows or columns of a matrix, returns a vector)
  • Other versions of apply exist, but I rarely use them
• Applies are super handy when working with data

What's the key difference between the “apply family” and “for loops”?

• lapply “returns” stuff; loops just execute stuff without returning anything

Each and every source code file obeys to this procedure. We called this setup-ITO earlier.

• 1) Setup such as loading packages, setting variables, making database connection
• 2) Input (e.g., loading data)
• 3) Transformation
  • many options…: filtering, grouping and summarizing, creating new variables, etc.
  • mostly leads to a “different primary key than the input data” (–> unit of analysis!)
• 4) Output (e.g., saving, passing on to the next script)

To do all of this, we require knowledge on new programming concepts. So let's continue.

• Sometimes, we would like to re-use code. For this, we make use of self-made functions.
• For example, a function that “reduces” the size of the data set to the unit of analysis you're interested in.
• Start by prototyping your function.

my_dataset = datasets[[1]]
my_dataset %>% group_by(neighbourhood) %>% summarize(hosts = n_distinct(host_id))

DO:

1. Write a function that executes the data aggregation code above, for any dataset it receives as an input. Tip: start with prep_data <- function(dataset) { # your code here }
2. Try to test the function on datasets[[1]] and datasets[[2]]

prep_data <- function(dataset) {
  my_dataset = dataset %>% group_by(neighbourhood) %>% summarize(hosts = n_distinct(host_id))
  return(my_dataset)
}
prep_data(datasets[[1]])

Now let's continue with some more exercises.

1. Call the function on all datasets stored in datasets, using lapply. Save the result in a new variable, called edited_datasets.
2. The result still is available in three separate data sets. Can you use the bind_rows() function from the dplyr package to bind them together? Call it final_dataset.

prep_data <- function(dataset) {
  result = dataset %>% group_by(neighbourhood) %>% summarize(hosts = n_distinct(host_id))
  result = result %>% mutate(neighbourhood = as.character(neighbourhood))
  return(result)
}
edited_datasets <- lapply(datasets, prep_data)
final_dataset <- bind_rows(edited_datasets)

• setup-ito
• knitting markdown documents
• conditional logic
• controls
• downloading data
• loops and the apply family (lapply)
• functions

• We'll make use of the Google mobility dataset

DO:

1. Download the regional CSV from https://www.google.com/covid19/mobility/ & unzip
2. Load the data into R using read_csv()
3. Start exploring the data: use head(), tail(), and View()

• Unit of analysis: Fundamental concept in data analysis. Refers to the specific entity or level of observation at which you collect, process, and analyze data.

• Why is it important? It determines how you aggregate, summarize, and interpret your data. Without knowing your unit of analysis – at each part of your project – you will likely get lost.

• Example: The listing data is at the “city-listing” level. However, we may want to analyze the AirBnB data at the city-listing-date level. Aggregation may be necessary.

• We can generate summary statistics using the summary() command.
  • do values make sense?
  • are there any missing values?
• We can also zoom in on specific columns
  • let's start making sense of the retail_and_recreation_percent_change_from_baseline variable: what does it mean?
  • let's plot this variable over time!
• If our target unit of analysis is different from the unit of analysis in the current data set, we can also use aggregation.

• Typically, the raw data is always messy and likely not at the correct unit of analysis (e.g., city level).
• Also, we may still want to select columns, or rename them.

DO:

1. Drop these columns: cols_to_drop <- c('country_region_code', 'metro_area', 'iso_3166_2_code', 'census_fips_code')
2. Rename sub_region_1 to province, and sub_region_2 to city

We can also use more advanced functions, e.g., to wipe the percent_change... from the column names.

• class(mobility$date)
• Can only calculate with dates when they are actual dates!

• Can you explore the data a bit more? Which observations are measured at the country level?
• How can you select only these country-level observations, using %>% filter(...)?

1. Setup done/packages and other prerequisites loaded?
2. Adhered to input - transformation - output?
3. File callable from the command prompt? –> we will need this for automation in subsequent tutorials (Rscript <filename>)

• Work through the tutorial and complete exercises (all solutions available)
• Start working on team projects
  • pick a dataset: IMDB or InsideAirbnb
  • come up with a 'research question' (but note: this course is about engineering a research pipeline, not about rigorously answering the RQ!)
• Join our coaching session, following today's tutorial session (check room!)

<!–

Welcome to today's coaching session.

• Recap: Git workflow (using our Cheatsheet)

• Example application: contributing to open source projects using forks and pull requests (can demonstrate)

• How's your team work going?

  • Let's look at some of your repositories right now
  • Reality check: doubting whether to stay enrolled? tell your team how you feel!

• Today's focus: See grading details, section 2.1 (“Data exploration”)
  • Download raw data programmatically
  • Create RMarkdown reports for raw data/input files…
  • allowing users to understand content of such files, and the definition of variables.
  • properly formatted, rendered as HTML or PDF files
  • feature information in a variety of modes (e.g., running text, tables, or figures).
  • code irrelevant to understanding data, including warning messages, are hidden.