---
title: "Incorporating design information"
author: "Mike Blazanin"
output: 
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 4
vignette: >
  %\VignetteIndexEntry{Incorporating design information}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r global options, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
knitr::opts_knit$set(root.dir = tempdir())

#Note: to knit to pdf, word, and html simultaneously, run in R console:
#{rmarkdown::render("./vignettes/incorporate_designs.Rmd", rmarkdown::html_vignette(toc = TRUE, toc_dept = 4)); rmarkdown::render("./vignettes/incorporate_designs.Rmd", rmarkdown::pdf_document(toc = TRUE, toc_depth = 4)); rmarkdown::render("./vignettes/incorporate_designs.Rmd", rmarkdown::word_document(toc = TRUE, toc_dept = 4))}
```

# Where are we so far?

1. Introduction: `vignette("gcplyr")`
2. Importing and transforming data: `vignette("import_transform")`
3. **Incorporating design information:** `vignette("incorporate_designs")`
4. Pre-processing and plotting your data: `vignette("preprocess_plot")`
5. Processing your data: `vignette("process")`
6. Analyzing your data: `vignette("analyze")`
7. Dealing with noise: `vignette("noise")`
8. Statistics, merging other data, and other resources: `vignette("conclusion")`

So far, we've imported and transformed our measures data into `R`. Now we're going to address how to incorporate our experimental design.

If you haven't already, load the necessary packages.

```{r setup}
library(gcplyr)
```

```{r, include = FALSE}
print_df <- function(df, col.names = FALSE) {
  write.table(format(df, justify = "right"),
              row.names=FALSE, col.names = col.names, quote = F)
}
```

# Including design elements
We often want to combine information about the experimental design with our data. `gcplyr` enables incorporation of design elements in two ways:

1. Designs can be imported from files
2. Designs can be generated in `R` using `make_design`

# Reading design elements from files
Users can read block-shaped or tidy-shaped design files:

* If design files are block-shaped, they can be read with `import_blockdesigns`
* If design files are tidy-shaped, they can simply be read with `read_tidys`

## Importing block-shaped design files
To import block-shaped design files, use `import_blockdesigns`, which will return a tidy-shaped designs data frame (or list of data frames).

`import_blockdesigns` only requires a list of filenames (or relative file paths) and will return a data.frame (or list of data frames) in a **tidy format** that you can save in R.

### A basic example


Let's look at an example. First, we need to create an example file for the sake of this tutorial. **Don't worry how the below code works**, just imagine that you've created this file in Excel.

```{r}
write.csv(
  file = "mydesign.csv",
  x = matrix(rep(c("Tr1", "Tr2"), each = 48),
             nrow = 8, ncol = 12, dimnames = list(LETTERS[1:8], 1:12)))
```

Now let's take a look at what the file looks like:

```{r}
print_df(read.csv("mydesign.csv", header = FALSE, colClasses = "character"))
```

Here we can see that our design has Treatment 1 on the left-hand side of the plate (wells in columns 1 through 6), and Treatment 2 on the right-hand side of the plate (wells in columns 7 through 12). Let's import this design using `import_blockdesigns` saving it with the column name `Treatment_numbers`.

```{r}
my_design <- import_blockdesigns(files = "mydesign.csv", 
                                 block_names = "Treatment_numbers")
head(my_design, 20)
```

### Importing multiple block-shaped design elements


What do you do if you have multiple designs? For instance, what if you have several strains each in several treatments? In that case, simply save each design component as a separate file, and import them all in one go with `import_blockdesigns`.

First, let's create another example designs file. Again, **don't worry how the below code works**, just imagine that you've created this file in Excel.

```{r}
write.csv(
  file = "mydesign2.csv",
  x = matrix(rep(c("StrA", "StrB", "StrC", "StrD"), each = 24),
             nrow = 8, ncol = 12, dimnames = list(LETTERS[1:8], 1:12),
             byrow = TRUE))
```

Now let's take a look at what the file looks like:

```{r}
print_df(read.csv("mydesign2.csv", header = FALSE, colClasses = "character"))
```

Here we can see that our design has Strain A in the first two rows, Strain B in the next two rows, and so on.

Let's now import both designs using `import_blockdesigns`, saving them to columns named `Treatment_numbers` and `Strain_letters`.

```{r}
my_design <- 
  import_blockdesigns(files = c("mydesign.csv", "mydesign2.csv"), 
                      block_names = c("Treatment_numbers", "Strain_letters"))
head(my_design, 20)
```

## Importing tidy-shaped design files
You can import tidy-shaped designs with `read_tidys`.

`read_tidys` only requires a filename (or vector of filenames, or relative file paths) and will return a `data.frame` (or list of data.frames) that you can save in R.

Once these design elements have been read into the `R` environment, you won't need to transform them. So you can skip down to learning how to merge them with your data in the **[Merging spectrophotometric and design data]** section.

# Generating designs in R
If you'd rather make your design data.frames in R, `make_design` can create:

* block-shaped data.frames with your design information (for saving to files)
* tidy-shaped data.frames with your design information (for saving to files and merging with tidy-shaped data)

## An example with a single design

Let's start with a simple design.

Imagine you have a 96 well plate (12 columns and 8 rows) with a different bacterial strain in each row, leaving the first and last rows and columns empty.

Row names | Column 1 | Column 2  | Column 3  | ... | Column 11 | Column 12
--------- | -------- | --------  | --------  | --- | --------- | --------
Row A     | Blank    | Blank     | Blank     | ... | Blank     | Blank
Row B     | Blank    | Strain #1 | Strain #1 | ... | Strain #1 | Blank
Row B     | Blank    | Strain #2 | Strain #2 | ... | Strain #2 | Blank
...       |...       | ...       | ...       | ... | ...       | ... 
Row G     | Blank    | Strain #5 | Strain #5 | ... | Strain #5 | Blank
Row G     | Blank    | Strain #6 | Strain #6 | ... | Strain #6 | Blank
Row H     | Blank    | Blank     | Blank     | ... | Blank     | Blank

Typing a design like this manually into a spreadsheet can be tedious. But generating it with `make_design` is easier. 

`make_design` first needs some general information, like the `nrows` and `ncols` in the plate, and the `output_format` you'd like (typically `blocks` or `tidy`).

Then, for each different design component, `make_design` needs five different pieces of information:

* a vector containing the possible values
* a vector specifying which rows these values should be applied to
* a vector specifying which columns these values should be applied to
* a string or vector of the pattern of these values
* a Boolean for whether this pattern should be filled byrow (defaults to TRUE)

```{r}
my_design_blk <- make_design(
  output_format = "blocks",
  nrows = 8, ncols = 12, 
  Bacteria = list(c("Str1", "Str2", "Str3", "Str4", "Str5", "Str6"),
                  2:7,
                  2:11,
                  "123456",
                  FALSE)
)
```

So for our example above, we can see:

* the possible values are `c("Strain 1", "Strain 2", "Strain 3", "Strain 4", "Strain 5", "Strain 6")`
* the rows these values should be applied to are `2:7`
* the columns these values should be applied to are `2:11`
* the pattern these values should be filled in by is `"123456"`
* and these values should *not* be filled by row (they should be filled by column)

```{r}
my_design_blk
```

This produces a `data.frame` with `Bacteria` as the `block_name` in the metadata. If we save this design to a file or transform it to tidy-shaped, this `block_name` metadata will come in handy.

## A few notes on the pattern
The pattern in `make_design` is flexible to make it easy to input designs. 

**The "0" character is reserved for `NA` values**, and can be put into your pattern anywhere you'd like to have the value be `NA`

```{r}
my_design_blk <- make_design(
  output_format = "blocks",
  nrows = 8, ncols = 12, 
  Bacteria = list(c("Str1", "Str2", "Str3", 
                    "Str4", "Str5", "Str6"),
                  2:7,
                  2:11,
                  "123056",
                  FALSE)
)
my_design_blk
```

In the previous examples, I used the numbers 1 through 6 to correspond to our values. If you have more than 9 values, you can use letters too. By default, the order is numbers first, then uppercase letters, then lowercase letters (so "A" is the 10th index). However, if you'd like to only use letters, you can simply specify a different `lookup_tbl_start` so that `make_design` knows what letter you're using as the `1` index.

```{r}
my_design_blk <- make_design(
  output_format = "blocks",
  nrows = 8, ncols = 12, lookup_tbl_start = "A",
  Bacteria = list(
    c("Str1", "Str2", "Str3", "Str4", "Str5", "Str6"),
    2:7,
    2:11,
    "ABCDEF",
    FALSE)
)
```

You can also specify the pattern as a vector rather than a string.

```{r}
my_design_blk <- make_design(
  output_format = "blocks",
  nrows = 8, ncols = 12,
  Bacteria = list(
    c("Str1", "Str2", "Str3", "Str4", "Str5", "Str6"),
    2:7,
    2:11,
    c(1,2,3,4,5,6),
    FALSE)
)
```

## Continuing with the example: multiple designs
Now let's return to our example growth curve experiment. *In addition* to having a different bacterial strain in each row, we now also have a different media in each column of the plate. 

Row names | Column 1 | Column 2 | Column 3 | ... | Column 11 | Column 12
--------- | -------- | -------- | -------- | --- | --------- | --------
Row A     | Blank    | Blank    | Blank    | ... | Blank     | Blank
Row B     | Blank    | Media #1 | Media #2 | ... | Media #10 | Blank
...       |...       | ...      | ...      | ... | ...       | ... 
Row G     | Blank    | Media #1 | Media #2 | ... | Media #10 | Blank
Row H     | Blank    | Blank    | Blank    | ... | Blank     | Blank

We can generate both designs with `make_design`:

```{r}
my_design_blk <- make_design(
  output_format = "blocks",
  nrows = 8, ncols = 12, lookup_tbl_start = "a",
  Bacteria = list(c("Str1", "Str2", "Str3", 
                    "Str4", "Str5", "Str6"),
                  2:7,
                  2:11,
                  "abcdef",
                  FALSE),
  Media = list(c("Med1", "Med2", "Med3",
                 "Med4", "Med5", "Med6",
                 "Med7", "Med8", "Med9",
                 "Med10", "Med11", "Med12"),
               2:7,
               2:11,
               "abcdefghij")
  )

my_design_blk
```

However, the real strength of `make_design` is that it is not limited to simple alternating patterns. `make_design` can use irregular patterns too, replicating them as needed to fill all the wells.

```{r}
my_design_blk <- make_design(
  output_format = "blocks",
  nrows = 8, ncols = 12, lookup_tbl_start = "a",
  Bacteria = list(c("Str1", "Str2"),
                  2:7,
                  2:11,
                  "abaaabbbab",
                  FALSE),
  Media = list(c("Med1", "Med2", "Med3"),
               2:7,
               2:11,
               "aabbbc000abc"))

my_design_blk
```

There is also an optional helper function called `make_designpattern`. `make_designpattern` just reminds us what arguments are necessary for each design. For example:

```{r}
my_design_blk <- make_design(
  output_format = "blocks",
  nrows = 8, ncols = 12, lookup_tbl_start = "a",
  Bacteria = make_designpattern(
    values = c("Str1", "Str2", "Str3", 
               "Str4", "Str5", "Str6"),
    rows = 2:7, cols = 2:11, pattern = "abc0ef",
    byrow = FALSE),
  Media = make_designpattern(
    values = c("Med1", "Med2", "Med3",
               "Med4", "Med5", "Med6",
               "Med7", "Med8", "Med9",
               "Med10", "Med11", "Med12"),
    rows = 2:7, cols = 2:11, pattern = "abcde0ghij"))

my_design_blk
```

**For merging our designs with plate reader data, we need it tidy-shaped**, so we just need to change the `output_format` to `tidy`.

```{r}
my_design_tdy <- make_design(
  output_format = "tidy",
  nrows = 8, ncols = 12, lookup_tbl_start = "a",
  Bacteria = make_designpattern(
    values = c("Str1", "Str2", "Str3", 
               "Str4", "Str5", "Str6"),
    rows = 2:7, cols = 2:11, pattern = "abc0ef",
    byrow = FALSE),
  Media = make_designpattern(
    values = c("Med1", "Med2", "Med3",
               "Med4", "Med5", "Med6",
               "Med7", "Med8", "Med9",
               "Med10", "Med11", "Med12"),
    rows = 2:7, cols = 2:11, pattern = "abcde0ghij"))

head(my_design_tdy, 20)
```

## Saving designs to files
If you'd like to save the designs you've created with `make_design` to files, you just need to decide if you'd like them tidy-shaped or block-shaped. Both formats can easily be read back into `R` by `gcplyr`.

### Saving tidy-shaped designs
These design files will be less human-readable, but easier to import and merge. Additionally, tidy-shaped files are often better for data repositories, like Dryad. To save tidy-shaped designs, simply use the built-in `write.csv` function.

```{r}
#See the previous section where we created my_design_tdy
write.csv(x = my_design_tdy, file = "tidy_design.csv",
          row.names = FALSE)
```

### Saving block-shaped designs
These design files will be more human-readable but slightly more computationally involved to import and merge. For these, use the `gcplyr` function `write_blocks`. Typically, you'll use `write_blocks` to save files in one of two formats:

* `multiple` - each block will be saved to its own `.csv` file
* `single` - all the blocks will be saved to a single `.csv` file, with an empty row in between them

#### Saving block-shaped designs to multiple files


The default setting for `write_blocks` is `output_format = 'multiple'`. This creates one `csv` file for each block. If we set `file = NULL`, the default is to name the files according to the `block_names` in the metadata.

```{r}
# See the previous section where we created my_design_blk
write_blocks(my_design_blk, file = NULL)

# Let's see what the files look like
print_df(read.csv("Bacteria.csv", header = FALSE, colClasses = "character"))

print_df(read.csv("Media.csv", header = FALSE, colClasses = "character"))
```

#### Saving block-shaped designs to a single file


The other setting for `write_blocks` is `output_format = 'single'`. This creates a single `csv` file that contains all the blocks, putting metadata like `block_names` in rows that precede each block.

Let's take a look what the `single` output format looks like:

```{r}
# See the previous section where we created my_design_blk
write_blocks(my_design_blk, file = "Design.csv", output_format = "single")

# Let's see what the file looks like
print_df(read.csv("Design.csv", header = FALSE, colClasses = "character"))
```

Here we can see all our design information has been saved to a single file, and the metadata has been added in rows before each block.

# Merging spectrophotometric and design data
Once we have both our design and data in the`R`environment and tidy-shaped, we can merge them using `merge_dfs`.

For this, we'll use the data in the `example_widedata_noiseless` dataset that is included with `gcplyr`, and which was the source for our previous examples with `import_blockmeasures` and `read_wides`.

In the `example_widedata_noiseless` dataset, we have 48 different bacterial strains. The left side of the plate has all 48 strains in a single well each, and the right side of the plate also has all 48 strains in a single well each:

Row names | Column 1   | ... | Column 6   | Column 7   | ... | Column 12
--------- | --------   | --- | --------   | --------   | --- | --------
Row A     | Strain #1  | ... | Strain #6  | Strain #1  | ... | Strain #6
Row B     | Strain #7  | ... | Strain #12 | Strain #7  | ... | Strain #12
...       | ...        | ... | ...        | ...        | ... |  ...
Row G     | Strain #37 | ... | Strain #42 | Strain #37 | ... |  Strain #42
Row H     | Strain #43 | ... | Strain #48 | Strain #43 | ... |  Strain #48

Then, on the right hand side of the plate a phage was also inoculated (while the left hand side remained bacteria-only):

Row names |Column 1 | ... | Column 6 | Column 7    | ... | Column 12
--------- |-------- | --- | -------- | --------    | --- | --------
Row A     |No Phage | ... | No Phage | Phage Added | ... | Phage Added
Row B     |No Phage | ... | No Phage | Phage Added | ... | Phage Added
...       |...      | ... | ...      | ...         | ... |  ... 
Row G     |No Phage | ... | No Phage | Phage Added | ... |  Phage Added
Row H     |No Phage | ... | No Phage | Phage Added | ... |  Phage Added

Let's generate our design:

```{r}
example_design <- make_design(
  nrows = 8, ncols = 12,
  "Bacteria_strain" = make_designpattern(
    values = paste("Strain", 1:48),
    rows = 1:8, cols = 1:6,
    pattern = 1:48,
    byrow = TRUE),
  "Bacteria_strain" = make_designpattern(
    values = paste("Strain", 1:48),
    rows = 1:8, cols = 7:12,
    pattern = 1:48,
    byrow = TRUE),
  "Phage" = make_designpattern(
    values = c("No Phage"),
    rows = 1:8, cols = 1:6,
    pattern = "1"),
  "Phage" = make_designpattern(
    values = c("Phage Added"),
    rows = 1:8, cols = 7:12,
    pattern = "1"))
```

Here's what the resulting data.frame looks like:
```{r}
head(example_design, 20)
```

Now let's transform the `example_widedata_noiseless` to tidy-shaped.

```{r}
example_tidydata <- trans_wide_to_tidy(example_widedata_noiseless,
                                       id_cols = "Time")
```

And finally, we merge the two using `merge_dfs`, saving the result to `ex_dat_mrg`, short for example_data_merged. `merge_dfs` merges using columns with the same name between the two data.frames.

```{r}
ex_dat_mrg <- merge_dfs(example_tidydata, example_design)

head(ex_dat_mrg)
```

# What's next?

Now that you've merged your data and designs, you can pre-process and plot your data

1. Introduction: `vignette("gcplyr")`
2. Importing and transforming data: `vignette("import_transform")`
3. Incorporating design information: `vignette("incorporate_designs")`
4. **Pre-processing and plotting your data: `vignette("preprocess_plot")`**
5. Processing your data: `vignette("process")`
6. Analyzing your data: `vignette("analyze")`
7. Dealing with noise: `vignette("noise")`
8. Statistics, merging other data, and other resources: `vignette("conclusion")`
