-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy path21-eda.Rmd
More file actions
199 lines (103 loc) · 6.21 KB
/
Copy path21-eda.Rmd
File metadata and controls
199 lines (103 loc) · 6.21 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
# (PART) PART II EDA BASICS {-}
# Exploratory Data analysis {#eda}
## What is EDA (Posit Primers: [Visualise Data](https://posit.cloud/learn/primers/3.1))
1. EDA is an iterative cycle that helps you understand what your data says. When you do EDA, you:
2. Generate questions about your data
3. Search for answers by visualising, transforming, and/or modeling your data
Use what you learn to refine your questions and/or generate new questions
EDA is an important part of any data analysis. You can use EDA to make discoveries about the world; or you can use EDA to ensure the quality of your data, asking questions about whether the data meets your standards or not.
## EDA Workflow using R Studio
### EDA Step 0
1. Choose and clarify a topic to study.
2. List questions to study
3. Find data:
- link to data with a url: universal resource locator in a webpage
- download data in csv, Excel, etc.
Repeat the process during your EDA.
### EDA by R Studio: Step 1
In RStudio,
1.1. Project
* Create a new project: File > New Project; or
* Open a project: File > Open Project, Open Project in New Session, Open Recent Project
- It is easier to find an existing project from: File > Recent Project
* _Check there is a file `project_name.Rproj` in your project folder (directory)_
1.2. data folder (directory) `data`
* Create a data folder: Press New Folder at the right bottom pane; or
* Confirm the data folder previously created: Press Files at the right bottom pane
* _If you follow 1, the data folder exists in your project folder_
1.3. Move (or copy) data for the project to the data folder
* If you downloaded the data, it is in your Download folder. Move it to `data`.
* _Check in your RStudio that your data is in `data`: Press Files at the right bottom pane and click `data`, the data folder._
### EDA by R Studio: Step 2
2.1. Project Notebook: Memo
- Create an R Notebook: File > New File > R Notebook
+ You can use R Notebook template in Moodle by moving the template (template.Rmd or template.nb.Rmd) file in your project folder or copy and paste the text file into your new R Notebook.
+ If you use template.nb.Rmd (R Notebook File), choose Open in Editor.
- Add descriptive title.
2.2. Setup Code Chunk
- Create a code chunk and add packages to use in the project and RUN the code.
+ library(tidyverse)
+ library(WDI)
+ or any other packages
2.3. Choose `Source` or `Visual` editor mode, and start editing Project Notebook
- Set up Headings such as: About, Data, Analysis and Visualizations, Conclusions
- Under About or Data, paste url of the sites and/or the data
+ eg. [World Development Indicator](https://datatopics.worldbank.org/world-development-indicators/)
+ eg. [Public expenditure on education](https://data.un.org/_Docs/SYB/CSV/SYB65_245_202209_Public%20expenditure%20on%20education.csv)
2.4. Edit a new file by saving as for a report
- File > Save As...
### EDA by R Studio: Step 3 - Importing Data
Assign a name you can recall easily when you import data. You may need to reload the data with options.
3.1. Use a package:
* WDI, wir, eurostat, etc/
* `wdi_shortname <- WDI(indicator = "indicator's name", ... )
* Store the data and use it: `write_csv(wdi_shortname, "./data/wdi_shortname.csv")`
* `wdi_shortname <- read_csv("./data/wdi_shortname.csv")`
3.2. Use `readr` to read from `data`, your data folder
* `df1_shortname <- read_csv("./data/file_name.csv")`
3.3. Use `readr` to read using the url of the data
* `df2_shortname <- read_csv("url_of_the_data")`
* Store the data and use it: `write_csv(df2_shortname, "./data/df2_shortname.csv")`
* `df2_shortname <- read_csv("./data/df2_shortname.csv")`
3.5. Use `readxl` to read Excel data. Add `library(readxl)` in the setup and run.
* `df4 <- read_excel("./data/file_name.xlsx", sheet = 1)`
#### References
* Cheat Sheet - `readr`
* [readr](https://readr.tidyverse.org)
* [readxl](https://readxl.tidyverse.org)
### EDA by R Studio: Step 4 - Data Trasnformation
4.1. Look at the data: suppose `df` is the data frame
* It is a good option to change into a tibble: `dt <- as_tibble(df)`
* `head(df)`, `str(df)`, `summary(df)`, `dt`, `glimpse(dt)`
4.2. Look at each variable
* categorical? numerical?
* factor? - [forcats](https://forcats.tidyverse.org)
4.3. Variation of each data: suppose `x1` is a column name.
* `df %>% ggplot() + geom_histogram(aes(x1), bins = 30)`
* `df %>% drop_na(x1)`: see the rows with a value in `x1`. If the value is NA, the row is not shown.
- `df_wo_na <- df %>% drop_na(x1)` if you want to use only the rows without NA in `x1`
4.4. Use `dpylr` and `tidyr` to change column names, tidy data, and/or summarize data
* `rename`, `select`, `filter`, `arrange`, `mutate`, `pivot_longer()`, `pivot_wider()`, `group_by` and `summarize`
#### References
* Cheat Sheet - `dplyr` and `tidyr`
* [dplyr](https://dplyr.tidyverse.org)
* [tidyr](https://tidyr.tidyverse.org)
### EDA by R Studio: Step 5 - Visualize Data
5.1. In combination with Stap 4 - data transformation, try various data visualization.
* What type of variation occurs within my variables?
* What type of covariation occurs between my variables?
5.2. Keep a record of what you can observe by the visualization
5.3. Edit the list of questions by adding or polishing
5.4. Select several informative chart and add options
5.5. Look at examples from the textbooks or teaching site to have better visualization
#### References
* Cheat Sheet - `ggplot2`
* [ggplot2](https://ggplot2.tidyverse.org)
* [ggplot2 book](https://ggplot2-book.org)
### EDA by R Studio: Step 6 - Conclusions and Questions for Further Study
1. EDA is an iterative cycle that helps you understand what your data says. When you do EDA, you:
2. Generate questions about your data
3. Search for answers by visualising, transforming, and/or modeling your data
Use what you learn to refine your questions and/or generate new questions
EDA is an important part of any data analysis. You can use EDA to make discoveries about the world; or you can use EDA to ensure the quality of your data, asking questions about whether the data meets your standards or not.