You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
|Defining Data Science - _Sketchnote by [@nitya](https://twitter.com/nitya)_|
6
6
7
7
---
8
8
@@ -69,11 +69,11 @@ Vast amounts of data are incomprehensible for a human being, but once we create
69
69
70
70
As we have already mentioned - data is everywhere, we just need to capture it in the right way! It is useful to distinguish between **structured** and **unstructured** data. The former are typically represented in some well-structured form, often as a table or number of tables, while latter is just a collection of files. Sometimes we can also talk about **semistructured** data, that have some sort of a structure that may vary greatly.
71
71
72
-
| Structured | Semi-structured | Unstructured |
73
-
|-----------|-----------------|--------------|
74
-
| List of people with their phone numbers | Wikipedia pages with links | Text of Encyclopaedia Britannica |
75
-
| Temperature in all rooms of a building at every minute for the last 20 years | Collection of scientific papers in JSON format with authors, data of publication, and abstract | File share with corporate documents |
76
-
| Data for age and gender of all people entering the building | Internet pages | Raw video feed from surveillance camera |
| List of people with their phone numbers | Wikipedia pages with links | Text of Encyclopaedia Britannica|
75
+
| Temperature in all rooms of a building at every minute for the last 20 years | Collection of scientific papers in JSON format with authors, data of publication, and abstract | File share with corporate documents |
76
+
| Data for age and gender of all people entering the building | Internet pages| Raw video feed from surveillance camera |
77
77
78
78
## Where to get Data
79
79
@@ -107,7 +107,7 @@ First step is to collect the data. While in many cases it can be a straightforwa
107
107
Storing the data can be challenging, especially if we are talking about big data. When deciding how to store data, it makes sense to anticipate the way you would want later on to query them. There are several ways data can be stored:
108
108
<ul>
109
109
<li>Relational database stores a collection of tables, and uses a special language called SQL to query them. Typically, tables would be connected to each other using some schema. In many cases we need to convert the data from original form to fit the schema.</li>
110
-
<li><ahref="https://en.wikipedia.org/wiki/NoSQL">NoSQL</a> database, such as <ahref="https://azure.microsoft.com/services/cosmos-db/?WT.mc_id=acad-31812-dmitryso">CosmosDB</a>, does not enforce schema on data, and allows storing more complex data, for example, hierarchical JSON documents or graphs. However, NoSQL database does not have rich querying capabilities of SQL, and cannot enforce referential integrity between data.</li>
110
+
<li><ahref="https://en.wikipedia.org/wiki/NoSQL">NoSQL</a> database, such as <ahref="https://azure.microsoft.com/services/cosmos-db/?WT.mc_id=academic-31812-dmitryso">CosmosDB</a>, does not enforce schema on data, and allows storing more complex data, for example, hierarchical JSON documents or graphs. However, NoSQL database does not have rich querying capabilities of SQL, and cannot enforce referential integrity between data.</li>
111
111
<li><ahref="https://en.wikipedia.org/wiki/Data_lake">Data Lake</a> storage is used for large collections of data in raw form. Data lakes are often used with big data, where all data cannot fit into one machine, and has to be stored and processed by a cluster. <ahref="https://en.wikipedia.org/wiki/Apache_Parquet">Parquet</a> is the data format that is often used in conjunction with big data.</li>
@@ -16,7 +16,7 @@ Data processing can be programmed in any programming language, but there are cer
16
16
In this lesson, we will focus on using Python for simple data processing. We will assume basic familiarity with the language. If you want a deeper tour of Python, you can refer to one of the following resources:
17
17
18
18
*[Learn Python in a Fun Way with Turtle Graphics and Fractals](https://github.com/shwars/pycourse) - GitHub-based quick intro course into Python Programming
19
-
*[Take your First Steps with Python](https://docs.microsoft.com/en-us/learn/paths/python-first-steps/?WT.mc_id=acad-31812-dmitryso) Learning Path on [Microsoft Learn](http://learn.microsoft.com/?WT.mc_id=acad-31812-dmitryso)
19
+
*[Take your First Steps with Python](https://docs.microsoft.com/en-us/learn/paths/python-first-steps/?WT.mc_id=academic-31812-dmitryso) Learning Path on [Microsoft Learn](http://learn.microsoft.com/?WT.mc_id=academic-31812-dmitryso)
20
20
21
21
Data can come in many forms. In this lesson, we will consider three forms of data - **tabular data**, **text** and **images**.
22
22
@@ -97,28 +97,28 @@ b = pd.Series(["I","like","to","play","games","and","will","not","change"],index
97
97
df = pd.DataFrame([a,b])
98
98
```
99
99
This will create a horizontal table like this:
100
-
|| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
101
-
|---|---|---|---|---|---|---|---|---|---|
102
-
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
103
-
| 1 | I | like | to | use | Python | and | Pandas | very | much |
After operations above, we will end up with the following DataFrame:
156
156
157
-
|| A | B | DivA | LenB |
158
-
|---|---|---|---|---|
159
-
| 0 | 1 | I | -4.0 | 1 |
160
-
| 1 | 2 | like | -3.0 | 4 |
161
-
| 2 | 3 | to | -2.0 | 2 |
162
-
| 3 | 4 | use | -1.0 | 3 |
163
-
| 4 | 5 | Python | 0.0 | 6 |
164
-
| 5 | 6 | and | 1.0 | 3 |
165
-
| 6 | 7 | Pandas | 2.0 | 6 |
166
-
| 7 | 8 | very | 3.0 | 4 |
167
-
| 8 | 9 | much | 4.0 | 4 |
157
+
|| A | B| DivA | LenB |
158
+
|---|---|------ |---- |---- |
159
+
| 0 | 1 | I | -4.0 | 1|
160
+
| 1 | 2 | like | -3.0 | 4|
161
+
| 2 | 3 | to | -2.0 | 2|
162
+
| 3 | 4 | use | -1.0 | 3|
163
+
| 4 | 5 | Python | 0.0 | 6|
164
+
| 5 | 6 | and | 1.0 | 3|
165
+
| 6 | 7 | Pandas | 2.0 | 6|
166
+
| 7 | 8 | very | 3.0 | 4|
167
+
| 8 | 9 | much | 4.0 | 4|
168
168
169
169
**Selecting rows based on numbers** can be done using `iloc` construct. For example, to select first 5 rows from the DataFrame:
170
170
```python
@@ -183,13 +183,13 @@ df.groupby(by='LenB') \
183
183
```
184
184
This gives us the following table:
185
185
186
-
| LenB | Count | Mean |
187
-
|------|-------|------|
188
-
| 1 | 1 | 1.000000 |
189
-
| 2 | 1 | 3.000000 |
190
-
| 3 | 2 | 5.000000 |
191
-
| 4 | 3 | 6.333333 |
192
-
| 6 | 2 | 6.000000 |
186
+
| LenB | Count | Mean |
187
+
|----|-----| --------|
188
+
| 1 | 1| 1.000000 |
189
+
| 2 | 1| 3.000000 |
190
+
| 3 | 2| 5.000000 |
191
+
| 4 | 3| 6.333333 |
192
+
| 6 | 2| 6.000000 |
193
193
194
194
### Getting Data
195
195
@@ -230,7 +230,7 @@ While data very often comes in tabular form, in some cases we need to deal with
230
230
231
231
In this challenge, we will continue with the topic of COVID pandemic, and focus on processing scientific papers on the subject. There is [CORD-19 Dataset](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge) with more than 7000 (at the time of writing) papers on COVID, available with metadata and abstracts (and for about half of them there is also full text provided).
232
232
233
-
A full example of analyzing this dataset using [Text Analytics for Health](https://docs.microsoft.com/azure/cognitive-services/text-analytics/how-tos/text-analytics-for-health/?WT.mc_id=acad-31812-dmitryso) cognitive service is described [in this blog post](https://soshnikov.com/science/analyzing-medical-papers-with-azure-and-text-analytics-for-health/). We will discuss simplified version of this analysis.
233
+
A full example of analyzing this dataset using [Text Analytics for Health](https://docs.microsoft.com/azure/cognitive-services/text-analytics/how-tos/text-analytics-for-health/?WT.mc_id=academic-31812-dmitryso) cognitive service is described [in this blog post](https://soshnikov.com/science/analyzing-medical-papers-with-azure-and-text-analytics-for-health/). We will discuss simplified version of this analysis.
234
234
235
235
> **NOTE**: We do not provide a copy of the dataset as part of this repository. You may first need to download the [`metadata.csv`](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge?select=metadata.csv) file from [this dataset on Kaggle](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge). Registration with Kaggle may be required. You may also download the dataset without registration [from here](https://ai2-semanticscholar-cord-19.s3-us-west-2.amazonaws.com/historical_releases.html), but it will include all full texts in addition to metadata file.
236
236
@@ -242,15 +242,15 @@ Open [`notebook-papers.ipynb`](notebook-papers.ipynb) and read it from top to bo
242
242
243
243
Recently, very powerful AI models have been developed that allow us to understand images. There are many tasks that can be solved using pre-trained neural networks, or cloud services. Some examples include:
244
244
245
-
***Image Classification**, which can help you categorize the image into one of the pre-defined classes. You can easily train your own image classifiers using services such as [Custom Vision](https://azure.microsoft.com/services/cognitive-services/custom-vision-service/?WT.mc_id=acad-31812-dmitryso)
246
-
***Object Detection** to detect different objects in the image. Services such as [computer vision](https://azure.microsoft.com/services/cognitive-services/computer-vision/?WT.mc_id=acad-31812-dmitryso) can detect a number of common objects, and you can train [Custom Vision](https://azure.microsoft.com/services/cognitive-services/custom-vision-service/?WT.mc_id=acad-31812-dmitryso) model to detect some specific objects of interest.
247
-
***Face Detection**, including Age, Gender and Emotion detection. This can be done via [Face API](https://azure.microsoft.com/services/cognitive-services/face/?WT.mc_id=acad-31812-dmitryso).
245
+
***Image Classification**, which can help you categorize the image into one of the pre-defined classes. You can easily train your own image classifiers using services such as [Custom Vision](https://azure.microsoft.com/services/cognitive-services/custom-vision-service/?WT.mc_id=academic-31812-dmitryso)
246
+
***Object Detection** to detect different objects in the image. Services such as [computer vision](https://azure.microsoft.com/services/cognitive-services/computer-vision/?WT.mc_id=academic-31812-dmitryso) can detect a number of common objects, and you can train [Custom Vision](https://azure.microsoft.com/services/cognitive-services/custom-vision-service/?WT.mc_id=academic-31812-dmitryso) model to detect some specific objects of interest.
247
+
***Face Detection**, including Age, Gender and Emotion detection. This can be done via [Face API](https://azure.microsoft.com/services/cognitive-services/face/?WT.mc_id=academic-31812-dmitryso).
248
248
249
-
All those cloud services can be called using [Python SDKs](https://docs.microsoft.com/samples/azure-samples/cognitive-services-python-sdk-samples/cognitive-services-python-sdk-samples/?WT.mc_id=acad-31812-dmitryso), and thus can be easily incorporated into your data exploration workflow.
249
+
All those cloud services can be called using [Python SDKs](https://docs.microsoft.com/samples/azure-samples/cognitive-services-python-sdk-samples/cognitive-services-python-sdk-samples/?WT.mc_id=academic-31812-dmitryso), and thus can be easily incorporated into your data exploration workflow.
250
250
251
251
Here are some examples of exploring data from Image data sources:
252
-
* In the blog post [How to Learn Data Science without Coding](https://soshnikov.com/azure/how-to-learn-data-science-without-coding/) we explore Instagram photos, trying to understand what makes people give more likes to a photo. We first extract as much information from pictures as possible using [computer vision](https://azure.microsoft.com/services/cognitive-services/computer-vision/?WT.mc_id=acad-31812-dmitryso), and then use [Azure Machine Learning AutoML](https://docs.microsoft.com/azure/machine-learning/concept-automated-ml/?WT.mc_id=acad-31812-dmitryso) to build interpretable model.
253
-
* In [Facial Studies Workshop](https://github.com/CloudAdvocacy/FaceStudies) we use [Face API](https://azure.microsoft.com/services/cognitive-services/face/?WT.mc_id=acad-31812-dmitryso) to extract emotions on people on photographs from events, in order to try to understand what makes people happy.
252
+
* In the blog post [How to Learn Data Science without Coding](https://soshnikov.com/azure/how-to-learn-data-science-without-coding/) we explore Instagram photos, trying to understand what makes people give more likes to a photo. We first extract as much information from pictures as possible using [computer vision](https://azure.microsoft.com/services/cognitive-services/computer-vision/?WT.mc_id=academic-31812-dmitryso), and then use [Azure Machine Learning AutoML](https://docs.microsoft.com/azure/machine-learning/concept-automated-ml/?WT.mc_id=academic-31812-dmitryso) to build interpretable model.
253
+
* In [Facial Studies Workshop](https://github.com/CloudAdvocacy/FaceStudies) we use [Face API](https://azure.microsoft.com/services/cognitive-services/face/?WT.mc_id=academic-31812-dmitryso) to extract emotions on people on photographs from events, in order to try to understand what makes people happy.
254
254
255
255
## Conclusion
256
256
@@ -271,7 +271,7 @@ Whether you already have structured or unstructured data, using Python you can p
271
271
272
272
**Learning Python**
273
273
*[Learn Python in a Fun Way with Turtle Graphics and Fractals](https://github.com/shwars/pycourse)
274
-
*[Take your First Steps with Python](https://docs.microsoft.com/learn/paths/python-first-steps/?WT.mc_id=acad-31812-dmitryso) Learning Path on [Microsoft Learn](http://learn.microsoft.com/?WT.mc_id=acad-31812-dmitryso)
274
+
*[Take your First Steps with Python](https://docs.microsoft.com/learn/paths/python-first-steps/?WT.mc_id=academic-31812-dmitryso) Learning Path on [Microsoft Learn](http://learn.microsoft.com/?WT.mc_id=academic-31812-dmitryso)
Copy file name to clipboardExpand all lines: 2-Working-With-Data/07-python/notebook-papers.ipynb
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@
7
7
"\r\n",
8
8
"In this challenge, we will continue with the topic of COVID pandemic, and focus on processing scientific papers on the subject. There is [CORD-19 Dataset](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge) with more than 7000 (at the time of writing) papers on COVID, available with metadata and abstracts (and for about half of them there is also full text provided).\r\n",
9
9
"\r\n",
10
-
"A full example of analyzing this dataset using [Text Analytics for Health](https://docs.microsoft.com/azure/cognitive-services/text-analytics/how-tos/text-analytics-for-health/?WT.mc_id=acad-31812-dmitryso) cognitive service is described [in this blog post](https://soshnikov.com/science/analyzing-medical-papers-with-azure-and-text-analytics-for-health/). We will discuss simplified version of this analysis."
10
+
"A full example of analyzing this dataset using [Text Analytics for Health](https://docs.microsoft.com/azure/cognitive-services/text-analytics/how-tos/text-analytics-for-health/?WT.mc_id=academic-31812-dmitryso) cognitive service is described [in this blog post](https://soshnikov.com/science/analyzing-medical-papers-with-azure-and-text-analytics-for-health/). We will discuss simplified version of this analysis."
0 commit comments