You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/cognitive-services/big-data/cognitive-services-for-big-data.md
+10-10Lines changed: 10 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,17 +14,17 @@ ms.author: marhamil
14
14
15
15

16
16
17
-
The Azure Cognitive Services for big data lets users channel terabytes of data through Cognitive Services using [Apache Spark™](/dotnet/spark/what-is-spark). With the Cognitive Services for big data, it's easy to create large-scale intelligent applications with any datastore.
17
+
Azure Cognitive Services for big data lets users channel terabytes of data through Cognitive Services using [Apache Spark™](/dotnet/spark/what-is-spark) and open source libraries for distributed machine learning workloads. With Cognitive Services for big data, it's easy to create large-scale intelligent applications with any datastore.
18
18
19
-
With Cognitive Services for big data you can embed continuously improving, intelligent models directly into Apache Spark™ and SQL computations. These tools liberate developers from low-level networking details, so that they can focus on creating smart, distributed applications.
19
+
Using the resources and libraries described in this article, you can embed continuously improving, intelligent models directly into Apache Spark™ and SQL computations. These tools liberate developers from low-level networking details, so that they can focus on creating smart, distributed applications.
20
20
21
21
## Features and benefits
22
22
23
-
Cognitive Services for big data can use services from any region in the world, as well as [containerized Cognitive Services](../cognitive-services-container-support.md). Containers support low or no connectivity deployments with ultra-low latency responses. Containerized Cognitive Services can be run locally, directly on the worker nodes of your Spark cluster, or on an external orchestrator like Kubernetes.
23
+
Cognitive Services for big data can use resources from any [supported region](https://azure.microsoft.comglobal-infrastructure/services/?products=cognitive-services), as well as [containerized Cognitive Services](../cognitive-services-container-support.md). Containers support low or no connectivity deployments with ultra-low latency responses. Containerized Cognitive Services can be run locally, directly on the worker nodes of your Spark cluster, or on an external orchestrator like Kubernetes.
24
24
25
25
## Supported services
26
26
27
-
[Cognitive Services](../index.yml), accessed through APIs and SDKs, help developers build intelligent applications without having AI or data science skills. With Cognitive Services you can make your applications see, hear, speak, understand, and reason. To use the Cognitive Services, your application must send data to the service over the network. Once received, the service sends an intelligent response in return. The following services are available for big data workloads:
27
+
[Cognitive Services](../index.yml), accessed through APIs and SDKs, help developers build intelligent applications without having AI or data science skills. With Cognitive Services you can make your applications see, hear, speak, understand, and reason. To use Cognitive Services, your application must send data to the service over the network. Once received, the service sends an intelligent response in return. The following Cognitive Services resources are available for big data workloads:
28
28
29
29
### Vision
30
30
@@ -59,7 +59,7 @@ Cognitive Services for big data can use services from any region in the world, a
59
59
60
60
## Supported programming languages for Cognitive Services for big data
61
61
62
-
The Cognitive Services for big data are built on Apache Spark. Apache Spark is a distributed computing library that supports Java, Scala, Python, R, and many other languages. These languages are currently supported.
62
+
Cognitive Services for big data are built on Apache Spark. Apache Spark is a distributed computing library that supports Java, Scala, Python, R, and many other languages. These languages are currently supported.
63
63
64
64
### Python
65
65
@@ -71,7 +71,7 @@ We provide a Scala and Java-based Spark API in the `com.microsoft.ml.spark.cogni
71
71
72
72
## Supported platforms and connectors
73
73
74
-
The Cognitive Services for big data requires Apache Spark. There are several Apache Spark platforms that support the Cognitive Services for big data.
74
+
Big data scenarios require Apache Spark. There are several Apache Spark platforms that support Cognitive Services for big data.
75
75
76
76
### Azure Databricks
77
77
@@ -100,11 +100,11 @@ The basis of Spark is the DataFrame: a tabular collection of data distributed ac
100
100
- Do SQL-style computations such as join and filter tables.
101
101
- Apply functions to large datasets using MapReduce style parallelism.
102
102
- Apply Distributed Machine Learning using Microsoft Machine Learning for Apache Spark.
103
-
- Use the Cognitive Services for big data to enrich your data with ready-to-use intelligent services.
103
+
- Use Cognitive Services for big data to enrich your data with ready-to-use intelligent services.
104
104
105
105
### Microsoft Machine Learning for Apache Spark (MMLSpark)
106
106
107
-
[Microsoft Machine Learning for Apache Spark](https://mmlspark.blob.core.windows.net/website/index.html#install) (MMLSpark) is an open-source, distributed machine learning library (ML) built on Apache Spark. The Cognitive Services for big data is included in this package. Additionally, MMLSpark contains several other ML tools for Apache Spark, such as LightGBM, Vowpal Wabbit, OpenCV, LIME, and more. With MMLSpark, you can build powerful predictive and analytical models from any Spark datasource.
107
+
[Microsoft Machine Learning for Apache Spark](https://mmlspark.blob.core.windows.net/website/index.html#install) (MMLSpark) is an open-source, distributed machine learning library (ML) built on Apache Spark. Cognitive Services for big data is included in this package. Additionally, MMLSpark contains several other ML tools for Apache Spark, such as LightGBM, Vowpal Wabbit, OpenCV, LIME, and more. With MMLSpark, you can build powerful predictive and analytical models from any Spark datasource.
108
108
109
109
### HTTP on Spark
110
110
@@ -126,11 +126,11 @@ Cognitive Services for big data is an example of how we can integrate intelligen
126
126
127
127
-[The Azure Cognitive Services on Spark: Clusters with Embedded Intelligent Services](https://databricks.com/session/the-azure-cognitive-services-on-spark-clusters-with-embedded-intelligent-services)
128
128
-[Spark Summit Keynote: Scalable AI for Good](https://databricks.com/session_eu19/scalable-ai-for-good)
129
-
-[The Cognitive Services for big data in Cosmos DB](https://medius.studios.ms/Embed/Video-nc/B19-BRK3004?latestplayer=true&l=2571.208093)
129
+
-[Cognitive Services for big data in Cosmos DB](https://medius.studios.ms/Embed/Video-nc/B19-BRK3004?latestplayer=true&l=2571.208093)
130
130
-[Lightning Talk on Large Scale Intelligent Microservices](https://www.youtube.com/watch?v=BtuhmdIy9Fk&t=6s)
131
131
132
132
## Next steps
133
133
134
-
-[Getting Started with the Cognitive Services for big data](getting-started.md)
134
+
-[Getting Started with Cognitive Services for big data](getting-started.md)
Copy file name to clipboardExpand all lines: articles/cognitive-services/big-data/getting-started.md
+29-29Lines changed: 29 additions & 29 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -98,7 +98,7 @@ To get started on Azure Kubernetes Service, follow these steps:
98
98
99
99
## Try a sample
100
100
101
-
After you set up your Spark cluster and environment, you can run a short sample. This sample assumes Azure Databricks and the `mmlspark.cognitive` package.
101
+
After you set up your Spark cluster and environment, you can run a short sample. This sample assumes Azure Databricks and the `mmlspark.cognitive` package. For an example using `synapseml.cognitive`, see [Add search to AI-enriched data from Apache Spark using SynapseML](/search/search-synapseml-cognitive-services).
102
102
103
103
First, you can create a notebook in Azure Databricks. For other Spark cluster providers, use their notebooks or Spark Submit.
104
104
@@ -114,34 +114,34 @@ First, you can create a notebook in Azure Databricks. For other Spark cluster pr
114
114
115
115
1. Paste this code snippet into your new notebook.
116
116
117
-
```python
118
-
from mmlspark.cognitive import*
119
-
from pyspark.sql.functions import col
120
-
121
-
# Add your region and subscription key from the Language service (or a general Cognitive Service key)
122
-
# If using a multi-region Cognitive Services resource, delete the placeholder text: service_region = ""
123
-
service_key ="ADD-SUBSCRIPTION-KEY-HERE"
124
-
service_region ="ADD-SERVICE-REGION-HERE"
125
-
126
-
df = spark.createDataFrame([
127
-
("I am so happy today, its sunny!", "en-US"),
128
-
("I am frustrated by this rush hour traffic", "en-US"),
129
-
("The cognitive services on spark aint bad", "en-US"),
| br |[{"page":1,"pages":1,"per_page":"50","total":1},[{"id":"BRA","iso2Code":"BR","name":"Brazil","region":{"id":"LCN","iso2code":"ZJ","value":"Latin America & Caribbean "},"adminregion":{"id":"LAC","iso2code":"XJ","value":"Latin America & Caribbean (excluding high income)"},"incomeLevel":{"id":"UMC","iso2code":"XT","value":"Upper middle income"},"lendingType":{"id":"IBD","iso2code":"XF","value":"IBRD"},"capitalCity":"Brasilia","longitude":"-47.9292","latitude":"-15.7801"}]]|
298
-
| usa |[{"page":1,"pages":1,"per_page":"50","total":1},[{"id":"USA","iso2Code":"US","name":"United States","region":{"id":"NAC","iso2code":"XU","value":"North America"},"adminregion":{"id":"","iso2code":"","value":""},"incomeLevel":{"id":"HIC","iso2code":"XD","value":"High income"},"lendingType":{"id":"LNX","iso2code":"XX","value":"Not classified"},"capitalCity":"Washington D.C.","longitude":"-77.032","latitude":"38.8895"}]]|
297
+
| br |`[{"page":1,"pages":1,"per_page":"50","total":1},[{"id":"BRA","iso2Code":"BR","name":"Brazil","region":{"id":"LCN","iso2code":"ZJ","value":"Latin America & Caribbean "},"adminregion":{"id":"LAC","iso2code":"XJ","value":"Latin America & Caribbean (excluding high income)"},"incomeLevel":{"id":"UMC","iso2code":"XT","value":"Upper middle income"},"lendingType":{"id":"IBD","iso2code":"XF","value":"IBRD"},"capitalCity":"Brasilia","longitude":"-47.9292","latitude":"-15.7801"}]]`|
298
+
| usa |`[{"page":1,"pages":1,"per_page":"50","total":1},[{"id":"USA","iso2Code":"US","name":"United States","region":{"id":"NAC","iso2code":"XU","value":"North America"},"adminregion":{"id":"","iso2code":"","value":""},"incomeLevel":{"id":"HIC","iso2code":"XD","value":"High income"},"lendingType":{"id":"LNX","iso2code":"XX","value":"Not classified"},"capitalCity":"Washington D.C.","longitude":"-77.032","latitude":"38.8895"}]]`|
0 commit comments