You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/cognitive-services/big-data/cognitive-services-for-big-data.md
+32-18Lines changed: 32 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
---
2
-
title: "Cognitive Services for Big Data"
3
-
description: Learn how to leverage Azure Cognitive Services on large datasets using Python, Java, and Scala. With Cognitive Services for Big Data you can embed continuously improving, intelligent models directly into Apache Spark™ and SQL computations.
2
+
title: "Cognitive Services for big data"
3
+
description: Learn how to leverage Azure Cognitive Services on large datasets using Python, Java, and Scala. With Cognitive Services for big data you can embed continuously improving, intelligent models directly into Apache Spark™ and SQL computations.
4
4
services: cognitive-services
5
5
author: mhamilton723
6
6
manager: nitinme
@@ -10,21 +10,21 @@ ms.date: 10/28/2021
10
10
ms.author: marhamil
11
11
---
12
12
13
-
# Azure Cognitive Services for Big Data
13
+
# Azure Cognitive Services for big data
14
14
15
-

15
+

16
16
17
-
The Azure Cognitive Services for Big Data lets users channel terabytes of data through Cognitive Services using [Apache Spark™](/dotnet/spark/what-is-spark). With the Cognitive Services for Big Data, it's easy to create large-scale intelligent applications with any datastore.
17
+
Azure Cognitive Services for big data lets users channel terabytes of data through Cognitive Services using [Apache Spark™](/dotnet/spark/what-is-spark) and open source libraries for distributed machine learning workloads. With Cognitive Services for big data, it's easy to create large-scale intelligent applications with any datastore.
18
18
19
-
With Cognitive Services for Big Data you can embed continuously improving, intelligent models directly into Apache Spark™ and SQL computations. These tools liberate developers from low-level networking details, so that they can focus on creating smart, distributed applications.
19
+
Using the resources and libraries described in this article, you can embed continuously improving, intelligent models directly into Apache Spark™ and SQL computations. These tools liberate developers from low-level networking details, so that they can focus on creating smart, distributed applications.
20
20
21
21
## Features and benefits
22
22
23
-
Cognitive Services for Big Data can use services from any region in the world, as well as [containerized Cognitive Services](../cognitive-services-container-support.md). Containers support low or no connectivity deployments with ultra-low latency responses. Containerized Cognitive Services can be run locally, directly on the worker nodes of your Spark cluster, or on an external orchestrator like Kubernetes.
23
+
Cognitive Services for big data can use resources from any [supported region](https://azure.microsoft.comglobal-infrastructure/services/?products=cognitive-services), as well as [containerized Cognitive Services](../cognitive-services-container-support.md). Containers support low or no connectivity deployments with ultra-low latency responses. Containerized Cognitive Services can be run locally, directly on the worker nodes of your Spark cluster, or on an external orchestrator like Kubernetes.
24
24
25
25
## Supported services
26
26
27
-
[Cognitive Services](../index.yml), accessed through APIs and SDKs, help developers build intelligent applications without having AI or data science skills. With Cognitive Services you can make your applications see, hear, speak, understand, and reason. To use the Cognitive Services, your application must send data to the service over the network. Once received, the service sends an intelligent response in return. The following services are available for big data workloads:
27
+
[Cognitive Services](../index.yml), accessed through APIs and SDKs, help developers build intelligent applications without having AI or data science skills. With Cognitive Services you can make your applications see, hear, speak, understand, and reason. To use Cognitive Services, your application must send data to the service over the network. Once received, the service sends an intelligent response in return. The following Cognitive Services resources are available for big data workloads:
28
28
29
29
### Vision
30
30
@@ -57,21 +57,35 @@ Cognitive Services for Big Data can use services from any region in the world, a
57
57
|:-----------|:------------------|
58
58
|[Bing Image Search](/azure/cognitive-services/bing-image-search"Bing Image Search")|The Bing Image Search service returns a display of images determined to be relevant to the user's query.|
59
59
60
-
## Supported programming languages for Cognitive Services for Big Data
60
+
## Supported programming languages for Cognitive Services for big data
61
61
62
-
The Cognitive Services for Big Data are built on Apache Spark. Apache Spark is a distributed computing library that supports Java, Scala, Python, R, and many other languages. These languages are currently supported.
62
+
Cognitive Services for big data are built on Apache Spark. Apache Spark is a distributed computing library that supports Java, Scala, Python, R, and many other languages. See [SynapseML](https://microsoft.github.io/SynapseML) for documentation, samples, and blog posts.
63
+
64
+
The following languages are currently supported.
63
65
64
66
### Python
65
67
66
-
We provide a PySpark API in the `mmlspark.cognitive` namespace of [Microsoft ML for Apache Spark](https://aka.ms/spark). For more information, see the [Python Developer API](https://mmlspark.blob.core.windows.net/docs/1.0.0-rc1/pyspark/mmlspark.cognitive.html). For usage examples, see the [Python Samples](samples-python.md).
68
+
We provide a PySpark API for current and legacy libraries:
For more information, see the [Python Developer API](https://mmlspark.blob.core.windows.net/docs/1.0.0-rc1/pyspark/mmlspark.cognitive.html). For usage examples, see the [Python Samples](samples-python.md).
67
75
68
76
### Scala and Java
69
77
70
-
We provide a Scala and Java-based Spark API in the `com.microsoft.ml.spark.cognitive` namespace of [Microsoft ML for Apache Spark](https://aka.ms/spark). For more information, see the [Scala Developer API](https://mmlspark.blob.core.windows.net/docs/1.0.0-rc1/scala/index.html#package). For usage examples, see the [Scala Samples](samples-scala.md).
78
+
We provide a Scala and Java-based Spark API for current and legacy libraries:
For more information, see the [Scala Developer API](https://mmlspark.blob.core.windows.net/docs/1.0.0-rc1/scala/index.html#package). For usage examples, see the [Scala Samples](samples-scala.md).
71
85
72
86
## Supported platforms and connectors
73
87
74
-
The Cognitive Services for Big Data requires Apache Spark. There are several Apache Spark platforms that support the Cognitive Services for Big Data.
88
+
Big data scenarios require Apache Spark. There are several Apache Spark platforms that support Cognitive Services for big data.
75
89
76
90
### Azure Databricks
77
91
@@ -100,15 +114,15 @@ The basis of Spark is the DataFrame: a tabular collection of data distributed ac
100
114
- Do SQL-style computations such as join and filter tables.
101
115
- Apply functions to large datasets using MapReduce style parallelism.
102
116
- Apply Distributed Machine Learning using Microsoft Machine Learning for Apache Spark.
103
-
- Use the Cognitive Services for Big Data to enrich your data with ready-to-use intelligent services.
117
+
- Use Cognitive Services for big data to enrich your data with ready-to-use intelligent services.
104
118
105
119
### Microsoft Machine Learning for Apache Spark (MMLSpark)
106
120
107
-
[Microsoft Machine Learning for Apache Spark](https://mmlspark.blob.core.windows.net/website/index.html#install) (MMLSpark) is an open-source, distributed machine learning library (ML) built on Apache Spark. The Cognitive Services for Big Data is included in this package. Additionally, MMLSpark contains several other ML tools for Apache Spark, such as LightGBM, Vowpal Wabbit, OpenCV, LIME, and more. With MMLSpark, you can build powerful predictive and analytical models from any Spark datasource.
121
+
[Microsoft Machine Learning for Apache Spark](https://mmlspark.blob.core.windows.net/website/index.html#install) (MMLSpark) is an open-source, distributed machine learning library (ML) built on Apache Spark. Cognitive Services for big data is included in this package. Additionally, MMLSpark contains several other ML tools for Apache Spark, such as LightGBM, Vowpal Wabbit, OpenCV, LIME, and more. With MMLSpark, you can build powerful predictive and analytical models from any Spark datasource.
108
122
109
123
### HTTP on Spark
110
124
111
-
Cognitive Services for Big Data is an example of how we can integrate intelligent web services with big data. Web services power many applications across the globe and most services communicate through the Hypertext Transfer Protocol (HTTP). To work with *arbitrary* web services at large scales, we provide HTTP on Spark. With HTTP on Spark, you can pass terabytes of data through any web service. Under the hood, we use this technology to power Cognitive Services for Big Data.
125
+
Cognitive Services for big data is an example of how we can integrate intelligent web services with big data. Web services power many applications across the globe and most services communicate through the Hypertext Transfer Protocol (HTTP). To work with *arbitrary* web services at large scales, we provide HTTP on Spark. With HTTP on Spark, you can pass terabytes of data through any web service. Under the hood, we use this technology to power Cognitive Services for big data.
112
126
113
127
## Developer samples
114
128
@@ -126,11 +140,11 @@ Cognitive Services for Big Data is an example of how we can integrate intelligen
126
140
127
141
-[The Azure Cognitive Services on Spark: Clusters with Embedded Intelligent Services](https://databricks.com/session/the-azure-cognitive-services-on-spark-clusters-with-embedded-intelligent-services)
128
142
-[Spark Summit Keynote: Scalable AI for Good](https://databricks.com/session_eu19/scalable-ai-for-good)
129
-
-[The Cognitive Services for Big Data in Cosmos DB](https://medius.studios.ms/Embed/Video-nc/B19-BRK3004?latestplayer=true&l=2571.208093)
143
+
-[Cognitive Services for big data in Cosmos DB](https://medius.studios.ms/Embed/Video-nc/B19-BRK3004?latestplayer=true&l=2571.208093)
130
144
-[Lightning Talk on Large Scale Intelligent Microservices](https://www.youtube.com/watch?v=BtuhmdIy9Fk&t=6s)
131
145
132
146
## Next steps
133
147
134
-
-[Getting Started with the Cognitive Services for Big Data](getting-started.md)
148
+
-[Getting Started with Cognitive Services for big data](getting-started.md)
0 commit comments