|
| 1 | +--- |
| 2 | +title: Apache Kafka bindings for Azure Functions |
| 3 | +description: Learn to integrate Azure Functions with an Apache Kafka stream. |
| 4 | +author: ggailey777 |
| 5 | +ms.topic: reference |
| 6 | +ms.date: 05/14/2022 |
| 7 | +ms.author: glenga |
| 8 | +zone_pivot_groups: programming-languages-set-functions-lang-workers |
| 9 | +--- |
| 10 | + |
| 11 | +# Apache Kafka bindings for Azure Functions overview |
| 12 | + |
| 13 | +The Kafka extension for Azure Functions lets you write values out to [Apache Kafka](https://kafka.apache.org/) topics by using an output binding. You can also use a trigger to invoke your functions in response to messages in Kafka topics. |
| 14 | + |
| 15 | +> [!IMPORTANT] |
| 16 | +> Kafka bindings are only available for Functions on the [Elastic Premium Plan](functions-premium-plan.md) and [Dedicated (App Service) plan](dedicated-plan.md). They are only supported on version 3.x and later version of the Functions runtime. |
| 17 | +
|
| 18 | +| Action | Type | |
| 19 | +|---------|---------| |
| 20 | +| Run a function based on a new Kafka event. | [Trigger](./functions-bindings-kafka-trigger.md) | |
| 21 | +| Write to the Kafka event stream. |[Output binding](./functions-bindings-kafka-output.md) | |
| 22 | + |
| 23 | +::: zone pivot="programming-language-csharp" |
| 24 | + |
| 25 | +## Install extension |
| 26 | + |
| 27 | +The extension NuGet package you install depends on the C# mode you're using in your function app: |
| 28 | + |
| 29 | +# [In-process](#tab/in-process) |
| 30 | + |
| 31 | +Functions execute in the same process as the Functions host. To learn more, see [Develop C# class library functions using Azure Functions](functions-dotnet-class-library.md). |
| 32 | + |
| 33 | +Add the extension to your project by installing this [NuGet package](https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.Kafka). |
| 34 | + |
| 35 | +# [Isolated process](#tab/isolated-process) |
| 36 | + |
| 37 | +Functions execute in an isolated C# worker process. To learn more, see [Guide for running C# Azure Functions in an isolated process](dotnet-isolated-process-guide.md). |
| 38 | + |
| 39 | +Add the extension to your project by installing this [NuGet package](https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.Extensions.Kafka). |
| 40 | + |
| 41 | +<!-- |
| 42 | +# [C# script](#tab/csharp-script) |
| 43 | +
|
| 44 | +Functions run as C# script, which is supported primarily for C# portal editing. To update existing binding extensions for C# script apps running in the portal without having to republish your function app, see [Update your extensions]. |
| 45 | +
|
| 46 | +The Kafka extension is part of an [extension bundle], which is specified in your host.json project file. When you create a project that targets version 2.x or later, you should already have this bundle installed. To learn more, see [extension bundle]. |
| 47 | +--> |
| 48 | + |
| 49 | +--- |
| 50 | + |
| 51 | +::: zone-end |
| 52 | + |
| 53 | +::: zone pivot="programming-language-javascript,programming-language-python,programming-language-java,programming-language-powershell" |
| 54 | + |
| 55 | +## Install bundle |
| 56 | + |
| 57 | +The Kafka extension is part of an [extension bundle], which is specified in your host.json project file. When you create a project that targets Functions version 3.x or later, you should already have this bundle installed. To learn more, see [extension bundle]. |
| 58 | + |
| 59 | +::: zone-end |
| 60 | + |
| 61 | +## Enable runtime scaling |
| 62 | + |
| 63 | +To allow your functions to scale properly on the Premium plan when using Kafka triggers and bindings, you need to enable runtime scale monitoring. |
| 64 | + |
| 65 | +# [Azure portal](#tab/portal) |
| 66 | + |
| 67 | +In the Azure portal, in your function app, choose **Configuration** and on the **Function runtime settings** tab turn **Runtime scale monitoring** to **On**. |
| 68 | + |
| 69 | +:::image type="content" source="media/functions-create-vnet/11-enable-runtime-scaling.png" alt-text="Enable runtime scaling in the Azure portal."::: |
| 70 | + |
| 71 | +# [Azure CLI](#tab/azure-cli) |
| 72 | + |
| 73 | +Use the following Azure CLI command to enable runtime scale monitoring: |
| 74 | + |
| 75 | +```azurecli-interactive |
| 76 | +az resource update -g <RESOURCE_GROUP> -n <FUNCTION_APP_NAME>/config/web --set properties.functionsRuntimeScaleMonitoringEnabled=1 --resource-type Microsoft.Web/sites |
| 77 | +``` |
| 78 | + |
| 79 | +--- |
| 80 | + |
| 81 | +## host.json settings |
| 82 | + |
| 83 | +This section describes the configuration settings available for this binding in versions 3.x and higher. Settings in the host.json file apply to all functions in a function app instance. For more information about function app configuration settings in versions 3.x and later versions, see the [host.json reference for Azure Functions](functions-host-json.md). |
| 84 | + |
| 85 | +```json |
| 86 | +{ |
| 87 | + "version": "2.0", |
| 88 | + "extensions": { |
| 89 | + "kafka": { |
| 90 | + "maxBatchSize": 64, |
| 91 | + "SubscriberIntervalInSeconds": 1, |
| 92 | + "ExecutorChannelCapacity": 1, |
| 93 | + "ChannelFullRetryIntervalInMs": 50 |
| 94 | + } |
| 95 | + } |
| 96 | +} |
| 97 | + |
| 98 | +``` |
| 99 | + |
| 100 | +|Property |Default | Type | Description | |
| 101 | +|---------|---------|---------| ---- | |
| 102 | +| ChannelFullRetryIntervalInMs | 50 | Trigger | Defines the subscriber retry interval, in milliseconds, used when attempting to add items to an at-capacity channel. | |
| 103 | +| ExecutorChannelCapacity | 1| Both| Defines the channel message capacity. Once capacity is reached, the Kafka subscriber pauses until the function catches up. | |
| 104 | +| MaxBatchSize | 64 | Trigger | Maximum batch size when calling a Kafka triggered function. | |
| 105 | +| SubscriberIntervalInSeconds | 1 | Trigger | Defines the minimum frequency incoming messages are executed, per function in seconds. Only when the message volume is less than `MaxBatchSize` / `SubscriberIntervalInSeconds`| |
| 106 | + |
| 107 | +The following properties, which are inherited from the [Apache Kafka C/C++ client library](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md), are also supported in the `kafka` section of host.json, for either triggers or both output bindings and triggers: |
| 108 | + |
| 109 | +|Property | Applies to | librdkafka equivalent | |
| 110 | +|---------|---------|---------| |
| 111 | +| AutoCommitIntervalMs | Trigger | `auto.commit.interval.ms` | |
| 112 | +| FetchMaxBytes | Trigger | `fetch.max.bytes` | |
| 113 | +| LibkafkaDebug | Both | `debug` | |
| 114 | +| MaxPartitionFetchBytes | Trigger | `max.partition.fetch.bytes` | |
| 115 | +| MaxPollIntervalMs | Trigger | `max.poll.interval.ms` | |
| 116 | +| MetadataMaxAgeMs | Both | `metadata.max.age.ms` | |
| 117 | +| QueuedMinMessages | Trigger | `queued.min.messages` | |
| 118 | +| QueuedMaxMessagesKbytes | Trigger | `queued.max.messages.kbytes` | |
| 119 | +| ReconnectBackoffMs | Trigger | `reconnect.backoff.max.ms` | |
| 120 | +| ReconnectBackoffMaxMs | Trigger | `reconnect.backoff.max.ms` | |
| 121 | +| SessionTimeoutMs | Trigger | `session.timeout.ms` | |
| 122 | +| SocketKeepaliveEnable | Both | `socket.keepalive.enable` | |
| 123 | +| StatisticsIntervalMs | Trigger | `statistics.interval.ms` | |
| 124 | + |
| 125 | + |
| 126 | +## Next steps |
| 127 | + |
| 128 | +- [Run a function from an Apache Kafka event stream](./functions-bindings-kafka-trigger.md) |
| 129 | + |
| 130 | +[extension bundle]: ./functions-bindings-register.md#extension-bundles |
0 commit comments