Add support for AWS IAM authentication

[korsosa]

AWS MSK has an optional IAM auth feature that enables to use AWS IAM policies for managing authentication/authorization to Kafka resources instead of Kafka ACLs. More information in the AWS MSK documentation.

This change

• includes the AWS library for MSK IAM auth that provides a new SASL mechanism AWS_MSK_IAM
• adds a config option sasl.client.callback.handler.class to allow providing custom SASL callback handlers
• in previous plugin versions the transitive Jar dependencies were not exported. The AWS MSK IAM auth library is a jar without dependencies, this caused issues with finding various AWS classes. Instead of listing all the transitive AWS dependencies, this change also modifies build.gradle to export all Jar dependencies and add them to the jars file. If a transitive dependency should not be exported, the transitive=false flag can be used.

Tested both input and output on an AWS EC2 machine with AWS MSK (Kafka version 2.6.2) and Logstash 8.2.3.

Usage as input/output:

 kafka {
    topic_id => "yourtopic"
    bootstrap_servers => "yourbroker"
    security_protocol => "SASL_SSL"
    sasl_mechanism => "AWS_MSK_IAM"
    sasl_jaas_config => "software.amazon.msk.auth.iam.IAMLoginModule required;"
    sasl_client_callback_handler_class => "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
  }

This PR also fixes the issue #124

Closes: #124

[korsosa]

Hey @andsel & @jsvd, could you please review this PR? Is there anything I can do to help review/merge it? :)

[otavioprado]

Hey @andsel & @jsvd, any idea when this PR will be merged?

[denes16]

Any news on this?

[mvignes89]

Any update on this ?

[korsosa]

Hey @andsel, is there anything I can do to get this merged? I'd be happy to resolve the conflicts and update the PR but not sure if there's any intention from your side to merge it. Unfortunately, I haven't received any feedback since I opened the PR, although I believe this would be a useful addition to the plugin.

[andsel]

Hi @korsosa
the PR seem promising, however would be better to split in 2:

• the first PR would introduce just the setting sasl.client.callback.handler.class, which is the abstract way to provide any SASL client handler
• another one for the specific dependencies related to AWS AIM

[jjatinggoyal]

Hi @korsosa, will you be working on this?

[amiraminb]

Hi @korsosa, Do we have any estimation on when this feature will be merged?

[andsel]

This PR can be closed because split in:

• Exposes sasl.client.callback.handler.class Kafka client setting to select a specific SASL Handler class #177 (merged)
• AWS IAM authentication #178 (waiting to release the uber jar to be referenced by the code in the PR)