Problem Statement

When Redpanda is configured with SASL, a Broadway based application fails to interact with the cluster. A typical error message in the interactive shell looks like this:

 

State: {:state, RedpandaSaslExperiment.Broadway.Broadway.Producer_0.Client, [{"broker.com", 9092}], :undefined, [], :undefined, :undefined, [sasl: {:scram_sha_256, "admin", #Function<11.74114486/0 in :brod_utils.init_sasl_opt/1>}, connect_timeout: 15000, ssl: [verify: :verify_none]], RedpandaSaslExperiment.Broadway.Broadway.Producer_0.Client

 

Solution

1. Clone the kafka_protocol repository locally. Say it is /src/kafka_protocol.

git clone git@github.com:kafka4beam/kafka_protocol.git

 

2. Apply the following patch in the repository

diff --git a/rebar.config.script b/rebar.config.script
index e7dbb8b..1670fcf 100644
--- a/rebar.config.script
+++ b/rebar.config.script
@@ -18,8 +18,8 @@ case IsRebar3 of
   true ->
     CONFIG1;
   false ->
-    URLs = [ {snappyer, "https://github.com/zmstone/snappyer"}
-           , {crc32cer, "https://github.com/zmstone/crc32cer"}
+    URLs = [
+           {crc32cer, "https://github.com/zmstone/crc32cer"}
            ],
     Rebar3Deps = proplists:get_value(deps, CONFIG1),
     Rebar2Deps =
diff --git a/src/kpro_scram.erl b/src/kpro_scram.erl
index 066d9bf..25ad709 100644
--- a/src/kpro_scram.erl
+++ b/src/kpro_scram.erl
@@ -67,7 +67,7 @@ parse(#{ sha := Sha
   <> = ServerNonce,
   Salt = base64:decode(Salt0),
   SaltedPassword = hi(Sha, Password, Salt, Iterations),
-  FinalNoProof = bin([?CHANNEL_BINDING, ",r=", Salt]),
+  FinalNoProof = bin([?CHANNEL_BINDING, ",r=", ServerNonce]),
   AuthMsg = [ClientFirstMsgBare, ",", ServerFirstMsg, ",", FinalNoProof],
   #{ sha => Sha
    , salted_password => SaltedPassword

 

3. Edit mix.exs in your application and add the line # 28 below to specify your custom kafka_protocol location

 22   # Run "mix help deps" to learn about dependencies.
 23   defp deps do
 24     [
 25       {:broadway_kafka, "~> 0.4.0"},
 26       {:certifi, "~> 2.10.0"},
 27       {:credo, "~> 1.6", runtime: false},
 28       {:kafka_protocol, path: "/src/kafka_protocol", override: true},
 29       {:plug_cowboy, "~> 2.5"}
 30       # {:dep_from_hexpm, "~> 0.3.0"},
 31       # {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"}
 32     ]

 

4. Clear all, and run the interactive shell.

mix deps.clean --all; rm mix.lock;
mix deps.get
iex -S mix

 

 

Notes

This is a bug in the underlying library, kafka4beam/kafka_protocol, and the following patch needs to be applied to your environment to make it work.

 

https://github.com/kafka4beam/kafka_protocol/pull/112

 

At the time of writing(2023-04-08), the patch isn't yet committed to the master branch.