Error search_command.py", line 934 : line contains NUL

[G0g0Gadget]

| makeresults
| eval string_b64="L2V0Yy9wYXNzd2QA"
| decrypt2 field=string_b64 atob emit('string')

| eval hello="hello"
| decrypt2 field=hello btoa

---

Error at "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 934 : line contains NUL

[gjanders]

Hi, I ran the decrypt command I.e. | decrypt field=string_b64 atob emit('string') And did not see an issue. On Splunk 8.0.8 I tested with python 2 and 3. Regards, Gareth

…

On Thu, 15 Jul 2021 at 10:57 pm, G0g0Gadget ***@***.***> wrote: | makeresults | eval string_b64="L2V0Yy9wYXNzd2QA" | decrypt2 field=string_b64 atob emit('string') | eval hello="hello" | decrypt2 field=hello btoa ------------------------------ Error at "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 934 : line contains NUL — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD2AWJT3G42JZ7ICCSG2WYLTX3LJ3ANCNFSM5ANP3SNQ> .

[G0g0Gadget]

Hello Gareth,

Sorry i forgot the context:
I run these commands with:

• Splunk 8.1.4
• decrypt2 (v2.3.3) in Python3 runtime with splunklib v1.6.16
  Regards,
  Rudy

[gjanders]

Thanks, I'll try again tomorrow but on my work PC it worked fine... I believe my test box is 8.2.x so I'll test that when I have time

…

On Fri, 16 Jul 2021 at 5:10 pm, G0g0Gadget ***@***.***> wrote: Hello Gareth, Sorry i forgot the context: I run these commands with: - Splunk 8.1.4 - decrypt2 (v2.3.3) in Python3 runtime with splunklib v1.6.16 Regards, Rudy — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD2AWJVJQIEEERCO35HZL6DTX7LQBANCNFSM5ANP3SNQ> .

[gjanders]

Hi, I tested: | makeresults | eval string_b64="L2V0Yy9wYXNzd2QA" | decrypt field=string_b64 atob emit('string') That results in a string of /etc/passwd For: | makeresults | eval hello="hello" | decrypt field=hello btoa Since you did not specify a field a field of decrypted was created with value: aGVsbG8= Decrypt2 2.3.3 and Splunk 8.2.0... Regards, Gareth On Fri, 16 Jul 2021 at 18:46, Gareth Anderson ***@***.***> wrote:

…

Thanks, I'll try again tomorrow but on my work PC it worked fine... I believe my test box is 8.2.x so I'll test that when I have time On Fri, 16 Jul 2021 at 5:10 pm, G0g0Gadget ***@***.***> wrote: > Hello Gareth, > > Sorry i forgot the context: > I run these commands with: > > - Splunk 8.1.4 > - decrypt2 (v2.3.3) in Python3 runtime with splunklib v1.6.16 > Regards, > Rudy > > — > You are receiving this because you commented. > > > Reply to this email directly, view it on GitHub > <#1 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AD2AWJVJQIEEERCO35HZL6DTX7LQBANCNFSM5ANP3SNQ> > . >

[gjanders]

Did you remove the old decrypt app before installing decrypt2?

I'm wondering if there is some kind of clash on the splunklib files, can you test from another instance?

[G0g0Gadget]

Hello Gareth,

The encoding of the string b '/etc/passwd' is L2V0Yy9wYXNzd2Q =
On the other hand L2V0Yy9wYXNzd2QA corresponds to the string b '/etc/passwd\x00' which ends with the character NUL (0x00)
With Splunk 8.1.4 / decrypt 2.3.3 / splunklib 1.6.16

If we run this: (there is no error)
| makeresults
| eval string_b64 = "L2V0Yy9wYXNzd2QA"
| decrypt field = string_b64 atob emit ('string')

But if we chain the decrypt command twice with the following code:
| makeresults
| eval string_b64 = "L2V0Yy9wYXNzd2QA"
| decrypt field = string_b64 atob emit ('string')
| eval hello = "hello"
| decrypt field = "hello" btoa emit ('result')

The following error is displayed:
Error at "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 934: line contains NUL

We have several platforms on which we were able to reproduce the problem.
For information, we have two versions of decrypt:

• the latest version of decrypt (the original)
• version 2.3.2 of decrypt which we modified to handle multivalue fields.

As soon as I have time I will try to test with an 8.2.x version of splunk.

[gjanders]

Strange, I just ran:
| makeresults | eval string_b64 = "L2V0Yy9wYXNzd2QA" | decrypt field=string_b64 atob emit('string') | eval hello = "hello" | decrypt field="hello" btoa emit('result')

And that resulted in:

_time	string_b64	string	.decrypt_failure__	hello	result
20/07/2021 19:02	L2V0Yy9wYXNzd2QA	/etc/passwd		hello	aGVsbG8=

On Splunk 8.0.8, so there must be something different between our instances, I'm not quite sure what the difference is...

[malvidin]

I have had issues with NUL in strings and with leading whitespace. I will check later if IAAAIAAA has a consistent output.
I think that the eval urldecode handles NUL in output, but that isn't helpful.

[thehelpseeker]

Hi @gjanders :) I am also facing the same issue as @G0g0Gadget ... when using a single [ | decrypt ] the functions seems to work fine but when 2 or more are used then the following error is seen. On version 2.3.8 :

/decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 959 : line contains NUL

I was wondering if you guys have found a fix for this please or a workaround that could be used ~ Thank you :)

[malvidin]

@thehelpseeker , can you share a makeresults command like @gjanders did with the input values and and decrypt commands you used, as well as the version of Splunk you used?

Strange, I just ran: | makeresults | eval string_b64 = "L2V0Yy9wYXNzd2QA" | decrypt field=string_b64 atob emit('string') | eval hello = "hello" | decrypt field="hello" btoa emit('result')

And that resulted in:

_time	string_b64	string	.decrypt_failure__	hello	result
20/07/2021 19:02	L2V0Yy9wYXNzd2QA	/etc/passwd		hello	aGVsbG8=

On Splunk 8.0.8, so there must be something different between our instances, I'm not quite sure what the difference is...

[thehelpseeker]

Hi @malvidin ! thanks for the quick response ; I tried the Decrypt command @gjanders used :

Search : | makeresults | eval string_b64 = "L2V0Yy9wYXNzd2QA" | decrypt field=string_b64 atob emit('string') | eval hello = "hello" | decrypt field="hello" btoa emit('result')

Version : 8.1

Error : /decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 959 : line contains NUL

[thehelpseeker]

PS if i remove the second | decrypt it works fine; its the second one that breaks it :/

[malvidin]

Kind of. The first command output breaks the second command input.

Splunk will accept null bytes from a Python based app, but it appears it will not read the same bytes in a Python based app. This is because the Splunk SDK uses the Python 3.7 CSV reader. Python 3.11 should read CSV's that contain null bytes, but that will module update will probably not be added to Splunk for quite some time.

Looking at the search.log:

Error at "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 959 : line contains NUL
Traceback:
  File "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 805, in _process_protocol_v2
    self._execute(ifile, None)
  File "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/streaming_command.py", line 55, in _execute
    SearchCommand._execute(self, ifile, self.stream)
  File "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 880, in _execute
    self._execute_v2(ifile, process)
  File "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 986, in _execute_v2
    self._execute_chunk_v2(process, result)
  File "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 999, in _execute_chunk_v2
    self._record_writer.write_records(process(records))
  File "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/internals.py", line 573, in write_records
    records = list(records)
  File "/opt/splunk/etc/apps/decrypt2/bin/decrypt.py", line 22, in stream
    for record in records:
  File "/opt/splunk/etc/apps/decrypt2/bin/lib/splunklib/searchcommands/search_command.py", line 959, in _read_csv_records

The result for the first command includes the trailing null byte. Using the escape command, we can work around the issue:

| makeresults 
| eval string_b64 = "L2V0Yy9wYXNzd2QA" 
| decrypt field=string_b64  atob     escape  emit('string') 
| decrypt field="string"    unescape btoa    emit('string_b64_2')
| eval roundtrip_success = if( rtrim(string_b64, "=") = rtrim(string_b64_2, "="), "true", "false")

Splunk Docker Version: splunk/splunk:8.1
decrypt2 Version: 2.3.8

[malvidin]

The changes in #13 are not perfect, but should assist users to work around the issue.

Error at "/opt/splunk/etc/apps/decrypt2/bin/decrypt.py", line 43 : Splunk record contained NUL. Use eval/replace or sed beforehand to work around, or use the decrypt/escape function in the previous command. (fixed in Python 3.11)

I recommend explicitly cleaning the output with escape/ascii if was from this app, or with builtins like rex/sed and eval/replace. The escape function should be fully reversible with the unescape function, but the ascii function and the Splunk builtins may cause loss of information or unexpected output.

The full traceback should still be included in the search.log.