intel_adsp_cavs18 fails with heap errors on current Zephyr

[andyross]

@lyakh reports that the SOF application fails on intel_adsp_cavs18 hardware with current code. If I understand correctly:

1. The https://github.com/andyross/zephyr/tree/sof-rebased-19Oct20 test branch works on 1.8 hardware.

2. https://github.com/andyross/zephyr/tree/sof-rebased-22Oct20 does not, failing with an error that appears to be heap related.

The main delta between these trees (other than a Zephyr rebase) is
that the first contains a full reversion of the heap optimization
series from @npitre (commit 63ee851).

Manually reverting lib/os/heap to that state in the failing tree fixes
the issues (is that right @lyakh ?)

(This is a continuation of discussion started in thesofproject/sof#3544)

[andyross]

@lyakh @npitre I opened this bug in Zephyr to have a better place to discuss

[andyross]

OK, so to start, we should clean up the trees and stop using those intermediate ones I was pushing for integration/demo work. Virtually everything should be in zephyr/sof masters now. All you need is to add the SOF app to Zephyr and then apply one trivial build fixup to SOF:

Zephyr: https://github.com/zephyrproject-rtos/zephyr/pull/29952
SOF: https://github.com/thesofproject/sof/pull/3601

This works for me to get through SOF initialization on Up Squared (CAVS 1.5) hardware.

[andyross]

Next, since I remain relatively certain this is going to turn out to be heap corruption, can you try something like this on your failing SOF tree and see if it acts to suppress the error:

diff --git a/zephyr/wrapper.c b/zephyr/wrapper.c
index 298160312..39ef575cd 100644
--- a/zephyr/wrapper.c
+++ b/zephyr/wrapper.c
@@ -61,6 +61,9 @@ static void *heap_alloc_aligned(struct k_heap *h, size_t align, size_t bytes)
 {
        void *ret = NULL;
 
+       // FIXME: pad buffer to hunt for overflows
+       bytes += 16;
+
        k_spinlock_key_t key = k_spin_lock(&h->lock);
 
        ret = sys_heap_aligned_alloc(&h->heap, align, bytes);

It's easy and cheap, and if "works" then it tells us the bug is probably on the SOF side. We can then start digging through all the specific allocations at runtime and see if we can find the specific one that needs to be padded. It's tedious, but there actually aren't that many; this is the sort of thing I was doing hunting for the original alignment bug.

[andyross]

And finally: @lyakh can you push your current 1.8 changes somewhere I can look at them? The last versions I have are the ones you pushed to my 17Sep branch, and that's very stale now. At this stage, you should be able to submit directly to Zephyr master via pull request. We're ready finally, I promise. :)

[lyakh]

@andyross sorry, I was hacking away to test the Zephyr "master" and I've done that: took the current Zephyr master, applied your 2 SOF integration patches and my cAVS 1.8 work. To that I've taken your SOF tree thesofproject/sof#3601 and also added my stuff on top. That still didn't work. I think I better stop for today, but I'll do what you asked about tomorrow.