Any ARM Linux support

[srikanth007m]

Hi,
I haven't tried the latency tool. But just did a walk through. Is there any ARM support as well?

[jdesfossez]

Hi,

I haven't tested to build on ARM, I guess the kprobes would need a little work to be arch independant, but besides that, it should work. I don't have any ARM board right now, but if you happen to test it and see problems, let me know and I'll make sure to fix them.

Thanks,

Julien

[srikanth007m]

Cross compiling is failing at

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-
make -C /lib/modules/3.15.0-031500rc2-generic/build SUBDIRS=/local/mnt2/workspace2/tools/latency_tracker modules
make[1]: Entering directory `/usr/src/linux-headers-3.15.0-031500rc2-generic'
  CC [M]  /local/mnt2/workspace2/tools/latency_tracker/tracker.o
In file included from include/asm-generic/int-ll64.h:10:0,
                 from /usr/src/linux-headers-3.15.0-031500rc2-generic/arch/arm/include/asm/types.h:4,
                 from include/uapi/linux/types.h:4,
                 from include/linux/types.h:5,
                 from include/linux/list.h:4,
                 from include/linux/module.h:9,
                 from /local/mnt2/workspace2/tools/latency_tracker/tracker.c:23:
include/uapi/asm-generic/int-ll64.h:11:29: fatal error: asm/bitsperlong.h: No such file or directory
compilation terminated.
make[2]: *** [/local/mnt2/workspace2/tools/latency_tracker/tracker.o] Error 1
make[1]: *** [_module_/local/mnt2/workspace2/tools/latency_tracker] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-3.15.0-031500rc2-generic'
make: *** [default] Error 2

[sgkimsm]

Hi,

Is this still open issue?
If it is the case, then do you have plan to support other architecture?

Thank you.

[jdesfossez]

Hi,

Yes we definitely plan to support other architectures.
We haven't had the time to look into this yet, but we will.

Most of the work is located in the rculfhash* files. It is a port from the userspace-rcu code to a kernel module and still uses per-architecture wrappers instead of the ones already in the kernel, so it should not be too hard to make it work, just remove these wrappers and let the kernel do it.

Also, some module extract arguments from kretprobes and right now only x86 support code is done, but it should be easy to make it generic.

[jdesfossez]

The fixes for ARM are now in the master branch !
I have tested them on ARMv7 (Freescale i.MX6).

Please test it and let me know if it works for you or if you encounter any issue.

Here is a way to test the interrupt to user-space tracker with ftrace:

modprobe latency_tracker                                                        
modprobe latency_tracker_rt                                                     
                                                                                
echo 0 >/sys/kernel/debug/tracing/events/enable # disable already enabled events (if necessary)
echo 1 >/sys/kernel/debug/tracing/events/latency_tracker/latency_tracker_rt/enable
echo 1 >/sys/kernel/debug/tracing/tracing_on                                    
                                          
                                      
echo 1 >/sys/kernel/debug/latency/rt/threshold # threshold set to 1ns for testing
echo 1 >/sys/kernel/debug/latency/rt/tracking_on                                
                                                                                
echo 1 >/sys/kernel/debug/latency/rt/filters/text_breakdown                     
                                                                                
cat /sys/kernel/debug/tracing/trace_pipe                                        
                                                   
# to see when the user-space process returns to a blocking state after          
# being woken up by an interrupt (might become very verbose)                    
echo 0 >/sys/kernel/debug/latency/rt/filters/enter_userspace```

[sgkimsm]

Thank you for the help.

I tried it without success.
Do you think there is kernel version dependency with it? I'm using 3.10.17 which is quite old.

The only thing I need to do additionally was adding 'llist_for_each_entry_safe' define since it was missing in 3.10.17.
I should use 'default' also as below,

echo 1 >/sys/kernel/debug/latency/rt/default/threshold
echo 1 >/sys/kernel/debug/latency/rt/default/tracking_on
echo 1 >/sys/kernel/debug/latency/rt/default/filters/text_breakdown
echo 0 >/sys/kernel/debug/latency/rt/default/filters/enter_userspace

but 'cat trace_pipe' doesn't show anything, and other trackers like wakeup, offcpu and syscall don't work also.

/sys/kernel/debug/tracing# cat set_event
latency_tracker:latency_tracker_wakeup
latency_tracker:latency_tracker_offcpu_sched_switch
latency_tracker:latency_tracker_offcpu_sched_wakeup
latency_tracker:latency_tracker_syscall_stack
latency_tracker:latency_tracker_syscall
latency_tracker:latency_tracker_rt

/sys/kernel/debug/tracing# cat trace_pipe
^C

/sys/kernel/debug/tracing#

FYI.

/sys/kernel/debug/tracing# cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 10 (v7l)
BogoMIPS : 790.52
Features : swp half thumb fastmult vfp edsp neon vfpv3 tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc09
CPU revision : 10

processor : 1
model name : ARMv7 Processor rev 10 (v7l)
BogoMIPS : 790.52
Features : swp half thumb fastmult vfp edsp neon vfpv3 tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc09
CPU revision : 10

processor : 2
model name : ARMv7 Processor rev 10 (v7l)
BogoMIPS : 790.52
Features : swp half thumb fastmult vfp edsp neon vfpv3 tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc09
CPU revision : 10

processor : 3
model name : ARMv7 Processor rev 10 (v7l)
BogoMIPS : 790.52
Features : swp half thumb fastmult vfp edsp neon vfpv3 tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc09
CPU revision : 10

Hardware : Freescale i.MX6 Quad/DualLite (Device Tree)
Revision : 0000
Serial : 0000000000000000

[jdesfossez]

Hi,

Thanks for trying it out.

Yes, I added the dummy session "default" since my reply here to allow having multiple trackers loaded at the same time eventually without breaking the old examples.

Do you have a way to test on a more recent kernel (> 3.12) ?

[sgkimsm]

Hi,

I tried it with wand board and kernel version 4.1.15, but still have issues.
Do you have recommended kernel config for proper working of trackers?

wakeup:
[ 105.500174] ------------[ cut here ]------------
[ 105.500248] WARNING: CPU: 0 PID: 949 at /mnt/build/work/seokim/lttng/latency-tracker/trackers/wakeup_latency.c:169 wakeup_latency_init+0x8c/0xe0 latency_tracker_wakeup
[ 105.500270] Modules linked in: latency_tracker_wakeup(O+) latency_tracker_syscalls(O) latency_tracker(O)
[ 105.500339] CPU: 0 PID: 949 Comm: modprobe Tainted: G O 4.1.15-1.1.0_ga-wandboard #3
[ 105.500361] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[ 105.500428] [<80017ed8>] (unwind_backtrace) from [<80013e30>] (show_stack+0x20/0x24)
[ 105.500470] [<80013e30>] (show_stack) from [<80965968>] (dump_stack+0x8c/0xcc)
[ 105.500513] [<80965968>] (dump_stack) from [<80037f1c>] (warn_slowpath_common+0x98/0xc8)
[ 105.500551] [<80037f1c>] (warn_slowpath_common) from [<80038008>] (warn_slowpath_null+0x2c/0x34)
[ 105.500596] [<80038008>] (warn_slowpath_null) from [<7f02008c>] (wakeup_latency_init+0x8c/0xe0 [latency_tracker_wakeup])
[ 105.500644] [<7f02008c>] (wakeup_latency_init [latency_tracker_wakeup]) from [<80009820>] (do_one_initcall+0x9c/0x1e8)
[ 105.500677] [<80009820>] (do_one_initcall) from [<809636c8>] (do_init_module+0x70/0x1bc)
[ 105.500713] [<809636c8>] (do_init_module) from [<800aef18>] (load_module+0x181c/0x1e84)
[ 105.500745] [<800aef18>] (load_module) from [<800af724>] (SyS_finit_module+0x74/0x84)
[ 105.500780] [<800af724>] (SyS_finit_module) from [<8000ff00>] (__sys_trace_return+0x0/0x20)
[ 105.500801] ---[ end trace 33f0cfa85c75b22e ]---

offcpu:
modprobe: ERROR: could not insert 'latency_tracker_offcpu': Unknown symbol in module, or unknown parameter (see dmesg)
root@wandboard:/lib/modules/4.1.15-1.1.0_ga-wandboard# [805605.391285] Unable to handle kernel paging request at virtual address 7f026654
[805605.398633] pgd = 86cd0000
[805605.401446] [7f026654] *pgd=16d3f811, *pte=00000000, *ppte=00000000
[805605.407893] Internal error: Oops: 80000007 [#1] PREEMPT SMP ARM
[805605.413918] Modules linked in: latency_tracker_wakeup(O) latency_tracker(O)
[805605.421061] CPU: 0 PID: 433 Comm: avahi-daemon Tainted: G W O 4.1.15-1.1.0_ga-wandboard #3
[805605.430385] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[805605.437021] task: 863edb00 ti: 86b34000 task.ti: 86b34000
[805605.442527] PC is at 0x7f026654
[805605.445786] LR is at __schedule+0x370/0x6f8
[805605.450078] pc : [<7f026654>] lr : [<80969c78>] psr: 20010193
[805605.450078] sp : 86b35f28 ip : 00000003 fp : 86b35f74
[805605.461748] r10: 863edeac r9 : 80dcb190 r8 : 8681d480
[805605.467076] r7 : 8bb24980 r6 : 8738b618 r5 : 863edb00 r4 : 8bb24980
[805605.473706] r3 : 7f026654 r2 : 8681d480 r1 : 863edb00 r0 : 00000000
[805605.480340] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[805605.487666] Control: 10c53c7d Table: 16cd0059 DAC: 00000015
[805605.493516] Process avahi-daemon (pid: 433, stack limit = 0x86b34210)
[805605.500060] Stack: (0x86b35f28 to 0x86b36000)
[805605.504528] 5f20: 80dc4d78 00000000 00000010 00000000 86b35f54 86b35f48
[805605.512817] 5f40: 8038ccf8 8038cb90 86b35f84 86b34000 86b34010 00000000 86b35fb0 00000000
[805605.521105] 5f60: 86b34000 00000000 86b35f8c 86b35f78 8096a04c 80969914 86b35f90 00000002
[805605.529392] 5f80: 86b35fac 86b35f90 80013648 8096a00c 76ee61c6 20010030 ffffffff 10c53c7d
[805605.537680] 5fa0: 00000000 86b35fb0 8000fda4 80013624 7e941acc 61000000 7e941ad0 00000061
[805605.545967] 5fc0: 00000000 de5e496f 7e941acc 7e941ac4 000264b0 00000000 00000000 00000000
[805605.554253] 5fe0: 76ef2080 7e941ac0 76ee5897 76ee61c6 20010030 ffffffff 00000000 00000000
[805605.562560] [<80969c78>] (__schedule) from [<8096a04c>] (schedule+0x4c/0xa4)
[805605.569737] [<8096a04c>] (schedule) from [<80013648>] (do_work_pending+0x30/0xc4)
[805605.577344] [<80013648>] (do_work_pending) from [<8000fda4>] (work_pending+0xc/0x20)
[805605.585195] Code: bad PC value
[805605.588360] ---[ end trace 25d915145b8df3e2 ]---
[805605.593086] note: avahi-daemon[433] exited with preempt_count 3

rt:
modprobe: ERROR: could not insert 'latency_tracker_rt': Unknown symbol in module, or unknown parameter (see dmesg)

[805493.333571] latency-tracker: Error, required tracepoint sched_waking not available
[805493.333885] latency_tracker: 0 events were still pending at destruction

syscall:
root@wandboard:/sys/kernel/debug/tracing# cat set_event
latency_tracker:latency_tracker_syscall_stack
latency_tracker:latency_tracker_syscall
root@wandboard:/sys/kernel/debug/tracing# cat trace_pipe
^C
root@wandboard:/sys/kernel/debug/tracing#

[jdesfossez]

I see, the rt, wakeup_latency and offcpu trackers require the sched_waking event that is only available starting at 4.3 mainline and 4.1.10.11 PREEMPT_RT.

We could maybe add a workaround to use a kprobe on ttwu (try to wakeup) on older kernels, but I'm not sure if the kprobe support on ARM is good enough on these versions.

[sgkimsm]

Thank you for the answer.

From my testing, kprobe is working fine for simple case even with 3.10.17, so I think it's worth to try. :-)

[jdesfossez]

Ok, I don't have the time right now to work on that, but I can review if you make the patch.
Otherwise, I'll try to do it next week.

[sgkimsm]

Or do you think I can make it to work if I add trace point to old kernel as below?

smp_mb__before_spinlock();
raw_spin_lock_irqsave(&p->pi_lock, flags);
if (!(p->state & state))
	goto out;
trace_sched_waking(p);
success = 1; /* we're going to change ->state */
cpu = task_cpu(p);

[jdesfossez]

Yes you could backport the Linux kernel patch fbd705a0c6184580d0e2fbcbd47a37b6e5822511 to older kernels, it should be easy to do, but it would be nice to have the workaround directly in the latency-tracker to avoid having to patch the kernel to use it.