[PW_SID:1028324] riscv: mm: Introduce lazy tlb flush #1164
Conversation
This feature avoids unnecessary TLB Flush IPIs. After memory mapping modifications on certain mm_struct, instead of sending IPIs, this feature records the TLB Flush information on percpu buffer, defer the TLB Flush to the moment when target CPUs really load this mm_struct. Signed-off-by: Xu Lu <[email protected]> Signed-off-by: Linux RISC-V bot <[email protected]>
Since each CPU has limited TLB entries, there exist limited active ASIDs in each CPU's TLB at the same time. Thus we apply a threshold here. When a mm_struct is loaded, we mark its ASID as active. If the number of active ASIDs exceeds the threshold, we evict the mm_struct that has not been used for the longest time, flush its TLB entries, mark its ASID inactive, and clear current CPU in its mm_cpumask. Signed-off-by: Xu Lu <[email protected]> Signed-off-by: Linux RISC-V bot <[email protected]>
We maintain an array of mm_structs whose ASIDs are active on the current CPU. To avoid these mm_structs getting released, we grab their mm_count before loaded them into the array. And drop their mm_count via tasklet when they are evicted out of the array. Signed-off-by: Xu Lu <[email protected]> Signed-off-by: Linux RISC-V bot <[email protected]>
When both CONFIG_RISCV_LAZY_TLB_FLUSH and CONFIG_MMU_LSYZ_TLB_SHOOTDOWN is enabled, riscv needs an arch special method to free the mm that needs to be shot down. Thus we add arch override for do_shoot_lazy_tlb(). Signed-off-by: Xu Lu <[email protected]> Signed-off-by: Linux RISC-V bot <[email protected]>
When an active_mm is shot down, we switch it to the init_mm, evict it out of percpu active mm array. Signed-off-by: Xu Lu <[email protected]> Signed-off-by: Linux RISC-V bot <[email protected]>
When memory mapping of a mm is modified, instead of sending IPI to all CPUs recorded in its mm_cpumask, we check whether each target CPU is using this mm right now. If not, we just store the TLB Flush information in target CPU's percpu TLB Flush queue, avoiding the IPI. Signed-off-by: Xu Lu <[email protected]> Signed-off-by: Linux RISC-V bot <[email protected]>
Since the TLB Flush IPI is avoided when the modified mm is not running on the target CPU, next time when target CPU switches to the modified mm, it has to check percpu TLB Flush queue and perform TLB FLush for the midified mm. Signed-off-by: Xu Lu <[email protected]> Signed-off-by: Linux RISC-V bot <[email protected]>
When local_flush_tlb_all_asid() is called, the target mm's TLB entries are all flushed out, then we can clear current CPU in its mm_cpumask so that next time the mm's memory mapping is modified, no IPI will be sent to current CPU. Signed-off-by: Xu Lu <[email protected]> Signed-off-by: Linux RISC-V bot <[email protected]>
Now that we maintain an array of active mms on each CPU, when local_flush_tlb_all() is called, we can clear current CPU in the mm_cpumask of all active mms on current CPU. Signed-off-by: Xu Lu <[email protected]> Signed-off-by: Linux RISC-V bot <[email protected]>
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 1: "[RFC,v2,1/9] riscv: Introduce RISCV_LAZY_TLB_FLUSH config" |
|
Patch 2: "[RFC,v2,2/9] riscv: mm: Apply a threshold to the number of active ASIDs on each CPU" |
|
Patch 2: "[RFC,v2,2/9] riscv: mm: Apply a threshold to the number of active ASIDs on each CPU" |
|
Patch 2: "[RFC,v2,2/9] riscv: mm: Apply a threshold to the number of active ASIDs on each CPU" |
|
Patch 2: "[RFC,v2,2/9] riscv: mm: Apply a threshold to the number of active ASIDs on each CPU" |
|
Patch 2: "[RFC,v2,2/9] riscv: mm: Apply a threshold to the number of active ASIDs on each CPU" |
|
Patch 2: "[RFC,v2,2/9] riscv: mm: Apply a threshold to the number of active ASIDs on each CPU" |
|
Patch 2: "[RFC,v2,2/9] riscv: mm: Apply a threshold to the number of active ASIDs on each CPU" |
|
Patch 2: "[RFC,v2,2/9] riscv: mm: Apply a threshold to the number of active ASIDs on each CPU" |
|
Patch 2: "[RFC,v2,2/9] riscv: mm: Apply a threshold to the number of active ASIDs on each CPU" |
|
Patch 7: "[RFC,v2,7/9] riscv: mm: Defer the TLB Flush to switch_mm" |
|
Patch 7: "[RFC,v2,7/9] riscv: mm: Defer the TLB Flush to switch_mm" |
|
Patch 7: "[RFC,v2,7/9] riscv: mm: Defer the TLB Flush to switch_mm" |
|
Patch 7: "[RFC,v2,7/9] riscv: mm: Defer the TLB Flush to switch_mm" |
|
Patch 7: "[RFC,v2,7/9] riscv: mm: Defer the TLB Flush to switch_mm" |
|
Patch 7: "[RFC,v2,7/9] riscv: mm: Defer the TLB Flush to switch_mm" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 8: "[RFC,v2,8/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all_asid()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
|
Patch 9: "[RFC,v2,9/9] riscv: mm: Clear mm_cpumask during local_flush_tlb_all()" |
PR for series 1028324 applied to workflow__riscv__fixes
Name: riscv: mm: Introduce lazy tlb flush
URL: https://patchwork.kernel.org/project/linux-riscv/list/?series=1028324
Version: 2