Pwx 48389 main by sreymondjohn · Pull Request #409 · portworx/px-fuse

sreymondjohn · 2025-11-19T18:17:18Z

commit 1a3a8297447a4e495cdee325683bfccea8a9e5b7 (HEAD -> PWX-48389, origin/PWX-48389)
Author: Sebas <sreymondjohnson@purestorage.com>
Date:   Thu Nov 6 00:48:23 2025 +0000

    PWX-48389: avoid acquiring spinlock while calling blk_mq_quiesce_queue

[73739.625809] pxd: blk-mq driver loaded version v3.5.0:1a3a8297447a4e495cdee325683bfccea8a9e5b7, features 0x7

Create 10 local fastpath volumes

for i in $(seq 1 1 10)
> do
> pxctl v c -s 10 -r 1 --fastpath=true --nodes=LocalNode v$i
> done
Volume successfully created: 1085811283747021855
Volume successfully created: 148278043961889093
Volume successfully created: 1032853105029774008
Volume successfully created: 972521549642283381
Volume successfully created: 986766375919628484
Volume successfully created: 393728088692654157
Volume successfully created: 106329257484681621
Volume successfully created: 456598983547578251
Volume successfully created: 123094058532683999
Volume successfully created: 933643067508159751

for i in $(seq 1 1 10)
> do
> pxctl host attach v$i
> done
Volume successfully attached at: /dev/pxd/pxd1085811283747021855
Volume successfully attached at: /dev/pxd/pxd148278043961889093
Volume successfully attached at: /dev/pxd/pxd1032853105029774008
Volume successfully attached at: /dev/pxd/pxd972521549642283381
Volume successfully attached at: /dev/pxd/pxd986766375919628484
Volume successfully attached at: /dev/pxd/pxd393728088692654157
Volume successfully attached at: /dev/pxd/pxd106329257484681621
Volume successfully attached at: /dev/pxd/pxd456598983547578251
Volume successfully attached at: /dev/pxd/pxd123094058532683999
Volume successfully attached at: /dev/pxd/pxd933643067508159751

for i in $(seq 1 1 10); do pxctl v i v$i | grep -i attached; done
	State           	 :  Attached: feb0166b-1a78-42f7-aac8-e7bd6c0b1c5c (10.13.168.150) (fastpath active)
	Last Attached   	 :  Nov 6 01:15:48 UTC 2025
	State           	 :  Attached: feb0166b-1a78-42f7-aac8-e7bd6c0b1c5c (10.13.168.150) (fastpath active)
	Last Attached   	 :  Nov 6 01:15:48 UTC 2025
	State           	 :  Attached: feb0166b-1a78-42f7-aac8-e7bd6c0b1c5c (10.13.168.150) (fastpath active)
	Last Attached   	 :  Nov 6 01:15:49 UTC 2025
	State           	 :  Attached: feb0166b-1a78-42f7-aac8-e7bd6c0b1c5c (10.13.168.150) (fastpath active)
	Last Attached   	 :  Nov 6 01:15:49 UTC 2025
	State           	 :  Attached: feb0166b-1a78-42f7-aac8-e7bd6c0b1c5c (10.13.168.150) (fastpath active)
	Last Attached   	 :  Nov 6 01:15:50 UTC 2025
	State           	 :  Attached: feb0166b-1a78-42f7-aac8-e7bd6c0b1c5c (10.13.168.150) (fastpath active)
	Last Attached   	 :  Nov 6 01:15:50 UTC 2025
	State           	 :  Attached: feb0166b-1a78-42f7-aac8-e7bd6c0b1c5c (10.13.168.150) (fastpath active)
	Last Attached   	 :  Nov 6 01:15:50 UTC 2025
	State           	 :  Attached: feb0166b-1a78-42f7-aac8-e7bd6c0b1c5c (10.13.168.150) (fastpath active)
	Last Attached   	 :  Nov 6 01:15:50 UTC 2025
	State           	 :  Attached: feb0166b-1a78-42f7-aac8-e7bd6c0b1c5c (10.13.168.150) (fastpath active)
	Last Attached   	 :  Nov 6 01:15:52 UTC 2025
	State           	 :  Attached: feb0166b-1a78-42f7-aac8-e7bd6c0b1c5c (10.13.168.150) (fastpath active)
	Last Attached   	 :  Nov 6 01:15:52 UTC 2025

change timeout to 1 min instead of 10 mins

echo 60 > /sys/devices/pxd/1/timeout

2GB randwrite on all the 10 fastpath devices in parallel and also after sometime stop portworx on the node

for i in $(seq 1 10); do pxctl v i v$i | awk '/Device Path/ {print $NF}'; done | xargs -n1 -P10 -I{} fio   --name="randw_$(basename {})"   --filename="{}"   --rw=randwrite --bs=4k --size=2G   --ioengine=libaio --iodepth=8 --direct=1   --group_reporting=1
randw_/dev/pxd/pxd1085811283747021855: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.16
randw_/dev/pxd/pxd148278043961889093: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.16
randw_/dev/pxd/pxd1032853105029774008: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.16
Starting 1 process
Starting 1 process
Starting 1 process
randw_/dev/pxd/pxd972521549642283381: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.16
Starting 1 process
randw_/dev/pxd/pxd986766375919628484: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.16
randw_/dev/pxd/pxd393728088692654157: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.16
Starting 1 process
Starting 1 process
randw_/dev/pxd/pxd106329257484681621: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.16
Starting 1 process
randw_/dev/pxd/pxd456598983547578251: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.16
Starting 1 process
randw_/dev/pxd/pxd123094058532683999: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.16
Starting 1 process
randw_/dev/pxd/pxd933643067508159751: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][4.1%][w=9809KiB/s][w=2452 IOPS][eta 03m:55s]

... after PX stopped and 60 seconds has expired, return IO error
randw_/dev/pxd/pxd1032853105029774008: (groupid=0, jobs=1): err= 5 (file:io_u.c:1787, func=io_u error, error=Input/output error): pid=1671622: Thu Nov  6 01:30:27 2025
  write: IOPS=6668, BW=26.0MiB/s (27.3MB/s)(1962MiB/75309msec); 0 zone resets
    slat (usec): min=2, max=13775, avg=18.23, stdev=60.27
    clat (usec): min=20, max=37056, avg=1177.57, stdev=1365.91
     lat (usec): min=233, max=37065, avg=1196.36, stdev=1366.65
    clat percentiles (usec):
     |  1.00th=[  371],  5.00th=[  441], 10.00th=[  486], 20.00th=[  562],
     | 30.00th=[  627], 40.00th=[  709], 50.00th=[  816], 60.00th=[  963],
     | 70.00th=[ 1156], 80.00th=[ 1401], 90.00th=[ 1860], 95.00th=[ 2606],
     | 99.00th=[ 8225], 99.50th=[ 9765], 99.90th=[14091], 99.95th=[16188],
     | 99.99th=[26084]
   bw (  KiB/s): min= 6688, max=36360, per=99.95%, avg=26660.43, stdev=8453.93, samples=150
   iops        : min= 1672, max= 9090, avg=6665.10, stdev=2113.48, samples=150
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.02%, 500=11.69%, 750=32.28%
  lat (usec)   : 1000=17.95%
  lat (msec)   : 2=29.56%, 4=5.54%, 10=2.49%, 20=0.43%, 50=0.03%
  cpu          : usr=3.12%, sys=9.49%, ctx=491647, majf=0, minf=24
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,502221,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=26.0MiB/s (27.3MB/s), 26.0MiB/s-26.0MiB/s (27.3MB/s-27.3MB/s), io=1962MiB (2057MB), run=75309-75309msec

Disk stats (read/write):
  pxd!pxd1032853105029774008: ios=48/502221, merge=0/0, ticks=23/557019, in_queue=80824, util=99.61%

randw_/dev/pxd/pxd393728088692654157: (groupid=0, jobs=1): err= 5 (file:io_u.c:1787, func=io_u error, error=Input/output error): pid=1671675: Thu Nov  6 01:30:27 2025
  write: IOPS=6657, BW=26.0MiB/s (27.3MB/s)(1945MiB/74785msec); 0 zone resets
    slat (nsec): min=1949, max=35210k, avg=15915.72, stdev=74415.07
    clat (usec): min=2, max=36536, avg=1182.60, stdev=1392.87
     lat (usec): min=240, max=36544, avg=1198.95, stdev=1394.15
    clat percentiles (usec):
     |  1.00th=[  371],  5.00th=[  441], 10.00th=[  486], 20.00th=[  553],
     | 30.00th=[  627], 40.00th=[  709], 50.00th=[  816], 60.00th=[  963],
     | 70.00th=[ 1156], 80.00th=[ 1401], 90.00th=[ 1860], 95.00th=[ 2638],
     | 99.00th=[ 8225], 99.50th=[ 9896], 99.90th=[14353], 99.95th=[17695],
     | 99.99th=[26084]
   bw (  KiB/s): min= 5800, max=37232, per=99.92%, avg=26609.02, stdev=8516.27, samples=149
   iops        : min= 1450, max= 9308, avg=6652.22, stdev=2129.08, samples=149
  lat (usec)   : 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=11.87%, 750=32.16%, 1000=17.79%
  lat (msec)   : 2=29.71%, 4=5.40%, 10=2.58%, 20=0.43%, 50=0.04%
  cpu          : usr=3.67%, sys=9.45%, ctx=387225, majf=0, minf=24
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,497874,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

dmesg output

[Nov 6 01:30] PXD_TIMEOUT (pxd/pxd-control-0:0): Aborting all requests...
[  +0.000088] blk_update_request: I/O error, dev pxd/pxd1032853105029774008, sector 743984 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[  +0.000044] blk_update_request: I/O error, dev pxd/pxd933643067508159751, sector 45888 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[  +0.000004] blk_update_request: I/O error, dev pxd/pxd986766375919628484, sector 4099712 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[  +0.000054] blk_update_request: I/O error, dev pxd/pxd986766375919628484, sector 1479520 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[  +0.000010] blk_update_request: I/O error, dev pxd/pxd393728088692654157, sector 290968 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[  +0.000002] blk_update_request: I/O error, dev pxd/pxd986766375919628484, sector 4168704 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[  +0.000008] blk_update_request: I/O error, dev pxd/pxd986766375919628484, sector 1143584 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[  +0.000007] blk_update_request: I/O error, dev pxd/pxd393728088692654157, sector 74928 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[  +0.000002] blk_update_request: I/O error, dev pxd/pxd986766375919628484, sector 1891048 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[  +0.000009] blk_update_request: I/O error, dev pxd/pxd393728088692654157, sector 4163440 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0

UT run

./test/pxd_test --gtest_filter=*Abort*
Note: Google Test filter = *Abort*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from PxdTest
[ RUN      ] PxdTest.AbortContextWithInFlightIO
SetUp
Opening control dev: /dev/pxd/pxd-control
dev_export: PXD_EXPORT completed, wrote 24 bytes
dev_export: device ID = 1
dev_export: expected device path = /dev/pxd/pxd1
dev_export: device file appeared after 1 seconds
validating pxd device: /dev/pxd/pxd1
reading sysfs path: /sys/block/pxd!pxd1/queue/minimum_io_size
reading sysfs path: /sys/block/pxd!pxd1/queue/optimal_io_size
reading sysfs path: /sys/block/pxd!pxd1/queue/logical_block_size
reading sysfs path: /sys/block/pxd!pxd1/queue/physical_block_size
reading sysfs path: /sys/block/pxd!pxd1/queue/max_segments
reading sysfs path: /sys/block/pxd!pxd1/queue/max_segment_size
reading sysfs path: /sys/block/pxd!pxd1/queue/discard_granularity
reading sysfs path: /sys/block/pxd!pxd1/queue/discard_max_bytes
reading sysfs path: /sys/block/pxd!pxd1/queue/max_discard_segments
reading sysfs path: /sys/block/pxd!pxd1/queue/nr_requests
reading sysfs path: /sys/block/pxd!pxd1/queue/read_ahead_kb
reading sysfs path: /sys/block/pxd!pxd1/queue/fua
FUA validation passed
minor = 1
opcode : 8194, offset: 4096, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
write thread: waiting for attach IO to complete for 10 seconds
opcode : 8194, offset: 0, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 16384, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 32768, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 65536, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 131072, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 262144, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 524288, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 12288, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 28672, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 61440, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 8192, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
opcode : 8194, offset: 1044480, len = 4096
finish_io: reply to kernel: status: 0 iovcnt: 1
about to write : write_len = 16384 test_off = 32768, name = /dev/pxd/pxd1
found the test write: offset: 32768, len = 16384
write_thread_timeout: bytes written: -1 (failure expected) write_len = 16384 test_off = 16384
Test: Wait complete.
dev_remove: device removing 1
cleaner thread active
initiating dev cleanup
device removal success
dev_remove: device 1 removed after 0 secs
prepping to stop background cleaner
cleaner thread done
TearDown
took 1 seconds to perform rmmod
[       OK ] PxdTest.AbortContextWithInFlightIO (48195 ms)
[----------] 1 test from PxdTest (48195 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (48195 ms total)
[  PASSED  ] 1 test.

For the scenario where kcalloc can fail, tested the same workflow as above with snap_list as NULL explicitly.

What this PR does / why we need it:
Described in detail here : https://purestorage.atlassian.net/browse/PWX-48389?focusedCommentId=768923

Which issue(s) this PR fixes (optional)
Closes #

Special notes for your reviewer:

Sebas added 2 commits November 19, 2025 18:12

PWX-48389: avoid acquiring spinlock while calling blk_mq_quiesce_queue

0ac67bd

address review comments

00bf080

sreymondjohn requested review from kagrwl2, ppadalia, sulakshm and talakad November 19, 2025 18:17

talakad approved these changes Nov 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pwx 48389 main#409

Pwx 48389 main#409
sreymondjohn wants to merge 2 commits intomainfrom
PWX-48389_main

sreymondjohn commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sreymondjohn commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants