blktests ublk io uring update #153

yizhanglinux · 2024-12-09T10:31:11Z

No description provided.

kawasaki · 2024-12-11T12:07:18Z

@ming1 FYI.
@yizhanglinux I will take closer look tomorrow. With a quick look, the changes look nice! I guess scsi/008 may need the same change.

ming1 · 2024-12-12T02:22:23Z

Looks fine, nice change!

kawasaki

Yi Zhang, thanks again for this PR.
I made two comments on the patches. Could you revise them and confirm with you environment?

common/rc

common/ublk

block/035 and ublk tests need to run with io_uring eanbled, add two common functions to use for the tests Signed-off-by: Yi Zhang <yi.zhang@redhat.com>

Signed-off-by: Yi Zhang <yi.zhang@redhat.com>

yizhanglinux · 2024-12-13T04:40:38Z

Thanks for the review, updated.

yizhanglinux · 2024-12-13T04:59:03Z

I tried the tests on 6.13-rc2, the tests passed, but I can saw "miniublk[14209]: segfault" during ublk/001, will check it.

[root@rdma-dev-33 blktests ((ce2766b...))]$ uname -r
6.13.0-0.rc2.22.eln144.x86_64
[root@rdma-dev-33 blktests ((ce2766b...))]$ ./check ublk/
ublk/001 (test ublk delete)                                  [passed]
    runtime  5.051s  ...  5.518s
ublk/002 (test ublk crash with delete after dead confirmation) [passed]
    runtime  6.278s  ...  6.280s
ublk/003 (test mounting block device exported by ublk)       [passed]
    runtime  3.836s  ...  3.453s
ublk/004 (test ublk crash with delete just after daemon kill) [passed]
    runtime  4.344s  ...  4.309s
ublk/005 (test ublk recovery with one time daemon kill)      [passed]
    runtime  7.399s  ...  7.411s
ublk/006 (test ublk recovery with two times daemon kill)     [passed]
    runtime  6.362s  ...  6.411s

[ 1704.826052] run blktests ublk/001 at 2024-12-12 23:46:38
[ 1704.837868] miniublk[14209]: segfault at 9c ip 0000000000404481 sp 00007ffd24e091d0 error 6 in miniublk[4481,402000+3000] likely on CPU 44 (core 26, socket 0)
[ 1704.852155] Code: 57 41 89 f7 41 56 41 55 41 54 41 89 fc 53 48 83 ec 48 89 74 24 0c 4c 8d 74 24 10 e8 69 f1 ff ff c5 f9 ef c0 4c 89 f6 48 89 c3 <44> 89 a0 9c 00 00 00 48 8b 05 e1 1b 00 00 c5 fe 7f 44 24 18 48 89
[ 1707.106119] fio: attempt to access beyond end of device
               ublkb0: rw=2048, sector=126120384, nr_sectors = 8 limit=0
[ 1707.106119] fio: attempt to access beyond end of device
               ublkb0: rw=2048, sector=498666656, nr_sectors = 8 limit=0
[ 1710.398697] run blktests ublk/002 at 2024-12-12 23:46:43
[ 1712.509896] blk_print_req_error: 38 callbacks suppressed
[ 1712.509903] I/O error, dev ublkb0, sector 523449176 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1712.510889] I/O error, dev ublkb0, sector 352895736 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1712.515272] I/O error, dev ublkb0, sector 334406264 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1712.524553] I/O error, dev ublkb0, sector 373051904 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1712.533846] I/O error, dev ublkb0, sector 410538792 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1716.731862] run blktests ublk/003 at 2024-12-12 23:46:50
[ 1719.493878] EXT4-fs (ublkb0): mounted filesystem 172144b4-60a3-4014-bd4c-362af74e2e87 r/w with ordered data mode. Quota mode: none.
[ 1719.511509] EXT4-fs (ublkb0): unmounting filesystem 172144b4-60a3-4014-bd4c-362af74e2e87.
[ 1720.237491] run blktests ublk/004 at 2024-12-12 23:46:53
[ 1724.390594] I/O error, dev ublkb0, sector 1539888 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1724.598382] run blktests ublk/005 at 2024-12-12 23:46:58
[ 1726.700212] I/O error, dev ublkb0, sector 55595472 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1726.709420] I/O error, dev ublkb0, sector 3978952 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1726.711209] I/O error, dev ublkb0, sector 92105568 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1726.718532] I/O error, dev ublkb0, sector 507387608 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1726.727721] I/O error, dev ublkb0, sector 266731472 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1726.737004] I/O error, dev ublkb0, sector 504203928 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1726.746281] I/O error, dev ublkb0, sector 378841096 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1726.746287] I/O error, dev ublkb0, sector 209308184 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1726.755567] I/O error, dev ublkb0, sector 418050888 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1730.317311] blk_print_req_error: 32 callbacks suppressed
[ 1730.317318] I/O error, dev ublkb0, sector 2068352 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1730.317321] I/O error, dev ublkb0, sector 701376 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1730.317330] I/O error, dev ublkb0, sector 158336 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1730.322687] I/O error, dev ublkb0, sector 868840 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1730.331792] I/O error, dev ublkb0, sector 2062200 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1730.340815] I/O error, dev ublkb0, sector 1662200 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1730.349826] I/O error, dev ublkb0, sector 704904 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1730.358845] I/O error, dev ublkb0, sector 873768 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1730.367950] I/O error, dev ublkb0, sector 1310568 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1730.377239] I/O error, dev ublkb0, sector 1605608 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1732.060990] run blktests ublk/006 at 2024-12-12 23:47:05
[ 1737.294465] blk_print_req_error: 78 callbacks suppressed
[ 1737.294470] I/O error, dev ublkb0, sector 1952416 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1737.295447] I/O error, dev ublkb0, sector 485976 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1737.299840] I/O error, dev ublkb0, sector 292608 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1737.308933] I/O error, dev ublkb0, sector 1031200 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1737.308940] I/O error, dev ublkb0, sector 1843440 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1737.317960] I/O error, dev ublkb0, sector 1614016 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1737.326975] I/O error, dev ublkb0, sector 704864 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1737.326980] I/O error, dev ublkb0, sector 1906824 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1737.336089] I/O error, dev ublkb0, sector 1569344 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 1737.345188] I/O error, dev ublkb0, sector 7040 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0

yizhanglinux · 2024-12-13T05:27:44Z

So the segfault was from _remove_ublk_devices which calls ./src/miniublk del -a before _io_uring_enable.

$ ./src/miniublk del -a
queue_init: Operation not permitted
Segmentation fault (core dumped)

_init_ublk() {
        _remove_ublk_devices > /dev/null 2>&1

        modprobe -rq ublk_drv
        if _module_file_exists ublk_drv && ! modprobe ublk_drv; then
                return 1
        fi

        udevadm settle

        _io_uring_enable

        return 0
}

Signed-off-by: Yi Zhang <yi.zhang@redhat.com>

yizhanglinux · 2024-12-13T05:55:59Z

I pushed another patch to enable io_uring at the beginning of _init_ublk.

yizhanglinux · 2024-12-13T05:58:18Z

@ming1 Do we need to fix the "[ 1704.837868] miniublk[14209]: segfault" when io_uring disabled?

I tried the tests on 6.13-rc2, the tests passed, but I can saw "miniublk[14209]: segfault" during ublk/001, will check it.

[root@rdma-dev-33 blktests ((ce2766b...))]$ uname -r
6.13.0-0.rc2.22.eln144.x86_64
[root@rdma-dev-33 blktests ((ce2766b...))]$ ./check ublk/
ublk/001 (test ublk delete)                                  [passed]
    runtime  5.051s  ...  5.518s
ublk/002 (test ublk crash with delete after dead confirmation) [passed]
    runtime  6.278s  ...  6.280s
ublk/003 (test mounting block device exported by ublk)       [passed]
    runtime  3.836s  ...  3.453s
ublk/004 (test ublk crash with delete just after daemon kill) [passed]
    runtime  4.344s  ...  4.309s
ublk/005 (test ublk recovery with one time daemon kill)      [passed]
    runtime  7.399s  ...  7.411s
ublk/006 (test ublk recovery with two times daemon kill)     [passed]
    runtime  6.362s  ...  6.411s

[ 1704.826052] run blktests ublk/001 at 2024-12-12 23:46:38
[ 1704.837868] miniublk[14209]: segfault at 9c ip 0000000000404481 sp 00007ffd24e091d0 error 6 in miniublk[4481,402000+3000] likely on CPU 44 (core 26, socket 0)
[ 1704.852155] Code: 57 41 89 f7 41 56 41 55 41 54 41 89 fc 53 48 83 ec 48 89 74 24 0c 4c 8d 74 24 10 e8 69 f1 ff ff c5 f9 ef c0 4c 89 f6 48 89 c3 <44> 89 a0 9c 00 00 00 48 8b 05 e1 1b 00 00 c5 fe 7f 44 24 18 48 89
[ 1707.106119] fio: attempt to access beyond end of device
               ublkb0: rw=2048, sector=126120384, nr_sectors = 8 limit=0
[ 1707.106119] fio: attempt to access beyond end of device
               ublkb0: rw=2048, sector=498666656, nr_sectors = 8 limit=0

ming1 · 2024-12-13T07:38:29Z

Hi @yizhanglinux ,

Please try the following patch:

https://lore.kernel.org/linux-block/20241213073645.2347787-1-ming.lei@redhat.com/

Thanks,

kawasaki · 2024-12-16T03:47:59Z

@yizhanglinux Thanks for updating the patches. They all look good to me.

As for the segmentation fault, the fix by @ming1 looks good to me. But the segmentation fault is not observed on my test system and I can not confirm the fix. Let me wait for your fix confirmation.

yizhanglinux · 2024-12-16T07:48:17Z

With @ming1's patch, when io_uring disabled, ./miniublk del -a still can be Segmentation.
After check with @ming1, it should be from liburing.

[root@storageqe-40 blktests]# cat /proc/sys/kernel/io_uring_disabled
2
[root@storageqe-40 blktests]# ./src/miniublk del -a
queue_init: Operation not permitted
del dev 0 failed
Segmentation fault (core dumped)
[root@storageqe-40 blktests]# dmesg
[ 2519.452124] miniublk[24422]: segfault at 251c ip 00000000004045c7 sp 00007fffeec3b4c0 error 4 in miniublk[45c7,402000+3000] likely on CPU 13 (core 9, socket 0)
[ 2519.466403] Code: 31 ed c7 44 24 34 00 00 00 00 c5 f9 ef c0 4c 89 f6 48 89 df c7 44 24 10 05 00 00 00 c5 fe 7f 44 24 14 c5 f8 77 e8 39 fa ff ff <8b> bb 1c 25 00 00 e8 6e db ff ff 48 89 df e8 66 da ff ff 48 83 c4

[root@storageqe-40 blktests]# journalctl -r
Dec 16 02:44:38 storageqe-40.fast.eng.rdu2.dc.redhat.com systemd[1]: systemd-coredump@61-24511-0.service: Deactivated successfully.
Dec 16 02:44:38 storageqe-40.fast.eng.rdu2.dc.redhat.com systemd-coredump[24512]: [🡕] Process 24510 (miniublk) of user 0 dumped core.

                                                                                  Module liburing.so.2 from rpm liburing-2.5-5.el10.x86_64
                                                                                  Stack trace of thread 24510:
                                                                                  #0  0x00000000004045c7 n/a (n/a + 0x0)
                                                                                  #1  0x000000000040266e n/a (n/a + 0x0)
                                                                                  #2  0x00007f398f64d30e __libc_start_call_main (libc.so.6 + 0x2a30e)
                                                                                  #3  0x00007f398f64d3c9 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a3c9)
                                                                                  #4  0x0000000000402a65 n/a (n/a + 0x0)
                                                                                  ELF object binary architecture: AMD x86-64

yizhanglinux · 2024-12-16T07:52:49Z

File axboe/liburing#1305 to track the liburing issue.

block/035 and ublk tests need to run with io_uring enabled, add two common functions to use for the tests Link: #153 Signed-off-by: Yi Zhang <yi.zhang@redhat.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

Link: #153 Signed-off-by: Yi Zhang <yi.zhang@redhat.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

Call the common functions _io_uring_enable/_io_uring_restore to ensure that io_uring is enabled during ublk test runs. Link: #153 Signed-off-by: Yi Zhang <yi.zhang@redhat.com> [Shin'ichiro: squashed the following fix into this commit] Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

kawasaki · 2024-12-20T10:32:43Z

I took the liberty to apply the patches as separated commits, since I wanted to squash the 4th fix patch into the 3rd patch. Anyway, the fixes @yizhanglinux suggested are now upstreamed. Thanks!

kawasaki requested changes Dec 12, 2024

View reviewed changes

common/rc Outdated Show resolved Hide resolved

common/ublk Outdated Show resolved Hide resolved

yizhanglinux added 3 commits December 12, 2024 23:23

common/rc: introduce _io_uring_enable/_io_uring_restore

52151d0

block/035 and ublk tests need to run with io_uring eanbled, add two common functions to use for the tests Signed-off-by: Yi Zhang <yi.zhang@redhat.com>

block/035: use the common functions _io_uring_enable/_io_uring_restore

cdea0ac

Signed-off-by: Yi Zhang <yi.zhang@redhat.com>

common/ublk: update io_uring setting during the tests

ce2766b

Signed-off-by: Yi Zhang <yi.zhang@redhat.com>

yizhanglinux force-pushed the blktests-ublk-io-uring-update branch from f002694 to ce2766b Compare December 13, 2024 04:37

common/ublk: move _io_uring_enable at the beginning of _init_ublk

f663dad

Signed-off-by: Yi Zhang <yi.zhang@redhat.com>

kawasaki pushed a commit that referenced this pull request Dec 20, 2024

block/035: use the common functions _io_uring_enable/_io_uring_restore

b19924b

Link: #153 Signed-off-by: Yi Zhang <yi.zhang@redhat.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

kawasaki closed this Dec 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

blktests ublk io uring update #153

blktests ublk io uring update #153

yizhanglinux commented Dec 9, 2024

kawasaki commented Dec 11, 2024

ming1 commented Dec 12, 2024

kawasaki left a comment

yizhanglinux commented Dec 13, 2024

yizhanglinux commented Dec 13, 2024

yizhanglinux commented Dec 13, 2024 •

edited

Loading

yizhanglinux commented Dec 13, 2024

yizhanglinux commented Dec 13, 2024

ming1 commented Dec 13, 2024

kawasaki commented Dec 16, 2024

yizhanglinux commented Dec 16, 2024

yizhanglinux commented Dec 16, 2024

kawasaki commented Dec 20, 2024

blktests ublk io uring update #153

blktests ublk io uring update #153

Conversation

yizhanglinux commented Dec 9, 2024

kawasaki commented Dec 11, 2024

ming1 commented Dec 12, 2024

kawasaki left a comment

Choose a reason for hiding this comment

yizhanglinux commented Dec 13, 2024

yizhanglinux commented Dec 13, 2024

yizhanglinux commented Dec 13, 2024 • edited Loading

yizhanglinux commented Dec 13, 2024

yizhanglinux commented Dec 13, 2024

ming1 commented Dec 13, 2024

kawasaki commented Dec 16, 2024

yizhanglinux commented Dec 16, 2024

yizhanglinux commented Dec 16, 2024

kawasaki commented Dec 20, 2024

yizhanglinux commented Dec 13, 2024 •

edited

Loading