GEOM_ELI: Crypto READ request failed (error=22)

mr44er

moderater Moderator
Teammitglied
GEOM_ELI: Crypto READ request failed (error=22). da3.eli[READ(offset=62105124864, length=28672)]
GEOM_ELI: Crypto READ request failed (error=22). da3.eli[READ(offset=62133731328, length=28672)]
GEOM_ELI: Crypto READ request failed (error=22). da2.eli[READ(offset=62173143040, length=28672)]
GEOM_ELI: Crypto READ request failed (error=22). da1.eli[READ(offset=62253531136, length=28672)]
GEOM_ELI: Crypto READ request failed (error=22). da6.eli[READ(offset=62276472832, length=28672)]
GEOM_ELI: Crypto READ request failed (error=22). da4.eli[READ(offset=62323597312, length=28672)]
GEOM_ELI: Crypto READ request failed (error=22). da5.eli[READ(offset=62374088704, length=28672)]
GEOM_ELI: Crypto READ request failed (error=22). da3.eli[READ(offset=62375321600, length=28672)]
uswusf.

Während eines resilverns floodet mir dmesg damit zu und zwar auf allen Platten. Dementsprechend steigen auch die READs in meinem Pool, währenddessen aber die ECC-Korrekturen der SAS-Platten nicht ansteigen.

Muss also an geli liegen. Kann man da was tun?

Code:
pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Jun 19 14:00:04 2018
        341G scanned out of 6.63T at 87.3M/s, 21h2m to go
        42.7G resilvered, 5.02% done
config:

    NAME                       STATE     READ WRITE CKSUM
    tank                       DEGRADED     0     0     0
      raidz3-0                 DEGRADED     0     0     0
        replacing-0            REMOVED      0     0     0
          2255941816842510515  REMOVED      0     0     0  was /dev/label/K5GWGJVA-hd1
          label/YFJJM1YD-hd1   ONLINE       0     0     0  (resilvering)
        label/K5GWGN5A-hd2     ONLINE      44     0     0  (resilvering)
        label/K5GW31WA-hd3     ONLINE      53     0     0  (resilvering)
        label/N4G2N9AY-hd4     ONLINE      44     0     0  (resilvering)
        label/K5GWAZTA-hd5     ONLINE     104     0     0  (resilvering)
        label/N4G1XV1K-hd6     ONLINE      55     0     0  (resilvering)
        label/K5GW6SGA-hd7     ONLINE      47     0     0  (resilvering)
        label/K5GWGL2A-hd8     ONLINE      55     0     0  (resilvering)
 
Mhhhhhm...mysteriös. Nach einem manuellen Reboot hat er nen kerneldump gemacht (funktioniert wohl nun *g*) und nach dem zweiten autoreboot funktionierts resilvern einwandfrei und die Fehlermeldungen GEOM_ELI: Crypto READ request failed (error=22) sind weg.

Code:
smain dumped core - see /var/crash/vmcore.1

Wed Jun 20 13:33:59 CEST 2018

FreeBSD smain 11.1-RELEASE-p10 FreeBSD 11.1-RELEASE-p10 #0 r334297: Tue May 29 00:28:19 CEST 2018     mr44er@smain:/usr/obj/usr/src/sys/SMAINKERNEL  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 14
fault virtual address    = 0x800e25000
fault code        = supervisor write data, page not present
instruction pointer    = 0x20:0xffffffff80ed570f
stack pointer            = 0x28:0xfffffe0861c40740
frame pointer            = 0x28:0xfffffe0861c40740
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 879 (logger)
trap number        = 12
panic: page fault
cpuid = 4
KDB: stack backtrace:
#0 0xffffffff80aa0897 at kdb_backtrace+0x67
#1 0xffffffff80a5ee36 at vpanic+0x186
#2 0xffffffff80a5eca3 at panic+0x43
#3 0xffffffff80ed7c12 at trap_fatal+0x322
#4 0xffffffff80ed7c6b at trap_pfault+0x4b
#5 0xffffffff80ed73c8 at trap+0x2a8
#6 0xffffffff80eb8810 at calltrap+0x8
#7 0xffffffff80ab61db at uiomove_faultflag+0x10b
#8 0xffffffff80abe211 at pipe_read+0x3b1
#9 0xffffffff80abb1ba at dofileread+0xba
#10 0xffffffff80abacbb at sys_read+0xdb
#11 0xffffffff80ed87f4 at amd64_syscall+0x6c4
#12 0xffffffff80eb9042 at fast_syscall_common+0x106
Uptime: 49s
Dumping 1295 out of 32714 MB:..2%..12%..22%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
done.
Loaded symbols for /boot/kernel/geom_eli.ko
Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_ether.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_ether.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /usr/lib/debug//boot/kernel/netgraph.ko.debug...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /usr/lib/debug//boot/kernel/accf_http.ko.debug...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/amdtemp.ko...Reading symbols from /usr/lib/debug//boot/kernel/amdtemp.ko.debug...done.
done.
Loaded symbols for /boot/kernel/amdtemp.ko
Reading symbols from /boot/kernel/aesni.ko...Reading symbols from /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
done.
Loaded symbols for /boot/kernel/aesni.ko
Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /usr/lib/debug//boot/kernel/ipmi.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ipmi.ko
Reading symbols from /boot/kernel/smbus.ko...Reading symbols from /usr/lib/debug//boot/kernel/smbus.ko.debug...done.
done.
Loaded symbols for /boot/kernel/smbus.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /usr/lib/debug//boot/kernel/pflog.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /usr/lib/debug//boot/kernel/pf.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/kernel/cpuctl.ko...Reading symbols from /usr/lib/debug//boot/kernel/cpuctl.ko.debug...done.
done.
Loaded symbols for /boot/kernel/cpuctl.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:229
229    pcpu.h: No such file or directory.
    in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:229
#1  0xffffffff80a5e9b1 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80a5ee70 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80a5eca3 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff80ed7c12 in trap_fatal (frame=0xfffffe0861c40680, 
    eva=34374569984) at /usr/src/sys/amd64/amd64/trap.c:878
#5  0xffffffff80ed7c6b in trap_pfault (frame=0xfffffe0861c40680, usermode=0)
    at pcpu.h:229
#6  0xffffffff80ed73c8 in trap (frame=0xfffffe0861c40680)
    at /usr/src/sys/amd64/amd64/trap.c:422
#7  0xffffffff80eb8810 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:231
#8  0xffffffff80ed570f in copyout ()
#9  0xffffffff80ab61db in uiomove_faultflag (cp=<value optimized out>, n=105, 
    uio=0xfffffe0861c40840, nofault=<value optimized out>)
    at /usr/src/sys/kern/subr_uio.c:251
#10 0xffffffff80abe211 in pipe_read (fp=0xfffff800381915f0, 
    uio=0xfffffe0861c40840, active_cred=<value optimized out>, 
    flags=<value optimized out>, td=<value optimized out>)
    at /usr/src/sys/kern/sys_pipe.c:705
#11 0xffffffff80abb1ba in dofileread (td=<value optimized out>, 
    fd=<value optimized out>, fp=<value optimized out>, 
    auio=0xfffffe0861c40840, offset=<value optimized out>, 
    flags=<value optimized out>) at file.h:299
#12 0xffffffff80abacbb in sys_read (td=0xfffff800295ad560, 
    uap=<value optimized out>) at /usr/src/sys/kern/sys_generic.c:291
#13 0xffffffff80ed87f4 in amd64_syscall (td=0xfffff800295ad560, traced=0)
    at subr_syscall.c:135
#14 0xffffffff80eb9042 in fast_syscall_common ()
    at /usr/src/sys/amd64/amd64/exception.S:464
#15 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb) 

------------------------------------------------------------------------
ps -axlww

UID PID PPID CPU PRI NI   VSZ  RSS MWCHAN   STAT TT     TIME COMMAND
  0   0    0   0 -16  0     0    0 swapin   DLs   -  0:01.30 [kernel]
  0   1    0   0  20  0  5384  432 wait     DLs   -  0:00.11 [init]
  0   2    0   0 -16  0     0    0 crypto_w DL    -  0:00.00 [crypto]
  0   3    0   0 -16  0     0    0 crypto_r DL    -  0:00.00 [crypto returns]
  0   4    0   0 -16  0     0    0 -        RL    -  0:00.02 [cam]
  0   5    0   0 -16  0     0    0 -        DL    -  0:00.00 [soaiod1]
  0   6    0   0 -16  0     0    0 -        DL    -  0:00.00 [soaiod2]
  0   7    0   0 -16  0     0    0 -        DL    -  0:00.00 [soaiod3]
  0   8    0   0 -16  0     0    0 -        DL    -  0:00.00 [soaiod4]
  0   9    0   0  -8  0     0    0 arc_recl DL    -  0:00.05 [zfskern]
  0  10    0   0 -16  0     0    0 audit_wo DL    -  0:00.00 [audit]
  0  11    0   0 155  0     0    0 -        RL    -  4:43.22 [idle]
  0  12    0   0 -64  0     0    0 -        WL    -  0:01.66 [intr]
  0  13    0   0 -16  0     0    0 sleep    DL    -  0:00.00 [ng_queue]
  0  14    0   0  -8  0     0    0 -        DL    -  0:26.79 [geom]
  0  15    0   0 -68  0     0    0 -        DL    -  0:00.00 [usb]
  0  16    0   0 -16  0     0    0 waiting_ DL    -  0:00.00 [sctp_iterator]
  0  17    0   0 -16  0     0    0 -        DL    -  0:00.00 [rand_harvestq]
  0  18    0   0 -16  0     0    0 ipmireq  DL    -  0:00.00 [ipmi0: kcs]
  0  19    0   0 -16  0     0    0 psleep   DL    -  0:00.00 [pagedaemon]
  0  20    0   0 -16  0     0    0 psleep   DL    -  0:00.00 [vmdaemon]
  0  21    0   0 155  0     0    0 pgzero   DL    -  0:00.00 [pagezero]
  0  22    0   0 -16  0     0    0 -        DL    -  0:00.00 [bufspacedaemon]
  0  23    0   0 -16  0     0    0 psleep   DL    -  0:00.00 [bufdaemon]
  0  24    0   0 -16  0     0    0 vlruwt   DL    -  0:00.00 [vnlru]
  0  25    0   0  16  0     0    0 syncer   DL    -  0:00.00 [syncer]
  0  26    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[0] ada0p3]
  0  27    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[1] ada0p3]
  0  28    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[2] ada0p3]
  0  29    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[3] ada0p3]
  0  30    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[4] ada0p3]
  0  31    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[5] ada0p3]
  0  32    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[6] ada0p3]
  0  33    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[7] ada0p3]
  0  34    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[0] da0]
  0  35    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[1] da0]
  0  36    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[2] da0]
  0  37    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[3] da0]
  0  38    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[4] da0]
  0  39    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[5] da0]
  0  40    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[6] da0]
  0  41    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[7] da0]
  0  42    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[0] da1]
  0  43    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[1] da1]
  0  44    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[2] da1]
  0  45    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[3] da1]
  0  46    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[4] da1]
  0  47    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[5] da1]
  0  48    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[6] da1]
  0  49    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[7] da1]
  0  50    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[0] da2]
  0  51    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[1] da2]
  0  52    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[2] da2]
  0  53    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[3] da2]
  0  54    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[4] da2]
  0  55    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[5] da2]
  0  56    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[6] da2]
  0  57    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[7] da2]
  0  58    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[0] da3]
  0  59    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[1] da3]
  0  60    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[2] da3]
  0  61    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[3] da3]
  0  62    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[4] da3]
  0  63    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[5] da3]
  0  64    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[6] da3]
  0  65    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[7] da3]
  0  66    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[0] da4]
  0  67    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[1] da4]
  0  68    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[2] da4]
  0  69    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[3] da4]
  0  70    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[4] da4]
  0  71    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[5] da4]
  0  72    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[6] da4]
  0  73    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[7] da4]
  0  74    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[0] da5]
  0  75    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[1] da5]
  0  76    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[2] da5]
  0  77    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[3] da5]
  0  78    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[4] da5]
  0  79    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[5] da5]
  0  80    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[6] da5]
  0  81    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[7] da5]
  0  82    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[0] da6]
  0  83    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[1] da6]
  0  84    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[2] da6]
  0  85    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[3] da6]
  0  86    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[4] da6]
  0  87    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[5] da6]
  0  88    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[6] da6]
  0  89    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[7] da6]
  0  90    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[0] da7]
  0  91    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[1] da7]
  0  92    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[2] da7]
  0  93    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[3] da7]
  0  94    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[4] da7]
  0  95    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[5] da7]
  0  96    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[6] da7]
  0  97    0   0  20  0     0    0 geli:w   DL    -  0:00.00 [g_eli[7] da7]
  0  98    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[0] ada1p3]
  0  99    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[1] ada1p3]
  0 100    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[2] ada1p3]
  0 101    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[3] ada1p3]
  0 102    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[4] ada1p3]
  0 103    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[5] ada1p3]
  0 104    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[6] ada1p3]
  0 105    0   0  20  0     0    0 geli:w   DL    -  0:00.01 [g_eli[7] ada1p3]
  0 106    1   0  52  0 13132 1756 wait     Ds+   -  0:00.02 [sh]
  0 672    1   0  52  0 12672 1476 select   Ds    -  0:00.00 [moused]
  0 689    1   0  52  0  9540 4764 select   Ds    -  0:00.00 [devd]
  0 696    0   0 -16  0     0    0 pftm     DL    -  0:00.00 [pf purge]
  0 708    1   0  52  0 10684 1816 sbwait   Ds    -  0:00.00 [pflogd]
 64 712  708   0  52  0 10684 1820 bpf      D     -  0:00.03 [pflogd]
  0 790    1   0   1  0 10460 1860 select   Ds    -  0:00.01 [syslogd]
  0 799    1   0  52  0 10408 1856 select   Ds    -  0:00.00 [rpcbind]
  0 873  106   0  52  0 13132 1912 wait     D+    -  0:00.00 [sh]
  0 878  873   0  52  0  8296 1468 pipelk   D+    -  0:00.00 [cpucontrol]
  0 879  873   0  72  0  8284 1468 -        R+    -  0:00.00 [logger]

------------------------------------------------------------------------
vmstat -s

   164753 cpu context switches
     4648 device interrupts
     1009 software interrupts
    88094 traps
   463394 system calls
      106 kernel threads created
      549  fork() calls
      224 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
     1204 vnode pager pageins
     1532 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
        0 clean page reclamation shortfalls
        0 pages reactivated by the page daemon
    21216 copy-on-write faults
      137 copy-on-write optimized faults
    30772 zero fill pages zeroed
        0 zero fill pages prezeroed
        0 intransit blocking page faults
    60028 total VM faults taken
     1117 page faults requiring I/O
        0 pages affected by kernel thread creation
    19970 pages affected by  fork()
     7840 pages affected by vfork()
        0 pages affected by rfork()
   122765 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
     3015 pages active
      563 pages inactive
        0 pages in the laundry queue
   113591 pages wired down
  8046077 pages free
     4096 bytes per page
        0 total name lookups
          cache hits (0% pos + 0% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
       feeder     7     1K       -        7  32
     acpiintr     1     1K       -        1  64
         UART     3     3K       -        3  16,1024
       acpica  3995   402K       -    68227  16,32,64,128,256,512,1024
     acpitask     1    64K       -        1  65536
          USB    71    82K       -       76  16,32,128,256,512,1024,2048,4096
       USBdev    55    11K       -       55  32,64,128,256,512,4096
      acpisem    38     5K       -       38  128
      acpidev    41     3K       -       41  64
      CAM SIM     8     2K       -        8  256
      CAM XPT   105    10K       -      319  16,32,64,128,256,512,1024,2048,8192
      CAM DEV    18    36K       -       49  2048
      CAM CCB     0     0K       -     4904  2048
     CAM path    28     1K       -      176  32
   CAM periph    22     6K       -       61  16,32,64,128,256
CAM I/O Scheduler    10     2K       -       10  128
        vtbuf    24  1968K       -       46  4096
           vt    11     6K       -       11  512
       DEVFS3   183    46K       -      186  256
       DEVFS1   160    80K       -      162  512
   DEVFS_RULE    55    26K       -       55  64,512
        DEVFS    21     1K       -       22  16,128
       DEVFSP     2     1K       -        2  64
NFSD V4client     1     1K       -        1  256
 NFSD lckfile     1     1K       -        1  256
 NFSD session     1     1K       -        1  1024
    pfs_nodes    21     6K       -       21  256
         GEOM   342    57K       -     1910  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
    raid_data     0     0K       -      228  32,128,256
       isadev     8     1K       -        8  128
     pci_link    16     2K       -       16  32,128
    acpi_perf     8     1K       -        8  128
         cdev     3     1K       -        3  256
     filedesc     1     4K       -        1  4096
        sigio     1     1K       -        1  64
     filecaps     0     0K       -        1  64
      kdtrace   854   192K       -     2999  64,256
         kenv   123    13K       -      128  16,32,64,128,8192
       kqueue   117     8K       -      880  64
    proc-args    10     1K       -      406  16,32,64,128,256
        hhook    13     4K       -       13  256
      ithread   142    26K       -      142  32,128,256
       KTRACE   100    13K       -      100  128
    CAM queue    26   149K       -      129  16,32,64,128,256,512,16384
       linker   292   230K       -      323  16,32,64,128,256,512,1024,2048,4096,8192,16384
        lockf    14     2K       -       18  64,128
   loginclass     2     1K       -        2  64
       devbuf 17706 35817K       -    17942  16,32,64,128,256,512,1024,2048,4096,8192,32768,65536
         temp    46    20K       -     1404  16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
       module   514    65K       -      515  128
     mtx_pool     2    16K       -        2  8192
          osd     6     1K       -       81  16,32,64,128,256
     pmchooks     1     1K       -        1  128
         pgrp     7     1K       -        7  128
      session     7     1K       -        7  128
         proc     2   128K       -        2  65536
      subproc   298   559K       -     1061  512,4096
         cred    28     7K       -       30  256
CAM dev queue     8     1K       -        8  64
       plimit     6     2K       -       76  256
      uidinfo     3    17K       -        3  128,16384
       dumper     1     1K       -        1  512
      scsi_da     0     0K       -       40  32,64
       sysctl     0     0K       -      746  16,32,64
    sysctloid  6241   322K       -     6350  16,32,64,128
    sysctltmp     0     0K       -       33  16,64,256,1024
       kbdmux     8    22K       -        8  16,512,1024,2048,16384
      tidhash     1   128K       -        1  
      callout     9  3208K       -        9  
         umtx  1568   196K       -     1568  128
     p1003.1b     1     1K       -        1  16
         SWAP     4   281K       -        4  64
          bus  1316   114K       -     7228  16,32,64,128,256,1024
       bus-sc    90   461K       -     4632  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
      devstat    18    37K       -       18  32,4096
 eventhandler   123    10K       -      123  64,128
    taskqueue    29    19K       -       29  16,32,256,8192
         kobj   345  1380K       -      714  4096
      Per-cpu     1     1K       -        1  32
         rman   311    36K       -      667  16,32,128
         sbuf     0     0K       -     1136  16,32,64,128,256
    toponodes    30     4K       -       30  128
          LED     8     1K       -        8  16,128
    taskqueue   309    35K       -      516  16,32,64,128,256
     terminal    11     3K       -       11  256
       Unitno    22     2K       -      164  32,64
         vmem     2   144K       -        3  8192,16384
     ioctlops     0     0K       -      922  256,512,1024,2048,4096
       select     5     1K       -        5  128
          iov     0     0K       -    22889  16,64,128,256,512
          msg     4    30K       -        4  2048,4096,8192,16384
          sem     4   106K       -        4  2048,4096
          shm     1    32K       -        1  32768
          tty    13    13K       -       13  1024
         accf     1     1K       -        1  64
     mbuf_tag     0     0K       -        6  32
        shmfd     1     8K       -        1  8192
       soname     5     1K       -    17272  16,32,128
          pcb    19  4247K       -       25  16,32,128,1024,2048,8192
     vfscache     4  8353K       -        4  512,32768
     vfs_hash     1  4096K       -        1  
       vnodes     1     1K       -        1  256
        mount   236     8K       -      587  16,32,64,128,256
       statfs     0     0K       -       53  512
  vnodemarker     0     0K       -        9  512
   arc4random     1     4K       -        1  4096
          BPF    10  1026K       -       11  16,128,512
        ifnet     7    13K       -        7  128,2048
       ifaddr    87    19K       -       87  16,32,64,128,256,512,2048,4096
  ether_multi    29     2K       -       29  16,32,64
        clone     8     1K       -        8  128
        ipsec     1     1K       -        1  256
      lltable    27     8K       -       27  256,512
     routetbl    76    15K       -       99  32,64,128,256,512
         igmp     6     1K       -        6  128
     in_multi     4     1K       -        4  256
encap_export_host     2     2K       -        2  1024
    sctp_a_it     0     0K       -        7  16
     sctp_vrf     1     1K       -        1  64
     sctp_ifa     8     1K       -        8  128
     sctp_ifn     4     1K       -        4  128
    sctp_iter     0     0K       -        7  256
    hostcache     1    32K       -        1  32768
      tcpfunc     1     1K       -        1  32
     syncache     1    68K       -        1  
  in6_mfilter     1     1K       -        1  1024
    in6_multi    16     2K       -       16  32,256
 ip6_moptions     2     1K       -        2  32,256
          mld     5     1K       -        5  128
       ip6ndp     6     1K       -        6  64,256
  inpcbpolicy     9     1K       -      124  32
     secasvar     1     1K       -        1  1024
       sahead     1     1K       -        1  1024
  ipsecpolicy     2     2K       -        2  256,1024
    ipsec-saq     2     2K       -        2  1024
       crypto     2     1K       -       22  64,256,512
          rpc     2    16K       -        2  8192
audit_evclass   187     6K       -      232  32
    ufs_quota     1  4096K       -        1  
    vm_pgdata     3  4097K       -        3  128
      UMAHash     9     5K       -       10  512,1024
  fpukern_ctx     8    16K       -        8  2048
      memdesc     1     4K       -        1  4096
          mps    21   643K       -      106  16,32,64,128,256,2048,4096,32768,65536
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        1  4096
      entropy     0     0K       -       11  4096
      io_apic     2     6K       -        2  2048,4096
          MCA     9     2K       -        9  128
          msi     9     2K       -        9  128
     nexusdev     6     1K       -        6  16
      solaris 35728 106907K       -   285781  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
   kstat_data     6     1K       -        6  64
    sfs_nodes    46    23K       -       46  512
     eli data  4168  1035K       -     8435  64,256,512,1024,2048,4096,8192,16384,32768,65536
     netgraph     4     1K       -        4  64
netgraph_node     6     2K       -        6  128,256
   aesni_data    82    82K       -       82  64,1024
      pf_hash     3  2880K       -        3  
     pf_ifnet    16     6K       -       42  128,256,2048
      pf_osfp  1184   122K       -     1184  64,128
       pf_tag     4     1K       -        4  128
      pf_altq     6     2K       -        6  256
      pf_rule    26    22K       -       26  128,1024
     pf_table     9    18K       -       14  2048
       cpuctl     1     1K       -        2  64,4096

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     246,       4,     246,   0,   0
UMA Zones:             1664,      0,     263,       1,     263,   0,   0
UMA Slabs:               80,      0,    7839,      11,    8461,   0,   0
UMA Hash:               256,      0,      97,      23,     106,   0,   0
4 Bucket:                32,      0,     460,    1915,    4350,   0,   0
6 Bucket:                48,      0,     118,    1957,     806,   0,   0
8 Bucket:                64,      0,      40,    2006,    1951,  11,   0
12 Bucket:               96,      0,     165,     860,     969,   0,   0
16 Bucket:              128,      0,     113,     910,    1466,   0,   0
32 Bucket:              256,      0,     143,     352,     660, 137,   0
64 Bucket:              512,      0,     131,     197,     563, 137,   0
128 Bucket:            1024,      0,      98,      82,     225,   0,   0
256 Bucket:            2048,      0,     424,      46,    1682, 152,   0
vmem btag:               56,      0,   12250,     956,   12675,  94,   0
VM OBJECT:              240,      0,    1267,     445,   11133,   0,   0
RADIX NODE:             144,      0,    6325,     425,   29293,   0,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             128,      0,       9,     518,       9,   0,   0
MAP ENTRY:              128,      0,     167,    1476,   23911,   0,   0
VMSPACE:               2520,      0,      11,      34,     775,   0,   0
fakepg:                 104,      0,       0,       0,       0,   0,   0
mt_zone:              16400,      0,     425,       0,     425,   0,   0
16:                      16,      0,    4110,    1161,   70120,   0,   0
32:                      32,      0,    4198,    1427,   23988,   0,   0
64:                      64,      0,   37894,   31980,  171863,   0,   0
128:                    128,      0,   10318,     563,   85853,   0,   0
256:                    256,      0,    6503,     637,   27514,   0,   0
512:                    512,      0,     833,     271,   44407,   0,   0
1024:                  1024,      0,    4880,     204,   19457,   0,   0
2048:                  2048,      0,     123,      63,    5934,   0,   0
4096:                  4096,      0,     723,       7,    2763,   0,   0
8192:                  8192,      0,      22,      57,    2052,   0,   0
16384:                16384,      0,      28,      25,    1127,   0,   0
32768:                32768,      0,      38,      12,     618,   0,   0
65536:                65536,      0,      19,      11,     485,   0,   0
64 pcpu:                  8,      0,    3089,    1007,    3089,   0,   0
SLEEPQUEUE:              80,      0,     785,     610,     785,   0,   0
Files:                   80,      0,      50,     950,    7079,   0,   0
filedesc0:             1104,      0,     117,     108,     880,   0,   0
rl_entry:                40,      0,      33,    2067,      33,   0,   0
TURNSTILE:              136,      0,     785,     255,     785,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
umtx_shm:                88,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1416,      0,     116,      64,     879,   0,   0
THREAD:                1368,      0,     759,      25,    1378,   0,   0
cpuset:                  96,      0,     731,     786,    1363,   0,   0
audit_record:          1248,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 13061205,    3072,    1738,    3086,   0,   0
mbuf:                   256, 13061205,       1,    1267,      67,   0,   0
mbuf_cluster:          2048, 2040812,    4807,       9,    4807,   0,   0
mbuf_jumbo_page:       4096, 1020405,       0,       0,       0,   0,   0
mbuf_jumbo_9k:         9216, 907026,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 680268,       0,       0,       0,   0,   0
NetGraph items:          72,   4123,       0,       0,       0,   0,   0
NetGraph data items:     72,   4123,       0,       0,       0,   0,   0
g_bio:                  376,      0,       0,     380,   18478,   0,   0
DMAR_MAP_ENTRY:         120,      0,       0,       0,       0,   0,   0
ttyinq:                 160,      0,      15,     360,     180,   0,   0
ttyoutq:                256,      0,       8,     367,      96,   0,   0
nvme_request:           128,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          960,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    2075,     120,   0,   0
VNODE:                  472,      0,     560,     136,     598,   0,   0
VNODEPOLL:              120,      0,       0,       0,       0,   0,   0
BUF TRIE:               144,      0,       0,  105948,       0,   0,   0
NAMEI:                 1024,      0,       0,     104,   18592,   0,   0
rentr:                   24,      0,       0,       0,       0,   0,   0
S VFS Cache:            108,      0,     480,     675,     968,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,       0,       0,       0,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       0,       0,       0,   0,   0
AIO:                    224,      0,       0,       0,       0,   0,   0
AIOP:                    32,      0,       0,       0,       0,   0,   0
AIOCB:                  752,      0,       0,       0,       0,   0,   0
AIOL:                   128,      0,       0,       0,       0,   0,   0
AIOLIO:                 280,      0,       0,       0,       0,   0,   0
Mountpoints:            816,      0,      24,      71,      24,   0,   0
pipe:                   760,      0,       1,      94,     353,   0,   0
range_seg_cache:         64,      0,   66090,   23004,  177671,   0,   0
zio_cache:              984,      0,      16,    1076,   69585,   0,   0
zio_link_cache:          48,      0,       0,    3320,   42183,   0,   0
zio_buf_512:            512,      0,     841,      87,   17210,   0,   0
zio_data_buf_512:       512,      0,      54,     154,     774,   0,   0
zio_buf_1024:          1024,      0,      35,      69,      57,   0,   0
zio_data_buf_1024:     1024,      0,      54,      38,      60,   0,   0
zio_buf_1536:          1536,      0,      25,      39,      33,   0,   0
zio_data_buf_1536:     1536,      0,      22,      24,      26,   0,   0
zio_buf_2048:          2048,      0,      34,      22,     112,   0,   0
zio_data_buf_2048:     2048,      0,      45,      15,      51,   0,   0
zio_buf_2560:          2560,      0,       1,       8,       6,   0,   0
zio_data_buf_2560:     2560,      0,      21,       6,      22,   0,   0
zio_buf_3072:          3072,      0,       1,       5,       2,   0,   0
zio_data_buf_3072:     3072,      0,      14,      10,      14,   0,   0
zio_buf_3584:          3584,      0,       4,       0,       4,   0,   0
zio_data_buf_3584:     3584,      0,       6,       0,       6,   0,   0
zio_buf_4096:          4096,      0,     569,     705,    8718,   0,   0
zio_data_buf_4096:     4096,      0,      38,      26,     192,   0,   0
zio_buf_5120:          5120,      0,       1,       0,       1,   0,   0
zio_data_buf_5120:     5120,      0,       7,       0,       7,   0,   0
zio_buf_6144:          6144,      0,       1,       0,       1,   0,   0
zio_data_buf_6144:     6144,      0,       5,       0,       5,   0,   0
zio_buf_7168:          7168,      0,       0,       2,       2,   0,   0
zio_data_buf_7168:     7168,      0,       6,       0,       8,   0,   0
zio_buf_8192:          8192,      0,      11,      11,     517,   0,   0
zio_data_buf_8192:     8192,      0,      27,       0,      35,   0,   0
zio_buf_10240:        10240,      0,       3,       0,       3,   0,   0
zio_data_buf_10240:   10240,      0,      10,       0,      10,   0,   0
zio_buf_12288:        12288,      0,       8,      12,     209,   0,   0
zio_data_buf_12288:   12288,      0,      15,       0,      17,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       8,       0,       8,   0,   0
zio_buf_16384:        16384,      0,     262,       8,     607,   0,   0
zio_data_buf_16384:   16384,      0,      21,       1,      24,   0,   0
zio_buf_20480:        20480,      0,       4,      14,     119,   0,   0
zio_data_buf_20480:   20480,      0,      10,       1,      13,   0,   0
zio_buf_24576:        24576,      0,       3,      11,      65,   0,   0
zio_data_buf_24576:   24576,      0,      15,       0,      17,   0,   0
zio_buf_28672:        28672,      0,       2,      30,     594,   0,   0
zio_data_buf_28672:   28672,      0,      14,       0,      15,   0,   0
zio_buf_32768:        32768,      0,       1,      11,      41,   0,   0
zio_data_buf_32768:   32768,      0,       8,       0,       8,   0,   0
zio_buf_40960:        40960,      0,       1,      12,      72,   0,   0
zio_data_buf_40960:   40960,      0,      13,       0,      13,   0,   0
zio_buf_49152:        49152,      0,      10,       7,      52,   0,   0
zio_data_buf_49152:   49152,      0,      14,       0,      15,   0,   0
zio_buf_57344:        57344,      0,       1,       8,      32,   0,   0
zio_data_buf_57344:   57344,      0,       7,       0,       8,   0,   0
zio_buf_65536:        65536,      0,       0,       8,      17,   0,   0
zio_data_buf_65536:   65536,      0,      12,       0,      12,   0,   0
zio_buf_81920:        81920,      0,       0,       7,      12,   0,   0
zio_data_buf_81920:   81920,      0,      25,       0,      25,   0,   0
zio_buf_98304:        98304,      0,       0,      14,      49,   0,   0
zio_data_buf_98304:   98304,      0,      31,       0,      31,   0,   0
zio_buf_114688:      114688,      0,       0,      12,      55,   0,   0
zio_data_buf_114688: 114688,      0,       4,       2,      30,   0,   0
zio_buf_131072:      131072,      0,     174,       7,     640,   0,   0
zio_data_buf_131072: 131072,      0,      72,       1,      87,   0,   0
zio_buf_163840:      163840,      0,       0,       0,       0,   0,   0
zio_data_buf_163840: 163840,      0,       0,       0,       0,   0,   0
zio_buf_196608:      196608,      0,       0,       0,       0,   0,   0
zio_data_buf_196608: 196608,      0,       0,       0,       0,   0,   0
zio_buf_229376:      229376,      0,       0,       0,       0,   0,   0
zio_data_buf_229376: 229376,      0,       0,       0,       0,   0,   0
zio_buf_262144:      262144,      0,       0,       0,       0,   0,   0
zio_data_buf_262144: 262144,      0,       0,       0,       0,   0,   0
zio_buf_327680:      327680,      0,       0,       0,       0,   0,   0
zio_data_buf_327680: 327680,      0,       0,       0,       0,   0,   0
zio_buf_393216:      393216,      0,       0,       0,       0,   0,   0
zio_data_buf_393216: 393216,      0,       0,       0,       0,   0,   0
zio_buf_458752:      458752,      0,       0,       0,       0,   0,   0
zio_data_buf_458752: 458752,      0,       0,       0,       0,   0,   0
zio_buf_524288:      524288,      0,       0,       0,       0,   0,   0
zio_data_buf_524288: 524288,      0,       0,       0,       0,   0,   0
zio_buf_655360:      655360,      0,       0,       0,       0,   0,   0
zio_data_buf_655360: 655360,      0,       0,       0,       0,   0,   0
zio_buf_786432:      786432,      0,       0,       0,       0,   0,   0
zio_data_buf_786432: 786432,      0,       0,       0,       0,   0,   0
zio_buf_917504:      917504,      0,       0,       0,       0,   0,   0
zio_data_buf_917504: 917504,      0,       0,       0,       0,   0,   0
zio_buf_1048576:     1048576,      0,       0,       0,       0,   0,   0
zio_data_buf_1048576: 1048576,      0,       0,       0,       0,   0,   0
zio_buf_1310720:     1310720,      0,       0,       0,       0,   0,   0
zio_data_buf_1310720: 1310720,      0,       0,       0,       0,   0,   0
zio_buf_1572864:     1572864,      0,       0,       0,       0,   0,   0
zio_data_buf_1572864: 1572864,      0,       0,       0,       0,   0,   0
zio_buf_1835008:     1835008,      0,       0,       0,       0,   0,   0
zio_data_buf_1835008: 1835008,      0,       0,       0,       0,   0,   0
zio_buf_2097152:     2097152,      0,       0,       0,       0,   0,   0
zio_data_buf_2097152: 2097152,      0,       0,       0,       0,   0,   0
zio_buf_2621440:     2621440,      0,       0,       0,       0,   0,   0
zio_data_buf_2621440: 2621440,      0,       0,       0,       0,   0,   0
zio_buf_3145728:     3145728,      0,       0,       0,       0,   0,   0
zio_data_buf_3145728: 3145728,      0,       0,       0,       0,   0,   0
zio_buf_3670016:     3670016,      0,       0,       0,       0,   0,   0
zio_data_buf_3670016: 3670016,      0,       0,       0,       0,   0,   0
zio_buf_4194304:     4194304,      0,       0,       0,       0,   0,   0
zio_data_buf_4194304: 4194304,      0,       0,       0,       0,   0,   0
zio_buf_5242880:     5242880,      0,       0,       0,       0,   0,   0
zio_data_buf_5242880: 5242880,      0,       0,       0,       0,   0,   0
zio_buf_6291456:     6291456,      0,       0,       0,       0,   0,   0
zio_data_buf_6291456: 6291456,      0,       0,       0,       0,   0,   0
zio_buf_7340032:     7340032,      0,       0,       0,       0,   0,   0
zio_data_buf_7340032: 7340032,      0,       0,       0,       0,   0,   0
zio_buf_8388608:     8388608,      0,       0,       0,       0,   0,   0
zio_data_buf_8388608: 8388608,      0,       0,       0,       0,   0,   0
zio_buf_10485760:    10485760,      0,       0,       0,       0,   0,   0
zio_data_buf_10485760: 10485760,      0,       0,       0,       0,   0,   0
zio_buf_12582912:    12582912,      0,       0,       0,       0,   0,   0
zio_data_buf_12582912: 12582912,      0,       0,       0,       0,   0,   0
zio_buf_14680064:    14680064,      0,       0,       0,       0,   0,   0
zio_data_buf_14680064: 14680064,      0,       0,       0,       0,   0,   0
zio_buf_16777216:    16777216,      0,       0,       0,       0,   0,   0
zio_data_buf_16777216: 16777216,      0,       0,       0,       0,   0,   0
lz4_ctx:              16384,      0,       0,      13,     722,   0,   0
sa_cache:               136,      0,     503,     309,     540,   0,   0
dnode_t:                736,      0,    1188,      77,    2184,   0,   0
arc_buf_hdr_t_full:     224,      0,    1134,     277,   10008,   0,   0
arc_buf_hdr_t_l2only:     88,      0,       0,       0,       0,   0,   0
arc_buf_t:               56,      0,    1052,    1007,   10438,   0,   0
dmu_buf_impl_t:         232,      0,    1857,     217,   10733,   0,   0
zil_lwb_cache:          200,      0,       6,     134,       9,   0,   0
zfs_znode_cache:        264,      0,     503,     112,     540,   0,   0
procdesc:               136,      0,       0,       0,       0,   0,   0
ksiginfo:               112,      0,     124,     926,     130,   0,   0
itimer:                 352,      0,       0,       0,       0,   0,   0
KNOTE:                  128,      0,       0,       0,       0,   0,   0
socket:                 864, 1046872,      17,      87,    3025,   0,   0
unpcb:                  240, 1046880,       8,     488,    2895,   0,   0
ipq:                     56,  63829,       0,       0,       0,   0,   0
udp_inpcb:              464, 1046872,       7,     225,     122,   0,   0
udpcb:                   32, 1046875,       7,    2118,     122,   0,   0
tcp_inpcb:              464, 1046872,       2,      30,       2,   0,   0
tcpcb:                 1040, 1046874,       2,       7,       2,   0,   0
tcptw:                   88,  27810,       0,       0,       0,   0,   0
syncache:               168,  15364,       0,       0,       0,   0,   0
hostcache:              128,  15376,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0
tcpreass:                40, 127600,       0,       0,       0,   0,   0
sctp_ep:               1488, 1046872,       0,       0,       0,   0,   0
sctp_asoc:             2408,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,    1328,       7,   0,   0
sctp_raddr:             736,  80000,       0,       0,       0,   0,   0
sctp_chunk:             152, 400010,       0,       0,       0,   0,   0
sctp_readq:             152, 400010,       0,       0,       0,   0,   0
sctp_stream_msg_out:    112, 400015,       0,       0,       0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
udplite_inpcb:          464, 1046872,       0,       0,       0,   0,   0
ripcb:                  464, 1046872,       0,       0,       0,   0,   0
IPsec SA lft_c:          16,      0,       0,       0,       0,   0,   0
rtentry:                208,      0,      17,     420,      19,   0,   0
selfd:                   64,      0,      19,    1283,     801,   0,   0
SWAPMETA:               288, 4081623,       0,       0,       0,   0,   0
pf mtags:                48,      0,       0,       0,       0,   0,   0
pf states:              296,  50011,       0,       0,       0,   0,   0
pf state keys:           88,      0,       0,       0,       0,   0,   0
pf source nodes:        136,  10005,       0,       0,       0,   0,   0
pf table entries:       160, 200000,      10,     115,      10,   0,   0
pf table counters:       64,      0,       0,       0,       0,   0,   0
pf frags:               112,      0,       0,       0,       0,   0,   0
pf frag entries:         40,   5000,       0,       0,       0,   0,   0
pf state scrubs:         40,      0,       0,       0,       0,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq16: ohci0 ohci1                    22          0
irq17: ehci0                         112          1
irq18: ohci2 ohci3+                    4          0
irq22: ahci0                        2386         29
cpu0:timer                          6632         80
cpu1:timer                          2033         24
cpu7:timer                          3688         44
cpu4:timer                          6965         84
cpu5:timer                          3774         45
cpu6:timer                          2309         28
cpu3:timer                         26031        313
cpu2:timer                          1966         24
irq256: mps0                        2101         25
irq257: em0:rx0                        5          0
irq258: em0:tx0                        3          0
irq259: em0:link                       2          0
irq260: em1:rx0                        5          0
irq261: em1:tx0                        1          0
irq262: em1:link                       1          0
irq263: em2                            6          0
Total                              58046        697

------------------------------------------------------------------------
pstat -T

 50/1046869 files
0M/0M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity

------------------------------------------------------------------------
iostat

       tty            ada0             ada1              da0             cpu
 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   0    67 18.23  29  0.51  20.81  23  0.47  23.87   7  0.16   0  0  9  0 90

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
    msgmax:        16384    (max characters in a message)
    msgmni:           40    (# of message queues)
    msgmnb:         2048    (max characters in a message queue)
    msgtql:           40    (max # of messages in system)
    msgssz:            8    (size of a message segment)
    msgseg:         2048    (# of message segments in system)

shminfo:
    shmmax:    536870912    (max shared memory segment size)
    shmmin:            1    (min shared memory segment size)
    shmmni:          192    (max number of shared memory identifiers)
    shmseg:          128    (max shared memory segments per process)
    shmall:       131072    (max amount of shared memory in pages)

seminfo:
    semmni:           50    (# of semaphore identifiers)
    semmns:          340    (# of semaphores in system)
    semmnu:          150    (# of undo structures in system)
    semmsl:          340    (max # of semaphores per id)
    semopm:          100    (max # of operations per semop call)
    semume:           50    (max # of undo entries per process)
    semusz:          632    (size in bytes of undo structure)
    semvmx:        32767    (semaphore maximum value)
    semaem:        16384    (adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
    0 packets sent
        0 data packets (0 bytes)
        0 data packets (0 bytes) retransmitted
        0 data packets unnecessarily retransmitted
        0 resends initiated by MTU discovery
        0 ack-only packets (0 delayed)
        0 URG only packets
        0 window probe packets
        0 window update packets
        0 control packets
    0 packets received
        0 acks (for 0 bytes)
        0 duplicate acks
        0 acks for unsent data
        0 packets (0 bytes) received in-sequence
        0 completely duplicate packets (0 bytes)
        0 old duplicate packets
        0 packets with some dup. data (0 bytes duped)
        0 out-of-order packets (0 bytes)
        0 packets (0 bytes) of data after window
        0 window probes
        0 window update packets
        0 packets received after close
        0 discarded for bad checksums
        0 discarded for bad header offset fields
        0 discarded because packet too short
        0 discarded due to memory problems
    0 connection requests
    0 connection accepts
    0 bad connection attempts
    0 listen queue overflows
    0 ignored RSTs in the windows
    0 connections established (including accepts)
        0 times used RTT from hostcache
        0 times used RTT variance from hostcache
        0 times used slow-start threshold from hostcache
    0 connections closed (including 0 drops)
        0 connections updated cached RTT on close
        0 connections updated cached RTT variance on close
        0 connections updated cached ssthresh on close
    0 embryonic connections dropped
    0 segments updated rtt (of 0 attempts)
    0 retransmit timeouts
        0 connections dropped by rexmit timeout
    0 persist timeouts
        0 connections dropped by persist timeout
    0 Connections (fin_wait_2) dropped because of timeout
    0 keepalive timeouts
        0 keepalive probes sent
        0 connections dropped by keepalive
    0 correct ACK header predictions
    0 correct data packet header predictions
    0 syncache entries added
        0 retransmitted
        0 dupsyn
        0 dropped
        0 completed
        0 bucket overflow
        0 cache overflow
        0 reset
        0 stale
        0 aborted
        0 badack
        0 unreach
        0 zone failures
    0 cookies sent
    0 cookies received
    0 hostcache entries added
        0 bucket overflow
    0 SACK recovery episodes
    0 segment rexmits in SACK recovery episodes
    0 byte rexmits in SACK recovery episodes
    0 SACK options (SACK blocks) received
    0 SACK options (SACK blocks) sent
    0 SACK scoreboard overflow
    0 packets with ECN CE bit set
    0 packets with ECN ECT(0) bit set
    0 packets with ECN ECT(1) bit set
    0 successful ECN handshakes
    0 times ECN reduced the congestion window
    0 packets with matching signature received
    0 packets with bad signature received
    0 times failed to make signature due to no SA
    0 times unexpected signature received
    0 times no signature provided by segment
TCP connection count by state:
    0 connections in CLOSED state
    2 connections in LISTEN state
    0 connections in SYN_SENT state
    0 connections in SYN_RCVD state
    0 connections in ESTABLISHED state
    0 connections in CLOSE_WAIT state
    0 connections in FIN_WAIT_1 state
    0 connections in CLOSING state
    0 connections in LAST_ACK state
    0 connections in FIN_WAIT_2 state
    0 connections in TIME_WAIT state
udp:
    0 datagrams received
    0 with incomplete header
    0 with bad data length field
    0 with bad checksum
    0 with no checksum
    0 dropped due to no socket
    0 broadcast/multicast datagrams undelivered
    0 dropped due to full socket buffers
    0 not for hashed pcb
    0 delivered
    0 datagrams output
    0 times multicast source filter matched
ip:
    0 total packets received
    0 bad header checksums
    0 with size smaller than minimum
    0 with data size < data length
    0 with ip length > max ip packet size
    0 with header length < data size
    0 with data length < header length
    0 with bad options
    0 with incorrect version number
    0 fragments received
    0 fragments dropped (dup or out of space)
    0 fragments dropped after timeout
    0 packets reassembled ok
    0 packets for this host
    0 packets for unknown/unsupported protocol
    0 packets forwarded (0 packets fast forwarded)
    0 packets not forwardable
    0 packets received for unknown multicast group
    0 redirects sent
    0 packets sent from this host
    0 packets sent with fabricated ip header
    0 output packets dropped due to no bufs, etc.
    0 output packets discarded due to no route
    0 output datagrams fragmented
    0 fragments created
    0 datagrams that can't be fragmented
    0 tunneling packets that can't find gif
    0 datagrams with bad address in header
icmp:
    0 calls to icmp_error
    0 errors not generated in response to an icmp message
    0 messages with bad code fields
    0 messages less than the minimum length
    0 messages with bad checksum
    0 messages with bad length
    0 multicast echo requests ignored
    0 multicast timestamp requests ignored
    0 message responses generated
    0 invalid return addresses
    0 no return routes
ipsec:
    0 inbound packets violated process security policy
    0 inbound packets failed due to insufficient memory
    0 invalid inbound packets
    0 outbound packets violated process security policy
    0 outbound packets with no SA available
    0 outbound packets failed due to insufficient memory
    0 outbound packets with no route available
    0 invalid outbound packets
    0 outbound packets with bundled SAs
    0 mbufs coalesced during clone
    0 clusters coalesced during clone
    0 clusters copied during clone
    0 mbufs inserted during makespace
ah:
    0 packets shorter than header shows
    0 packets dropped; protocol family not supported
    0 packets dropped; no TDB
    0 packets dropped; bad KCR
    0 packets dropped; queue full
    0 packets dropped; no transform
    0 replay counter wraps
    0 packets dropped; bad authentication detected
    0 packets dropped; bad authentication length
    0 possible replay packets detected
    0 packets in
    0 packets out
    0 packets dropped; invalid TDB
    0 bytes in
    0 bytes out
    0 packets dropped; larger than IP_MAXPACKET
    0 packets blocked due to policy
    0 crypto processing failures
    0 tunnel sanity check failures
esp:
    0 packets shorter than header shows
    0 packets dropped; protocol family not supported
    0 packets dropped; no TDB
    0 packets dropped; bad KCR
    0 packets dropped; queue full
    0 packets dropped; no transform
    0 packets dropped; bad ilen
    0 replay counter wraps
    0 packets dropped; bad encryption detected
    0 packets dropped; bad authentication detected
    0 possible replay packets detected
    0 packets in
    0 packets out
    0 packets dropped; invalid TDB
    0 bytes in
    0 bytes out
    0 packets dropped; larger than IP_MAXPACKET
    0 packets blocked due to policy
    0 crypto processing failures
    0 tunnel sanity check failures
ipcomp:
    0 packets shorter than header shows
    0 packets dropped; protocol family not supported
    0 packets dropped; no TDB
    0 packets dropped; bad KCR
    0 packets dropped; queue full
    0 packets dropped; no transform
    0 replay counter wraps
    0 packets in
    0 packets out
    0 packets dropped; invalid TDB
    0 bytes in
    0 bytes out
    0 packets dropped; larger than IP_MAXPACKET
    0 packets blocked due to policy
    0 crypto processing failures
    0 packets sent uncompressed; size < compr. algo. threshold
    0 packets sent uncompressed; compression was useless
arp:
    5 ARP requests sent
    0 ARP replies sent
    0 ARP requests received
    0 ARP replies received
    0 ARP packets received
    0 total packets dropped due to no ARP entry
    0 ARP entrys timed out
    0 Duplicate IPs seen
ip6:
    0 total packets received
    0 with size smaller than minimum
    0 with data size < data length
    0 with bad options
    0 with incorrect version number
    0 fragments received
    0 fragments dropped (dup or out of space)
    0 fragments dropped after timeout
    0 fragments that exceeded limit
    0 packets reassembled ok
    0 packets for this host
    0 packets forwarded
    0 packets not forwardable
    0 redirects sent
    0 packets sent from this host
    0 packets sent with fabricated ip header
    0 output packets dropped due to no bufs, etc.
    0 output packets discarded due to no route
    0 output datagrams fragmented
    0 fragments created
    0 datagrams that can't be fragmented
    0 packets that violated scope rules
    0 multicast packets which we don't join
    Mbuf statistics:
        0 one mbuf
        0 one ext mbuf
        0 two or more ext mbuf
    0 packets whose headers are not contiguous
    0 tunneling packets that can't find gif
    0 packets discarded because of too many headers
    0 failures of source address selection
icmp6:
    0 calls to icmp6_error
    0 errors not generated in response to an icmp6 message
    0 errors not generated because of rate limitation
    0 messages with bad code fields
    0 messages < minimum length
    0 bad checksums
    0 messages with bad length
    Histogram of error messages to be generated:
        0 no route
        0 administratively prohibited
        0 beyond scope
        0 address unreachable
        0 port unreachable
        0 packet too big
        0 time exceed transit
        0 time exceed reassembly
        0 erroneous header field
        0 unrecognized next header
        0 unrecognized option
        0 redirect
        0 unknown
    0 message responses generated
    0 messages with too many ND options
    0 messages with bad ND options
    0 bad neighbor solicitation messages
    0 bad neighbor advertisement messages
    0 bad router solicitation messages
    0 bad router advertisement messages
    0 bad redirect messages
    0 path MTU changes
ipsec6:
    0 inbound packets violated process security policy
    0 inbound packets failed due to insufficient memory
    0 invalid inbound packets
    0 outbound packets violated process security policy
    0 outbound packets with no SA available
    0 outbound packets failed due to insufficient memory
    0 outbound packets with no route available
    0 invalid outbound packets
    0 outbound packets with bundled SAs
    0 mbufs coalesced during clone
    0 clusters coalesced during clone
    0 clusters copied during clone
    0 mbufs inserted during makespace
rip6:
    0 messages received
    0 checksum calculations on inbound
    0 messages with bad checksum
    0 messages dropped due to no socket
    0 multicast messages dropped due to no socket
    0 messages dropped due to full socket buffers
    0 delivered
    0 datagrams output
pfkey:
    0 requests sent from userland
    0 bytes sent from userland
    0 messages with invalid length field
    0 messages with invalid version field
    0 messages with invalid message type field
    0 messages too short
    0 messages with memory allocation failure
    0 messages with duplicate extension
    0 messages with invalid extension type
    0 messages with invalid sa type
    0 messages with invalid address extension
    0 requests sent to userland
    0 bytes sent to userland
    0 messages toward single socket
    0 messages toward all sockets
    0 messages toward registered sockets
    0 messages with memory allocation failure

------------------------------------------------------------------------
netstat -m

3073/3005/6078 mbufs in use (current/cache/total)
3069/1747/4816/2040812 mbuf clusters in use (current/cache/total/max)
3072/1738 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/1020405 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/907026 9k jumbo clusters in use (current/cache/total/max)
0/0/0/680268 16k jumbo clusters in use (current/cache/total/max)
6906K/4245K/11151K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 sendfile syscalls
0 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
0 pages were valid at time of a sendfile request
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
fffff8010a15e410 tcp4       0      0 *.111              *.*                LISTEN
fffff8010a15e820 tcp6       0      0 *.111              *.*                LISTEN
fffff80038230740 udp6       0      0 *.*                *.*                
fffff80038262570 udp4       0      0 *.713              *.*                
fffff80038262740 udp4       0      0 *.111              *.*                
fffff80038262910 udp6       0      0 *.793              *.*                
fffff80038262ae0 udp6       0      0 *.111              *.*                
fffff80038077cb0 udp4       0      0 *.514              *.*                
fffff8003826c000 udp6       0      0 *.514              *.*                
Active UNIX domain sockets
Address          Type   Recv-Q Send-Q            Inode             Conn             Refs          Nextref Addr
fffff80038c49c30 stream      0      0 fffff8003817d000                0                0                0 /var/run/rpcbind.sock
fffff80038bde000 stream      0      0                0 fffff80038bde0f0                0                0
fffff80038bde0f0 stream      0      0                0 fffff80038bde000                0                0
fffff80038bde2d0 stream      0      0 fffff8002b80d3b0                0                0                0 /var/run/devd.pipe
fffff80038c49b40 dgram       0      0                0 fffff80038bdbd20                0                0
fffff80038bdbd20 dgram       0      0 fffff80038c1c000                0 fffff80038c49b40                0 /var/run/logpriv
fffff80038bdbe10 dgram       0      0 fffff80038c1c1d8                0                0                0 /var/run/log
fffff80038bde1e0 seqpac      0      0 fffff80038c1db10                0                0                0 /var/run/devd.seqpacket.pipe

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address         
tcp4  0/0/128                          *.sunrpc               
tcp6  0/0/128                          *.sunrpc               
unix  0/0/128                          /var/run/rpcbind.sock
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/4                            /var/run/devd.seqpacket.pipe

------------------------------------------------------------------------
fstat

fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x200000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x200000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x400000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read file 11 at 0x400000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x400000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read file 11 at 0x400000000
fstat: can't read file 13 at 0x20007ffffffffff
fstat: can't read file 14 at 0x4000000001fffff
fstat: can't read file 16 at 0x780000ffff
fstat: can't read file 17 at 0x400000000
fstat: can't read file 19 at 0x20007ffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x400000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x800000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x800000000
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0x20007ffffffffff
fstat: can't read file 2 at 0x4000000001fffff
fstat: can't read file 4 at 0x780000ffff
fstat: can't read file 5 at 0x200000000
fstat: can't read file 7 at 0x20007ffffffffff
fstat: can't read file 8 at 0x4000000001fffff
fstat: can't read file 10 at 0x780000ffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     logger       879 root -         -       error    -
root     logger       879   wd -         -       error    -
root     logger       879 text -         -       error    -
root     logger       879 ctty /dev         11 crw-------  console rw
root     logger       879    0* pipe fffff8002b7292f8 <-> fffff8002b729460    105 rw
root     cpucontrol   878 root -         -       error    -
root     cpucontrol   878   wd -         -       error    -
root     cpucontrol   878 text -         -       error    -
root     cpucontrol   878 ctty /dev         11 crw-------  console rw
root     cpucontrol   878    0 /dev         11 crw-------  console rw
root     sh           873 root -         -       error    -
root     sh           873   wd -         -       error    -
root     sh           873 text -         -       error    -
root     sh           873 ctty /dev         11 crw-------  console rw
root     sh           873    0 /dev         11 crw-------  console rw
root     sh           873    6 /dev         11 crw-------  console rw
root     rpcbind      799 root -         -       error    -
root     rpcbind      799   wd -         -       error    -
root     rpcbind      799 text -         -       error    -
root     rpcbind      799    0 /dev         26 crw-rw-rw-    null rw
root     rpcbind      799    6 /dev         26 crw-rw-rw-    null rw
root     syslogd      790 root -         -       error    -
root     syslogd      790   wd -         -       error    -
root     syslogd      790 text -         -       error    -
root     syslogd      790    0 /dev         26 crw-rw-rw-    null rw
root     syslogd      790    6 /dev         26 crw-rw-rw-    null rw
root     syslogd      790   12 /dev         26 crw-rw-rw-    null rw
root     syslogd      790   18 -         -       error    -
_pflogd  pflogd       712 root -         -       error    -
_pflogd  pflogd       712   wd -         -       error    -
_pflogd  pflogd       712 jail -         -       error    -
_pflogd  pflogd       712 text -         -       error    -
_pflogd  pflogd       712    0 /dev         26 crw-rw-rw-    null rw
root     pflogd       708 root -         -       error    -
root     pflogd       708   wd -         -       error    -
root     pflogd       708 text -         -       error    -
root     pflogd       708    0 /dev         26 crw-rw-rw-    null rw
root     pf purge     696 root -         -       error    -
root     pf purge     696   wd -         -       error    -
root     devd         689 root -         -       error    -
root     devd         689   wd -         -       error    -
root     devd         689 text -         -       error    -
root     devd         689    0 /dev         26 crw-rw-rw-    null rw
root     devd         689    6 /dev         26 crw-rw-rw-    null rw
root     moused       672 root -         -       error    -
root     moused       672   wd -         -       error    -
root     moused       672 text -         -       error    -
root     moused       672    0 /dev         26 crw-rw-rw-    null rw
root     sh           106 root -         -       error    -
root     sh           106   wd -         -       error    -
root     sh           106 text -         -       error    -
root     sh           106 ctty /dev         11 crw-------  console rw
root     sh           106    0 /dev         11 crw-------  console rw
root     sh           106    6 /dev         11 crw-------  console rw
root     zfskern        9 root -         -       error    -
root     zfskern        9   wd -         -       error    -
root     init           1 root -         -       error    -
root     init           1   wd -         -       error    -
root     init           1 text -         -       error    -
root     kernel         0 root -         -       error    -
root     kernel         0   wd -         -       error    -

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p10 #0 r334297: Tue May 29 00:28:19 CEST 2018
    mr44er@smain:/usr/obj/usr/src/sys/SMAINKERNEL amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
VT(vga): resolution 640x480
CPU: AMD Opteron(tm) Processor 4284                  (3000.07-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x600f12  Family=0x15  Model=0x1  Stepping=2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x1e98220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x1c9bfff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,XOP,SKINIT,WDT,LWP,FMA4,NodeId,Topology,PCXC,PNXC>
  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=65536
  TSC: P-state invariant, performance statistics
real memory  = 34359738368 (32768 MB)
avail memory = 33297387520 (31754 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <022513 APIC1142>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s)
random: unblocking device.
ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20170303/tbfadt-796)
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 64/32 (20170303/tbfadt-748)
ioapic0 <Version 2.1> irqs 0-23 on motherboard
ioapic1 <Version 2.1> irqs 24-55 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1500033267 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80f53120, 0) error 19
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
acpi0: <022513 XSDT1142> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 52 at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib1
mps0: <Avago Technologies (LSI) SAS2308> port 0xe000-0xe0ff mem 0xfeab0000-0xfeabffff,0xfeac0000-0xfeafffff irq 28 at device 0.0 on pci1
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib2: <ACPI PCI-PCI bridge> irq 53 at device 9.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xd800-0xd81f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 48 at device 0.0 on pci2
em0: Using MSIX interrupts with 3 vectors
em0: Ethernet address: a0:42:3f:00:86:d7
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib3: <ACPI PCI-PCI bridge> irq 54 at device 10.0 on pci0
pci3: <ACPI PCI bus> on pcib3
em1: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xc800-0xc81f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 47 at device 0.0 on pci3
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: a0:42:3f:00:86:d6
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <ACPI PCI-PCI bridge> irq 54 at device 11.0 on pci0
pci4: <ACPI PCI bus> on pcib4
em2: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xb400-0xb41f mem 0xfe780000-0xfe79ffff,0xfe760000-0xfe77ffff irq 32 at device 0.0 on pci4
em2: Using an MSI interrupt
em2: Ethernet address: 00:15:17:d6:43:ca
em2: netmap queues/slots: TX 1/1024, RX 1/1024
em3: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xb800-0xb81f mem 0xfe7e0000-0xfe7fffff,0xfe7c0000-0xfe7dffff irq 33 at device 0.1 on pci4
em3: Using an MSI interrupt
em3: Ethernet address: 00:15:17:d6:43:cb
em3: netmap queues/slots: TX 1/1024, RX 1/1024
ahci0: <AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller> port 0x9400-0x9407,0x9c00-0x9c03,0x9880-0x9887,0x9800-0x9803,0x9480-0x948f mem 0xfe6fa400-0xfe6fa7ff irq 22 at device 17.0 on pci0
ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
ahci0: quirks=0x22000<ATI_PMP_BUG,1MSI>
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ohci0: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe6f6000-0xfe6f6fff irq 16 at device 18.0 on pci0
usbus0 on ohci0
usbus0: 12Mbps Full Speed USB v1.0
ohci1: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe6f7000-0xfe6f7fff irq 16 at device 18.1 on pci0
usbus1 on ohci1
usbus1: 12Mbps Full Speed USB v1.0
ehci0: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfe6fa800-0xfe6fa8ff irq 17 at device 18.2 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci0
usbus2: 480Mbps High Speed USB v2.0
ohci2: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe6f8000-0xfe6f8fff irq 18 at device 19.0 on pci0
usbus3 on ohci2
usbus3: 12Mbps Full Speed USB v1.0
ohci3: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe6f9000-0xfe6f9fff irq 18 at device 19.1 on pci0
usbus4 on ohci3
usbus4: 12Mbps Full Speed USB v1.0
ehci1: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfe6fac00-0xfe6facff irq 19 at device 19.2 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci1
usbus5: 480Mbps High Speed USB v2.0
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib5: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci5: <ACPI PCI bus> on pcib5
vgapci0: <VGA-compatible display> port 0xa800-0xa87f mem 0xfd800000-0xfdffffff,0xfd7e0000-0xfd7fffff irq 23 at device 9.0 on pci5
vgapci0: Boot video device
ohci4: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe6fb000-0xfe6fbfff irq 18 at device 20.5 on pci0
usbus6 on ohci4
usbus6: 12Mbps Full Speed USB v1.0
amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb4
acpi_button0: <Power Button> on acpi0
ipmi0: <IPMI System Interface> port 0xca8,0xcac on acpi0
ipmi0: KCS mode found at io 0xca8 on acpi
uart1: <Non-standard ns8250 class UART with FIFOs> port 0x2f8-0x2ff irq 3 on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xce000-0xcefff,0xcf000-0xcffff on isa0
ppc0: cannot reserve I/O port range
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
nvme cam probe device init
ugen0.1: <ATI OHCI root HUB> at usbus0
ugen2.1: <ATI EHCI root HUB> at usbus2
ugen6.1: <ATI OHCI root HUB> at usbus6
ugen3.1: <ATI OHCI root HUB> at usbus3
ugen1.1: <ATI OHCI root HUB> at usbus1
ugen4.1: <ATI OHCI root HUB> at usbus4
ugen5.1: <ATI EHCI root HUB> at usbus5
uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub1: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub3: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub5: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ipmi0: IPMI device rev. 1, firmware rev. 1.00, version 2.0
ipmi0: Number of channels 2
ipmi0: Attached watchdog
uhub1: 3 ports with 3 removable, self powered
uhub5: 3 ports with 3 removable, self powered
uhub0: 3 ports with 3 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub4: 3 ports with 3 removable, self powered
uhub3: 6 ports with 6 removable, self powered
uhub2: 6 ports with 6 removable, self powered
ugen0.2: <CHICONY USB Keyboard> at usbus0
ukbd0 on uhub0
ukbd0: <CHICONY USB Keyboard, class 0/0, rev 1.10/3.00, addr 2> on usbus0
kbd2 at ukbd0
ugen2.2: <American Megatrends Inc. Generic Hub> at usbus2
uhub7 on uhub3
uhub7: <Hub Interface> on usbus2
uhub7: 3 ports with 3 removable, self powered
ugen2.3: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus2
ukbd1 on uhub7
ukbd1: <Keyboard Interface> on usbus2
kbd3 at ukbd1
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <ST3750640NS 3.BAJ> ATA-7 SATA 2.x device
ada0: Serial Number 5QD17ZR2
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 715404MB (1465149168 512 byte sectors)
ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
ada1: <ST3750640NS 3.BAJ> ATA-7 SATA 2.x device
ada1: Serial Number 5QD18179
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 715404MB (1465149168 512 byte sectors)
da0 at mps0 bus 0 scbus0 target 2 lun 0
da3 at mps0 bus 0 scbus0 target 9 lun 0
da0: <HGST HUS726020AL5210 AD05> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number K5GWGL2A
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 1907729MB (3907029168 512 byte sectors)
da1 at mps0 bus 0 scbus0 target 4 lun 0
da1: <HITACHI HUS723020ALS641 MS06> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number YFJJM1YD
da3: <HGST HUS726020AL5210 AD05> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number N4G1XV1K
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 1907729MB (3907029168 512 byte sectors)
da5 at mps0 bus 0 scbus0 target 11 lun 0
da5: da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 1907729MB (3907029168 512 byte sectors)
da2 at mps0 bus 0 scbus0 target 8 lun 0
da2: <HGST HUS726020AL5210 AD05> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number K5GW6SGA
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 1907729MB (3907029168 512 byte sectors)
<HGST HUS726020AL5210 AD05> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number N4G2N9AY
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 1907729MB (3907029168 512 byte sectors)
da7 at mps0 bus 0 scbus0 target 13 lun 0
da7: <HGST HUS726020AL5210 AD05> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number K5GWGN5A
da7: 600.000MB/s transfersda4 at mps0 bus 0 scbus0 target 10 lun 0
da4: <HGST HUS726020AL5210 AD05> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number K5GWAZTA
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 1907729MB (3907029168 512 byte sectors)
da6 at mps0 bus 0 scbus0 target 12 lun 0
da6: 
da7: Command Queueing enabled
<HGST HUS726020AL5210 AD05> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number K5GW31WA
da7: 1907729MB (3907029168 512 byte sectors)
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 1907729MB (3907029168 512 byte sectors)
Trying to mount root from zfs:zroot/ROOT/default []...
GEOM_ELI: Device ada0p3.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da0.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da2.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da3.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da4.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da5.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da6.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da7.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device ada1p3.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
Setting hostuuid: 2b41f897-6279-11e8-b7cd-a0423f0086d7.
Setting hostid: 0xa2d4d2aa.
Starting file system checks:
Mounting local filesystems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/perl5/5.26/mach/CORE
32-bit compatibility ldconfig path: /usr/lib32
Setting hostname: smain.
Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
em0: link state changed to UP
em2: link state changed to UP
Starting Network: lo0 em0 em1 em2 em3.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 
    inet 127.0.0.1 netmask 0xff000000 
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
    ether a0:42:3f:00:86:d7
    hwaddr a0:42:3f:00:86:d7
    inet 192.168.100.5 netmask 0xffffff00 broadcast 192.168.100.255 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
    ether a0:42:3f:00:86:d6
    hwaddr a0:42:3f:00:86:d6
    inet 10.0.8.1 netmask 0xfffffe00 broadcast 10.0.9.255 
    inet 10.0.8.10 netmask 0xfffffffe broadcast 255.255.255.255 
    inet 10.0.8.11 netmask 0xfffffffe broadcast 255.255.255.255 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: no carrier
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
    ether 00:15:17:d6:43:ca
    hwaddr 00:15:17:d6:43:ca
    inet 10.0.6.1 netmask 0xfffffe00 broadcast 10.0.7.255 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
em3: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
    ether 00:15:17:d6:43:cb
    hwaddr 00:15:17:d6:43:cb
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: no carrier
Starting devd.
Starting Network: em3.
em3: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
    ether 00:15:17:d6:43:cb
    hwaddr 00:15:17:d6:43:cb
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: no carrier
Configuring vt: keymap.
ums0 on uhub7
ums0: <Mouse Interface> on usbus2
ums0: 3 buttons and [Z] coordinates ID=0
em1: link state changed to UP
Starting ums0 moused.
Starting pflog.
pflog0: promiscuous mode enabled
Enabling pfJun 20 13:30:39 pflogd[708]: [priv]: msg PRIV_OPEN_LOG received
.
add host 127.0.0.1: gateway lo0 fib 0: route already in table
Additional inet routing options: gateway=YES.
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Creating and/or trimming log files.
Starting syslogd.
Starting rpcbind.
Updating CPU Microcode...


Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 14
fault virtual address    = 0x800e25000
fault code        = supervisor write data, page not present
instruction pointer    = 0x20:0xffffffff80ed570f
stack pointer            = 0x28:0xfffffe0861c40740
frame pointer            = 0x28:0xfffffe0861c40740
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 879 (logger)
trap number        = 12
panic: page fault
cpuid = 4
KDB: stack backtrace:
#0 0xffffffff80aa0897 at kdb_backtrace+0x67
#1 0xffffffff80a5ee36 at vpanic+0x186
#2 0xffffffff80a5eca3 at panic+0x43
#3 0xffffffff80ed7c12 at trap_fatal+0x322
#4 0xffffffff80ed7c6b at trap_pfault+0x4b
#5 0xffffffff80ed73c8 at trap+0x2a8
#6 0xffffffff80eb8810 at calltrap+0x8
#7 0xffffffff80ab61db at uiomove_faultflag+0x10b
#8 0xffffffff80abe211 at pipe_read+0x3b1
#9 0xffffffff80abb1ba at dofileread+0xba
#10 0xffffffff80abacbb at sys_read+0xdb
#11 0xffffffff80ed87f4 at amd64_syscall+0x6c4
#12 0xffffffff80eb9042 at fast_syscall_common+0x106
Uptime: 49s
Dumping 1295 out of 32714 MB:..2%..12%..22%..31%..41%..51%..61%..71%..81%..91%

------------------------------------------------------------------------
kernel config

options    CONFIG_AUTOGENERATED
ident    GENERIC
machine    amd64
cpu    HAMMER
makeoptions    WITH_CTF=1
makeoptions    DEBUG=-g
options    ALTQ_PRIQ
options    ALTQ_HFSC
options    ALTQ_RIO
options    ALTQ_RED
options    ALTQ_CBQ
options    ALTQ
options    XENHVM
options    USB_DEBUG
options    ATH_ENABLE_11N
options    AH_AR5416_INTERRUPT_MITIGATION
options    AH_SUPPORT_AR5416
options    IEEE80211_SUPPORT_MESH
options    IEEE80211_AMPDU_AGE
options    IEEE80211_DEBUG
options    SC_PIXEL_MODE
options    VESA
options    AHD_REG_PRETTY_PRINT
options    AHC_REG_PRETTY_PRINT
options    PCI_IOV
options    PCI_HP
options    ACPI_DMAR
options    EARLY_AP_STARTUP
options    DEVICE_NUMA
options    SMP
options    KDB_TRACE
options    KDB
options    RCTL
options    RACCT_DEFAULT_TO_DISABLED
options    RACCT
options    INCLUDE_CONFIG_FILE
options    DDB_CTF
options    KDTRACE_HOOKS
options    KDTRACE_FRAME
options    MAC
options    CAPABILITIES
options    CAPABILITY_MODE
options    AUDIT
options    HWPMC_HOOKS
options    KBD_INSTALL_CDEV
options    PRINTF_BUFR_SIZE=128
options    _KPOSIX_PRIORITY_SCHEDULING
options    SYSVSEM
options    SYSVMSG
options    SYSVSHM
options    STACK
options    KTRACE
options    SCSI_DELAY=5000
options    COMPAT_FREEBSD10
options    COMPAT_FREEBSD9
options    COMPAT_FREEBSD7
options    COMPAT_FREEBSD6
options    COMPAT_FREEBSD5
options    COMPAT_FREEBSD4
options    COMPAT_FREEBSD32
options    GEOM_LABEL
options    GEOM_RAID
options    GEOM_PART_GPT
options    PSEUDOFS
options    PROCFS
options    CD9660
options    MSDOSFS
options    NFS_ROOT
options    NFSLOCKD
options    NFSD
options    NFSCL
options    MD_ROOT
options    QUOTA
options    UFS_GJOURNAL
options    UFS_DIRHASH
options    UFS_ACL
options    SOFTUPDATES
options    FFS
options    SCTP
options    TCP_OFFLOAD
options    IPSEC
options    INET6
options    INET
options    PREEMPTION
options    SCHED_ULE
options    NEW_PCIB
options    GEOM_PART_MBR
options    GEOM_PART_EBR_COMPAT
options    GEOM_PART_EBR
options    GEOM_PART_BSD
device    isa
device    mem
device    io
device    uart_ns8250
device    cpufreq
device    acpi
device    pci
device    fdc
device    ahci
device    ata
device    mvs
device    siis
device    ahc
device    ahd
device    esp
device    hptiop
device    isp
device    mpt
device    mps
device    mpr
device    sym
device    trm
device    adv
device    adw
device    aic
device    bt
device    isci
device    scbus
device    ch
device    da
device    sa
device    cd
device    pass
device    ses
device    amr
device    arcmsr
device    ciss
device    dpt
device    hptmv
device    hptnr
device    hptrr
device    hpt27xx
device    iir
device    ips
device    mly
device    twa
device    tws
device    aac
device    aacp
device    aacraid
device    ida
device    mfi
device    mlx
device    mrsas
device    pmspcv
device    twe
device    nvme
device    nvd
device    atkbdc
device    atkbd
device    psm
device    kbdmux
device    vga
device    splash
device    sc
device    vt
device    vt_vga
device    vt_efifb
device    agp
device    cbb
device    pccard
device    cardbus
device    uart
device    ppc
device    ppbus
device    lpt
device    ppi
device    puc
device    bxe
device    de
device    em
device    igb
device    ix
device    ixv
device    ixl
device    ixlv
device    le
device    ti
device    txp
device    vx
device    miibus
device    ae
device    age
device    alc
device    ale
device    bce
device    bfe
device    bge
device    cas
device    dc
device    et
device    fxp
device    gem
device    hme
device    jme
device    lge
device    msk
device    nfe
device    nge
device    pcn
device    re
device    rl
device    sf
device    sge
device    sis
device    sk
device    ste
device    stge
device    tl
device    tx
device    vge
device    vr
device    wb
device    xl
device    wlan
device    wlan_wep
device    wlan_ccmp
device    wlan_tkip
device    wlan_amrr
device    an
device    ath
device    ath_pci
device    ath_hal
device    ath_rate_sample
device    ipw
device    iwi
device    iwn
device    malo
device    mwl
device    ral
device    wi
device    wpi
device    loop
device    random
device    padlock_rng
device    rdrand_rng
device    ether
device    vlan
device    tun
device    md
device    gif
device    firmware
device    bpf
device    uhci
device    ohci
device    ehci
device    xhci
device    usb
device    ukbd
device    umass
device    sound
device    snd_cmi
device    snd_csa
device    snd_emu10kx
device    snd_es137x
device    snd_hda
device    snd_ich
device    snd_via8233
device    mmc
device    mmcsd
device    sdhci
device    virtio
device    virtio_pci
device    vtnet
device    virtio_blk
device    virtio_scsi
device    virtio_balloon
device    hyperv
device    xenpci
device    vmx
device    netmap
device    crypto

------------------------------------------------------------------------
ddb capture buffer

ddb: ddb_capture: kvm_nlist

Kann man da was rauslesen?
 
Code:
Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 14
fault virtual address    = 0x800e25000
fault code        = supervisor write data, page not present
Das ist sozusagen die Standard-Panic. Es wurde versucht auf eine Speicheradresse zuzugreifen, die das System nicht kennt. Im Userland führt das zu einem Segmentation Fault des jeweiligen Prozesses, wenn es im Kernel passiert zu Panic. Das kann mindestens 1001 Gründe haben. Angefangen vom Alphateilchen, über klassische Programmfehler bis hin zu einem inkonsistenten System durch Hardwaredefekt. Kurz gesagt, da lässt sich Post Mortem nichts mehr erkennen. Wenn es einmal war, würde ich es als "Pech" abhaken. Passiert sowas öfter sollte man sich anfangen Gedanken zu machen.
 
powerdxx ist gestern weggebröselt, habs dann einfach neu gestartet. Kann es irgendwie sein, dass sich das mit den microcode updates aus devcpu-data nicht verträgt?
 
Weg gebröselt? Es ist eigentlich nicht sonderlich komplex und verwendet nur das sysctl Interface.
 
Es stürzte zusammen mit anderen Dienstan ab, als geswapped wurde.
Mehr hab ich aus der Erinnerung raus nicht.
 
Nach einem Tausch des HBAs sind die Fehlermeldungen 'GEOM_ELI: Crypto READ request failed (error=22)' nun nicht mehr aufgetaucht. :D
 
Ich muss das hier an der Stelle doch nochmal aufrollen, da es wieder auftritt.

Die 'weggebröselten' Dienste lagen am nicht limitierten ARC, powerdxx läuft wie es soll. Da ist alles gut.

Jedoch dieser Error taucht wieder auf: GEOM_ELI: Crypto READ request failed (error=22)

Code:
Aug 11 13:42:12 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da1.eli[READ(offset=1821762646016, length=28672)]
Aug 11 13:42:15 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da0.eli[READ(offset=1822054469632, length=28672)]
Aug 11 13:42:15 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da5.eli[READ(offset=1822057996288, length=28672)]
Aug 11 13:42:15 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da1.eli[READ(offset=1822059233280, length=28672)]
Aug 11 13:42:15 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da2.eli[READ(offset=1822074810368, length=28672)]
Aug 11 13:42:15 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=753553677821058681
Aug 11 13:42:15 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=16269308682429721136
Aug 11 13:42:16 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=14531015339997244915
Aug 11 13:42:16 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=8935334454069511531
Aug 11 13:42:33 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da0.eli[READ(offset=1823935234048, length=28672)]
Aug 11 13:42:36 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=8935334454069511531
Aug 11 13:42:38 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da3.eli[READ(offset=1824397213696, length=28672)]
Aug 11 13:42:39 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da1.eli[READ(offset=1824474816512, length=28672)]
Aug 11 13:42:42 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=753553677821058681
Aug 11 13:42:42 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=15458723054115801944
Aug 11 13:42:43 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da5.eli[READ(offset=1824916123648, length=28672)]
Aug 11 13:42:47 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=14531015339997244915
Aug 11 13:42:47 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da5.eli[READ(offset=1825341333504, length=28672)]
Aug 11 13:42:50 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da3.eli[READ(offset=1825646137344, length=28672)]
Aug 11 13:42:52 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=14531015339997244915
Aug 11 13:42:52 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=15458723054115801944
Aug 11 13:43:02 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da0.eli[READ(offset=1826836549632, length=28672)]
Aug 11 13:43:03 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=8935334454069511531
Aug 11 13:43:07 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da2.eli[READ(offset=1827361468416, length=28672)]
Aug 11 13:43:08 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=16269308682429721136
Aug 11 13:43:11 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da0.eli[READ(offset=1829954146304, length=28672)]
Aug 11 13:43:13 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da5.eli[READ(offset=1830128898048, length=28672)]
Aug 11 13:43:13 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=14531015339997244915
Aug 11 13:43:13 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=8935334454069511531
Aug 11 13:43:17 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da1.eli[READ(offset=1830584135680, length=28672)]
Aug 11 13:43:17 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da0.eli[READ(offset=1830586281984, length=28672)]
Aug 11 13:43:18 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=753553677821058681
Aug 11 13:43:18 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=8935334454069511531
Aug 11 13:43:33 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da4.eli[READ(offset=1832131252224, length=28672)]
Aug 11 13:43:34 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=8582248611295115976
Aug 11 13:43:35 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da5.eli[READ(offset=1832410112000, length=28672)]
Aug 11 13:43:37 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da0.eli[READ(offset=1832544899072, length=28672)]
Aug 11 13:43:37 aserver kernel: GEOM_ELI: Crypto READ request failed (error=22). da5.eli[READ(offset=1832601538560, length=28672)]
Aug 11 13:43:39 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=14531015339997244915
Aug 11 13:43:39 aserver ZFS: vdev state changed, pool_guid=13797102652085798891 vdev_guid=8935334454069511531

Es tritt auf an beiden Maschinen nach längerer Uptime, sobald Zugriff auf die Platten erfolgt.
Direkt reproduzieren kann ich es, wenn ich einen scrub starte und die Maschine hat eine gewisse Uptime.
dmesg wird gefloodet damit und zfs listet wie verrückt Lesefehler auf.

Maschine1 hat den SAS2308 und Maschine2 den SAS2008. Beide sind auf IT-Version 20.00.07.00.
Beide Maschinen haben 11.2-RELEASE.

Sobald dieser Fehler 1x auftaucht, kann ich die Maschine nichtmal mehr sauber herunterfahren.

Ist eine der Kisten frisch gebootet und starte ich einen scrub, dann funktioniert alles ohne Fehlermeldungen.

Hat irgendjemand dazu eine Idee?
 
Wenn das SSDs sind könnte es das alte Problem mit den TRIM Batch Sizes sein. Aber das sind mechanische Platten, oder?
 
Moin,
nein sind keine SSDs. 3x SATA am Mainboard (port 1-3) im zfs mirror als zroot und 8x SAS raidz3 am HBA.
Könnte es der selfscrub der Platten sein oder eine Inkompatibilität, die nicht sein dürfte? Es sind 12G-Platten, die BP und der Controller können nur 6G.
 
Zuletzt bearbeitet:
https://github.com/zfsonlinux/zfs/issues/4713

Genau wie es dort mehrere Personen beschreiben, hab ich den Fehler.

Mit deaktiviertem sysutils/spindown läuft alles rund und es gibt keine Hänger/Fehlermeldungen. (Ich hatte am Samstag abend mal blind geraten, bevor ich zur Freundin fuhr, den Server mit deaktiviertem spindown rebootet und mich eben vom smartphone aus eingeloggt und mal nen scrub gestartet). Dh. längere uptime kreiert und dann probiert. Es klappt also und der spinup löst das Problem aus oder eben der daraus resultierende timeout. Platte zu langsam?

Hat jemand eine Idee, wie ich die Platten trotzdem schlafen legen kann? Mir ist der Verschleiß bewusst, allerdings erfolgt am Wochenende gar kein Zugriff und an Werktagen zu selten, dass ein Dauerlauf lohnen würde. Die Stromersparnis würde ich daher gerne mitnehmen.
 
Code:
dev.mps.0.spinup_wait_time=60

Den hab ich mal so in /boot/loader.conf gesetzt (standard sind 5 Sekunden), rebootet (live setzen geht mit dieser Variable nicht!) und einen scrub nach dem Schlafenlegen der Platten gestartet. Funktionierte anstandslos ohne Fehlermeldungen, aber ich behalte das mal im Auge, weil ich nicht glauben kann, dass es diesmal so einfach gewesen sein soll. ;)

Ich habe mal einen spinup der Platten (HGST 7K6000 HUS726020AL5210) gemessen. -> zwischen 17 und 19 Sekunden. Wenn alle Platten schlafen und bis ein 'zpool scrub' mich wieder zum prompt lässt (staggered spinup), vergehen gemessene 91 Sekunden (Schnittwert von ~11,38 Sekunden je Platte bei 8 Stück).
Wichtig zu wissen wäre noch, dass camcontrol idle, standby und sleep nicht bei SAS-Platten greifen. sysutils/spindown führt ein 'camcontrol stop' auf der jeweiligen SAS-Platte aus.
 
Das riecht nach "Staggered Spinup", als das der Controller nicht alle Platten auf einmal startet, sondern schön der Reihe nach. Das kann schwachen Netzteilen helfen, da mehrere andrehende Spindeln schon recht viel Energie ziehen. Aber es dauert eben auch deutlich länger. Meistens lässt sich das im BIOS oder im Controller umstellen. Manchmal auch mit Jumpern an der Backplane.
 
Das riecht nach "Staggered Spinup", als das der Controller nicht alle Platten auf einmal startet, sondern schön der Reihe nach
Macht er auch so, sehe ich wie die Lämpchen anfangen zu blinken (alle nacheinander). Ich hab da auch gar nichts auszusetzen, wenn ich ein wenig warten muss und das Netzteil dadurch keine Lastspitze abbekommt...es soll mir nur keine timeouts werfen, wo keine sein sollen und sind. (geli, zfs). :)
 
Bisher kann ich sagen, dass der hochgesetzte timeout beim SAS2308 funktioniert, beim SAS2008 irgendwie nicht. Richtig gesetzt ist aber alles, daher kann das Zufall sein.
Maschine1:
root@aserver:/usr/home/mr44er # sysctl -a | grep mps
device mps
kstat.zfs.misc.zcompstats.skipped_insufficient_gain: 49
kstat.zfs.misc.zcompstats.empty: 167
kstat.zfs.misc.zcompstats.attempts: 48414
dev.mps.0.use_phy_num: 1
dev.mps.0.encl_table_dump:
dev.mps.0.mapping_table_dump:
dev.mps.0.spinup_wait_time: 60
dev.mps.0.chain_alloc_fail: 0
dev.mps.0.enable_ssu: 3
dev.mps.0.max_io_pages: -1
dev.mps.0.max_chains: 2048
dev.mps.0.chain_free_lowwater: 2047
dev.mps.0.chain_free: 2048
dev.mps.0.io_cmds_highwater: 80
dev.mps.0.io_cmds_active: 12
dev.mps.0.driver_version: 21.02.00.00-fbsd
dev.mps.0.firmware_version: 20.00.07.00
dev.mps.0.disable_msi: 0
dev.mps.0.disable_msix: 0
dev.mps.0.debug_level: 3
dev.mps.0.%parent: pci1
dev.mps.0.%pnpinfo: vendor=0x1000 device=0x0087 subvendor=0x1000 subdevice=0x3020 class=0x010700
dev.mps.0.%location: slot=0 function=0 dbsf=pci0:5:0:0
dev.mps.0.%driver: mps
dev.mps.0.%desc: Avago Technologies (LSI) SAS2308
dev.mps.%parent:

Maschine2:
GEOM_ELI: Crypto READ request failed (error=22). da6.eli[READ(offset=958590853120, length=28672)]
GEOM_ELI: Crypto READ request failed (error=22). da0.eli[READ(offset=945162608640, length=49152)]
GEOM_ELI: Crypto READ request failed (error=22). da5.eli[READ(offset=958692184064, length=28672)]
GEOM_ELI: Crypto READ request failed (error=22). da1.eli[READ(offset=958396235776, length=28672)]

root@dserver:/usr/home/mr44er # zpool status dtank
pool: dtank
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: scrub canceled on Mon Aug 13 16:24:41 2018
config:

NAME STATE READ WRITE CKSUM
dtank ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
label/YFJJM1YD-hd1 ONLINE 0 0 0
label/F326MXGD-hd2 ONLINE 0 0 0
label/F326NR2D-hd3 ONLINE 0 0 0
label/F3208GHG-hd4 ONLINE 0 0 0
label/YFJJAH9D-hd5 ONLINE 0 0 0
label/K5H4TPGA-hd6 ONLINE 0 0 0
label/F325S74D-hd7 ONLINE 0 0 0
label/F326PRAD-hd8 ONLINE 1 0 0
cache
label/2GBIntenso ONLINE 0 0 0

errors: No known data errors
root@dserver:/usr/home/mr44er # sysctl -a | grep mps
device mps
kstat.zfs.misc.zcompstats.skipped_insufficient_gain: 20
kstat.zfs.misc.zcompstats.empty: 145
kstat.zfs.misc.zcompstats.attempts: 22800
dev.mps.0.use_phy_num: 1
dev.mps.0.encl_table_dump:
dev.mps.0.mapping_table_dump:
dev.mps.0.spinup_wait_time: 60
dev.mps.0.chain_alloc_fail: 0
dev.mps.0.enable_ssu: 3
dev.mps.0.max_io_pages: -1
dev.mps.0.max_chains: 2048
dev.mps.0.chain_free_lowwater: 2040
dev.mps.0.chain_free: 2048
dev.mps.0.io_cmds_highwater: 80
dev.mps.0.io_cmds_active: 0
dev.mps.0.driver_version: 21.02.00.00-fbsd
dev.mps.0.firmware_version: 20.00.07.00
dev.mps.0.disable_msi: 0
dev.mps.0.disable_msix: 0
dev.mps.0.debug_level: 3
dev.mps.0.%parent: pci1
dev.mps.0.%pnpinfo: vendor=0x1000 device=0x0072 subvendor=0x1028 subdevice=0x1f1c class=0x010700
dev.mps.0.%location: slot=0 function=0 dbsf=pci0:4:0:0
dev.mps.0.%driver: mps
dev.mps.0.%desc: Avago Technologies (LSI) SAS2008
dev.mps.%parent:
 
Also der Fehler taucht beim SAS2308 als auch beim SAS2008 definitiv auf.

Ein gesetztes 'dev.mps.0.spinup_wait_time=120' brachte auch keinen Unterschied. 120 Sekunden sind definitiv über dem, was die Platten bei einem staggered spinup benötigen.

Maschine1 hat 8x die gleichen SAS-Platten, Maschine2 hat eine bunte Mischung von SATA und SAS. Daran kanns also auch nicht liegen.

Das einzige Auffällige war, dass wenn die Platten schlafen ein 'zpool scrub' sie per staggered spinup weckt. Ein Zugriff per nfs weckt sie alle auf einmal. Da ich die Controller ohne BIOS geflasht habe und somit auf dem Controller nicht die Art des spinups eingestellt ist, wieso ist das unterschiedlich bei div. Zugriffen?

Ich glaube fest, dass es das Problem hier ebenfalls ist: https://github.com/torvalds/linux/commit/ee6928d63e37e1b1e7eca898c5b2bac19f194ebd

Ein User von dort https://github.com/zfsonlinux/zfs/issues/4713 schreibt:
Your assessment seems correct sd 9:0:5:0: scsi_eh_9: Sending START_UNIT is done after the failed command. That's weird.
Ich kann es aber nicht greifen/bestätigen. Wie kann ich das bei FreeBSD nachprüfen?

Ein anderer schreibt:
i switched from a Broadcom SAS2008 to a marvel 88SE9485 chipset and the problem is gone.
Seems to be a kernel problem issue.
The problem described above only occurs if a controller which uses "mpt2sas" kernel mode is installed.
HBA controllers which use the "mvsas" kernel module work fine.
Tested with ubuntu 16.04. and kernel 4.4.0-112.

Ich überlege jetzt auch schon, mir einfach andere Controller zu holen, damit das funktioniert.
 
Ich überlege jetzt auch schon, mir einfach andere Controller zu holen, damit das funktioniert.
Ich sage mal so, diplomatisch: Die LSI / Avago / Broadcom / $AktuellerName Controller sind nicht schlecht. Es sind gute, robuste HBA. Aber es sind auch kleine Zicken. Darauf läuft eine wahnsinnig komplexe Firmware, die ja bei den aktuellen SAS3008 Modellen alleine schon ungefähr eine halbe Minute zum Booten braucht. Und die muss dann auch noch mit dem Treiber harmonieren, der Treiber wiederum mit dem SCSI-Framework des Kernels, etc. Das haben sie niemals wirklich zu 100% hinbekommen. Sobald man beginnt Dinge zu machen, die etwas Abseits des Mainstreams sind, läuft man schnell in Bugs... Daher würde ich mir, wenn S-ATA ausreicht, auf ZFS-Kisten den Ärger inzwischen eher sparen und einfach den Onboard-Controller oder einen anderen AHCI-Controller nehmen. Andererseits sagt es sich so einfach, ich habe gerade gestern trotzdem zwei neue Kisten mit 9300-8i gekauft.
 
Mhmmm, ich seh schon, dass einfach anders ist. :)
SATA würde ausreichen, aber die SAS-Platten hab ich jetzt eben schon. Die jetzt nochmal gegen SATA tauschen ist mir zu fummelig, zumal ich wegen zwei Maschinen das doppelt machen müsste. Das nächste Problem: Ich brauch 11 SATA Ports, das Board hat aber nur 8.

Da ists einfacher, den Controller zu tauschen. Hast du eine Ersatzempfehlung für mich, die mich nicht finanziell ruiniert? PCI-Express, 2x SFF8087, IT-Mode bzw. direktes Durchreichen und noch lauffähig unter FreeBSD und eben mit nicht krachendem spinup :p
 
Bei SFF8087 würde ich den 9300-8i empfehlen. :( Aber vielleicht hat noch jemand anders eine kluge Idee.
 
Den würd ich gerne nehmen...kannst du mir bestätigen, dass der mit sysutils/spindown nach einigen Stunden beim spinup keine Fehler wirft? :rolleyes::D

Edit: Der kostet knapp 300 Tacken :eek: Also dafür kann ich die Platten laaaaaange ohne spindown mit Strom versorgen. Und bist du dir sicher mit den Anschlüssen? Für mich sieht das nach 2x SFF-8643 aus. ;)
 
Ich muss nochmal nerven, ich brauche input. Es klappt nämlich immer fehlerfrei kurz nacheinander und das raffe ich nicht. :ugly:

Thema1:
https://www.freebsd.org/cgi/man.cgi?mps sagt:
The adapter can issue the StartStopUnit SCSI command to SATA direct-
access devices during shutdown. This allows the device to quiesce power-
ing down. To control this feature for all adapters, set the

hw.mps.enable_ssu

tunable in loader.conf(5) to one of these values:

0 Do not send SSU to either HDDs or SSDs.

1 Send SSU to SSDs, but not to HDDs. This is the default
value.

2 Send SSU to HDDs, but not to SSDs.

3 Send SSU to both HDDs and SSDs.
Den Wert habe ich auf 0 gesetzt. -> hw.mps.enable_ssu=0 in /boot/loader.conf und zur Sicherheit neu gestartet.
Verstehe ich das korrekt, dass dieser Wert nur Auswirkung bei einem system-shutdown hat? Was ist bei SAS-Platten? Kommt der Befehl vom Controller oder soll das eine 'Firewall' dem System gegenüber sein? (camcontrol stop geht trotzdem, egal was eingestellt ist)

Thema2:
Erstes Anhangsbild. Die kleine Platine ist jeweils per signal connector mit der Backplane und mit dem Mainboard verbunden.
Wer kann mir beantworten, wohin die Anweisung vom blauen SW4 genau geht? Zum Mainboard, Backplane oder zu den Platten direkt?
(er war auf on gestellt, also staggered spinup deaktiviert. 'zpool scrub' hat trotzdem einen staggered spinup gemacht, mit zugriff per nfs wurden alle platten auf einmal upgespinned)
Hab den jetzt mal testweise auf off gestellt, also staggered spinup aktiviert. 'zpool scrub' macht nach wie vor den staggered spinup, nfs-Zugriff macht nun jetzt auch einen staggered spinup. Daher frage ich mich, wohin die SW4-Anweisung hingeht.

Thema3:
In /etc/rc.conf habe ich:
spindown_enable="YES" spindown_flags="-b -t 600 -d da0 -d da1 -d da2 -d da3 -d da4 -d da5 -d da6 -d da7"

10 Minuten, damit ich nicht lange warten muß und das klappt auch. (während dem spindown blinken die Lampen an der Front und danach leuchten sie durchweg)
Ich habe dann sicherheitshalber 15 Minuten gewartet, bevor ich einen 'zpool scrub' startete. Klappt reibungslos bei allen meinen Versuchen. Die Platten drehen nacheinander hoch und der scrub startet dann nach ~90 Sekunden.
Spätestens nach 40GB gescannten Daten beim scrub kamen normalerweise die Fehlermeldungen beim zfs. In diesen Fällen aber nicht, dmesg ist auch sauber.

Ich komme daher zu der Erkenntnis, dass (wie weiter oben schon gesagt), der Faktor Zeit/längere Uptime etwas auslöst. Aber was? Gibt es sowas im mps-Treiber? SCSI-Stack? geli? Machen 'gestoppte' SAS-Platten irgendwas oder irgendwas nicht? Außer dass ihr selfcheck damit pausiert wird? Kann es das doch sein?

Ich will da noch weiter rumpopeln, jetzt könnte ich den selfcheck der Platten ausschalten. Hat da jemand Lektüre für mich oder einen Befehl? Im Netz fand ich leider nichts.

Die Letzten beiden Bilder: Könnten diese BIOS-Einstellungen dem SAS2308 was anhaben, zu wenig/zu langsam Power nach einer gewissen Zeit? Hat der SAS2308 intern einen sleep-mode?

Fragen über Fragen. :rolleyes:

Edit: Platten schlafen -> gute 30 Minuten vergehen lassen -> nochmal versucht, alles in Ordnung. Der 'kürzeste' Zeitraum wo der Terror (:p kleine Flaxrakete eingebaut) losging, waren 120 Minuten Uptime.
 

Anhänge

  • sw4.png
    sw4.png
    66,8 KB · Aufrufe: 461
  • power1.png
    power1.png
    16 KB · Aufrufe: 437
  • power2.png
    power2.png
    2,9 KB · Aufrufe: 251
Zuletzt bearbeitet:
Jup...ich kann es nochmal bestätigen. Nach uptime über 2 Stunden gehen die Probleme beim spinup los. Faktor Zeit knallt hier irgendwo dazwischen. Tricky!
 
Verstehe ich das korrekt, dass dieser Wert nur Auswirkung bei einem system-shutdown hat? Was ist bei SAS-Platten? Kommt der Befehl vom Controller oder soll das eine 'Firewall' dem System gegenüber sein? (camcontrol stop geht trotzdem, egal was eingestellt ist)
Der Befehl wird von dem Controller an die Festplatten geschickt, nachdem das Betriebssystem den Controller mitgeteilt hat, dass das System herunterfährt. Durch SSU bekommt die Festplatte die Gelegenheit sich sauber abzuschalten, bevor der Strom getrennt wird. Trennt man stattdessen einfach den Strom, ist das ein "Emergency Shutdown". Die Festplatte muss dann die kinetische Restenergie der Spindel und Strom aus einem Stützkondensator nutzen um den Cache auszuschreiben und die Schreibköpfe in Parkposition zu bringen.

Erstes Anhangsbild. Die kleine Platine ist jeweils per signal connector mit der Backplane und mit dem Mainboard verbunden.
Wer kann mir beantworten, wohin die Anweisung vom blauen SW4 genau geht? Zum Mainboard, Backplane oder zu den Platten direkt?
Keine Ahnung.


Die Letzten beiden Bilder: Könnten diese BIOS-Einstellungen dem SAS2308 was anhaben, zu wenig/zu langsam Power nach einer gewissen Zeit? Hat der SAS2308 intern einen sleep-mode?
Die beiden BIOS-Einstellungen werden damit nichts zu tun haben. Bild 1 ist die Frage ob PCIe v1 oder v2, da v2 rückkompatibel ist, ist es wohl die richtige Wahl. Bild 2 ist der Link zwischen der in der CPU integrierten Northbridge (Neudeutsch: Uncore) und der Southbridge. Wenn es da ein Problem gäbe, würde es fast alle Geräte im System betreffen...

Mein Tipp wäre auch, dass irgendwas in einen Tiefschlafmodus geht und nicht wieder aufwacht. Die meisten Festplatten haben zwei Modi. "Standby" hält die Spindel an und "Sleep" schaltet sie komplett ab. Es gibt immer mal wieder Platten, die ohne Reboot nicht mehr aus "Sleep" herauskommen. Könnte es sein, dass die Platten nach ca. 120 Minuten von "Standby" in "Sleep" wechseln?
 
Zurück
Oben