Having filesystem mounted can break zfs receive without any explicit modification

[rincebrain]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	10.10
Kernel Version	4.19.0-17-amd64
Architecture	x86_64
OpenZFS Version	2.0.3-8~bpo10-1

Describe the problem you're observing

(I suppose this might be known and a reason for people using zfs recv -u, but was certainly surprising to me, having not encountered it that I can recall, and I don't see any bug report or note about it, though the search terms are pretty generic...)

While I was shuffling around a large dataset to recompress it, I had the following odd experience:

# zfs send -RLe -I monolith/sorted/backups/urbackup@zfs-auto-snap_daily-2021-09-18-1554 monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-21-0317 | mbuffer -m 2G | zfs recv -o compress=zstd -o checksum=skein -o com.sun:auto-snapshot=false workspace/urbackup_zstd
in @  0.0 kiB/s, out @  176 MiB/s, 22.0 GiB total, buffer   1% full
summary: 22.0 GiByte in 31min 49.9sec - average of 11.8 MiB/s
cannot receive incremental stream: destination workspace/urbackup_zstd has been modified
since most recent snapshot
#

...huh, that's curious, nothing should be touching it, and the auto-snapshotting should be ignoring it...

# zfs list -o name,used,lused,refer,lrefer,guid -t all -r workspace/urbackup_zstd | tail -n 5; zfs list -o name,used,lused,refer,lrefer,guid -t all -r monolith/sorted/backups/urbackup | grep -A20 zfs-auto-snap_daily-2021-09-20-1632;
workspace/urbackup_zstd@zfs-auto-snap_daily-2021-09-20-1632       0B      -     2.79T   3.39T  10176713998951144728
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-1717   38.6M      -     2.79T   3.39T  5443784001748964352
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-1818   15.0M      -     2.79T   3.39T  10358248322821333959
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-1918    183M      -     2.78T   3.37T  11396114093498370378
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-2017    432K      -     2.08T   2.64T  15065966855315306501
monolith/sorted/backups/urbackup@zfs-auto-snap_daily-2021-09-20-1632       0B      -     2.91T   3.39T  10176713998951144728
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-20-1717   25.4M      -     2.91T   3.39T  5443784001748964352
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-20-1818   6.22M      -     2.91T   3.39T  10358248322821333959
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-20-1918    173M      -     2.90T   3.37T  11396114093498370378
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-20-2017      0B      -     2.19T   2.64T  15065966855315306501
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-20-2117      0B      -     2.19T   2.64T  8328899955974102268
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-20-2217      0B      -     2.19T   2.64T  1657122139991726938
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-20-2317      0B      -     2.19T   2.64T  11471628937756454123
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-21-0017      0B      -     2.19T   2.64T  11846511501685479153
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-21-0117      0B      -     2.19T   2.64T  2973001046682453498
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-21-0217      0B      -     2.19T   2.64T  4463158981212865079
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-21-0317      0B      -     2.19T   2.64T  15126046486804738953
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-21-0507      0B      -     2.19T   2.64T  10243722194683015519
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-21-0541      0B      -     2.19T   2.64T  17223118383772674624
monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-21-0627      0B      -     2.19T   2.64T  14809183698294611028

...huh, the snapshot shows 432K changed...

# zfs diff workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-2017
#

...but no diff.
Okay, let's roll back...

# zfs rollback workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-2017
# zfs list -o name,used,lused,refer,lrefer,guid -t all -r workspace/urbackup_zstd | tail -n 5; zfs list -o name,used,lused,refer,lrefer,guid -t all -r monolith/sorted/backups/urbackup | grep -A20 zfs-auto-snap_daily-2021-09-20-1632;
workspace/urbackup_zstd@zfs-auto-snap_daily-2021-09-20-1632       0B      -     2.79T   3.39T  10176713998951144728
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-1717   38.6M      -     2.79T   3.39T  5443784001748964352
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-1818   15.0M      -     2.79T   3.39T  10358248322821333959
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-1918    183M      -     2.78T   3.37T  11396114093498370378
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-2017    432K      -     2.08T   2.64T  15065966855315306501

...what.

Then I sighed, and wildly guessed that something was in a queue being processed when it was mounted.

# zfs umount workspace/urbackup_zstd
# zfs rollback workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-2017
# zfs list -o name,used,lused,refer,lrefer,guid -t all -r workspace/urbackup_zstd | tail -n 5;
workspace/urbackup_zstd@zfs-auto-snap_daily-2021-09-20-1632       0B      -     2.79T   3.39T  10176713998951144728
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-1717   38.6M      -     2.79T   3.39T  5443784001748964352
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-1818   15.0M      -     2.79T   3.39T  10358248322821333959
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-1918    183M      -     2.78T   3.37T  11396114093498370378
workspace/urbackup_zstd@zfs-auto-snap_hourly-2021-09-20-2017      0B      -     2.08T   2.64T  15065966855315306501
# 

...okay...let's try again.

# zfs send -RLe -I monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-20-2017 monolith/sorted/backups/urbackup@zfs-auto-snap_hourly-2021-09-21-0317 | mbuffer -m 2G | zfs recv -o compress=zstd -o checksum=skein -o com.sun:auto-snapshot=false workspace/urbackup_zstd
summary: 17.0 kiByte in  1.5sec - average of 11.2 kiB/s
# 

Describe how to reproduce the problem

My speculation is inline above.

Include any warning/errors/backtraces from the system logs

...

[riyad]

This is definitely surprising! I had this happen to me on different occasions and always dismissed it as "probably something silly my distro does".

The annoying part is this only happens (or becomes visible ❗) when you have datasets that definitely should not have data written to them on top of snapshots (e.g. datasets that are only received into).
I didn't investigate further then and learned to live with workarounds. 😐

My workaround is to always rollback to the last snapshot before receiving. It's stupid but it works. 🤡

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[rincebrain]

Noticed this breaking again at one point recently, saved the snapshots around it happening.

It appears to be something very strange happening.

Specifically, if I look at the received dataset before I mount it:
Deadlist: 0 (0/0 comp), 4 entries
...and after?
Deadlist: 594M (297M/297M comp), 4 entries

So I'm guessing we've got a delete queue captured in here somewhere that's getting played back on mount...

[rincebrain]

...well, that was easy.

before mount:

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         3    3   128K    16K   776M     512   772M   99.94  ZFS delete queue (K=inherit) (Z=inherit=uncompressed)
        dnode flags: USED_BYTES USERUSED_ACCOUNTED
        dnode maxblkid: 49430
        Fat ZAP stats:
                Pointer table:
                        65536 elements
                        zt_blk: 32800
                        zt_numblks: 32
                        zt_shift: 16
                        zt_blks_copied: 0
                        zt_nextblk: 0
                ZAP entries: 6930102
                Leaf blocks: 49367
                Total blocks: 49431
                zap_block_type: 0x8000000000000001
                zap_magic: 0x2f52ab2ab
                zap_salt: 0xcd1f1bcc7
                Leafs with 2^n pointers:
                          0:  33198 ****************************************
                          1:  16169 ********************
                Blocks with n*5 entries:
                          9:  49367 ****************************************
                Blocks n/10 full:
                          4:      3 *
                          5:  16240 ***************************************
                          6:  16891 ****************************************
                          7:    202 *
                          8:   4222 **********
                          9:  11809 ****************************
                Entries with n chunks:
                          3: 6930102 ****************************************
                Buckets with n entries:
                          0: 19441728 ****************************************
                          1: 4852618 **********
                          2: 873774 **
                          3: 101453 *
                          4:   6081 *
                          5:    247 *
                          6:      3 *

after mount and waiting a couple minutes:

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         3    3   128K    16K   751M     512   772M   99.94  ZFS delete queue
        dnode flags: USED_BYTES USERUSED_ACCOUNTED
        dnode maxblkid: 49430
        Fat ZAP stats:
                Pointer table:
                        65536 elements
                        zt_blk: 32800
                        zt_numblks: 32
                        zt_shift: 16
                        zt_blks_copied: 0
                        zt_nextblk: 0
                ZAP entries: 6478534
                Leaf blocks: 49367
                Total blocks: 49431
                zap_block_type: 0x8000000000000001
                zap_magic: 0x2f52ab2ab
                zap_salt: 0xcd1f1bcc7
                Leafs with 2^n pointers:
                          0:  33198 ****************************************
                          1:  16169 ********************
                Blocks with n*5 entries:
                          0:   3201 ***
                          1:      0
                          2:      0
                          3:      0
                          4:      0
                          5:      0
                          6:      0
                          7:      1 *
                          8:      0
                          9:  46165 ****************************************
                Blocks n/10 full:
                          1:   3201 *********
                          2:      1 *
                          3:      0
                          4:      3 *
                          5:  15215 ***************************************
                          6:  15788 ****************************************
                          7:    190 *
                          8:   3949 ***********
                          9:  11020 ****************************
                Entries with n chunks:
                          3: 6478534 ****************************************
                Buckets with n entries:
                          0: 19821803 ****************************************
                          1: 4536492 **********
                          2: 816922 **
                          3:  94782 *
                          4:   5675 *
                          5:    228 *
                          6:      2 *

and much later:

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         3    3   128K    16K   391M     512   772M   99.94  ZFS delete queue
        dnode flags: USED_BYTES USERUSED_ACCOUNTED
        dnode maxblkid: 49430
        Fat ZAP stats:
                Pointer table:
                        65536 elements
                        zt_blk: 32800
                        zt_numblks: 32
                        zt_shift: 16
                        zt_blks_copied: 0
                        zt_nextblk: 0
                ZAP entries: 0
                Leaf blocks: 49367
                Total blocks: 49431
                zap_block_type: 0x8000000000000001
                zap_magic: 0x2f52ab2ab
                zap_salt: 0xcd1f1bcc7
                Leafs with 2^n pointers:
                          0:  33198 ****************************************
                          1:  16169 ********************
                Blocks with n*5 entries:
                          0:  49367 ****************************************
                Blocks n/10 full:
                          1:  49367 ****************************************
                Entries with n chunks:
                Buckets with n entries:
                          0: 25275904 ****************************************

So it appears to be evicting a 7 million(!) element delete queue, if I can read? Spicy.

Wonder if there's some cutoff where it assumes you shouldn't bother blocking on the delete queue draining and just takes the snapshot, so you don't hit this most times...

[digitalsignalperson]

Linking in the discussion from here #15853

• I'm consistently seeing data written immediately after mounting received snapshots
• the diff between snapshots shows + /mnt/0010(on_delete_queue) or M /mnt/0010(on_delete_queue) for a xpool/enc/0010 dataset.
• the data is not written if I temporarily zfs set readonly=on before mounting, and zfs set readonly=off after mounting

Edit: actually don't see that "on_delete_queue" in the zfs diff in more than one place

[digitalsignalperson]

Here's a reproducer. I'm using a clean VM per incus-arch-zfs.sh. It only reproduces when using encryption.

dd if=/dev/zero of=/root/zpool bs=1M count=128
zpool create testpool /root/zpool -m /mnt/zpool
echo "12345678" | zfs create -o canmount=off -o encryption=on -o keylocation=prompt -o keyformat=passphrase testpool/enc
zfs create testpool/enc/data1
zfs snapshot -r testpool/enc@1
zfs send -Rw testpool/enc@1 | zfs recv testpool/enc_copy
echo "12345678" | zfs load-key testpool/enc_copy
zfs mount -a
zfs get written -t filesystem

NAME                     PROPERTY  VALUE    SOURCE
testpool                 written   25K      -
testpool/enc             written   0        -
testpool/enc/data1       written   0        -
testpool/enc_copy        written   0        -
testpool/enc_copy/data1  written   85K      -

# dump this written data for analysis
zfs snapshot testpool/enc_copy/data1@2
zfs send -I @1 testpool/enc_copy/data1@2 | zstream dump -d > data1.txt

So simply receiving an empty dataset and mounting it results in data being written when mounting the received dataset. Here's the zstream dump of the written data data1.txt

Maybe this object it contains is relevent?

OBJECT object = 34 type = 20 bonustype = 44 blksz = 512 bonuslen = 176 dn_slots = 1 raw_bonuslen = 0 flags = 0 maxblkid = 0 indblkshift = 0 nlevels = 0 nblkptr = 0
 5a 50 2f 00  02 04 18 00  ed 41 00 00  00 00 00 00   ZP/. .... .A.. ....
 02 00 00 00  00 00 00 00  0b 00 00 00  00 00 00 00   .... .... .... ....
 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   .... .... .... ....
 22 00 00 00  00 00 00 00  44 01 00 00  08 84 00 00   "... .... D... ....
 38 6f c1 65  00 00 00 00  3a c9 66 07  00 00 00 00   8o.e .... :.f. ....
 38 6f c1 65  00 00 00 00  3a c9 66 07  00 00 00 00   8o.e .... :.f. ....
 38 6f c1 65  00 00 00 00  3a c9 66 07  00 00 00 00   8o.e .... :.f. ....
 38 6f c1 65  00 00 00 00  3a c9 66 07  00 00 00 00   8o.e .... :.f. ....
 02 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   .... .... .... ....
 03 00 00 00  00 00 00 00  00 00 00 10  ff 01 1e 00   .... .... .... ....
 00 00 40 20  a9 00 12 00  00 00 00 40  a9 00 12 00   ..@  .... ...@ ....

The issue doesn't reproduce if not using encryption

zpool destroy testpool
rm /root/zpool
dd if=/dev/zero of=/root/zpool bs=1M count=128
zpool create testpool /root/zpool -m /mnt/zpool
zfs create -o canmount=off testpool/nenc
zfs create testpool/nenc/data1
zfs snapshot -r testpool/nenc@1
zfs send -Rw testpool/nenc@1 | zfs recv testpool/nenc_copy
zfs mount -a
zfs get written -t filesystem

zfs get written -t filesystem
NAME                      PROPERTY  VALUE    SOURCE
testpool                  written   24K      -
testpool/nenc             written   0        -
testpool/nenc/data1       written   0        -
testpool/nenc_copy        written   0        -
testpool/nenc_copy/data1  written   0        -

And to demo if readonly=on during mount:

dd if=/dev/zero of=/root/zpool bs=1M count=128
zpool create testpool /root/zpool -m /mnt/zpool
echo "12345678" | zfs create -o canmount=off -o encryption=on -o keylocation=prompt -o keyformat=passphrase testpool/enc
zfs create testpool/enc/data1
zfs snapshot -r testpool/enc@1
zfs send -Rw testpool/enc@1 | zfs recv -o readonly=on testpool/enc_copy
echo "12345678" | zfs load-key testpool/enc_copy
zfs mount -a
zfs get written -t filesystem
NAME                     PROPERTY  VALUE    SOURCE
testpool                 written   24K      -
testpool/enc             written   0        -
testpool/enc/data1       written   0        -
testpool/enc_copy        written   0        -
testpool/enc_copy/data1  written   0        -
zfs set readonly=off testpool/enc_copy
zfs umount -a
zfs mount -a
zfs get written -t filesystem
NAME                     PROPERTY  VALUE    SOURCE
testpool                 written   25K      -
testpool/enc             written   0        -
testpool/enc/data1       written   0        -
testpool/enc_copy        written   0        -
testpool/enc_copy/data1  written   85K      -

[digitalsignalperson]

16 Exabytes written for a moment?

zfs rollback xpool/enc/0010@test
zfs get written xpool/enc/0010
zfs umount xpool/enc/0010
zfs mount xpool/enc/0010
for i in (seq 1 15)
    echo "t=$i"
    zfs get written xpool/enc/0010
    sleep 1
end

NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=1
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=2
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=3
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=4
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=5
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=6
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   16.0E    -         <----------------- woah
t=7
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   645M     -
t=8
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   787M     -
t=9
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   787M     -
t=10
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   787M     -
t=11
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   787M     -
t=12
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   787M     -
t=13
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   787M     -
t=14
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   787M     -
t=15
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   787M     -

Can we capture a snapshot with those exabytes? nope

zfs rollback xpool/enc/0010@test
zfs get written xpool/enc/0010
zfs umount xpool/enc/0010
zfs mount xpool/enc/0010
for i in (seq 1 15)
    echo "t=$i"
    zfs get written xpool/enc/0010
    zfs snapshot xpool/enc/0010@t$i
    sleep 1
end

NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=1
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=2
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=3
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=4
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=5
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=6
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=7
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=8
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   13M      -                <------------ curious tho
t=9
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=10
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=11
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=12
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=13
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=14
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=15
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -

zfs list -t snapshot xpool/enc/0010
NAME                                    USED  AVAIL  REFER  MOUNTPOINT
...
xpool/enc/0010@test                     312K      -  6.35G  -
xpool/enc/0010@t1                       176K      -  6.35G  -
xpool/enc/0010@t2                       144K      -  6.35G  -
xpool/enc/0010@t3                       120K      -  6.35G  -
xpool/enc/0010@t4                       136K      -  6.35G  -
xpool/enc/0010@t5                       120K      -  6.35G  -
xpool/enc/0010@t6                       104K      -  6.35G  -
xpool/enc/0010@t7                        88K      -  6.35G  -
xpool/enc/0010@t8                         0B      -  6.35G  -
xpool/enc/0010@t9                         0B      -  6.35G  -
xpool/enc/0010@t10                        0B      -  6.35G  -
xpool/enc/0010@t11                        0B      -  6.35G  -
xpool/enc/0010@t12                        0B      -  6.35G  -
xpool/enc/0010@t13                        0B      -  6.35G  -
xpool/enc/0010@t14                        0B      -  6.35G  -
xpool/enc/0010@t15                        0B      -  6.35G  -

from here written stays zero. it I touch a file it will increase to 248K. which is an improvement from 787M

touch /mnt/0010/test
echo "touched"
for i in (seq 1 15)
    echo "t=$i"
    zfs get written xpool/enc/0010
    sleep 1
end

touched
t=1
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=2
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=3
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   0        -
t=4
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=5
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=6
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=7
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=8
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=9
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=10
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=11
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=12
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=13
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=14
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -
t=15
NAME            PROPERTY  VALUE    SOURCE
xpool/enc/0010  written   248K     -

[rincebrain]

I would assume it's that encrypted send/recv means you have to throw out and recalculate all the quota metadata on first mount or something in that vein.

[digitalsignalperson]

Darn I though I had a work around, but this "extra" data being generated really wants to stick around. This has potential for a significant (or explosive runaway) amount of storage space being wasted. (Still specifically testing with raw send/receive, where I haven't encountered the issue with non-encrypted).

In a VM test setup I created 128k files, each 8kB in size, captured in one snapshot. After incremental send and receive, 128MB is written within a few seconds after the next time mounting it. For 1024 files 1MB in size it was ~1.3MB (seems to scale linearly with number of files, not file size). This still an order of magnitude better than my real world example where I had a ~300MB snapshot with ~11k files created, ~9k deleted, 600 modified, that resulted in 648M extra data written after snapshot receive/mount. Maybe there is some difference related to compression, which I kept disabled in the VM.

In the VM setup after that initial 128MB encounter, passing nearly empty (touched a file) snapshots back and forth a couple times, from an initial 1.1G REFER, the USED quickly grows to 1.3G, 1.4G, 1.5... the extra 128MB seeming to apply every time. For this reason I think I must assume raw encrypted send is a strictly one-way/read-only trip until the issue is resolved.

To not make this a 500 line reply, I dumped my notes and code here: https://gist.github.com/digitalsignalperson/709e2adf847473c5301d67644d4f4e19

Maybe there is some use to seeing the extra data to trace the source. For the 131072 ("128k") files, there's 131078 "OBJECT" lines in the extra data's zstream dump, and 131073 of them start with "ZP/"