zpool import -T is broken

[ryao]

zpool import -T is broken. Here are simple steps to reproduce:

truncate -s 1G /tmp/zfs
zpool create test /tmp/zfs
zpool export test
# This should show that the highest transaction group is 8.
zdb -lu /tmp/zfs 
zpool import -d /tmp test
zpool export test
# This should show that the transaction group increased
zdb -lu /tmp/zfs 
zpool import -T 8 -d /tmp test

The output will be cannot import 'test': one or more devices is currently unavailable. This is a problem when -F is needed and does not work. I am filing this as a note to myself to fix this.

[FransUrbo]

@ryao Any new on this?

[FransUrbo]

Comparing with Illumos, it almost seems like it's broken there as well... ?

[FransUrbo]

Adding a couple of fprintf()s, it seems like it's the

while ((ret = zfs_ioctl(hdl, ZFS_IOC_POOL_IMPORT, &zc)) != 0 && errno == ENOMEM)

in libzfs_pool.c:zpool_import_props() that fails. We never enter the loop, so apparently the first ioctl fails...

[FransUrbo]

This seems to affect zdb as well:

# zdb -e -t 8 -p /tmp test3

Configuration for import:
        vdev_children: 1
        version: 5000
        pool_guid: 18071166951262728880
        name: 'test3'
        state: 1
        hostid: 2831168069
        hostname: 'DebianZFS-Wheezy-SCST2'
        vdev_tree:
            type: 'root'
            id: 0
            guid: 18071166951262728880
            children[0]:
                type: 'file'
                id: 0
                guid: 2863442196652816775
                metaslab_array: 34
                metaslab_shift: 23
                ashift: 9
                asize: 1069023232
                is_log: 0
                create_txg: 4
                path: '/tmp/zfs'
zdb: can't open 'test3': No such device or address

[FransUrbo]

Further investigation:

if (nv && nvlist_lookup_nvlist(nv,
       ZPOOL_CONFIG_LOAD_INFO, &nvinfo) == 0 &&
       nvlist_lookup_nvlist(nvinfo,
       ZPOOL_CONFIG_MISSING_DEVICES, &missing) == 0) {

Only the first and second part is TRUE, the ZPOOL_CONFIG_MISSING_DEVICES is not. So obviously no missing devices... Which we already knew.

[FransUrbo]

strace:

open("/tmp/zfs", O_RDONLY)              = 7
close(7)                                = 0
ioctl(3, 0x5a05, 0x7fff7c870b60)        = -1 ENOENT (No such file or directory)
ioctl(3, 0x5a06, 0x7fff7c870bd0)        = -1 ENOMEM (Cannot allocate memory)
ioctl(3, 0x5a06, 0x7fff7c870bd0)        = 0
ioctl(3, 0x5a02, 0x7fff7c871ce0)        = -1 ENXIO (No such device or address)
write(2, "cannot import 'test3': one or mo"..., 68cannot import 'test3': one or more devices is currently unavailable
) = 68

[FransUrbo]

WEIRD!! I managed to get it imported by modifying the test case a little. Instead of creating the file in /tmp, I created it in /var/tmp and made sure it was the only entry/file/dir in there. Then ran it through strace (might not be significant). And it worked!

Exporting the pool and trying again, gave the usual message!

No, strace was not significant:

# /tmp/test_import.sh 
cannot import 'test3': no such pool available
# /usr/local/sbin/zpool import -T 8 -d /var/tmp test3
# zpool export test3
# /usr/local/sbin/zpool import -T 8 -d /var/tmp test3
cannot import 'test3': one or more devices is currently unavailable
# 

My test script is only slightly different from yours:

#!/bin/sh                                                                                                                                                     

set -e

cd /usr/src/zfs

rm -f /var/tmp/zfs*
truncate -s 1G /var/tmp/zfs1

./cmd/zpool/zpool create test3 /var/tmp/zfs1
./cmd/zpool/zpool export test3

./cmd/zpool/zpool import -d /tmp/var test3
./cmd/zpool/zpool export test3

./cmd/zpool/zpool import -T 8 -d /var/tmp test3

[FransUrbo]

Oups! /tmp/var isn't quite right :). Fixing that, I'm back to not being able to import.

WIthout the first import, I can get it to import. Is this significant?!

[FransUrbo]

After finally figured out how to get the dprintf messages, I managed to see that it was spa.c:spa_import(). The line

        error = spa_load_best(spa, state, B_TRUE, policy.zrp_txg,
            policy.zrp_request);

Returns 6.

[FransUrbo]

And in spa.c:spa_load_best() it's the

        load_error = rewind_error = spa_load(spa, state, SPA_IMPORT_EXISTING, mosconfig);

that sets load_error=6.

[FransUrbo]

And in spa.c:spa_load()

                error = spa_load_impl(spa, pool_guid, config, state, type, mosconfig, &ereport);

[FransUrbo]

EDIT: It's in spa.c:spa_load_impl() that the code block

        /*                                                                                                                                                    
         * Find the best uberblock.                                                                                                                           
         */
        vdev_uberblock_load(rvd, ub, &label);

        /*
         * If we weren't able to find a single valid uberblock, return failure.
         */
        if (ub->ub_txg == 0) {
                nvlist_free(label);
                return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, ENXIO));
        }

that gets triggered...

[FransUrbo]

And in vdev_label.c:vdev_uberblock_load()

        /*                                                                                                                                                    
         * It's possible that the best uberblock was discovered on a label                                                                                    
         * that has a configuration which was written in a future txg.                                                                                        
         * Search all labels on this vdev to find the configuration that                                                                                      
         * matches the txg for our uberblock.                                                                                                                 
         */
        if (cb.ubl_vd != NULL)
                *config = vdev_label_read_config(cb.ubl_vd, ub->ub_txg);
        spa_config_exit(spa, SCL_ALL, FTAG);

that fails (the vdev_label_read_config() part)

[FransUrbo]

In vdev_label.c:vdev_label_read_config() it seems like it's

                        /*                                                                                                                                    
                         * Auxiliary vdevs won't have txg values in their                                                                                     
                         * labels and newly added vdevs may not have been                                                                                     
                         * completely initialized so just return the                                                                                          
                         * configuration from the first valid label we                                                                                        
                         * encounter.                                                                                                                         
                         */
                        error = nvlist_lookup_uint64(label, ZPOOL_CONFIG_POOL_TXG, &label_txg);

nvlist_lookup_uint64() returns 0, but label_txg is NULL..