omicron-dev -- db-run not working on Helios

[smklein]

Since #427 landed, if I attempt to run:

$ cargo run --bin=omicron-dev -- db-run

I see the following:

   Compiling omicron-test-utils v0.1.0 (/home/smklein/omicron/test-utils)
    Finished dev [unoptimized + debuginfo] target(s) in 5.49s
     Running `target/debug/omicron-dev db-run`
omicron-dev: using temporary directory for database store (cleaned up on clean exit)
omicron-dev: will run this to start CockroachDB:
cockroach start-single-node --insecure --http-addr=:0 --store /tmp/.tmpPAeMXd/data --listen-addr 127.0.0.1:32221 --listening-url-file /tmp/.tmpPAeMXd/listen-url
omicron-dev: temporary directory: /tmp/.tmpPAeMXd
omicron-dev: wrong version of CockroachDB installed. expected 'v21.1.10', found: 'Err(error Some(4) when checking CockroachDB version)

This is true even after re-running the ./tools/ci_download_cockroachdb script.

[smklein]

FYI @teisenbe

[teisenbe]

I can go take a look, sorry about that

[teisenbe]

(I'll revert the version check for now)

[smklein]

No worries, I'll see if I can scrape together some extra info for debugging:

Running ./tools/ci_download_cockroachdb eventually emits the following:

Build Tag:        v20.2.5
Build Time:       2021/03/17 21:00:51
Distribution:     OSS
Platform:         illumos amd64 (x86_64-pc-solaris2.11)
Go Version:       go1.16.2
C Compiler:       gcc 9.3.0
Build Commit ID:  162c5ac4968cf31c0ed54cd29aa8aeccd66247bb
Build Type:       release

Notably, the build tag is not 21.1.10 - it's 20.2.5.

Also, running: $ ./cockroachdb/bin/cockroach version --build-tag shows the following:

Usage:
  cockroach version [flags]

Flags:
  -h, --help   help for version

Global Flags:
      --logtostderr Severity[=DEFAULT]   logs at or above this threshold go to stderr (default NONE)
      --no-color                         disable standard error log colorization
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging (significantly hurts performance)

ERROR: unknown flag: --build-tag
Failed running "version"

... so, although the check might be sorta broken right now, there is actually a mismatch in CRDB versions here!

[teisenbe]

oh, huh, I wonder if build tag is a new argument in CRDB

[teisenbe]

Oh, huh. Off by just one minor rev:
https://www.cockroachlabs.com/docs/releases/v20.2.6#command-line-changes. So it looks like the logic likely works, and Helios is just so far stale that the version check doesn't execute as expected

[teisenbe]

I can't reproduce over here. Running tools/ci_download_cockroachdb from main on my Helios VM gets me

Build Tag:        v21.1.10
Build Time:       2021/10/12 22:52:33
Distribution:     OSS
Platform:         illumos amd64 (x86_64-pc-solaris2.11)
Go Version:       go1.16.5
C Compiler:       gcc 10.3.0
Build Commit ID:  a6daab16ee8a1c15abc4c4a8a425e46d12033b5c
Build Type:       release

I'm not sure how it'd end up trying to fetch such an old version given how recent the version check logic is. Any chance you have local changes to tools/ci_download_cockroachdb? The error message from db-run suggests that tools/cockroachdb_version is present and correct

[davepacheco]

@smklein can you paste the full output of ci_download_cockroachdb? When I run it from commit 41c10d6 (tip of main) I get:

$ ./tools/ci_download_cockroachdb 
+ set -o errexit
+++ dirname ./tools/ci_download_cockroachdb
++ cd ./tools
++ pwd
+ SOURCE_DIR=/home/dap/omicron/tools
++ basename ./tools/ci_download_cockroachdb
+ ARG0=ci_download_cockroachdb
++ cat /home/dap/omicron/tools/cockroachdb_version
+ CIDL_VERSION=v21.1.10
+ source /home/dap/omicron/tools/cockroachdb_checksums
++ CIDL_MD5_DARWIN=cab1998101cc7138c52c312aff0776dc
++ CIDL_MD5_LINUX=d0aab99810ddf37b2f6d183077df262d
++ CIDL_MD5_ILLUMOS=499b7f9a80028c993b87808556c29425
+ CIDL_ASSEMBLE_DIR=./cockroachdb
+ CIDL_URL_COCKROACH=https://binaries.cockroachdb.com
+ CIDL_URL_ILLUMOS=https://illumos.org/downloads
+ main
+ [[ 0 != 0 ]]
+ CIDL_OS=solaris2.11
+ [[ 0 != 0 ]]
+ configure_os solaris2.11
+ echo 'current directory: /home/dap/omicron'
current directory: /home/dap/omicron
+ echo 'configuring based on OS: "solaris2.11"'
configuring based on OS: "solaris2.11"
+ case "$1" in
+ CIDL_BUILD=illumos
+ CIDL_SUFFIX=tar.gz
+ CIDL_MD5=499b7f9a80028c993b87808556c29425
+ CIDL_MD5FUNC=do_md5sum
+ CIDL_URL_BASE=https://illumos.org/downloads
+ CIDL_ASSEMBLE=do_assemble_illumos
+ CIDL_DIR=cockroach-v21.1.10.illumos
+ CIDL_FILE=cockroach-v21.1.10.illumos.tar.gz
+ CIDL_URL=https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
+ echo 'URL: https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz'
URL: https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
+ echo 'Local file: cockroach-v21.1.10.illumos.tar.gz'
Local file: cockroach-v21.1.10.illumos.tar.gz
+ do_download_curl https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz cockroach-v21.1.10.illumos.tar.gz
+ curl --silent --show-error --fail --location --output cockroach-v21.1.10.illumos.tar.gz https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
++ do_md5sum cockroach-v21.1.10.illumos.tar.gz
++ md5sum
++ awk '{print $1}'
+ calculated_md5=499b7f9a80028c993b87808556c29425
+ [[ 499b7f9a80028c993b87808556c29425 != \4\9\9\b\7\f\9\a\8\0\0\2\8\c\9\9\3\b\8\7\8\0\8\5\5\6\c\2\9\4\2\5 ]]
+ do_untar cockroach-v21.1.10.illumos.tar.gz
+ tar xzf cockroach-v21.1.10.illumos.tar.gz
+ do_assemble_illumos cockroach-v21.1.10.illumos
+ cp -r cockroach-v21.1.10 ./cockroachdb
+ ./cockroachdb/bin/cockroach version
Build Tag:        v21.1.10
Build Time:       2021/10/12 22:52:33
Distribution:     OSS
Platform:         illumos amd64 (x86_64-pc-solaris2.11)
Go Version:       go1.16.5
C Compiler:       gcc 10.3.0
Build Commit ID:  a6daab16ee8a1c15abc4c4a8a425e46d12033b5c
Build Type:       release

That looks right to me but it's different than you saw.

[smklein]

$ ./tools/ci_download_cockroachdb 
+ set -o errexit
+++ dirname ./tools/ci_download_cockroachdb
++ cd ./tools
++ pwd
+ SOURCE_DIR=/home/smklein/omicron/tools
++ basename ./tools/ci_download_cockroachdb
+ ARG0=ci_download_cockroachdb
++ cat /home/smklein/omicron/tools/cockroachdb_version
+ CIDL_VERSION=v21.1.10
+ source /home/smklein/omicron/tools/cockroachdb_checksums
++ CIDL_MD5_DARWIN=cab1998101cc7138c52c312aff0776dc
++ CIDL_MD5_LINUX=d0aab99810ddf37b2f6d183077df262d
++ CIDL_MD5_ILLUMOS=499b7f9a80028c993b87808556c29425
+ CIDL_ASSEMBLE_DIR=./cockroachdb
+ CIDL_URL_COCKROACH=https://binaries.cockroachdb.com
+ CIDL_URL_ILLUMOS=https://illumos.org/downloads
+ main
+ [[ 0 != 0 ]]
+ CIDL_OS=solaris2.11
+ [[ 0 != 0 ]]
+ configure_os solaris2.11
+ echo 'current directory: /home/smklein/omicron'
current directory: /home/smklein/omicron
+ echo 'configuring based on OS: "solaris2.11"'
configuring based on OS: "solaris2.11"
+ case "$1" in
+ CIDL_BUILD=illumos
+ CIDL_SUFFIX=tar.gz
+ CIDL_MD5=499b7f9a80028c993b87808556c29425
+ CIDL_MD5FUNC=do_md5sum
+ CIDL_URL_BASE=https://illumos.org/downloads
+ CIDL_ASSEMBLE=do_assemble_illumos
+ CIDL_DIR=cockroach-v21.1.10.illumos
+ CIDL_FILE=cockroach-v21.1.10.illumos.tar.gz
+ CIDL_URL=https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
+ echo 'URL: https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz'
URL: https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
+ echo 'Local file: cockroach-v21.1.10.illumos.tar.gz'
Local file: cockroach-v21.1.10.illumos.tar.gz
+ do_download_curl https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz cockroach-v21.1.10.illumos.tar.gz
+ curl --silent --show-error --fail --location --output cockroach-v21.1.10.illumos.tar.gz https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
++ do_md5sum cockroach-v21.1.10.illumos.tar.gz
++ md5sum
++ awk '{print $1}'
+ calculated_md5=499b7f9a80028c993b87808556c29425
+ [[ 499b7f9a80028c993b87808556c29425 != \4\9\9\b\7\f\9\a\8\0\0\2\8\c\9\9\3\b\8\7\8\0\8\5\5\6\c\2\9\4\2\5 ]]
+ do_untar cockroach-v21.1.10.illumos.tar.gz
+ tar xzf cockroach-v21.1.10.illumos.tar.gz
+ do_assemble_illumos cockroach-v21.1.10.illumos
+ cp -r cockroach-v21.1.10 ./cockroachdb
+ ./cockroachdb/bin/cockroach version
Build Tag:        v20.2.5
Build Time:       2021/03/17 21:00:51
Distribution:     OSS
Platform:         illumos amd64 (x86_64-pc-solaris2.11)
Go Version:       go1.16.2
C Compiler:       gcc 9.3.0
Build Commit ID:  162c5ac4968cf31c0ed54cd29aa8aeccd66247bb
Build Type:       release

[teisenbe]

Oh interesting. It tried downloading 21.1.10, but got 20.2.5? I wonder if the last bits of the script behave incorrectly on Helios

[teisenbe]

I'm gonna check out an old rev to get 20.2.5 and then go back to main and try to repro

Edit: Didn't even have to get that far. Just checking out an old rev is enough. The same issue happens, just that it fails to replace 21.1.10 with the older 20.2.5

[smklein]

Ugh, okay, I just rm -rf'd cockroachdb and re-ran, and I'm seeing the 21.1.10 version now. I'm still not sure why this is the case.

[leftwo]

I saw this last night on broadway.
If the ./cockroach directory exists when you run ./tools/ci_download_cockroachdb it will put the new version inside instead of replacing the existing version.

[teisenbe]

Just saw the same behavior leftwo describes. I'll send a patch for the script