Skip to content

Pin the version of cockroach for tests to a specific version#2971

Merged
mhutchinson merged 1 commit intogoogle:masterfrom
mhutchinson:pinnedRoach
Apr 18, 2023
Merged

Pin the version of cockroach for tests to a specific version#2971
mhutchinson merged 1 commit intogoogle:masterfrom
mhutchinson:pinnedRoach

Conversation

@mhutchinson
Copy link
Copy Markdown
Contributor

./storage/crdb tests have started failing in GCP recently. Investigation shows that this is pulling the latest tagged version, which breaks hermetic builds. This PR is a test to see if using the previous version fixes the issue, and if so, I propose merging this to unblock things and then discussing with the community to look for better solutions.

./storage/crdb tests have started failing in GCP recently. Investigation shows that this is pulling the latest tagged version, which breaks hermetic builds. This PR is a test to see if using the previous version fixes the issue, and if so, I propose merging this to unblock things and then discussing with the community to look for better solutions.
@mhutchinson mhutchinson requested a review from a team as a code owner April 18, 2023 15:16
@mhutchinson
Copy link
Copy Markdown
Contributor Author

@JAORMX FYI

@mhutchinson
Copy link
Copy Markdown
Contributor Author

Failing snippet for those that want the gory details:

...
2023/04/17 22:02:29 waiting for download of /tmp/cockroach-22-2-8
2023/04/17 22:02:29 waiting for download of /tmp/cockroach-22-2-8
2023/04/17 22:02:29 Using automatically-downloaded binary: /tmp/cockroach-22-2-8
2023/04/17 22:02:30 executing: /tmp/cockroach-22-2-8 start-single-node --logtostderr --insecure --host=localhost --port=0 --http-port=0 --store=type=mem,size=0.20 --listening-url-file=/tmp/cockroach-testserver898281996/0/listen-url --external-io-dir=disabled
2023/04/17 22:02:30 process 10042 started. env=[COCKROACH_MAX_OFFSET=1ns COCKROACH_TRUST_CLIENT_PROVIDED_SQL_REMOTE_ADDR=true GID=0 HOME=/root PATH=/usr/local/go/bin:/go/bin:/testbase/protoc/bin:/usr/local/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin USER=root UID=0]; cmd: /tmp/cockroach-22-2-8 start-single-node --logtostderr --insecure --host=localhost --port=0 --http-port=0 --store=type=mem,size=0.20 --listening-url-file=/tmp/cockroach-testserver898281996/0/listen-url --external-io-dir=disabled
W0417 22:02:31.310299    9780 tree_storage.go:86] failed verifying database connection: missing "=" after "&bogus*:::?" in connection info string"
W0417 22:02:44.230324    9780 tree_storage.go:363] TX commit error: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/732/2/1): "sql txn" meta={id=222e8833 key=/Table/616/1/7040283176384859160/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.00107903 epo=0 ts=1681768961.179821613,2 min=1681768956.877110741,0 seq=3000} lock=true stat=PENDING rts=1681768956.877110741,0 wto=false gul=1681768957.377110741,0, stack:
goroutine 84 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x72
github.com/google/trillian/storage/crdb.(*treeTX).Commit(0xc000185a20, {0x1184a10, 0xc00003e068})
	/workspace/storage/crdb/tree_storage.go:363 +0x305
github.com/google/trillian/storage/crdb.(*crdbLogStorage).QueueLeaves(0x11861e8?, {0x1184a10, 0xc00003e068}, 0x0?, {0xc000834800, 0x3e8, 0x500}, {0x0, 0xecfb6864b, 0x0})
	/workspace/storage/crdb/log_storage.go:281 +0x1a8
github.com/google/trillian/storage/crdb.TestQueueLeavesDuplicateBigBatch(0xc00048f860)
	/workspace/storage/crdb/log_storage_test.go:246 +0x418
testing.tRunner(0xc00048f860, 0x106f730)
	/usr/local/go/src/testing/testing.go:1446 +0x217
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1493 +0x75e
--- FAIL: TestQueueLeavesDuplicateBigBatch (12.92s)
    log_storage_test.go:247: Failed to queue leaves: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/732/2/1): "sql txn" meta={id=222e8833 key=/Table/616/1/7040283176384859160/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.00107903 epo=0 ts=1681768961.179821613,2 min=1681768956.877110741,0 seq=3000} lock=true stat=PENDING rts=1681768956.877110741,0 wto=false gul=1681768957.377110741,0
FAIL
I0417 22:02:48.610299    9780 common_test.go:70] Cleaning up database for test TestCRDBAdminStorage
I0417 22:02:48.942298    9780 common_test.go:70] Cleaning up database for test TestNodeRoundTrip/store-4-read-4
...

@pavelkalinnikov in case this is interesting to you. No expectation or pressure :-)

@pav-kv
Copy link
Copy Markdown
Contributor

pav-kv commented Apr 20, 2023

@pavelkalinnikov in case this is interesting to you. No expectation or pressure :-)

@mhutchinson There are a couple of CRDB issues with similar errors:

See if your symptoms match, or transaction does anything similar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants