Skip to content

Bump CockroachDB to 22.2.17#3301

Merged
roger2hk merged 2 commits intogoogle:masterfrom
roger2hk:fix-crdb-test
Jan 19, 2024
Merged

Bump CockroachDB to 22.2.17#3301
roger2hk merged 2 commits intogoogle:masterfrom
roger2hk:fix-crdb-test

Conversation

@roger2hk
Copy link
Copy Markdown
Contributor

Fixes #3300.

Checklist

@roger2hk roger2hk changed the title Bump CockroachDB to 23.1.13 Bump CockroachDB to 22.2.17 Jan 18, 2024
@JAORMX
Copy link
Copy Markdown
Contributor

JAORMX commented Jan 18, 2024

=== RUN   TestLogSuite/DequeueLeavesNoneQueued
W0118 01:18:25.752295    3636 tree_storage.go:375] TX commit error: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/474/2/1): "sql txn" meta={id=392f48e9 key=/Table/421/1/6041628873873342746/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.01106136 epo=0 ts=1705540702.707518406,2 min=1705540701.649429217,0 seq=3000} lock=true stat=PENDING rts=1705540701.649429217,0 wto=false gul=1705540702.149429217,0, stack:
goroutine 50 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.21.6/x64/src/runtime/debug/stack.go:24 +0x5e
github.com/google/trillian/storage/crdb.(*treeTX).Commit(0xc000201080, {0xd36db0, 0x123c280})
	/home/runner/work/trillian/trillian/storage/crdb/tree_storage.go:375 +0x22d
github.com/google/trillian/storage/crdb.(*crdbLogStorage).QueueLeaves(0xd38328?, {0xd36db0, 0x123c280}, 0x0?, {0xc00099c000, 0x3e8, 0x500}, {0x0, 0xecfb6864b, 0x0})
	/home/runner/work/trillian/trillian/storage/crdb/log_storage.go:301 +0x225
github.com/google/trillian/storage/crdb.TestQueueLeavesDuplicateBigBatch(0xc000b56340)
	/home/runner/work/trillian/trillian/storage/crdb/log_storage_test.go:246 +0x250
testing.tRunner(0xc000b56340, 0xc74738)
	/opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1648 +0x3ad
=== NAME  TestQueueLeavesDuplicateBigBatch
    log_storage_test.go:247: Failed to queue leaves: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/474/2/1): "sql txn" meta={id=392f48e9 key=/Table/421/1/6041628873873342746/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.01106136 epo=0 ts=1705540702.707518406,2 min=1705540701.649429217,0 seq=3000} lock=true stat=PENDING rts=1705540701.649429217,0 wto=false gul=1705540702.149429217,0
--- FAIL: TestQueueLeavesDuplicateBigBatch (4.95s)

🤔

@roger2hk
Copy link
Copy Markdown
Contributor Author

=== RUN   TestLogSuite/DequeueLeavesNoneQueued
W0118 01:18:25.752295    3636 tree_storage.go:375] TX commit error: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/474/2/1): "sql txn" meta={id=392f48e9 key=/Table/421/1/6041628873873342746/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.01106136 epo=0 ts=1705540702.707518406,2 min=1705540701.649429217,0 seq=3000} lock=true stat=PENDING rts=1705540701.649429217,0 wto=false gul=1705540702.149429217,0, stack:
goroutine 50 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.21.6/x64/src/runtime/debug/stack.go:24 +0x5e
github.com/google/trillian/storage/crdb.(*treeTX).Commit(0xc000201080, {0xd36db0, 0x123c280})
	/home/runner/work/trillian/trillian/storage/crdb/tree_storage.go:375 +0x22d
github.com/google/trillian/storage/crdb.(*crdbLogStorage).QueueLeaves(0xd38328?, {0xd36db0, 0x123c280}, 0x0?, {0xc00099c000, 0x3e8, 0x500}, {0x0, 0xecfb6864b, 0x0})
	/home/runner/work/trillian/trillian/storage/crdb/log_storage.go:301 +0x225
github.com/google/trillian/storage/crdb.TestQueueLeavesDuplicateBigBatch(0xc000b56340)
	/home/runner/work/trillian/trillian/storage/crdb/log_storage_test.go:246 +0x250
testing.tRunner(0xc000b56340, 0xc74738)
	/opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1648 +0x3ad
=== NAME  TestQueueLeavesDuplicateBigBatch
    log_storage_test.go:247: Failed to queue leaves: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/474/2/1): "sql txn" meta={id=392f48e9 key=/Table/421/1/6041628873873342746/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.01106136 epo=0 ts=1705540702.707518406,2 min=1705540701.649429217,0 seq=3000} lock=true stat=PENDING rts=1705540701.649429217,0 wto=false gul=1705540702.149429217,0
--- FAIL: TestQueueLeavesDuplicateBigBatch (4.95s)

🤔

As go test runs concurrently and both of the crdb testservers now use the same version, I doubt if both of testservers are sharing the same data file behind the scene causing the unexpected error.

@roger2hk roger2hk marked this pull request as ready for review January 18, 2024 18:39
@roger2hk roger2hk requested a review from a team as a code owner January 18, 2024 18:39
@roger2hk
Copy link
Copy Markdown
Contributor Author

The default configuration of testserver stores the data in memory. I suspected that's the root cause failing the tests when two testservers are concurrently reading/writing in the same memory database. Thus, I've added testserver.StoreOnDiskOpt() into the latest commit to store the test data in two different on-disk database. The database is saved at /tmp/cockroach-testserverxxxxxxxx, with randomly generated xxxxxxxx postfix.

@JAORMX
Copy link
Copy Markdown
Contributor

JAORMX commented Jan 19, 2024

@roger2hk that makes sense. Thanks for taking a look!

@roger2hk roger2hk merged commit 2f829cf into google:master Jan 19, 2024
@roger2hk roger2hk deleted the fix-crdb-test branch January 19, 2024 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CRDB Check Broken due to missing the latest (v23.1.14) binary

3 participants