Refactor: remove ckb-sdk-utils and repalce with lumos by zhangyouxin · Pull Request #2734 · nervosnetwork/neuron

zhangyouxin · 2023-06-20T10:04:57Z

Description

Due to merge of ckb-sdk-x to lumos, we need to gradually deprecate ckb-sdk-x, this PR removes ckb-sdk-utils in packages/neuron-wallet, It's not a small PR and may be not that easy to review, so I plan to seperate this task to two PRs, will create another PR to remove it in packages/neuron-ui

This PR is not not compilable, because I have updated lumos to 0.20, some snake_case fields has changed to camelCase, the task is in another PR: #2683, only when after that PR is merged, this PR can be ready for review.

Resolve Magickbase/neuron-public-issues#201

changed

scriptToHash(script) should be replaced with utils.computeScriptHash(script).
scriptToAddress(script, isMainet) should be replaced with helpers.encodeToAddress(script, lumosOptions). We should translate isMainet to lumosOptions here.
addressToScript(address) should be replaced with helpers.addressToScript(address, lumosOptions). We should detect if isMainet by checking if the address starts with 'ckb' or not. Alternatively, we could use NetworksService.getInstance().isMainnet().
src/models/blake2b.ts has been deleted and replaced with @ckb-lumos/hd and lumos helpers.
null values are not compatible with @ckb-lumos/base types.
Some test cases have been changed:
- args should be a valid hex string. Values like "0xuuid" have been deleted.
- Optional hex values should be 0x or undefined. Empty strings have been deleted.
- Hex strings are not hex numbers. Values like 0x1 have been changed to 0x01.

homura · 2023-06-27T07:23:37Z

@zhangyouxin the conflicts is needed to resolve

github-actions · 2023-06-27T13:54:17Z

Packaging for test is done in 5390477118 for commit 8e89627 .

github-actions · 2023-06-28T02:56:42Z

Packaging for test is done in 5396537909 for commit 7ffd05c .

github-actions · 2023-06-28T03:31:03Z

Packaging for test is done in 5396866191 for commit dd6a4de .

homura · 2023-06-28T06:25:08Z

    const outputsResult: [boolean, Output[], AnyoneCanPayInfo[]] = this.selectOutputs()
    const isMainnet = NetworksService.getInstance().isMainnet()
-    const outputAddresses: string[] = outputsResult[1].map(output => scriptToAddress(output.lock, isMainnet))
+    const lumosOptions = isMainnet ? { config: config.predefined.LINA } : { config: config.predefined.AGGRON4 }


there are so many isMainnet ? LINA : AGGRON4 in the PR, such a refactoring would only increase the complexity, maybe it is better to extract a helper to cover the case

Agree, we can init the config after ckb node starts and change the config when the network changes.

homura · 2023-06-28T06:28:11Z

a migration that has already been released should not be refactored, which may pose some risks

homura · 2023-06-28T06:32:06Z

+    const rawTransaction = this.toSDKRawTransaction()
+    return utils.ckbHash(
+      blockchain.RawTransaction.pack({
+        ...rawTransaction,
+        outputs: rawTransaction.outputs.map(o => ({
+          ...o,
+          type: o.type || undefined,
+        })),
+        cellDeps: rawTransaction.cellDeps.map(cd => ({
+          ...cd,
+          outPoint: cd.outPoint!,
+        })),
+        inputs: rawTransaction.inputs.map(i => ({
+          ...i,
+          previousOutput: i.previousOutput!,
+        })),
+      })
+    )


If it is found that refactoring increases complexity, this refactoring should not be done, and if there is no export method in lumos to calculate transaction hash, it should be marked as TODO instead of refactoring as such

homura · 2023-06-28T06:35:38Z

+    codeHash: config.predefined.LINA.SCRIPTS.DAO.CODE_HASH,
+    hashType: config.predefined.LINA.SCRIPTS.DAO.HASH_TYPE,


SystemScriptInfo may not same with the LINA's

The codeHashes of systemScripts are the same in mainnet/testnet/devnet

homura · 2023-06-28T06:36:15Z

+    const cellOutput = output.toSDK()
+    const bytes = bytesUtils.hexify(
+      blockchain.CellOutput.pack({
+        capacity: cellOutput.capacity,
+        lock: cellOutput.lock,
+        type: cellOutput.type || undefined, // TODO: null is not a packable type
+      })
+    )


what is the TODO means here?

cellOutput.type could be null, so maybe it's better to provide a translator to convert CKBCompont types with nulls to lumos compatible types with undefineds so that the code may look nicer.

homura · 2023-06-28T07:03:31Z

-    const bytes = serializeFixVec(wit)
-    return HexUtils.byteLength(bytes) + TransactionSize.SERIALIZED_OFFSET_BYTESIZE
+    if (typeof witness === 'string') {
+      const fixvecCodec = fixvec(number.Uint8)


witness is not uint8, it is bytes, and I think the refactor just makes the code hard to understand

the blockchain.mol describes the Transaction as

table Transaction { raw: RawTransaction, witnesses: BytesVec, } vector Bytes <byte>; vector BytesVec <Bytes>;

So it should be BytesVec.pack(Bytes.pack(WitnessArgs.pack({lock:'...'}))) here, but it is not. I think this method only calculates a part of the size, now this change will only make this method more difficult to understand, I suggest not to change this method first

Yes this part of code is hard to understand. So I just try to keep the logic as it is.

The original code just wraps witness in Uint8Array by using serializeFixVec.

The idea you provided here is changing original logic, which may cause unintended bugs, please refer to :

neuron/packages/neuron-wallet/src/models/transaction-size.ts

Lines 68 to 72 in 55e0db3

public static witness(witness: WitnessArgs | string): number {

const wit: string = typeof witness === 'string' ? witness : serializeWitnessArgs(witness.toSDK())

const bytes = serializeFixVec(wit)

return HexUtils.byteLength(bytes) + TransactionSize.SERIALIZED_OFFSET_BYTESIZE

}

homura · 2023-06-28T07:10:01Z

              args,
-              codeHash: systemScripts.SECP256K1_BLAKE160.codeHash,
-              hashType: systemScripts.SECP256K1_BLAKE160.hashType,
+              codeHash: config.predefined.AGGRON4.SCRIPTS.SECP256K1_BLAKE160.CODE_HASH,


I found there are so many AGGRON4 in the refactor, it is better to extract them to an independent module to manage the config, that would make the config easy to maintain when the config has changed or the code hash is not the same with the predefined.

For this part, we are just using SECP256K1_BLAKE160 code hash, which is the same in mainnet/testnet/devnet.

Because lumos utils like helpers.encodeToAddress, parseAddress needs to receive a config, so I plan to create a static method to provide config like NeuronInstance.getLumosConfig()

homura · 2023-06-28T07:11:20Z

      '10000',
-      Script.fromObject({ codeHash: '', args: '', hashType: ScriptHashType.Type }),
-      Script.fromObject({ codeHash: '', args: '', hashType: ScriptHashType.Type }),
+      Script.fromObject({ codeHash: `0x${'00'.repeat(32)}`, args: '0x', hashType: ScriptHashType.Type }),


it is better to add some comment to describe why the empty codeHash is required here

I have described the reason in the PR comment, please refer to:#2734 (comment)

homura · 2023-06-28T07:16:59Z

      describe('with public key info exist for the public key', () => {
-        const publicKey = 'public key'
+        // const publicKey = 'public key' // not valid public key
+        const publicKey = '0x' + '00'.repeat(65) // 65 bytes pub key


Just keep it as it is, unit tests can't always construct complete test data, as compared to semanticization, which makes unit tests easier to maintain

It can't pass test after refactor, because publicKey is used to generate secp256k1 args, in lumos we can call utils.ckbHash(pubkey).slice(0, 42), which requires publicKey to be a valid BytesLike value.

Keith-CY · 2023-06-29T06:46:25Z

 import Multisig from '../models/multisig'
 import SystemScriptInfo from '../models/system-script-info'
 import NetworksService from '../services/networks'
+import { config as lumosConig, helpers, utils, config } from '@ckb-lumos/lumos'


What's the difference between config as lumosConfig and config

setting lumosConfig as alias for config , because there is another viarable named config in the scope

I didn't get it, lumosConfig and config are both from @ckb-lumos/lumos.config, pointing to the same one.

BTW, lumosConig is a typo

neuron/packages/neuron-wallet/src/controllers/multisig.ts

Lines 110 to 117 in b4f7de2

const saveConfigs = Object.values(configOutput.multisig_configs).map(config => ({

r: +config.require_first_n,

m: +config.threshold,

n: config.sighash_addresses.length,

blake160s: config.sighash_addresses.map(v => addressToScript(v).args),

walletId,

alias: config.alias,

}))

config is delared here in line 110, if import { config } from '@ckb-lumos/lumos', then there are two configs in the mapFunction, I'll fix the typo

Suggested change

import { config as lumosConig, helpers, utils, config } from '@ckb-lumos/lumos'

import { config as lumosConig, helpers, utils } from '@ckb-lumos/lumos'

zhangyouxin · 2023-06-29T07:39:14Z

I find that totally removal of ckb-sdk-utils may be not as easy as I thougt of, and some changes needs to be discussed, so is it a better idea to seperate the task to more small tasks? @homura

After creating this PR, I can see some sub-tasks.

So I would like to close this PR and create each PR for:

config management
- (discussion) how to manage a lumos config, or we don't need a full lumos config, just part of it
- convert Script to Address
- convert Address to Script
refactor Script class: packages/neuron-wallet/src/models/chain/script.ts
- plan to remove this class, the methods of this class can be non-state, so it can be replaced by util functions
- computeHash -> utils.computeScriptHash()
- calculateBytesize -> blockchain.Script.pack(script).bytelength
Serialization/Deserialization for Script, RawTransaction, WitnessArgs
refactor: src/models/blake2b.ts
- replace with CKBHasher in @ckb-lumos/base
Test cases
- (discussion) some old conventions are not sutable for lumos style , such as script with args: "0xnot-hex" is not allowed when we use blockChain.Script.pack(script)

With the tasks seperated we can push the diliveration of this PR more smoothly

yanguoyu · 2023-06-29T07:06:14Z

    const outputsResult: [boolean, Output[], AnyoneCanPayInfo[]] = this.selectOutputs()
    const isMainnet = NetworksService.getInstance().isMainnet()
-    const outputAddresses: string[] = outputsResult[1].map(output => scriptToAddress(output.lock, isMainnet))
+    const lumosOptions = isMainnet ? { config: config.predefined.LINA } : { config: config.predefined.AGGRON4 }


Agree, we can init the config after ckb node starts and change the config when the network changes.

yanguoyu · 2023-06-29T07:10:59Z

-      const script = addressToScript(address)
+      const isMainnet = address.startsWith('ckb')
+      const script = isMainnet
+        ? helpers.addressToScript(address, { config: config.predefined.LINA })


Can we move the address.startsWith('ckb') logic to helpers.addressToScript function?

Good idea, it's common way to determine if isMainnet by address prefix, in the mean time, when we need to translate Script to Address, we still need to get a isMainnet env from somewhere else, so I try to solve the two convertion problem together, maybe more common way is to call config.initializeConfig(isMainnet ? LINA : AGGRON4) when network changes? @homura

Can we move the address.startsWith('ckb') logic to helpers.addressToScript function?

OK, we'll implement it in the Lumos

yanguoyu · 2023-06-29T07:30:55Z

-          outputType: '',
-        })
+        witnessEntry.witness = bytes.hexify(
+          blockchain.WitnessArgs.pack({


Refer to https://github.com/nervosnetwork/neuron/pull/2734/files#diff-7703c04c04ff2f19a47617fbb9bd430893f7cb702bb9ef8185998fd2cdcb259dR73,
blockchain.WitnessArgs.pack can not accept spaces as inputType or outputType parameters, so should we change to

witnessEntry.witnessArgs.inputType ?? undefined

yanguoyu · 2023-06-29T08:08:35Z

+    //  input_type: undefined
+    //  output_type: undefined
+    // }  
+    expect(result).toEqual(HexUtils.byteLength('0x1400000010000000100000001400000000000000') + 4 + 4)


It may cause here https://github.com/ckb-js/lumos/blob/develop/packages/codec/src/blockchain.ts#L64-L66 Lumos set all fields as option. But ckb-sdk-js does not set fields as option

I feel confused with https://github.com/ckb-js/lumos/blob/develop/packages/codec/src/blockchain.ts#L64-L66.
When I call bytes.hexify(base.blockchain.WitnessArgs.pack({ lock: '0x', input_type: '0x', output_type: '0x'})), it will output 0x1400000010000000140000001400000000000000, why input_type and output_type set empty, but lock set '0x00000000'.

here '0x00000000' is args, check the test case setup:

neuron/packages/neuron-wallet/tests/models/transaction-size.test.ts

Line 19 in 1afdb2f

const witnessArgs = new WitnessArgs('', '0x', '')

yanguoyu · 2023-06-29T08:42:30Z

            '0'
          )
-          tx.witnesses[0] = serializeWitnessArgs(WitnessArgs.emptyLock().toSDK())
+          // tx.witnesses[0] = bytes.hexify(blockchain.WitnessArgs.pack(WitnessArgs.emptyLock().toSDK())


Please remove annotation

remove ckb-sdk-utils and repalce with lumos

63aac4a

zhangyouxin added 2 commits June 27, 2023 16:25

Merge branch 'develop' into ckb-sdk-utils

69bef32

fix tests

8e89627

zhangyouxin added 2 commits June 27, 2023 22:05

fix unit test

892d274

fix test

7ffd05c

zhangyouxin marked this pull request as ready for review June 28, 2023 03:16

Merge branch 'develop' into ckb-sdk-utils

dd6a4de

zhangyouxin requested review from FrederLu, Keith-CY, WhiteMinds, alexsupa597, devchenyan, homura and yanguoyu June 28, 2023 03:24

homura reviewed Jun 28, 2023

View reviewed changes

Keith-CY reviewed Jun 29, 2023

View reviewed changes

yanguoyu reviewed Jun 29, 2023

View reviewed changes

zhangyouxin mentioned this pull request Jul 6, 2023

Refactor: remove models/script and repalce with lumos #2762

Closed

zhangyouxin closed this Jul 6, 2023

zhangyouxin mentioned this pull request Aug 2, 2023

Refactor ckb-sdk-utils by lumos Magickbase/neuron-public-issues#201

Closed

		codeHash: config.predefined.LINA.SCRIPTS.DAO.CODE_HASH,
		hashType: config.predefined.LINA.SCRIPTS.DAO.HASH_TYPE,

	public static witness(witness: WitnessArgs \| string): number {
	const wit: string = typeof witness === 'string' ? witness : serializeWitnessArgs(witness.toSDK())
	const bytes = serializeFixVec(wit)
	return HexUtils.byteLength(bytes) + TransactionSize.SERIALIZED_OFFSET_BYTESIZE
	}

	const saveConfigs = Object.values(configOutput.multisig_configs).map(config => ({
	r: +config.require_first_n,
	m: +config.threshold,
	n: config.sighash_addresses.length,
	blake160s: config.sighash_addresses.map(v => addressToScript(v).args),
	walletId,
	alias: config.alias,
	}))

	import { config as lumosConig, helpers, utils, config } from '@ckb-lumos/lumos'
	import { config as lumosConig, helpers, utils } from '@ckb-lumos/lumos'

Uh oh!

Conversation

zhangyouxin commented Jun 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

changed

Uh oh!

homura commented Jun 27, 2023

Uh oh!

github-actions Bot commented Jun 27, 2023

Uh oh!

github-actions Bot commented Jun 28, 2023

Uh oh!

github-actions Bot commented Jun 28, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyouxin Jun 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyouxin commented Jun 29, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

homura Jun 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyouxin Jun 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

zhangyouxin commented Jun 20, 2023 •

edited

Loading

zhangyouxin Jun 29, 2023 •

edited

Loading

homura Jun 30, 2023 •

edited

Loading

zhangyouxin Jun 30, 2023 •

edited

Loading