new Wallet(privateKey) throws "invalid hexlify value" when privateKey is not 0x prefixed

[matYang]

While upgrading to Ethers V5, we ran into

Error: invalid hexlify value (argument="value", value="[not copied here]", code=INVALID_ARGUMENT, version=bytes/5.0.6)
I20201120-11:39:47.012(-5)?     at Logger.makeError (/Users/matthewyang/Desktop/blockchain/tokenfunder/tokenfunderapp/node_modules/@ethersproject/logger/src.ts/index.ts:205:28)
I20201120-11:39:47.013(-5)?     at Logger.throwError (/Users/matthewyang/Desktop/blockchain/tokenfunder/tokenfunderapp/node_modules/@ethersproject/logger/src.ts/index.ts:217:20)
I20201120-11:39:47.013(-5)?     at Logger.throwArgumentError (/Users/matthewyang/Desktop/blockchain/tokenfunder/tokenfunderapp/node_modules/@ethersproject/logger/src.ts/index.ts:221:21)
I20201120-11:39:47.013(-5)?     at Object.hexlify (/Users/matthewyang/Desktop/blockchain/tokenfunder/tokenfunderapp/node_modules/@ethersproject/bytes/src.ts/index.ts:242:19)
I20201120-11:39:47.013(-5)?     at new SigningKey (/Users/matthewyang/Desktop/blockchain/tokenfunder/tokenfunderapp/node_modules/@ethersproject/signing-key/src.ts/index.ts:35:44)
I20201120-11:39:47.032(-5)?     at new Wallet (/Users/matthewyang/Desktop/blockchain/tokenfunder/tokenfunderapp/node_modules/@ethersproject/wallet/src.ts/index.ts:81:36)

This issue was brought up in
https://www.gitmemory.com/issue/ethers-io/ethers.js/994/674504533

I assume this is not an intended feature, as all other tools accept non 0x prefixed private keys, and prefixing 0x to the private key causes issues with bunch of other libraries, so it's a change specific for Ethers V5

Regardless of it's intended or not, this should have been included in the breaking change log https://docs.ethers.io/v5/migration/ethers-v4/#migration-v4--contracts

[ricmoo]

Heya!

Ethers 100% requires all hex strings to be 0x-prefixed and this is intentional.

This was also required in v4 (I just installed v4 to verify). So, it is not a breaking change since v4.

/tmp/test-0x> npm install ethers@4.0.0
/tmp/test-0x> node
> e = require("ethers")
> new e.Wallet("123456789012345678901234567890123456789012345678901234567890123")
Thrown:
{ Error: hex string must have 0x prefix (arg="value", value="123456789012345678901234567890123456789012345678901234567890123")
    at Object.throwError (/private/tmp/test-0x/node_modules/ethers/errors.js:74:17)
    at Object.arrayify (/private/tmp/test-0x/node_modules/ethers/utils/bytes.js:56:20)
    at new SigningKey (/private/tmp/test-0x/node_modules/ethers/utils/signing-key.js:35:39)
    at new Wallet (/private/tmp/test-0x/node_modules/ethers/wallet.js:44:62)
  reason: 'hex string must have 0x prefix',
  code: 'INVALID_ARGUMENT',
  arg: 'value',
  value:
   '123456789012345678901234567890123456789012345678901234567890123' }

In fact, if I do the same with ethers@2.0.0 The same thing happens. I believe the requirement was present since 0.0.1, but my memory gets a little foggy that far back, cut I seem to recall people at the time complaining, but I believe in explicitly declaring intentions, and since strings are used for myriad things in JavaScript, it is the least worst way to indicate the string's intention as an octet-string.

[matYang]

I'm very certain this is not an issue in Ethers V4, as I personally have experienced, and the author of the linked post also experienced that, it was perfectly ok to use non-0x-prefixed private key in V4 to construct a wallet object.

The issue only showed up as we upgraded form v4 to v5. I was using 4.0.43.

[ricmoo]

Perhaps there is somewhere else the hex string is being generated that used to 0x-prefix it for you in your pre-upgrade code?

You can install v4 to try it out, maybe there is some other aspect that has changed I am not thinking of, but you can install v4 direct via a specific version or use the tag legacy (npm install ethers@legacy) and try it out... I can't (I don't think?) fiddle with the versions in npm, so you can verify/investigate for yourself...

[ricmoo]

(I just installed 4.0.43 and verified it throws the same error)

[matYang]

[matYang]

using a valid random private key and ethers@4.0.43, a wallet object was returned

[ricmoo]

What is the result of console.log(e.version)?

That seems very odd, because I know I have code in there to check the regex of the string coming in... :)

[matYang]

[matYang]

so as in the screenshot, it is Ethers 4.0.43. We had that piece of code working in production since last year so pretty sure it was working fine, and we know it was not 0x prefixed because the other libraries we use would throw

[ricmoo]

Can you include the npm install you used for that? Or your package.json? Because now I am super confused... Staring at this, my eyes are close to falling out. :p

[matYang]

[ricmoo]

Right, but I know that code has been there for years, and just now installed 4.0.43 locally from npm. So there is some disconnect. It was never supported, so the fact it is working for you means something we are not thinking of is going on...

Can you verify in the node_modules/ethers the following code:

    // Line 17 of ethers-wallet/signing-key.js
    privateKey = utils.arrayify(privateKey);

    // Line 41 of ethers-utils/convert.js
    if (isHexString(value)) {

    // Line 17 of ethers-utils/convert.js
    if (typeof(value) !== 'string' || !value.match(/^0x[0-9A-Fa-f]*$/)) {

[matYang]

I assume it should be reproducible with valid private keys, like

9573fa33a57fee662a23bf60f1b1674364d99fb8dd2166b4ae470ce7ab20ed9f
79ae42fce836c603bc3d6d90421fa3ab0b946e339735e5df3433d328ac870755
147fe9be399879bdeefbecb239e4dc37ffa1e01b2eaae151e77e9babba28cf88

[ricmoo]

Something weird just happened... It started doing what you have.

I will investigate further, because this is quite odd behaviour. Gimme a few minutes. I'll update this issue shortly.