refactor: Optimize input and output storage

[yanguoyu]

To search others' addresses at the history page, I create a table tx_lock to save tx_hash map lockHash

1. Only save local wallets' lock into input and output, and save others' cell lockHash to tx_lock
2. When searching by address, search addresses from input, output and tx_lock.
3. When getting transaction detail, get the transaction detail from RPC.
4. Record the last block number when caching tx to indexer_tx_hash_cache

[Keith-CY]

I'm not sure if I get the point.

Seems that all inputs and outputs are saved into inputs, outputs, others_input, and others_output.

It speeds searching up, but the storage is not optimized as mentioned at Magickbase/neuron-public-issues#77 (comment)

If a transaction includes many cells, they are still all saved into the database. The disk usage is not ameliorated.

[yanguoyu]

I'm not sure if I get the point.

Seems that all inputs and outputs are saved into inputs, outputs, others_input, and others_output.

It speeds searching up, but the storage is not optimized as mentioned at Magickbase/neuron-public-issues#77 (comment)

If a transaction includes many cells, they are still all saved into the database. The disk usage is not ameliorated.

Currently yes, I have not paid attention to the SQLite file's size, but only care about the search speed and SQL execution time. I want to use the data that has been got.

You remind me that we also pay attention to the SQLite file's size. Actually, I have tried another way to finish it, create a table that only saves tx hash and lockHash for searching addresses, and also works. Maybe it will reduce the disk cost by saving only two fields. I think saving two fields for searching addresses is cost-effective. But finally, for fewer RPC requests so I use currently schema.

For smaller SQLite'file size, there exist two schemas:

1. Create a new table to save tx hash and lockHash for searching addresses.
2. When searching others' addresses, get all transactions' detail in the current wallet and then filter with the address. I'm worry about the rpc's result is too big.

So, for reduce SQLite's file size, I suggestion for first solution. What do you think? @Keith-CY

[Keith-CY]

I'm not sure if I get the point.
Seems that all inputs and outputs are saved into inputs, outputs, others_input, and others_output.
It speeds searching up, but the storage is not optimized as mentioned at Magickbase/neuron-public-issues#77 (comment)
If a transaction includes many cells, they are still all saved into the database. The disk usage is not ameliorated.

Currently yes, I have not paid attention to the SQLite file's size, but only care about the search speed and SQL execution time. I want to use the data that has been got.

You remind me that we also pay attention to the SQLite file's size. Actually, I have tried another way to finish it, create a table that only saves tx hash and lockHash for searching addresses, and also works. Maybe it will reduce the disk cost by saving only two fields. I think saving two fields for searching addresses is cost-effective. But finally, for fewer RPC requests so I use currently schema.

For smaller SQLite'file size, there exist two schemas:

1. Create a new table to save tx hash and lockHash for searching addresses.
2. When searching others' addresses, get all transactions' detail in the current wallet and then filter with the address. I'm worry about the rpc's result is too big.

So, for reduce SQLite's file size, I suggestion for first solution. What do you think? @Keith-CY

Solution 2 has an alternative as mentioned at Magickbase/neuron-public-issues#77 (comment)

But I think solution 1 is good for our case, and it can be removed easily if a better solution is found

[yanguoyu]

Solution 2 has an alternative as mentioned at Magickbase/neuron-public-issues#77 (comment)

I'm worried about search one address that may belong to a miner.

[Keith-CY]

Solution 2 has an alternative as mentioned at Magickbase/neuron-public-issues#77 (comment)

I'm worried about search one address that may belong to a miner.

Indexer supports pagination so the count of transactions won't bother. But the pagination of indexer is cursor-style, while neuron uses offset-style it, it's a problem if we use indexer to list transactions.

So I think solution 1 is good for our case now

[yanguoyu]

I will reopen the PR after I changed it to Solution 1

[yanguoyu]

I will reopen the PR after I changed it to Solution 1

It's ready for review now.

[Keith-CY]

Please have a review @WhiteMinds @devchenyan @JeffreyMa597

[alexsupa597]

please resolve the conflicts.

[github-actions]

Packaging for test is done in 5249523509 for commit 3e9f263 .

[yanguoyu]

@FrederLu Please test it with the package.

[github-actions]

Packaging for test is done in 5308946921 for commit cf25d37 .

[github-actions]

Packaging for test is done in 5308951833 for commit 5e11fdf .

[Keith-CY]

Please resolve the conflicts and squash merge @yanguoyu

[github-actions]

Packaging for test is done in 5369493403 for commit 8b41c58 .