fix: amount calculation in the rgbpp address balance endpoint

[ShookLyngs]

Changes

• Fix Displaying incorrect balance when fetching RGB++ balance via BTC address #202

Fix details

The calculation in the /rgbpp/address/{btc_address}/balance endpoint has been refactored to:

• Confirmed UTXOs:

  1. Get all UTXOs
  2. Filter for all confirmed UTXOs
  3. Find the isomorphic XUDT cells corresponding to the UTXOs
  4. Add the amount of each target cell to the total_amount and available_amount fields

• Unconfirmed UTXOs:

  1. Get all UTXOs
  2. Filter for all unconfirmed UTXOs
  3. Find the isomorphic XUDT cells corresponding to the UTXOs
  4. Add the amount of each target cell to the pending_amount field

• Unconfirmed Spent Outputs:

  1. Get all transactions for the target BTC address
  2. Filter out all unconfirmed BTC transactions from the total transactions
  3. Find all RgbppLock XUDT cells in the inputs of the unconfirmed transactions
  4. Filter the above input cells to include only those belonging to the target BTC address
  5. Add the amount of each filtered cell to the total_amount field

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
btc-assets-api	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 20, 2024 3:20am

[Flouse]

Unconfirmed Spent Outputs:

1. Get all transactions for the target BTC address
2. Filter out all unconfirmed BTC transactions from the total transactions
3. Find all RgbppLock XUDT cells in the inputs of the unconfirmed transactions
4. Filter the above input cells to include only those belonging to the target BTC address
5. Add the amount of each filtered cell to the total_amount field

Good catch!
This calculation is a necessary step in the [BTC --(leap)--> CKB] process.

[Dawn-githup]

issues

curl -X 'GET' \
  'https://btc-assets-api-git-fix-202-unconfirmed-xudt-balance-cell-studio.vercel.app/rgbpp/v1/address/tb1qlmafss8vm2v82700j9ztzqp07nfqy6wwhazv9q/balance?type_script=%7B%0A%20%20%22codeHash%22%3A%20%220x25c29dc317811a6f6f3985a7a9ebc4838bd388d19d0feeecf0bcd60f6c0975bb%22%2C%0A%20%20%22hashType%22%3A%20%22type%22%2C%0A%20%20%22args%22%3A%20%22%22%0A%7D&no_cache=false' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJteS1hcHAiLCJhdWQiOiJidGMtYXNzZXRzLWFwaS1naXQtZml4LTIwMi11bmNvbmZpcm1lZC14dWR0LWJhbGFuY2UtY2VsbC1zdHVkaW8udmVyY2VsLmFwcCIsImp0aSI6IjIyMWY0MGU5LWQ3ZTQtNGM4YS05YzgyLWUyZDg5OGJhYjc0OSIsImlhdCI6MTcyNDA4MzU1M30.xSMoL0r9BEuqlj9dCv4SwtCMmMi5QF05H_o7ioxKT-o'

{
    "message": "ERR max request size exceeded. Limit: 1048576 bytes, Actual: 6064477 bytes. See https://upstash.com/docs/redis/troubleshooting/max_request_size_exceeded for details"
}

Is this caused by a limitation of Vercel?

[Dawn-githup]

get ... address/tb1q9mktcwlf0zs7ayajv7rsx4j6k959d8us7rg858/balance
The obtained xudt is null

{
  "address": "tb1q9mktcwlf0zs7ayajv7rsx4j6k959d8us7rg858",
  "xudt": []
}

[ShookLyngs]

get ... address/tb1q9mktcwlf0zs7ayajv7rsx4j6k959d8us7rg858/balance The obtained xudt is null

Because the corresponding ckbVirtualTxResult has only been added to my local queue, without this information you won’t be able to determine the current status. To test it locally, it’s better to start some new examples.

[ShookLyngs]

@Flouse @ahonn Another issue that affects the balance calculation is that, currently, if a job in the queue has been retried a certain number of times, it will be marked as failed. In the getPendingInputCellsByTxid() and getPendingOutputCellsByTxid() methods, if the target job is marked as failed, an empty list [] will be returned.

This means that if we're transferring some RGBPP assets and the BTC_TX gets stuck for a while, the job in the queue will be marked as failed and excluded from the balance calculation. We have roughly two ways to handle this issue:

1. It's expected: Failed jobs should be reactivated manually to be included in the balance calculation
2. It's unexpected: Failed jobs should be included in the balance calculation

If this is unexpected behavior, a possible resolution is to add a allowFailedJob: boolean = false parameter to the getPendingInputCellsByTxid and getPendingOutputCellsByTxid methods, allowing them to retrieve the input/output cells from the failed jobs.

Ref: https://github.com/ckb-cell/btc-assets-api/pull/206/files#diff-c69299e8c418dcefed9c4114412c428323e0060b87a31d3e8c6b3d5b92db8057R628-R632

[Flouse]

Another issue that affects the balance calculation is that, currently, if a job in the queue has been retried a certain number of times, it will be marked as failed. In the getPendingInputCellsByTxid() and getPendingOutputCellsByTxid() methods, if the target job is marked as failed, an empty list [] will be returned.

This means that if we're transferring some RGBPP assets and the BTC_TX gets stuck for a while, the job in the queue will be marked as failed and excluded from the balance calculation. We have roughly two ways to handle this issue:

1. It's expected: Failed jobs should be reactivated manually to be included in the balance calculation
2. It's unexpected: Failed jobs should be included in the balance calculation

If this is unexpected behavior, a possible resolution is to add a allowFailedJob: boolean = false parameter to the getPendingInputCellsByTxid and getPendingOutputCellsByTxid methods, allowing them to retrieve the input/output cells from the failed jobs.

Ref: https://github.com/ckb-cell/btc-assets-api/pull/206/files#diff-c69299e8c418dcefed9c4114412c428323e0060b87a31d3e8c6b3d5b92db8057R628-R632

#207 is created including the proposed resolutions. Let's discuss and handle it later.

• An issue about excluding Failed Jobs from Balance Calculation #207

[ahonn]

The reason why Unit Tests failed may be that redis flushall has not been executed yet, but the test has begun to be executed.
The test of mempool/elelctrs uses the same redis, so it may be affected.

Please try:

beforeAll(async () => {
  await container.cradle.redis.flushall();
});

[ShookLyngs]

@Flouse @ahonn Test-related fixes have been reverted since the error appears randomly in the test workflow but not in the local environment. We can skip the error and merge it.

Related issue: #208