Unable to adjust debt position when individual settlement fund is not empty

[abitmore]

Bug Description

Unable to adjust debt position when individual settlement fund is not empty

Assert Exception: enable_black_swan: Black swan was detected during a margin update which is not allowed to trigger a blackswan

3128356ms th_a       db_market.cpp:161             check_for_blackswan  ] *call_ptr: {"id":"1.8.117222","borrower":"1.2.458173","collateral":"32522288519","debt":449905151,"call_price":{"base":{"amount":1,"asset_id":"1.3.0"},"quote":{"amount":1,"asset_id":"1.3.113"}}}
3128357ms th_a       db_market.cpp:171             check_for_blackswan  ] Black Swan detected on asset CNY (1.3.113) at block 65662393:
   Least collateralized call: 72.28698859462491555  0.01383374822276602
   Settle Price:              68.18559658342681473  0.01466585393553717
   Max:                       73.61204268292682684  0.01358473374128951

3128358ms th_a       db_market.cpp:172             check_for_blackswan  ] enable_black_swan: false

  "median_feed": {
    "settlement_price": {
      "base": {
        "amount": 328,
        "asset_id": "1.3.113"
      },
      "quote": {
        "amount": 22995,
        "asset_id": "1.3.0"
      }
    },
    "maintenance_collateral_ratio": 1600,
    "maximum_short_squeeze_ratio": 1050,
    "core_exchange_rate": {
      "base": {
        "amount": 8506627,
        "asset_id": "1.3.113"
      },
      "quote": {
        "amount": 500000000,
        "asset_id": "1.3.0"
      }
    },
    "initial_collateral_ratio": 1750
  },
  "current_feed": {
    "settlement_price": {
      "base": {
        "amount": "504696031977",
        "asset_id": "1.3.113"
      },
      "quote": {
        "amount": "34413000033640",
        "asset_id": "1.3.0"
      }
    },
    "maintenance_collateral_ratio": 1600,
    "maximum_short_squeeze_ratio": 1050,
    "core_exchange_rate": {
      "base": {
        "amount": 8506627,
        "asset_id": "1.3.113"
      },
      "quote": {
        "amount": 500000000,
        "asset_id": "1.3.0"
      }
    },
    "initial_collateral_ratio": 1750
  },
  "current_feed_publication_time": "2022-01-22T16:40:12",
  "current_maintenance_collateralization": {
    "base": {
      "amount": 4599,
      "asset_id": "1.3.0"
    },
    "quote": {
      "amount": 41,
      "asset_id": "1.3.113"
    }
  },
  "current_initial_collateralization": {
    "base": {
      "amount": 160965,
      "asset_id": "1.3.0"
    },
    "quote": {
      "amount": 1312,
      "asset_id": "1.3.113"
    }
  },
  "options": {
    "feed_lifetime_sec": 86400,
    "minimum_feeds": 3,
    "force_settlement_delay_sec": 86400,
    "force_settlement_offset_percent": 200,
    "maximum_force_settlement_volume": 50,
    "short_backing_asset": "1.3.0",
    "extensions": {
      "initial_collateral_ratio": 1750,
      "maintenance_collateral_ratio": 1600,
      "maximum_short_squeeze_ratio": 1050,
      "margin_call_fee_ratio": 30,
      "force_settle_fee_percent": 300,
      "black_swan_response_method": 2
    }
  },
  "force_settled_volume": 0,
  "is_prediction_market": false,
  "settlement_price": {
    "base": {
      "amount": 0,
      "asset_id": "1.3.0"
    },
    "quote": {
      "amount": 0,
      "asset_id": "1.3.0"
    }
  },
  "settlement_fund": 0,
  "individual_settlement_debt": "49480003135",
  "individual_settlement_fund": "3441300003364",

Impacts
Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)

CORE TEAM TASK LIST

• Evaluate / Prioritize Bug Report
• Refine User Stories / Requirements
• Define Test Cases
• Design / Develop Solution
• Perform QA/Testing
• Update Documentation

[abitmore]

Found the reason:

bitshares-core/libraries/chain/asset_evaluator.cpp

Lines 1432 to 1433 in e52b9ff

	if( old_feed.margin_call_params_equal(bad.current_feed) )
	return void_result();

and (else)

bitshares-core/libraries/chain/asset_evaluator.cpp

Line 1467 in e52b9ff

d.check_call_orders( base, true, false, bitasset_ptr );

When the individual settlement fund is not empty, and the price of the backing asset continues to fall, after a new price feed is published, current_feed will not change, according to the code, return void_result() will be executed, this means that check_call_orders() will not be executed, so new individual settlement(s) will not happen although it (or they) should. In this case, when trying to update a debt position, the check_call_orders() call will fail.

bitshares-core/libraries/chain/market_evaluator.cpp

Line 383 in e52b9ff

bool called_some = d.check_call_orders( *_debt_asset, false, false, _bitasset_data );

Similar suspicious code:

bitshares-core/libraries/chain/db_update.cpp

Line 498 in e52b9ff

&& !b.current_feed.margin_call_params_equal( old_median_feed ) )

and

bitshares-core/libraries/chain/asset_evaluator.cpp

Line 1004 in e52b9ff

feed_actually_changed = ( after_hf_core_868_890 && !old_feed.margin_call_params_equal( bdo.current_feed ) );

[abitmore]

To work around this issue, asset owners can temporarily or regularly update MCR (to a slightly lower value) to trigger individual settlements then update it back. See https://bitsharestalk.org/index.php?topic=33637.0.

[abitmore]

When this occurs, these notes in the code are no longer true:

bitshares-core/libraries/chain/asset_evaluator.cpp

Lines 1291 to 1295 in 2c2e0ef

	// Note: theoretically, if the fund is still not empty, its new CR should be >= old CR,
	// in this case, calling check_call_orders() should not change anything.
	// Note: there should be no existing force settlements
	if( 0 == bitasset.individual_settlement_debt && old_feed_price != bitasset.current_feed.settlement_price )
	d.check_call_orders( asset_to_settle, true, false, &bitasset );

For the first note, although it is not true, but it just means that there is still a check_call_orders() call missing, so nothing changes.

For the second note, in short, it is not true but also not critical. The details are as follows.

• At first there was no force settlement. Due to the bug described in this issue, when a force settlement is requested, it is possible that the check_call_orders() call closes a debt position (or more) and moves its (or their) collateral to the individual settlement fund, so the fund becomes non-empty again. In case when the force settlement request is too large to be fully filled, a new force settlement order will be created, and apply_force_settlement() will be called (there is an issue here but it is not critical: Unable to settle more than total debt amount in individual settlement fund when no sufficient price feeds #2587). It is possible that the apply_force_settlement() call doesn't fill the force settlement either, in this case, the fund is not empty, and at the same time there is a force settlement.
  • Note: if the force settlement expires, it will be filled at price current_feed.settlement_price * (1+force_settlement_offset), in this case, no matter if current_feed.settlement_price is capped, the matching debt position should have sufficient collateral to pay at that price.
• After this, if median_feed.settlement_price falls lower, current_feed.settlement_price should not change (due to the bug). In this case, if check_call_orders() got called somewhere else (E.G. due to the workaround), the force settlement will get matched with no issue.
• If a new force settlement is requested before the first force settlement is gone (expired or filled due to margin calls), and if it triggers the check_call_orders() call again, the first force settlement will get matched with no issue.

[abitmore]

It should have been fixed by #2599. But missing test cases.

[abitmore]

Added tests in #2606.