Skip to content

Latest commit

 

History

History
88 lines (65 loc) · 7.82 KB

developer_faq.md

File metadata and controls

88 lines (65 loc) · 7.82 KB

DaLI Frequently Asked Questions (Developer)

Table of Contents

General Questions

What are the Contribution Guidelines?

Read the contribution guidelines section of the documentation.

What are the Design Guidelines?

Read the design guidelines section of the documentation.

What is the Best Way to Get Started on DaLI Development?

Read the contribution guidelines and the developer documentation (especially the Internal Design section). Then look for an unassigned issue that is marked as good first issue, or ask the project maintainers.

How to Develop a DaLI Data Loader Plugin?

Read the Internal Design section of the Developer Documentation (and in particular the Plugin subsections).

How to Fill the Unique Id Field?

Data loader plugins read data from CSV and/or REST sources and return them to DaLI as transaction instances. One of the most important jobs of a data loader plugin is to fill in the unique_id field. This field is used by the DaLI transaction resolver to match incomplete transaction and join them. E.g. a transfer from Coinbase to Trezor is initially represented as 2 transactions:

  • an IntraTransaction from Coinbase to UNKNOWN, generated by the Coinbase plugin (which doesn't know that the destination is a Trezor);
  • an IntraTransaction from UNKNOWN to Trezor, generated by the Trezor plugin (which doesn't know that the source is Coinbase).

These two transactions both share the same unique_id (which is typically the transaction hash and is filled in by the respective plugins): the transactions are fed to the transaction resolver, which uses unique_id to match them up and to join them into a single transaction without UNKNOWN fields.

This is why unique_id is so important in data loaders: if an incorrect value is used, the transaction resolver will be unable to match and join incomplete transactions.

Some data sources (especially exchange CSVs) have invalid or missing transaction identifier information: sometimes it's empty, other times (confusingly) it's an internal id of the exchange (which cannot be used by the transaction resolver) instead of the hash. In such cases (where transaction hash is not available) DaLI plugins must set the unique_id field to Keyword.UNKNOWN.

Some examples of this:

  • BlockFI CSVs don't contain the hash or any other transaction id;
  • Coinbase CSVs contain an internal transaction id, but it's not the hash.

There is one more nuanced detail to this topic. If the transaction is complete and DOESN'T need resolution (e.g. a BUY or a SELL transaction), then it's ok to use an internal exchange id as the unique_id: it will help document the transaction and can be helpful when looking it up in the logs.

Check the following for more details:

Should I Implement a CSV or a REST Data Loader Plugin?

REST endpoints typically provide more complete data compared to CSV files, so they are preferred if available. Often CSV files have incorrect or missing unique_id, spot_price, etc. Here is how to decide whether to write a CSV or a REST plugin for a given service (wallet or exchange):

  • given the choice, a REST plugin is always preferrable to a CSV one, if the endpoint is available;
  • if a REST plugin for the service is already implemented and the CSV files for this service are incomplete, then there is no need for a CSV plugin;
  • if a REST plugin for the service is already implemented and the CSV files for this service are complete (no missing / incorrect information), then it's OK to have a CSV plugin as an alternative option;
  • if a REST plugin for the service is NOT yet implemented and CSV files for this service are available, then a CSV plugin is good to have regardless of whether CSV files are complete or not (it's better than nothing). This CSV plugin may be decommissioned in the future, if a REST plugin is implemented.

How to Develop a DaLI Pair Converter Plugin?

Read the Internal Design section of the Developer Documentation (and in particular the Plugin subsections).

How to Develop a DaLI Country Plugin?

Read the Internal Design section of the Developer Documentation (and in particular the Plugin subsections).

How Does DaLI Merge Transactions Between Different Exchanges/Wallets?

Read about the transaction_resolver in the Internal Design section of the Developer Documentation.

Why the Strange Directory Structure with Src?

Because DaLI is a src-based project.

How to Use the Cache to Speed Up Development?

Use the -c command line option to enable the cache. This instructs DaLI to store transactions coming from cache-enabled plugins in the cache. The next time DaLI is run, transaction data is read directly from the cache instead of from the plugin native source. To make a plugin cache-enabled, just define its cache_key() method (for an example, look at the Coinbase plugin). Note that the code doesn't yet check for newer data: if it finds cached data, it reads it but doesn't check the native source for more recent entries, so currently the cache is only useful to speed up development, not for regular use. The cache is stored in the .dali_cache/ directory: to reset the cache delete this directory.