Supporting Arctic Backend Provider & Orderbook, Tick Data Example #744
Conversation
|
|
|
@luocy16 Please follow the instructions to format your code. Thanks |
qlib/data/data.py
Outdated
| @@ -886,6 +955,7 @@ def dataset( | |||
| return_uri=False, | |||
| inst_processors=[], | |||
| ): | |||
| #print("disk_cache {}".format(disk_cache)) | |||
There was a problem hiding this comment.
Remove the useless code after testing.
qlib/config.py
Outdated
| @@ -122,7 +131,7 @@ def set_conf_from_C(self, config_c): | |||
| # default 1 hour | |||
| "mem_cache_expire": 60 * 60, | |||
| # memory cache space limit, default 5GB, only in used client | |||
| "mem_cache_space_limit": 1024 * 1024 * 1024 * 5, | |||
| "mem_cache_space_limit": 1024 * 1024 * 1024, | |||
There was a problem hiding this comment.
Why is it necessary to change the default memory size?
qlib/data/data.py
Outdated
| @@ -502,16 +505,18 @@ def parse_fields(fields): | |||
| return [ExpressionD.get_expression_instance(f) for f in fields] | |||
|
|
|||
| @staticmethod | |||
| def dataset_processor(instruments_d, column_names, start_time, end_time, freq, inst_processors=[]): | |||
| def dataset_processor(instruments_d, column_names, start_time, end_time, freq, type="qlib", inst_processors=[]): | |||
There was a problem hiding this comment.
Using constant string will be better.
qlib/data/data.py
Outdated
| from .ops import Operators | ||
| from .op_arctic import * |
There was a problem hiding this comment.
import * is not encouraged
qlib/data/data.py
Outdated
| try: | ||
| arctic = Arctic_Connection_List[0] | ||
| #print("arctic example: {}, task_index:{}".format(arctic, task_index)) | ||
| except Exception: |
There was a problem hiding this comment.
Please catch specific exception instead of except Exception:
qlib/data/ops.py
Outdated
| elif self.func == 'ffill': | ||
| # TODO: | ||
| # Not sure why dropping duplicated element is necessary | ||
| series = series[~series.index.duplicated(keep='first')] |
There was a problem hiding this comment.
@luocy16 It's very kind of you if you could give any hints why ffill should be processed in such a special approach
you-n-g
changed the title
…crosoft#744) * change weight_decay & batchsize * del weight_decay * big weight_decay * mid weight_decay * small layer * 2 layer * full layer * no weight decay * divide into two data source * change parse field * delete some debug * add Toperator * new format of arctic * fix cache bug to arctic read * fix connection problem * add some operator * final version for arcitc * clear HZ cache * remove not used function * add topswrappers * successfully import data and run first test * A simpler version to support arctic * Successfully run all high-freq expressions * Black format and fix add docs * Add docs for download and test data * update scripts and docs * Add docs * fix bug * Refine docs * fix test bug * fix CI error * clean code Co-authored-by: bxdd <bxddream@gmail.com> Co-authored-by: wangwenxi.handsome <wangwenxi.handsome@gmail.com> Co-authored-by: Young <afe.young@gmail.com>
Description
Motivation and Context
How Has This Been Tested?
pytest qlib/tests/test_all_pipeline.pyunder upper directory ofqlib.Screenshots of Test Results (if appropriate):
Types of changes