UnsortedIndexError in qlib/examples/highfreq/ #521
Description
🐛 Bug Description
When try qlib/examples/highfreq as its READEME, I got
An exception has been raised[UnsortedIndexError: 'MultiIndex slicing requires the index to be lexsorted: slicing on levels [1], lexsort depth 1']
To Reproduce
Steps to reproduce the behavior:
1.python workflow.py get_data
2.python workflow.py dump_and_load_dataset
Screenshot
(qlib.venv) MBP15:highfreq elife$ python workflow.py dump_and_load_dataset
2021-07-17 21:53:03.660 | WARNING | qlib.tests.data:qlib_data:149 - Data already exists: ~/.qlib/qlib_data/yahoo_cn_1min, the data download will be skipped
If downloading is required: exists_skip=False or change target_dir
[21940:MainThread](2021-07-17 21:53:03,661) INFO - qlib.Initialization - [config.py:287] - default_conf: client.
[21940:MainThread](2021-07-17 21:53:04,060) INFO - qlib.Initialization - [init.py:54] - qlib successfully initialized based on client settings.
[21940:MainThread](2021-07-17 21:53:04,061) INFO - qlib.Initialization - [init.py:55] - data_path=/Users/elife/.qlib/qlib_data/yahoo_cn_1min
[21940:MainThread](2021-07-17 21:54:49,057) INFO - qlib.timer - [log.py:113] - Time cost: 104.545s | Loading data Done
[21940:MainThread](2021-07-17 22:06:14,629) INFO - qlib.timer - [log.py:113] - Time cost: 685.515s | HighFreqNorm Done
[21940:MainThread](2021-07-17 22:06:14,919) INFO - qlib.timer - [log.py:113] - Time cost: 685.861s | fit & process data Done
[21940:MainThread](2021-07-17 22:06:14,919) INFO - qlib.timer - [log.py:113] - Time cost: 790.408s | Init data Done
[21940:MainThread](2021-07-17 22:07:27,595) INFO - qlib.timer - [log.py:113] - Time cost: 72.646s | Loading data Done
[21940:MainThread](2021-07-17 22:07:27,595) INFO - qlib.timer - [log.py:113] - Time cost: 72.647s | Init data Done
[21940:MainThread](2021-07-17 22:08:43,465) INFO - qlib.timer - [log.py:113] - Time cost: 75.169s | Loading data Done
[21940:MainThread](2021-07-17 22:08:46,157) INFO - qlib.timer - [log.py:113] - Time cost: 2.691s | HighFreqNorm Done
[21940:MainThread](2021-07-17 22:08:46,170) INFO - qlib.timer - [log.py:113] - Time cost: 2.704s | fit & process data Done
[21940:MainThread](2021-07-17 22:09:07,714) INFO - qlib.timer - [log.py:113] - Time cost: 21.544s | Loading data Done
[21940:MainThread](2021-07-17 22:09:07,718) ERROR - qlib.workflow - [utils.py:34] - An exception has been raised[UnsortedIndexError: 'MultiIndex slicing requires the index to be lexsorted: slicing on levels [1], lexsort depth 1'].
File "workflow.py", line 175, in
fire.Fire(HighfreqWorkflow)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/fire-0.4.0-py3.7.egg/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/fire-0.4.0-py3.7.egg/fire/core.py", line 471, in _Fire
target=component.name)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/fire-0.4.0-py3.7.egg/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "workflow.py", line 167, in dump_and_load_dataset
xtest = dataset.prepare("test")
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/pyqlib-0.7.0.99-py3.7-macosx-10.13-x86_64.egg/qlib/data/dataset/init.py", line 232, in prepare
return self._prepare_seg(slice(*self.segments[segments]), **fetch_kwargs)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/pyqlib-0.7.0.99-py3.7-macosx-10.13-x86_64.egg/qlib/data/dataset/init.py", line 176, in _prepare_seg
return self.handler.fetch(slc, **kwargs, **self.fetch_kwargs)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/pyqlib-0.7.0.99-py3.7-macosx-10.13-x86_64.egg/qlib/data/dataset/handler.py", line 529, in fetch
return fetch_df_by_index(df, selector, level, fetch_orig=self.fetch_orig)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/pyqlib-0.7.0.99-py3.7-macosx-10.13-x86_64.egg/qlib/data/dataset/utils.py", line 65, in fetch_df_by_index
pd.IndexSlice[idx_slc],
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/pandas-1.3.0-py3.7-macosx-10.13-x86_64.egg/pandas/core/indexing.py", line 925, in getitem
return self._getitem_tuple(key)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/pandas-1.3.0-py3.7-macosx-10.13-x86_64.egg/pandas/core/indexing.py", line 1100, in _getitem_tuple
return self._getitem_lowerdim(tup)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/pandas-1.3.0-py3.7-macosx-10.13-x86_64.egg/pandas/core/indexing.py", line 822, in _getitem_lowerdim
return self._getitem_nested_tuple(tup)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/pandas-1.3.0-py3.7-macosx-10.13-x86_64.egg/pandas/core/indexing.py", line 906, in _getitem_nested_tuple
obj = getattr(obj, self.name)._getitem_axis(key, axis=axis)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/pandas-1.3.0-py3.7-macosx-10.13-x86_64.egg/pandas/core/indexing.py", line 1157, in _getitem_axis
locs = labels.get_locs(key)
File "/Users/elife/work/lab/Trading/qlib.venv/lib/python3.7/site-packages/pandas-1.3.0-py3.7-macosx-10.13-x86_64.egg/pandas/core/indexes/multi.py", line 3265, in get_locs
"MultiIndex slicing requires the index to be lexsorted: slicing "
UnsortedIndexError: 'MultiIndex slicing requires the index to be lexsorted: slicing on levels [1], lexsort depth 1'
(qlib.venv) MBP15:highfreq elife$
Environment
Note: User could run cd scripts && python collect_info.py all under project directory to get system information
and paste them here directly.
Darwin
x86_64
Darwin-19.6.0-x86_64-i386-64bit
Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64
Python version: 3.7.3 (default, Mar 27 2019, 09:23:39) [Clang 10.0.0 (clang-1000.11.45.5)]
Qlib version: 0.7.0.99
numpy==1.21.0
pandas==1.3.0
scipy==1.7.0
requests==2.26.0
sacred==0.8.2
python-socketio==5.3.0
redis==3.5.3
python-redis-lock==3.7.0
schedule==1.1.0
cvxpy==1.0.21
hyperopt==0.1.1
fire==0.4.0
statsmodels==0.12.2
xlrd==2.0.1
plotly==4.12.0
matplotlib==3.1.3
tables==3.6.1
pyyaml==5.4.1
mlflow==1.19.0
tqdm==4.61.2
loguru==0.5.3
lightgbm==3.2.1
tornado==6.1
joblib==1.0.1
fire==0.4.0
ruamel.yaml==0.17.10