From 18f646fbd8918147b12e37824a283bfc148ccf98 Mon Sep 17 00:00:00 2001
From: lewwang <lwwang@microsoft.com>
Date: Mon, 11 Oct 2021 23:57:36 +0800
Subject: [PATCH 1/8] Update TCTS.

---
 examples/benchmarks/README.md                 |   3 +-
 examples/benchmarks/TCTS/README.md            |  57 +++++++-----------
 examples/benchmarks/TCTS/task_description.png | Bin 25240 -> 0 bytes
 .../TCTS/workflow_config_tcts_Alpha360.yaml   |  19 +++---
 qlib/contrib/model/pytorch_tcts.py            |  45 +++++++++-----
 5 files changed, 63 insertions(+), 61 deletions(-)
 delete mode 100644 examples/benchmarks/TCTS/task_description.png

diff --git a/examples/benchmarks/README.md b/examples/benchmarks/README.md
index a18b468543f..49ad3f04a49 100644
--- a/examples/benchmarks/README.md
+++ b/examples/benchmarks/README.md
@@ -47,9 +47,10 @@ The numbers shown below demonstrate the performance of the entire `workflow` of
 | ALSTM (Yao Qin, et al.)                   | Alpha360 | 0.0497±0.00 | 0.3829±0.04 | 0.0599±0.00 | 0.4736±0.03 | 0.0626±0.02       | 0.8651±0.31       | -0.0994±0.03 |
 | LSTM(Sepp Hochreiter, et al.)             | Alpha360 | 0.0448±0.00 | 0.3474±0.04 | 0.0549±0.00 | 0.4366±0.03 | 0.0647±0.03       | 0.8963±0.39       | -0.0875±0.02 |
 | GRU(Kyunghyun Cho, et al.)                | Alpha360 | 0.0493±0.00 | 0.3772±0.04 | 0.0584±0.00 | 0.4638±0.03 | 0.0720±0.02       | 0.9730±0.33       | -0.0821±0.02 |
-| TCTS(Xueqing Wu, et al.)                  | Alpha360 | 0.0454±0.01 | 0.3457±0.06 | 0.0566±0.01 | 0.4492±0.05 | 0.0744±0.03       | 1.0594±0.41       | -0.0761±0.03 |
 | GATs (Petar Velickovic, et al.)           | Alpha360 | 0.0476±0.00 | 0.3508±0.02 | 0.0598±0.00 | 0.4604±0.01 | 0.0824±0.02       | 1.1079±0.26       | -0.0894±0.03 |
+| TCTS(Xueqing Wu, et al.)                  | Alpha360 | 0.0508±0.00 | 0.3931±0.04 | 0.0599±0.00 | 0.4756±0.03 | 0.0893±0.03       | 1.2256±0.36       | -0.0857±0.02 |
 | TRA(Hengxu Lin, et al.)                   | Alpha360 | 0.0485±0.00 | 0.3787±0.03 | 0.0587±0.00 | 0.4756±0.03 | 0.0920±0.03       | 1.2789±0.42       | -0.0834±0.02 |
 
 - The selected 20 features are based on the feature importance of a lightgbm-based model.
 - The base model of DoubleEnsemble is LGBM.
+- The base model of TCTS is GRU.
diff --git a/examples/benchmarks/TCTS/README.md b/examples/benchmarks/TCTS/README.md
index ee67ffbeb1e..8a26ff91533 100644
--- a/examples/benchmarks/TCTS/README.md
+++ b/examples/benchmarks/TCTS/README.md
@@ -1,14 +1,7 @@
 # Temporally Correlated Task Scheduling for Sequence Learning
-We provide the [code](https://github.com/microsoft/qlib/blob/main/qlib/contrib/model/pytorch_tcts.py) for reproducing the stock trend forecasting experiments.
-
 ### Background
 Sequence learning has attracted much research attention from the machine learning community in recent years. In many applications, a sequence learning task is usually associated with multiple temporally correlated auxiliary tasks, which are different in terms of how much input information to use or which future step to predict. In stock trend forecasting, as demonstrated in Figure1, one can predict the price of a stock in different future days (e.g., tomorrow, the day after tomorrow). In this paper, we propose a framework to make use of those temporally correlated tasks to help each other. 
 
-<p align="center"> 
-<img src="task_description.png" width="600" height="200"/>
-</p>
-
-
 ### Method
 Given that there are usually multiple temporally correlated tasks, the key challenge lies in which tasks to use and when to use them in the training process. In this work, we introduce a learnable task scheduler for sequence learning, which adaptively selects temporally correlated tasks during the training process. The scheduler accesses the model status and the current training data (e.g., in current minibatch), and selects the best auxiliary task to help the training of the main task. The scheduler and the model for the main task are jointly trained through bi-level optimization: the scheduler is trained to maximize the validation performance of the model, and the model is trained to minimize the training loss guided by the scheduler. The process is demonstrated in Figure2.
 
@@ -16,37 +9,31 @@ Given that there are usually multiple temporally correlated tasks, the key chall
 <img src="workflow.png"/>
 </p>
 
-At step <img src="https://render.githubusercontent.com/render/math?math=s">, with training data <img src="https://render.githubusercontent.com/render/math?math=x_s,y_s">, the scheduler <img src="https://render.githubusercontent.com/render/math?math=\varphi"> chooses a suitable task <img src="https://render.githubusercontent.com/render/math?math=T_{i_s}"> (green solid lines) to update the model <img src="https://render.githubusercontent.com/render/math?math=f"> (blue solid lines). After <img src="https://render.githubusercontent.com/render/math?math=S"> steps, we evaluate the model <img src="https://render.githubusercontent.com/render/math?math=f"> on the validation set and update the scheduler <img src="https://render.githubusercontent.com/render/math?math=\varphi"> (green dashed lines).
-
-### DataSet
-* We use the historical transaction data for 300 stocks on [CSI300](http://www.csindex.com.cn/en/indices/index-detail/000300) from 01/01/2008 to 08/01/2020. 
-* We split the data into training (01/01/2008-12/31/2013), validation (01/01/2014-12/31/2015), and test sets (01/01/2016-08/01/2020) based on the transaction time. 
+At step <img src="https://latex.codecogs.com/png.latex?s" title="s" />, with training data <img src="https://latex.codecogs.com/png.latex?x_s,y_s" title="x_s,y_s" />, the scheduler <img src="https://latex.codecogs.com/png.latex?\varphi" title="\varphi" /> chooses a suitable task <img src="https://latex.codecogs.com/png.latex?T_{i_s}" title="T_{i_s}" /> (green solid lines) to update the model <img src="https://latex.codecogs.com/png.latex?f" title="f" /> (blue solid lines). After <img src="https://latex.codecogs.com/png.latex?S" title="S" /> steps, we evaluate the model <img src="https://latex.codecogs.com/png.latex?f" title="f" /> on the validation set and update the scheduler <img src="https://latex.codecogs.com/png.latex?\varphi" title="\varphi" /> (green dashed lines).
 
 ### Experiments
+Due to different data versions and different Qlib versions, the original data and data preprocessing methods of the experimental settings in the paper are different from those experimental settings in the existing Qlib version. Therefore, we provide two versions of the code according to the two kind of settings, 1) the code[https://github.com/lwwang1995/tcts] that can be used to reproduce the experimental results and 2) the code[https://github.com/microsoft/qlib/blob/main/qlib/contrib/model/pytorch_tcts.py] in the current Qlib baseline.
+
 #### Task Description
-* The main tasks <img src="https://render.githubusercontent.com/render/math?math=T_k"> (<img src="https://render.githubusercontent.com/render/math?math=task_k"> in Figure1) refers to forecasting return of stock <img src="https://render.githubusercontent.com/render/math?math=i"> as following,
+Setting1
+* Dataset: We use the historical transaction data for 300 stocks on [CSI300](http://www.csindex.com.cn/en/indices/index-detail/000300) from 01/01/2008 to 08/01/2020. We split the data into training (01/01/2008-12/31/2013), validation (01/01/2014-12/31/2015), and test sets (01/01/2016-08/01/2020) based on the transaction time. 
+
+* The main tasks <img src="https://latex.codecogs.com/png.latex?T_k" title="T_k" /> refers to forecasting return of stock <img src="https://latex.codecogs.com/png.latex?i" title="i" /> as following,
 <div align=center>
-<img src="https://render.githubusercontent.com/render/math?math=r_{i}^k = \frac{\price_i^{t+k}}{\price_i^{t+k-1}} - 1">
+<img src="https://latex.codecogs.com/png.image?\dpi{110}&space;r_{i}^{t,k}&space;=&space;\frac{price_i^{t&plus;k}}{price_i^{t&plus;k-1}}-1" title="r_{i}^{t,k} = \frac{price_i^{t+k}}{price_i^{t+k-1}}-1" />
 </div>
 
-* Temporally correlated task sets <img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_k = \{T_1, T_2, ... , T_k\}">, in this paper, <img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_3">, <img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_5"> and <img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_10"> are used.
-#### Baselines
-* GRU/MLP/LightGBM (LGB)/Graph Attention Networks (GAT)
-* Multi-task learning (MTL): In multi-task learning, multiple tasks are jointly trained and mutually boosted. Each task is treated equally, while in our setting, we focus on the main task.
-* Curriculum transfer learning (CL): Transfer learning also leverages auxiliary tasks to boost the main task. [Curriculum transfer learning](https://arxiv.org/pdf/1804.00810.pdf) is one kind of transfer learning which schedules auxiliary tasks according to certain rules. Our problem can also be regarded as a special kind of transfer learning, where the auxiliary tasks are temporally correlated with the main task. Our learning process is dynamically controlled by a scheduler rather than some pre-defined rules. In the CL baseline, we start from the task <img src="https://render.githubusercontent.com/render/math?math=T_1" >, then <img src="https://render.githubusercontent.com/render/math?math=T_2" >, and gradually move to the last one.
-#### Result
-| Methods | <img src="https://render.githubusercontent.com/render/math?math=T_1" > | <img src="https://render.githubusercontent.com/render/math?math=T_2"> | <img src="https://render.githubusercontent.com/render/math?math=T_3"> |
-| :----: | :----: | :----: | :----: |
-| GRU | 0.049 / 1.903 | 0.018 / 1.972 | 0.014 / 1.989 |
-| MLP | 0.023 / 1.961 | 0.022 / 1.962 | 0.015 / 1.978 |
-| LGB | 0.038 / 1.883 | 0.023 / 1.952 | 0.007 / 1.987 |
-| GAT | 0.052 / 1.898 | 0.024 / 1.954 | 0.015 / 1.973 |
-| MTL(<img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_3">)  | 0.061 / 1.862  | 0.023 / 1.942  | 0.012 / 1.956 |
-| CL(<img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_3">)  | 0.051 / 1.880  | 0.028 / 1.941  | 0.016 / 1.962 |
-| Ours(<img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_3">)  | 0.071 / 1.851  | 0.030 / 1.939  | 0.017 / 1.963 |
-| MTL(<img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_5">)  | 0.057 / 1.875  | 0.021 / 1.939  | 0.017 / 1.959 |
-| CL(<img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_5">)  | 0.056 / 1.877  | 0.028 / 1.942  | 0.015 / 1.962 |
-| Ours(<img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_5">)  | 0.075 / 1.849  | 0.032 /1.939  | 0.021 / 1.955  | 
-| MTL(<img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_{10}">)  | 0.052 / 1.882  | 0.020 / 1.947  | 0.019 / 1.952 |
-| CL(<img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_{10}">)  | 0.051 / 1.882  | 0.028 / 1.950  | 0.016 / 1.961 |
-| Ours(<img src="https://render.githubusercontent.com/render/math?math=\mathcal{T}_{10}">)  | 0.067 /  1.867  | 0.030 / 1.960  | 0.022 / 1.942|
\ No newline at end of file
+* Temporally correlated task sets <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_k&space;=&space;\{T_1,&space;T_2,&space;...&space;,&space;T_k\}" title="\mathcal{T}_k = \{T_1, T_2, ... , T_k\}" />, in this paper, <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_3" title="\mathcal{T}_3" />, <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_5" title="\mathcal{T}_5" /> and <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_{10}" title="\mathcal{T}_{10}" /> are used in <img src="https://latex.codecogs.com/png.latex?T_1" title="T_1" />, <img src="https://latex.codecogs.com/png.latex?T_2" title="T_2" />, and <img src="https://latex.codecogs.com/png.latex?T_3" title="T_3" />.
+
+Setting2
+* Dataset: We use the historical transaction data for 300 stocks on [CSI300](http://www.csindex.com.cn/en/indices/index-detail/000300) from 01/01/2008 to 08/01/2020. We split the data into training (01/01/2008-12/31/2014), validation (01/01/2015-12/31/2016), and test sets (01/01/2017-08/01/2020) based on the transaction time. 
+
+* The main tasks <img src="https://latex.codecogs.com/png.latex?T_k" title="T_k" /> refers to forecasting return of stock <img src="https://latex.codecogs.com/png.latex?i" title="i" /> as following,
+<div align=center>
+<img src="https://latex.codecogs.com/png.image?\dpi{110}&space;r_{i}^{t,k}&space;=&space;\frac{price_i^{t&plus;1&plus;k}}{price_i^{t&plus;1}}-1" title="r_{i}^{t,k} = \frac{price_i^{t+1+k}}{price_i^{t+1}}-1" />
+</div>
+
+* In Qlib baseline, <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_3" title="\mathcal{T}_3" />, is used in  <img src="https://latex.codecogs.com/png.latex?T_1" title="T_1" />.
+
+Experimental Result
+You can find the experimental result of setting1 in paper[http://proceedings.mlr.press/v139/wu21e/wu21e.pdf] and the experimental result of setting2 in this page[https://github.com/microsoft/qlib/tree/main/examples/benchmarks].
\ No newline at end of file
diff --git a/examples/benchmarks/TCTS/task_description.png b/examples/benchmarks/TCTS/task_description.png
deleted file mode 100644
index 7a9005bf24869c6c88bec34513a67704743492ee..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001

literal 25240
zcmdpd*I!f57cHI8n+ix5Lk(RafOLc}M7q*@N4oS9x*$po5J0+!fItYnccch}CN)&)
zz4vzG??1Q?_vJq1B<Fn2%$$8@&z`l{+KJXwS0*EVN{oetMW*tfqBa&5jy~r1GJpVc
zeDrGIKg<`lhqkgj76Qe%g*m~qlT(w!!m5ZTxwXK@oD;eIXW)T_CEWh+gUw}ah{nRQ
z<Wo_Ud+TeuH&2-63DJ5ua_s(;))Dh9gr2l0?XeK=dJOSr1Feo*+Q;ncpNQ!Mzld|b
z70k!6eN76m)#_k;I7^2Ppz99O>zfXKEME9+&t#>GaC++Lz0;c#_fP<^MNrEB-yN3!
zp1hg5kEHB0aL<5ka9<Tpl(+!|j6amXg88q^TrvHD1Mp_s^pAaj0_(9zS{a?=Mc{Cq
z|Ks!J8P)s7)pAvr<ybe$MZIGgpaABwJYd=Quo7t7(pG`FNA7%0Mm5OsjAwViMD@O9
zl~xs|B}1C@EGeHe<yE1WJiu;P#Po&%!SA`$6?fmVTmc*w?h{FlI0FFo&AycdHLZ#h
zs34J9c6?WJj={lYiUcKSjf}`Wx;9t6X93qY3tVH7Y!Pme0IY@b)g4vq^1dASs}Gbj
z>24m`C9BfFS=3l=?G2jR0Q8LJ@)~GCf2NzGE#G@zqIh#>pi;-;mjy#ILrTJKT=}l1
zsGTp9VHISHWzQqrl&s-UFTYS&Kyne<cZfPpovAk``F5?QO~%CXiev84fS`bxrGqA`
z4I|XFe;EGi$7(7h2%*^FUN1gu`k@I{2nT$}E2aQx5f<I`QKXLaBT2KGoTHbvy&%@u
zEQZHc!1+kH2_I7==KbQD=6CPvCAz~Oov)yPVW9MIP~E{WLoxsnRP`KxKZ>o--@kt`
zHSNVmN)(Yh@p28~?&ZqeaDH%CT(f_4?%TaDtS@(ajZ%<uZ9301$!-(~`3xSk`p288
z<iRY@+lhUe&M&kub3E@i?ikER$k9Ga0WtQGxxR>G0jb<DMXo_dI`E1yuQAT|L$o2S
z7b99z(yHeT;#wr<PqS}OQu~rxa|JHE!*pwNkvE>u>gSoxijSkKldM*=ti0?^50IG`
zCTSe+d8ntlxl|kGowS8c)rSL557QwSXUxPQj(yyaaV6a*Cc32SHb7ZP_POC9Z#7F5
z!Si|UG2H_sTC!2JgX%pm#6spRY-uTy!L%S^U#g;;dWxF)<sm*WSm*c><_bCj#{~1b
z>vX&uZZB!je%P@6oO;-U<GHs03^x7CzZ{iXPZYjzuh>tL^3Q247OJ{;FQ)X$Cs)+P
z&kS<jbru&(Ue{8~I3Gs>^{D7^oqLa58yWP_4K}8N7B?*1Dj{B7WI3AOq`+5R09L0J
zm&u@zbw0>u$$21&tFguY+i%b}f!0^>+>CwH_d<ccv7G@F{73Twr%IRwmsF4wdNyC8
zx8vSaXa3$;d$X`j#{X=4`{*$cVWw-t<Z5<I2y``hHUi`Rv%Vb3&|0T>1MS(K5Dw#!
zyZcgZI^WludWm3N<~ivA&Mv3wmcywA0}>8P%1q}!t?4e_!6!;6JEN1HU6iOTPC9#|
z^-+zkDgtPIe`F@cJcIBaE2clJSH`-_KS$fH)(zBSk35HA8>TH1{_t^LxW29$_Hk@q
z$hoVjzpP{aopj50kwAE>bVawqf6AtZvx{n>zuqm`Anu@@HEko_7d%@9GID=XQ+>6A
ztTH5%F9%n-Bix9ew>QpuwtxT6h$+!*<>5Cq1?!%T0P4;9wty<#Y7Z=rb-)K<$)qeZ
zCVyD0`(MJAotbp?cgIS+cSNNw<Wk4Cy!A<%f(LrItySG7Wf?f$=wj|l(D~3pedlSo
z{ydg>=F_Mgonv<((`r0jZ)Mnl&a;HON?17$5ymq+FN+yb>bYn2)CNCuc&J{y^x>m=
z)7KeOqfkj1w5<r2>$rHQKshUuv?7<ze0q5T%rZ$oYOVlLE%Zetd){yxr;^>B!@Oz~
zb3J5CkE4KDEe*f>;>Nh0i$0k!T&HtTuw=Q@d=V<$o6BSqNn?p5XN%C0|M9wKEkpHB
zce7Q$lcyNI{yxXs44a1KpUc;aO^WTS7fTBTiX4tiL`av<%DVMc#}J5gQ+GbD(%shz
zPv_zt6=3;xEYdJkbF-Yx24VeU62EBnfFIaXB`Jb$^?9-BP%475(<@Tz3EgFE)hISG
z#~jnflWRmF(GJEH!C|;!k9`N*c>~xPGkmxb8}374$)7Q^U~T@<`Et}Qw1Wwv1f_E7
z|Le;PH!P)8$MsUU5ICxm*zc*vBWT>{s9P2UIR-qLgrqq65B>*OmZiotJXK)k1I7$Q
zeU6>zD9YFhJ3unMG|Yf^cB)<FQV$PaPgBduovZ^Z4IKuC0NKpCmPFo)x(Ro6-7*VI
zjF`wSaqJJ=*FPTQ^x<jPfta5)0R;@H;fc^Au9#HNRW<dmzRxj_IZ3KTC!9Vq1b6Fe
z)iMB>r9SNr_#{7XpCx5FO(PMjm}`iT_OJddDVk$C)<#H=S*n(qo%|z7@%&fq)MvV+
zRkCtge1vrHqkz~j?T$Z}9uh&p!%UtOe14zAI-dpk9Da$rH(8|uW-z~f8dv!?$M9Kr
z0ai9Ze?mhVX+20BNvWQIr*FYB7|AEz*A1Y&Hc+yVnsuB9SG`DDOFy%(Q#+G$Xq%_9
zPn>*B@2~>?zuqzR@Y<w&Z(z5!YxkEzmd;|*9a6h-Ii3ykqixGro7rZRn0Vkuexw5o
z((M{{jJT_Jj;?JQ-+nipN|^lWYID&IOM~}nFh5uUU?(R`V?lGuPM_9wnDl&>Uq;!B
zF6i0D6Xl0U+}wc5BDM&TXV|*`_VZ4nBrM*4Pr4kT67p}4<vH$#UGVg4dx9ThCYDmZ
z{&TZue;0&V@53;)k6M(QB_{4;9wVhep?@kBvpe0Xx)@M*R{XCq+nie$6NFj!*+`?F
z3bO`H+-scDe_bn?B39Xe@qAYyFg(L$*5jD$s$W$>re&+mh&#zT!jxv`ei=XZ-9cy7
z)&|Vps&S|^{pCWyF9Q}(YWfj+bp$)$x!Rs6L3Xp3-_7^vs50vxmjNF(V=2AauqS7Z
z8G&uu$HXOi7;{j(@mvi)Dq*Ea8X_d>$=<^pP}l%-9l@f}a_++r&vR7|Gvm7w_y<2Q
z=G)x+e7W`PcTBOy!FVBC1d<&h_xLIu)&yVWK>qIzsdU_u9A3Y0STgE=O&CArs;W&0
zz~_b-{s)PceQ&k<=i~3m{&&Wf8BjUN$S;}wd(H>@?sdNm;~hh1!_qHE_Ri)a0|wL_
z#QpD1l-OroDLT!TUjhE?j3hEl$5p^K;|~~6WaWMXTNKF4VOfsF3~!-lonYFw`b)PQ
zA=9SY8Jabf^SseM(4_w%IdEiqVy%5xe!gK9RC&Fqe>@B<Z~h08*mnDPQvhxYUT53a
z$E84wM|8mkDxU`pz;Xnxb|*@9h7}-l&;m07T=?c@6d@%0W1R1+kiyT4@6UKJ7EUJ)
z^B!4^3bgrqLH$x-51^y2fz*$ZDYriT1~v)z)rx#*8$Zew6V>Vs1CZafly=9q2Ez0s
zB|jPW2a3X2P?Xtus@G9~sfIwu1$<EE>opFrT+b3M!*irCEueEO01Va^?h<OXk6!dx
zWeqM_t%Kmb9coIse0&=K*Z)lQs98Zpfc7`zZTn~&EzZV60q}zRNQ4<KQ_fk(m$)}-
z>v|ANf+9(IRk~WH+FG=F1ayn7j9?!&`u5|%Pc>Wnuh{mZViH(Tm9?Twi&M1u<rCn8
z)?djsP2v0OEa_9m9~wMYr7$~wza8q{_Qlj&^Jw7AzZd<hMHMVC5KipZ_V;U`ZND+a
z*(mVAv~wo#UB49s&j5b0P-?w818FUYIJrVlqX&Mv8Jp<}TZ#`HnOU4|+iSCdkC+QL
z>S;etfkuPE`lW-NmHyPn>Yr=BxFZBgKV-ouc&|EPgz&*4xnml(;CGe*dkE*n@7Ak)
zs^A+9qZWyu<pFLgIlU{Yr^Byh4^P%G*yPz^@P8^hn<elUp6gK6`&z}3vl`&%Tl_b|
zRkL1-8GW-qON0}VZJf!Qbu2veBEdPDfA8urUHzShoR&$~k_H+Gf5dpDdcqX{GS_~a
zf6XASDtM2AZ>x0sB-PE{#VDA_m9$TG<~|GJe?tqZxZ5l`3;=5IZz;c=zLx_A>mK(k
z)XRc>3?6mFzX9~3g`d?QH?+%%Dn<?`{k(Be-4i|D24*&6DJW^qe`Ou4June|#(LOV
zFs0;P3tDkap6g(s1$b#_fT%`fzKA_>yC_~Vi7Y--i`*K9RDT5Re6sXAz*y*d01h?~
zCOoYyc`*F2Gf~1`UD4kq`Ol~10VIJRh&E2)C3<4YCh);tjpS;PFq=*p{))FEc1_ax
z5x7ZS4tScCr0~F}K77BQ>7y!LLji<#273jn{Lh4W5hjt=L{nS-__^gPvDgN*2gBGq
zg{i*5rHM4MfV6dQZGa2fw3B-9PCt5W>Wgpc`krca;hP<59CkVBEc)`{%AC^Tn8m>P
z10(s3MccD&CA|0WM+{IK?U8{mIL#LVR}-+RB0W+}I1EJ&K5r5tkY*3Vo=FlXIqU?6
zv5VZMi}7gL0VQp=T<57jm&QVko~$52!^VGPb1QbYXD5F0Eq}$AHEZ$rV&9Gc(NI6c
z1H$m>!-i3(d~BOFp1{7bb>_d<o79lRN>f^@0e6Int|w6_h)FyDrF@PQ-5-&Tby1zd
zA6eEt)^1Z<8%z+!8#KiFzO}V`h^{gjGxeQ@VCDhsZ4LO+_dtZ8iTlB)0_o7src%@S
z%$t;=jH5dZzC^7wE%0DyKXKrG&BJ41*OeL7?1RlkkB>1?vm%fPX3YjX!aa$N0tra4
zuuwcG9T#Vy;(ZMs_4bVEcWWYunXpJacSY89emTfMUNetuv3YeFh+{;-U1RKLI+DHk
z))EJakm3UlAW>d6pT}!{^<p6DqnpR|Tbu*I!VzKP^z?jCo9Jf8idR-nOdL!HT7NkR
zov)XHJJ*<K)Cv5N6MXja-}EyLx24n%xMi-9V6|37;2!)v1qPny1961N0}**wiPUl*
zm%f1E7n5ux>>{hm<~EN}3<wR~?y)WHmIUvW5gQHg6;+M?wdH(OA9x!<NbOWS@Rf>n
zCQ7vOMNzF<*%Y;_;Ws6f#JV*L1OCv#i`3E*uf~PGRqo$DBEPhkIAy;Z!HbnQd9o&-
zxV+MjBAuv90c09qFn)&yMML#}B)>~BseT345FM%UfzwE3E12nj^E~;~<cdk0joCnU
z?yM|T{$3vb;yVnw<t~bXtd2{gq>%JtDs@N#(D!Lp)>F1KK9?rfC>zNeCX&w?dZ5I|
znm((UD65cP%Z|ehEnm^e_0sWmbhx0<h+M6unyuyB){|PGaCjK87<o+JIk<IJ6#brk
zxXZsEmfn;KK=_M~o$qf}-$4Ra7?-X=3XbK#O+1sd@Z%{ys!zXN_-uYCZI*kFxtypy
zX6BW5#Xq0Vo?RH*iD6Y`ny&wY04xM;G&v~%HbT5=u-P*4_5$%F5*jpf7i7{`)Va?T
z;nIe*cjf<0?`9^^3i{%O7I64(pXuj!j`oMRYpx8Msil@iG<AX2+WF2=9r*<2Go`5m
z`-Xq_<v*Up`7v+hXeYWmfaij6>PunXbVt|us@6BR(>0|@-XHcf0gH>vr4!x7?{ok%
z1mEUet_u#bb-onDZ&><s1zu!=40xwa&2G?l?vtAV*ttzNPdn}cnC@<Z{Ur$^W^dRb
zb9u!CkdW`K!%X2w&+RKGG80%!C)j|J)TpOm`0hV~fx2s?mkHMN^_KDG4=d3ye#okD
z9A>p3DB;3j@`0!DXt2pk&Ak`a>39a23520R=KbZV00PiQ_x1PcuNb~0C2?0caZzp$
zekzT_g48;aHiJHjpZgj$H_{&%M7o2pr?knw<nFz)MJ^z_uZ*N09~MQ!pj%BiHN488
z1S(kv_KvU((HV8VO!GGhH$Y6JaCVi6HHRb!$}o2+h&bOsflLW!>|hX|1pI66mF-8R
zC3A80#jMomEc1nz2L3t}Yom|$s~L9=HW5JSc=`&^hJl4}0en6C?YBu)vOK%@a!R{Y
zK&gh@q-zMJ7(poqS*6v=5MWjxX;VDHR8d(Dr?C4oNaU@?{D%uA=Y_#N5OTsEIrmPu
z4(KuJ3`jfz<rqKOq>jg53c<HD(VSP$2#38~p2MSdNxU!#MF$$)TqJ|{^}B4^7EL-W
z426v@RvI^6wtx5{L%4ryqluRXH&7<L!3QFYY`a9^G<<Rn6pQ07`?@-P<<59k?;H0`
zKEzYYMY8HzoY$L$ehrZg`J<oh#c6k-XI`;SuJ-E?XZgglE|2xyA$Q*x;@bF7*wR3w
zCxO4w3Yqf&{O>FqxOS9oemZ_G{P9%B+$LH}P@=XlYHypf?Zt>aLrp{&s3V1_BSz>?
z5_Ra|*s&AFLSC#w3(;YEfuZV_4H5b{lvBq8{L(fGBum41IT*BY@bY}?<SMPj4CTdy
zu>?oCOm{FV37)>bGcFybi|+IFl65rtI3pY|OM#<~zZ1rbrlA;rG=ehV81_8Ax{UCs
zzfnJYJ`N@vj%`}Cw&}UfJCWb%l8}<$LHj~y?Nns;y3Wt*RAl&QuLJK(hwV&Qw*!RR
zt`*2qi6RH}Fwf_<FFuSt`MOJEyAO=QKUuy!g<bqGQ^BwtR41U9m|gs{4^N_y2X5$^
zYSFI~Ruqz@GWbI3`r_fx=o5T0Rk-V4xNit!*I7Jf0=gSVv0DLYiU@Xf5YaYj1BT^V
z)aPAerBA^W!(RDehGG;cULx|_$bTe?(Odx+;4tG;_QdzvwGJReW1AeU)m=6c^d$cC
zAr@vF9G-!)&zm(XI*_^1hOGu$%s~I3^8eR|&a+i|#9hngD@Z>>%Bzj-2{N%;<Nieq
zn*X2a<Z`)}+Q49~<2E4X&AP;4o$f)h{V@1o0kQfk{$5HaJAYNG{Emz^)hM~yBD%FX
zpx-0K*gLy!)mfFltL7`Rl4ckKu)UEx<vPcvz*8=7He<j2cgMEM52kvgt&7}OD=;H}
z_FEPK!()7)hQL(`Y?KH?7Ca>1E^SZD9RC91tUh2UQT}I{$MMO5vi)|cw|f1qDRYgh
z!pEJ!9PQ41I^{aPnaHEdC}5k(hXB`15$Wdcn!pK6Yjfet79flEF+K3X_}CFx$4#`X
z`i$h*6=+t0!Tc8O50x0kah-JNUGJMipz1whwH(rewF87?8y;_!=ta5r!t5{%pMJ9!
zQ5QcArFy&`w|K0Oe0fm8>V1ixa-Zd$LO(;PsY!5l;i^sCz0_ko9AW7;1IrIcNQhS*
z?18`H?jT?#(|v;)>U}U#QnjoF<S-rzNWOLMH_<KU=g>8nUWzwqk$_^-L;n=)t(?M?
z=D*wTz>lSiB1U~rFbT|;x0O~Idl5F-n!<$oy|2yB)PMp=f3HA(h{=-QYwrt5h`Lq7
znmPvfAN!Z+ZG>Yo$<+K{jTBeDnV0R<$x!?Ls1&#Q<sZ#X(b4dNj)91N0;UG$l@)wQ
zYssqrv`+739Chp3RWntbZ;xYe{FkpJ*eFg6W^(J2ObJ|gd366N{e(C56|#GSfGWy0
zHY+rHnbKMmBsOg-aX$}PkLh<by>axzVeBhRIrYn2bJ`U?K>^F5rw7sPSRcnj*>2mZ
z>3hdz*R@19nh!03PGcfvrdcc?pZ?>AH~m)4R4$$Ks{Ck0h~`}f@WJpnnK~#oQ;EX>
z4Grx7Qfitd-9!IScB`lovh#Obz=&mkNCjViENm=M;O((8uubzDvubPyOY)1S_Zbjo
z6HEr!rgz*5)Bs(L!WxRS%Q`iZi&$Vm)fye>x@5q}QzmXe)t|G%1+M!7MO6a{xX$Pm
z=SOc3#Bhs$4byMiVjP06Yj0jxK=xeG+4ulzXGsab)O&`5l85Q)7Z*;z>j?V;FL0gL
z)H7di(2d(%q=oMQHFV__)TK#{e6pd*Wn(RPN*v{DTsS4oGOE5%7u@>{<)~z2PlMhQ
zX+MiTk*{mGH`z#QIq*RD&kZS!x$m9mV%*5N1iF4Q%4z!*+6t-n-mqmr^VgKu9;V;L
zqa~wyD!w1d)F|F~OYT`N?Ip!c2dShs4qZQObY4Gdj_??bY`K%}NT}UBuVU%-k(%v*
zT`E9a^uDn;jYH?AYva}0;5}C3==WmI6>4?613}94<!a&|W7<HIG3OH~Mf6yXbMHyR
z$#%uKMEg2wdh$kUy(Zq<DR5t^*R)e;7wx0O#4z(i1acY#rI>eZT?8{rxySW14cjrD
zH=fB$mO4s87reiF`7X3Bn!WEe@jHOI3M4w^;>uun{j^L^wRF+jPmbS)_7<<=8#k^`
zTP|vxcLv!oS!aBTJxi#rS?YDe$%~oB!?`>~PyR~-dCnl8=$gB!v!0ipgTe4V<H3$j
zDC7@M^X>KoRW0qS^tBhsuaO*Bjy6Yny(H_U(46P*_3>QW3@iKD1bc*CQA5ICN#|XY
z>vgdg^-@n?>UfF)E^iAVq&>Z<)3qYuF>r$f&r+PX9i^Sa1*9<gkGW?95T_djAkK(;
zbV6dAS$|G4AX$l2edd>Pa-c)>Ol>H;6}}f=j?wXVSTK*DTl80=saICAkQ+x5uMffr
zIM(rFtkMA!Fae%X{;<uShU;vnRb_afLA19!H1^IBcVpY*^+Z^}uf~i_j;O0>VC};)
zwf_$dJ`MfGt6G@&p~YO=kBF1=h8_1gZ(`Efn-p+sP^Ga8?C1d4vMOh{kk@_DTDfWm
z*+ejq_nZ_km}ws$!d$toPN*g3eF7!K#FB5VqYWx{Msqkor|a&(7_QB1=^wZerkmx?
zap9&Ct?qkYAKQ6QTB%K#Kz4UhRT=e+{O^Q-JDChI>?<cb5GPk6bPeEhZC+MetzyfR
zvdPWjS*vlh_X#umL7;WbUa;|p4^DgK0&Q*GUQnxkSeH(cb;BAPa8bu{ZZyMvcNh}U
z6Yp(>j!}@1Apqs*R90tl?HnShX}aS1-N}098?g#{;rBvui)GzjV}u(k#x);3YRcRl
z7Z)B%scGgoQ<254hJQ1eHUHia4=PW;Jy&jW(1`ES3Rn+dN2RPdXH^JDRF&oCkRcb9
z;b5P}<(dlDxd#3fovbp+Uo+BHULX>oQJ)~E5(b~sl8W&aSxGt)r__oYT_Akdn)>SL
zfrEhDSD;j{8^4QXy<LAElFzflFN=d&uiIAtr;lFCxs@Jz_%Oxkd&qFZq1rYFkV`an
zLX9pQ;auIn9I-6O_01ki6pu4PWIK{iBlP94RXsaV>4F|t%#t!z>jILh$@LkJ{F6t%
z{f@1EZ^wu4F;9CNxZ5iqv4cnWW!`oA3Azvna`uAhu+BS#`l?kbgr8-y;r80AUT6n-
zG0c|RImNIQ-dq6pJdY4S9PYZ`UH3!w)oXO-4!y$y?m<=|1;Nq{5uxKUBXKd6tc|H3
zhVJPk3BG!mMcr|$uk|qwyfmtONAg_O>Xxp7MPQV}D#R;?8eM}*dweI`4^E>uiYvEU
zWlRr&sXP5_TXvF0bc+ii8S`o<b!Yo;NVV$f?j%wA_5+(DgPGu7%WrJRLzBH;Ds-VJ
zwQOk8^*k&)-O(uSb5a1%_0jp-^a@i02L~d0)K0MG3Ju>3#%5Fg>%X^>q*}e^+Z7#D
zs1dW>xZC~7+~DDTylCqwMu>zrEOav9?%#XPH~mR-OY4bu@~IPmPQY;x&VWsPoY>Y1
zft4a53wP<T462TaoRsEC%tFUe@|hv_?0u8El$ifM%Lj2fs{a(XR=j;|vhvK+A}N+`
zv?mPrdK55sw%3R?(7Gjq{(H%x;Hi}`oJhPo^T>OKPI%qf%1ACh<W>&23%IKk@LxR<
zYX88vGTDm$L5#xJ=utJRJJ0jLX_=(oQ>$aj=X{poX57nyPK~|zdY{o8Xb~;>Zsxn`
z&=Ti#(vgsqtFa}HJ@wl3>?k|ev{fUAFU{_@ne;Xb#5mr4AZLUMvbfvQzqILdF2|Sa
z=$|uDGX85?SX#}uC%2z&bOTZ1H5Q;7y%}wOL%BPCq<&%x<Bg{m!hcF;tWZ2;K1lA*
zF-jNsXuj0BM|OE+h#3qM5@fKDWkQ##%+8;a4mR$^ldQdPoU@5VzZ4|4wXynXl(3#j
zw&^0mt&4(*Fpsb9)!X>2pRd{P7NX-V>`2$&d<*R^j=_F{N75=opdS;9EjIiHefV|H
zd<v^%l(SX9G1s-4J!)PlMC2<jNG?9+asH^Fe^!U}2tKcexvTBAek+<hkZ7+T40zvr
z|6;|H{~CeI6FFT9{lj}RsT((8z!VximFZ>osXG2(Mj(9nN(^E4;2r@D|73nW1?lm)
zXp8=IvMr4T^~uiEWDgPc-p!xkZ2y3GPrF6R#F#gZ(9o3Fx68pApw3q5p?gz7HruX0
zLm&m8cZez^Q#-E3*YANzKN=j~G(!)3n$B@%ifWAmgk%=>tBVlmBNV(n7BiqRT2S^R
z(eox7#f?fLG`O|&4;>yLuBCkO20CMUKqPz(UZG|)+QH1bC5dqrX#5iqp;W-xL`l9%
zL<~BUKE<zQ_=tSm3xuz4E`#rI`=8zK85Qp$XiW6P8;xi9tIm#inmvg`2h)>bcIe-K
zE@}m}p1f`A-=4T1Ve?Lpm2rmSfP!NQ?;{VMCK_0GO(*)59)!h+OEPe%^I+xWl1i_M
zg{LhmlF9abj-lH{DM&zj-k7`{64DIh9uflETZ;@HTZ?j0uc#%G<*K=nqZ}8#DJcz6
zZY^kJ8Mfn-SEiD^oM4Mj>))&#oYm>zZ^bo4gnASy`+e*5)u|Zinh_>4_W1|A^g@ys
zfLN1+bYvNpMZ~hlE4g9!CE2ss+GXpSVTnP+pG$WFVMau()y8wSqNTT6{nPxyX0soq
z<03!w_)%VqoEJL!IMIt>Gi^LCs)~5-ySJi$ZDY*n=pAM|bd3*BW>SXB63BX&xZmOM
zL=Jn&KY=ME;%`}9<*>crZ}k37l@{+6nI%Lgl5xYzy1G<NLMZX=4j)I@2IoP6GTCD?
z?D2{wCGSq{j{YhVf0Xg(;Ky_1^KKQVPT$lN;UK$H-WJ;DNC%XLWM53IKF=RLoDups
zvqO%hS)WX|&zUJ9nr2P>dlu=j9*cM_A=KgSnZYI)$J;mN>9_}cMed~vSo=(WYnZpT
ze)H7&W!@g+0L*!ZPbesc%Idy;gz1_bJKRys;+-yAeTI;dGrRrvAX$uZQk4tt>4h3d
zsp-4xbfOR)`tgjdXt)0RrdGTr^ADEk2P5uH<lZrNwzQeOpByK}PP?Y33FEIU!fhUR
zbz|^nW?WyY0u|}tCLXGAYLMZ;9h8<O?&O#rk!N+QxP%jR)V!xa$bfae>=7Q<Vd?`-
zR{N?y6q&o1Am?FBx*d0&x-D%>3Xaq^A;_QnHZqqhNVAhptCU|M057P%o7S@@bjySu
zkSGkf6IbHB!l6-W&idpOx>L*es`d*g&AUOlUv#57j{Gm4#OWKs!O+-VwlI#~4?$>q
zT+uoUyZ1N7_?2*(KhRTm>#f1uJ%#n97gYN8MiR(JSVjvpf7}pJ=508#=~Nwfg_dxc
zojD0>c~Hq$-&@OQ*)QSqlT%r-#$y(8f-S=NcI==+_n#-yEt7g4OGE?qo?fWa-7rTx
zimkK)Q+oP=Z4OEZ6iLGPbE)@i(4YXC5WCa=_fvF6?Ph!Q!f!!?R)*eB{X{Jj(uHyY
z7yM@>OIjf|!fsD$Tf>cL>{=a&ax&`saExldk4M=Kz(*c*znm_P(XBL9K+gTT&(E)l
zCQ8mmq%^RQPEZ?gF%~2wAKQimE>Ui$?%@s5Y^O_C8A>ms+PQu2Z7ogrcF|!ls0Gd@
zJbhQw((&rYZ-y$DGlR+QAyF|KG4+gWGS$U{Fu8Biij<NN1~}@QXI3{G;Jfue53sOG
zp5Q-ZH`Ty%-|i@LqgZ__-S>X3I!%mW=G3-TWG3rx;Bhr1OmwyRixKTE^A+A?isZlC
zYqH$uC0;(Gqq09WX}Ym=A}+En16vJvpewo9dm47d<`X`e(>|ISc<=lz{Fx8eFhI$;
z>fZZqUe8KxSQL+@r)pMuqgtSqSM!m+qtR5CH1l<?u74*pv7q%AijQi@nEC-~T69Mp
zXv>+<l2h$sV(d!Xr*|z_edjVG!5n$ZE7`)YsoVP;n=N5aCA>1Ug82{(=4TS*HYBc=
zym20@q~M_T9BdF+_VqMC2c(^`K@w=;NyNxq<y`qh28e+PXD62Bi?@t1Gv(@bSX?1b
zi4AyuZ=!fU&(P6T(r$L(k*8As{xD6L*C$a)64B&FXDb-fm|REy%ix=-LWYW;YgehR
zV6n8dw{K?3Q&K4FyPbnpkwmIH8<unNolos*ZvHKs7*3RqJ{wWEl>xjKVzq@@F{!Mm
zvxaW5jmVHseW2@gqBeiRbtBGxIJ?IGb+VyA^!o)|G%rYgU7l<x=0rmMVLz;Zw4)3(
zsg2y1vbq?VC4o8^OvUITYXvwo<A#L{&S^&UKhMC~w)$g&GV?gYysSQ<+=+LiSzStB
zhC9!%z2X<|xWH4`O$j?|Sg@h#@tW!?>awgej=z{0SKtWm`4s&;gPTh#xt7=2PU(${
z^_ZM1e!0tTnls^DwksISZ5>dNEWWy$^<`-pOoLF^0v&9+X4);vT4$C7taWY1dQ_7-
zPguU;U?jX*vrE>e8yC4IaDh(~uJPfN@|O!TzloI8DN@ouYT8{kz^k+S{R)fx*V2UV
z=QNvwYh6c&0KvE2!XaF`*8Am6TXcfIe_8xi6-=|0W&9*JTbH{1ALRMieEzp{rpT3m
z&t_f{`QOz$g{_A10vEL{39X`7bG6v5h?X_!?5DVn4^DSEK#$%0i8ZCX^T=8~s65^8
z5rvL3cu&gRP7l;ZveDNjwsK1w>a2G&Xpf)!(-o#|aY@;EzJKHw8D)E|S?F122<ly4
z3e;+)irCKop==52&r_PflY0U|4!av7R*fIskW@CB;Qhw=sJ1tzPwpWdlVCFxNke&g
z@m!u|N5jYWOjrUh(M5=5rz|eKqBcS$-eY6~BF9Xa$JE&4WVZRE_R*Uc2<Iu*FqDs5
zzSKN@L5_9B)Ux5`r#-xGkH~)QmZJ_AB#ZDmjxaWE)1TVD>AhYCkT@B=Aia;Ma&Gmb
zTz-oOoL||LBjP~pb1`+E3%Liw&h^b6qQlnulaVU`vL{NE;b7N$YN?2JoCWSSbFj}I
zH%~WXBkTv*cX{D`$z&Czh}EaQJ*D7$ufhs=?2Cj1YF`i<$;T_PtOw`pG$n<oA2oz|
z(S*|zQM???(1lIr#jEjD3kCCja>?Meu$&pMJYsV$cfM@qjg4Fbb4C9*>5*ZpQ2Zov
zj4Ro**MbnRZ)!F8N4>_w_7UBSP|B{q9!v)=3Es6!OL#gl+xlE6P4{k;a_E{b;VdAk
zXgbh=$eM;!1pfmcdh!%MB=eK^mZTXgT@Btyomja05)wpIoyfG+<oF3o!Tnox69ZOH
zdbMax>|Ij{tb>XXlc~Gp?|;Rwp<2i|NVfcs{%ub2JFE6;P6c$mH|n=2+;`4l+=qs*
z!(e(+oUa4xaxT}&8UZGhm;}>Kji+smV=or)W<D!?UX&w3Au>DX!>17o%|i{qK9f3F
znZ?wvP(lkRs=q|}ikgU2?Yn@zKA^@$P2L<Zu1jk^pYPR9<1tb`ctuf4mDZq+J+<2h
zwKMk4-+&kFmAfh|zWt6MA1F057hrWdjdvFjcnP4}3!n`zYVEGQjyA2|TR3|UIUso^
zre<D!cc(gwvM1b-kp7#?!n6X7m)&Z@RoiSdZ%QSui6spq>IP%y89~`2JqiL$19az$
zlEMPfp0<P1b8K5uq1$7=*%lXgJgAyIVxu2-bvChlRT7hz(<CF@XF@AOa?q$faHJB*
z<GMMgZ7L1V=b?!9eTJZGA#y=faVSwpe(^>~0xdXtXp&Sy1qGsuWhAGWD&+tA3ZqP>
zR0T2%NiU|$PY1-)72`I8d$U>OEWfQ7HuZS1yiu}l-<uMOWfHS|8TP)`fOWd>8cEbt
zI|GoMIM-wtlbI2`ejnDs5*u$VXnHofY2;JD;-PR!A|Vs^g(^p2ez2B&#&{1%(8tco
z&l*LbpUKb8Tf|7XN6lo;o#staLkJ-j#81u)U^kM#qSvwX%L~4@>lRKnIuqBZ>Nsl9
zC*yp!GlE^ln#}&VUVKB{faDpbt}lB>D*|krYfXyb-7`g=G6h(*;XI{}_c{|m4VS72
zl|l`EQi9?;C^k4*Pa8_LSRj)2v=FS!kGm7Ns_Wi$PCpJausLFG?U9qC*l=&buE&Af
zEJ}W)h^LK~@SabSdXQ&U9NmiY<`d5Ezagy|Ppt?AW_978MbZ-Sk*}O2gL`|m>;nkY
zSZ<7ara=ZHPNLgV<jNtl9qd`uF<W0u;Pm)fnQ<PTW#>EBy@RRHfZ;QVAGL`S({Cw&
zOO(kwymI)3>aKhT<Bs%y)d1J{;-^WtqvTd@_<LLA!W-W;l(R6Ii-Ie@^0kV&i2B>7
z{;Ysn7VMp(rZ9(&ypZ9I*}eu6EWDTn_EU$T-S(K;$<|$%(84vpZw_av2$Ok(`oQN|
z#q+@((|aNEnnX~mhp<D+N5<62EOxsck0g2eUH8ndtMp%_r(9R8?Yp0=vYb7)CWpf*
zJ(0Nca1L?~oYVhgQMUd$NC%G}#Bgg9p9|g_wiVIy5_gxy^Od>jryU<<M=zRP1Ca*T
z5%W)s19Du3<!1Q%C>x$&&+fY7afihp$(1shkCrKzo5es^tg5YgB6^pRQl}xUM1RSN
z5KrlOz26thD4mlbH~BI+(-vQnFruh1-f=U64R7bjl#yts?*8bewS~1qxxs>ZB3C+i
zs-Hi|PqcZuV`H(1wbNO}cSHS;vmfuOM3Ao4J8?<!p;xt-mcu@z;DYK-ehO8*&6Bn;
zB9_Jo8ej)J{ol{#<eaz%VWlwafo`bJ>T5d_lgWa09s>PIuL(H=w62wT%i=uWg3-@V
zasImz`7>E7J_RsK{f=fEpf)v|f~M0?8LCA)#ao23KKh%7QfF(FFbI7dVC{D}(xg00
zvj6?vMX^#+_VuE$_u^e}alh;<DZCG#d*<iubuD8}ju1^+*A3>&yMp=Je|}=U?WmQE
zcvRmA(Kay!&|>>cHjMtV;(Sz5a`Cpkj4n$(>I<&SC`Fs-mDzPc$d|~8yn>9_O<S2h
z@&8BiOsB;wqyECwTS$L!YO|I~y?7l6Q`2Q<($bh1kUF=&xhS&Mpep@iwL!1E^Yt&^
zz#D4l6*F*H=fjL&2kb(ifpqG(bf9^EnO-?6&W_9P3+yIkB!Bv&+nOI7W+#1p(fF^H
zOXv6xjP_XRQpWV>s-x<CK1P7R6CT2pAi4I(W0-P2Gu^+&cmvCp$KQZ5TE{3LPU!;(
zBbcOeHN-UQ_tPfd+7QjRIQNU1Hm(js?yFZ(3t+oI>u5*6K8*U|RzfYhH4n)g9#;)&
zTdmyI0vHK(CA_}A{wFM8Uv<M4`Kg>fW>|{y)MI#ng9nR#<#JP|9!L1AjnTH=zhsd6
z{n<q~|7O$Z=n)kw(tBH?sOjc+{VXYSHkg8|6HK*}=jlN)tI<UarYNn_G31N@$RQav
z%;|@bJbJ02=lmslQ!IN|)3D)xl{$Qo``p!c|74GzUORX_e5WHLxev#bBBlaU6nFVg
z_4w|=H#+Z#>AI6k&~UNd|1sQGzMs*YiB|6HBH7J)`8Gc%sA%;C2V1$$GkePREL!Q>
zRTGQ^^{%zXdI}4o$#EFiOPyZsrh}*TfYF*t^irdkSq^Iby&C&Y@67wph>_BDs|OI%
z4`)23f;wPs@GONug~ouVfR9>sopq$U-pR^|=|*rXtl*{hO~5@Ym9L`Z^qM4fP}8*P
z_y9BU%aL|GTL+ZIw{Dem%`Dm<{+0PJ2*IT*tW#N+C>N?%LBDY)EyjQ%0Wk0V=a+TV
za{D<cw=rU!YW0qo;Yks+o`>P8##0U#;)|R&y{@JdZ|IW00+679C>l&X4zt!XiH8m9
zt$8zns~Xr8j{{eFb=7b|z+DaEUJbbY!gH*TPgS}=as4$k1-h+R)fz8Ae^pCO50ZOK
zcO8|Q_Kw<%niC)YNp-Q<48EMT^7(cb-g5yACHscm_fg=}PsDc4l1~Z(7Vnz{_?jNI
zixRsypL_<w4<~Jq-@T?xzI6JEsNU4aiXkjCx#NQU`^{47E84}K4SD{!;8<_$22T~Y
z&8EGl*n5^D{TGU!X%kKSXh7V9zg2E*DY;i)k~qnUB8eF*frA$UMPyf?cL(4)fxJTY
zxSdO;zUKd;gWQJLua_UIRv(Og$3C5I5T6`K1=(a*e;kexaGpTN*L;dQa|dSL<RmwH
zM?0Uch-QjmvbmG2Gv$n{-4fN9m%kQF9W$Ia9noj!9!4&IXCQc5t&qaa!UIP_XkRNh
zM}w<kh*7`s{9d{x<HoK#C@ZIR?1{&Ss2%b&<1S8kn?`EF6ms^hhZ4%>Iq&5>ZnNRS
zSSRJi9M|xoqPpy23;4%=d#b0#&(%cNFvr16q7JbzA!MzU84eYQoua>~xwA{PAJAw#
z|GMW8^H57KrE8Z(e`cxERw=W~Fgy8Fa^?K)-W}e$dvcG_vO1OTKlBfq(K!3UHiKJH
zs$;InMMoa@`Q@W+*9$kEPPKJ6xp!?1iotgZDrU67r7WF97P7>M$pWC)&jJN>pgPg7
z+AOTyHdW6NC(`dD+@EtxnL@Ae59U42wgEX(KNLqqC46z;xvvtYg78J6zODaf&Sq{P
z=G(y<42i~4q;ycb<pv@q{L+7&n<&Th;HnP34{q%Ku#*IF!Vvy3=Y=T%Mw@5MW|lW~
z-1O+VhuIBURi}$$;B2pWJT@Hr2hw<coH69XD&z4*>RkD16U4YzIXNUr%P~J_pq&~p
z@B~Dr3w71~NoITz0LPRl<{+Zw_Wj$cx;LzHhs>wXyHw}W&p!fZ*Y<(2JJ22K4X(o=
z>*!~ecWuSNbgkw5S6^T(+==*)1MNNhuRVaIy8xy;1U0w2^AD-u;_u<ZlF9ZYH@9)r
z7zHkJ;$=vF%HG{F>_N{`GKnB3egh%~c7dK^DxD?TFeG~jZeSbrfjD#i)oY+_BBJBG
z2M89wI}rn-lo$P5h5*<18gP5gMo#<m9S=OF^8&6J_X%@<=V-hBCokUb1(mO!K%5?$
zFxqUnZY5cqPH>wKQHy3m*#TbbZs^5U>`9)_xy*J9QHMr~y<TYC^-vC<0WtAnb9dTz
z4KsspWp_-tKQQccyyrXFombNhqXZ5Q1*Eu=ON}7M_bUgyLi>~WH&V6Wmn+l9Nl=C)
zRAFze6?)Py(7k^V9;h3AWNDN%9*~T5rf50O-aQ3m>`AKP%(b@%(cA4GGFXJ45oTG8
zSMz1jbD|-el6TgD;yxu*ri+D&O%i%HS_c?o%Qnok#^%Z*S{t?2JAk!QKC_Q!M~_Rf
zmRKOuu|8bOk}+E6QNh&JM4toO&{|E*I7Y?$sv*BbiRg?(W8)P@dUjv7j7wi{wF1im
zoFeZ<lw=wr_Jm#LOoI9v>Is~kRrJixyei6;@%*Z&V+W5w7PYK5-9@ogw!|70+=rxH
zy`w`jj|m(Dhj$lS#Z^@0(@FgaZeHw<QPoJAp=^_IH!``iuzwpo*c0J$g;3f%2)wjH
zrT%5uCCy4FRN!#?5JSP|ZjxD%8<`hzH*iZ|&lq48pfm%bZ{2ZWykW)#k*8Dr(?Zt1
zg5MKV<LcgZUv2RcFvTRyq=necJJH4o2wp`}_gV&;l(frfRkMDkqBeI(!Vtf#g9-kJ
zl45c@n;<^5E0<yGnnGv1{Vcede9sOPl9^<lat|P;Gv3?2V-eW8lh7de;Mtj3zJejc
z%wOl&J1(vP@U+jlanTzfESf`a89ItK!*9-J^9&!crsjR~?}=fjHSO6|7?x-iwDnMz
z+V_T@qVfyMqEyB6RnJ?^s$`-JDr5IoVcjkYi&fniNe|%>5hStB|56{Ng=KG6HjKlY
z#0jk<ONtTtQnq(S#yOuhBA#>gN^PuqCQXWR)lq>4F2vg<G<HCqnOR~bsX<d`OOAZ=
z0uL~YD0gM3%?}S-k92V&MxK%w^*Kn~Po;omLL<*DWU_dhER3wOt!kWEcJftYoEvR3
ziAkZ-1o}P6vJ?8d9zy(J_YQqD%K?!oW1CkbGm$agiyh&bf05dX3|_%mL=7HM3F?`y
zQXLD8AD*cjeL{TCl!~a(+9i}4H=X@Be*&d>ZWV8$_xBswv{E{+)nIqG*FG%TNM82f
zYY&60I!S0#5bdkj+W3}$x6$si62)zA4yOfc&na8KN0<mlc5=(9a3_@Z#h|{AK(bh4
zOFdF!-ZT71(1m40*lQyh8HPQ&Y%BL@`-Phn%FdL>pfh(ZhOSDesYgZ9&Y(bjSi~i!
zncmFR5lfJm^kaw2FllRZcMzQnSfH0CBF-%)W|_{sW8UPS_L222lgWk!B1pDT3kjJJ
zTyGurrN6W$ttC4YZx?2;x$Rw3k$7GdR?560PAgoeUSCpe=L$)i`TdP3!M_3Py0Zb`
z54Hwe4n->Gzow;w_Wpg_9_`Bq-j`LvS_<fix~I>Iy4WhU>i-btPMh1%+Vg-K_ank#
z=*ab;!^NH2ZN%FdlE!|Ll~ho1P+`I<xK+y}gXV0fegyc?vm0LE>d&b;OTfv6#pIQ4
z>Oz7%#Vx$W#Z-X)(*CyBR;}1JrWW`9m2zL)R*6tM11`w2xlO5G+8)vo-tI(NWSu{S
zuOAy7u`-|^FL5FbToB81%y<U4$QB(LraN19h6G|JoFm{AQB_EW-bJ~~0jkOFzB=dk
ze}xk7*H%0-4-FbfBML*;)<OT2_hjrY#t_BcrN20zo=`_9dzXfj)ia~xu%B7Gp83Te
zL4`!$xTOkz=N!@wSu0<P3r>duSO&!g1`$Tb^AWq;EP~r6TuQ$r_{UDAnzAeYbWFP)
zo^05vp~CaCsXPt736rckf$z1p7(}Sad(3dVh2ev^?5`AvOxq$(1VvsnCGwjPlTKNf
zYriFTVyOM-k#`!)`r9dt?JFS@?qC0Q`V0pHbqiKL+TWAaeYg5w6Ms*N3M*vt;>vUa
zo-Q?++iuQ+9%2FJX9c+%;8VW6@8up$I5u(#?VVLA)&!tvue_q#l~VDv3X|qZeqn{U
zsyJy15vMUhI?wpl5nl37R+ce?m)GD*CDgBq%?E(V)c#@mihkrOh`z1oCJdskfJHd4
z5~dp06u%biuUNO?o!^@u<W6uJK>FrFjzgW&Zc?34*?I-f1-?b@MaNh$NRThU%c}Fd
zZ*hXdfr6eS-J+xRJ}<Qjh0@2d^1zD|fv=n<$(G$@tuWsZSAws9kFCcXamglIBo@pO
zCh^IYnD<;hb|PVBbVX0wu(y)~v>6&z1@K%^&Wh*tZBzb@<QvafZM&5Jpwl(`u)<lE
zmTblhU5S{)ly+C@M|NskoLR02<g#~isE|(uc4z7EkV8FwX#3N%Es~8nI*r{OHfbG}
zF>b5qMKH-F9<+eTw#`25ck9<Br2CEeoI^mZ%bzBCE0y*)wo=X2nQudW(PfB#2!xR9
zyd5O^DUvQzx3A%zYMUIx#O^98Kxy%&cb2!Ffa#4x=}RMKE6q||0Sj;2bw{OMgRhBP
zV=Ow0Q@FqTv@B8$aMz~1!x(42KG{Ht7gt$QO5BJ%jfGlF?>W8@bVG$Dtn3Du`S*w4
z18fJi@iq47=sY$Yyd`SCV7+eA$C-O^C^kwyCPZh8jgV%DS73}98FsG5&Oncd3vb`G
z77+C;XkYk(;!qWTT(dY34I=JU%<Xt`=jJH;-Oo_sr^Y#~vOStwCx}L%)%_-Qx=EZ7
z+6Qh!(}LIJspbTT8iH@Rl4m%7z1XU0H5}u8X@aA?Q7M4xr}<;@Wl4*5@9C5p$Vbln
zu%4RS8^yC1&gxw}v>QySv{?pfa2MO=Yl65=*klo8@fmLir$3;Wy?XZjE#HNzABtz-
zfY22yE_9L(m!jzp1=10v?2WfpdYc?9AOj@c1OKJ=#YH1<`;CnQen4tAhNvHey{?3_
zz+=eQI_HSK8Eyqs%@oiB!P+9PnFM63PG9ohiSx6&64>TS5?1-W9%pPP?ak%Vh^4XO
z$><D4_x>pOKAv0U2dP2;NvR?3F?}E@w?>T(n=I&(z4d2uuWZifTv&yz?7S=+SFLsa
zQoY3s`g66t4}f_ATi@6f`;`K#J)C((0ZAA6rFruE1S=-OxFnr#%8->$`G;4vF{2VO
zM)cw}mK^v&4(1LYFl-i<7atClm`9(i$D5VoW9qDeM<YXE>6>3qyxK|PEc8JAM4jT)
zzy92On^dkf<S;aU&T1zXU=U=sNqqBrl5e76pEyI7#fGo0!yEao3I$t<=~eS=tE3PK
z#Feppb<w@+|7^!KtmuBa-ca6rh^3x;>BVhAZK^MFw`G+<qKa#!je-Mo-ooEbomTa@
zr#M;&y52iA@!pI_ou^$GJ>i~}7~1#^QMbJ^7)r(2ANt?Bt_@py0?tB-Cy$;oY7f=2
z5H<|P;au}K+dMDWbRPc|x3a6jgC6(n4c)&j$kcE=EYIHw1`)Y*vGaiR0HSwHqj1|(
zM3UC|l<;Prrwq!creo=aCW3bC`udkwI=+WtX3kE59%^DEN2_x6MZ5=Lu_1nMrBl~`
zC0YqW71mPhbZd|5MzN#)tF;6y2+&>vq)+ZNd4*j!?i3v5yg!-}6HL5_s|hEQ5}fpR
zus1s`*{wGBGoGD$Ov82OqVjHtHtT!I%6MBU<*TcOu^|DI;rupnd3r_k*N?;M{k7M&
z%q+FEm!q`57IuWUAk&4iqtbss=^;Pwj=rvBote|dP(e2X@eHHBsV)78y?O<@^6ZIQ
zkjQ?sFei#rKJJhCBTHoU4C&%`rpxpKymkln`Vhl0TPKD4{|;64{KueU0MHe^!JCeS
znASYIuy*K@9iBeiFhdA|{AO*?iVNvA$E6ruBk7%{?3K$84PSmvr|$4zy7TyP2ALsB
zxJy&NY+t2OzZcTG<5?VpQs_R=HK@AX2dHcR7?ifj`-;edMZfni?s3v?T$~R$Z-_Y2
zqI9N*QO)A3{`9tH;S{TTY4!fc_N1GSq>I9N59y!dtT6A;g7)9-o?^_nMJ(1UbX4hb
zd#`G9JeELp_>7fAwoKQfOBYOX0$s(j^WyPf0t-zagZ(+e;?+^s&4F@0hxuc@vPUgd
zr;bVmCYe=|;H`0frJC}*Bbi>FkX()PJePPO_nW#wlB^FPlwnh8#;@2-7Jm*dK?1x1
z&ko_H()Uw!+(lxdAbPHZ0*|;gf4rGqq3*ITnDVC#8nD$=xA>4;cF-gWntH%?I`4f-
zj4IA~uzfI%KNA-DTyWELXpfx)IujQuJ3zd<`RNLa&P+YRBSzGHIolF1bSnM>E0zB6
zrqTA6H(LMqC;ewSyUu1kKlgk(&a+<~xUwT*a!fn8VnTK49=aEHL{J3PT-1Fl`by1Z
zv#fT+_U@8g--6ONgX?lDnVG!@xpuFBd+e;Qz9v%&2ec|TZx}bl{4KSIzRa`hZIm7x
zs}FgrTkANWj<eF|`|aVn-F-7ELtH6%HTb+QB6n{aQElQz`at4lf|g)efu2{o{+^Av
zYaj|O+gmYK0h=m?92<&KYR3AUY`(2q_aR=W&GHr#gvKk>DN%ghs0opGP7dYf1s*d6
z+sft}*(NkR4c>`4+TBpoq6_V$CXmT@dLqR$CyJ*_%9~%;PZT07E~BKUJIrb&%iUC(
zEL)t&B-C#02p_9|Pv5xEAwBf8y1414=fn={Cap3erbYR=+ivU5<X!7Hqzra3rzyCv
zAm_&dku72FugLVV!Xml7zQKw28u)zulPvQjgD}_1V^dlWGPTL#lS>>?lU4eo6hHh=
z-wQ9>Ft^GG<~Y=QBK3K%F)@*{eHJm*w8uWISfL3Z9wu+by<&<$Go>FE)`1AOWQz=;
z{QuM1x&Jfy`2RmbXbDN=oY#=^Ea&q%%^@8fYs}f?d<b)n)Qr%OGdYx+GdVQJh8!nh
zmQxd3bIgzl-@U(o!{^uQcKvw$@VGr5&-?w_K)$OtxajuMZP|<^6PL9?zFN5dd1q5r
zYaIcB6nIN^(?s5#s;$rcs9(3?_Onz}cz0u8!7H>-<%vgI0;`sIpUNX4=Ow3&DD06h
zhxKD^Eq_w_$)`u8uX^jg-;4#E^R<;U<!09ASdDQv`|xvO^W2@Bk4nZVP1r`7Wz{So
z(jBLZIy**a85?T5dXD}A{5#D3tf;eVzUae!i8FgeLaI$fHR;mNRxR1exz)4!#9Qj_
z{^}Cjf%3<488l}t=z6vDdf<}q>}{gYZpBXNg=owuE_ii)_Du1+ay)-XU%k2I*GO%q
zPhlSExDLgc(s?%tM0k^`a@z#IiXoi=;-`RoEd>+jYo+v52ndCc?{BBvh9;dz!Rihg
zEgZi;>54QTOEeoZKaC{+AUuk(l=6o=lFuKndAuZbTiYiy5*rNmPWqfwD6_jCM4D2d
zR?>w#zzQ^?U=2MFYkvLi_~}Xe`1S?ISSM#=(q$Px>l`FOF721#i~rocpySG$oIhEl
zVdg6TAqP4;`|5Im=i8$dH}n(*6B2@604VE&k;(0wZE|DrS=^n?r9UonIpe(43?LzK
z?qTi^)*@6S_+oX`&Bo;kOOj92AHDzfs-CoRy-8b%)dtZ0JUCNfMOnTSban1-<r)*P
zkf#Aa<#!waE4!JQ^#S`lJjlUJ7{nM^T~Ji8zk;TtA>QbO>%(do87|v}P^Tk$TE^$i
zwsa)CB$1k=e&zPnws}Erut#_+q06x`93;RPDDIA^WHHdg>U4C%#hG5a^Ew4`YpPfI
z)+%c|toBHECMbgL^NaLUj3O(QX2k8Q9UIlg#2jRu*|*asWU5mzS?ENcixQS7pSvU)
z+YC<B(o^yh0hIa%q54Pe)~Q~-c{F8?nE_FZ7DMW|w1!=%O5laD2fne9pFf(7&i>MX
zQ|aU{1#Qq^ji|L1#Zg<&SrO}G4CwFJ!>|a(2o%{3noS2FPyF*;?9J=m7b2b*L2;vO
zTIO3kr~Zx(o{Q@c1CLI8coxc;GhMqo6CpX5^burB7QYq69)x_j)}M?o4$K@dqv#h4
znJ1j!i+o{BFUOmSe|6e<AcxGX8?FRz$2j(|`}$#yZ@k)LBHf~Aiw{z@&*q>u9FT}(
zpXfs@2)r*xgfZ3@2ZB^qWT+txe*}MhK;Z7V?{Bk+D}i5W>bjh#=db5G<j=^gtS!0D
z(5}PYWG?b~`pt`$e8|2Ej$_4bxj9086Ls>BBr-fN;KXZQ4Lp}P3NvO+TTT!cD2E~s
zy~CZ(*_9myoRArvGGDK0#n}&_J~2gpX+dbAw570BE^94JIhJDo=A(0%Q)6CD#id5o
zo{RDo_AUQIMS6$CNeaGF#&F7={joo%io;1)-qZ*{+Zt^Zl8NROgB5<MYmh>>co)XY
zb#ni9oY;c#wtA%|riaOyY%w#F822vY9AKZOnMhLTHqidPh_>RL1I|628fnhZlYpv{
zlULYBvIQ5GJ7FA^v1`HDWne)pUw%#gI&+lz-J8tS4<CR-U)g4n-+0(=$5M`$@02=g
z2#%P@o5~isow@#6+eZf<mNTG|^v(2hVu#>7aPWEPsm%1I<Cw9=mW;}SsnLbHHL3GH
zoVv9>6dvB?f2C)lDc6tSnONFE*|p~mFThUUFzKQUoy7k!6yqfO#P$z-W<~Hb&&kD&
zofp93$A3J5<M$^!jfNIqNB{mG3-ScW==B+d?D93E8NE*{j<-(Uu2=xY9E0o!-clp$
zGsk5P^D~KPcwe|))5_UfMMFYsNOk$cqs#-w03lRVy21eVAN?5s#yXUZ&K@$n5i9l(
z05(?ka4}O|hC{uVDcW_Y>-E3TA;$e}&8h4m_IphqlqS$m`YF6u+_g+o=Fm(%dBdZj
zQ?lY)dwePk)W2=iwNSPq&)r>jvFcgeZ{u5Y`|G^W&sM3aNgK`IpZ1xobwRN3z2ESO
z%8+xkJ?&d$ylKlutL$NCW-CK1f+#wkTcq~XWHY)z2!sE>OZozNv_xK{a>7oufwA}h
z{99mGU(SP^RJ(E2iNk1R=BJw%%;~4Mtjop)lfv8f)eE?oQR51rN7iq06rM&730P8w
zTV*xVNE^M>?7^f0wMB>j4Kvq(U3Py4Gf%g1nk4&MEKCgbQ0TEs7i)xAQ)DQMZrQ^t
znu(z^yoP0G_M7l2%thaFZz@OxWbawVg}xBTIK30Z->5QxHj9Z)6}0)JX(MCLS%*rj
z7+93od|9HkK2AmJw5;4<7PWUX#8DZum2SaKdUog0@9&fbo4D_tZbSYInuu+o(Zmru
z^FNf|WGblY40U=z!+q2C&w1dCWH;|WhDU`(4?w16t)$&zQP*!~@HwyIB|MTDUJc4_
zFYw~^rwjjETb3ufFe^<eNo&`G;cTZaK!3)(a}p4v;#=GpdD*OJZEkV6cjJ~GhGE8G
zLFwQ7Gd#2M3u1ls$Sucau1jgh2MGTP$q_h3I4{~~zJ7jb<=L$mY!dTzRP?cL*@-94
zp(*6uxc}p8Ut5~I#K7nq8f=ZO#I6TJ!>>nvRQ1<)7`$NuJ8tbp{L0Mg1+1Kdh~8_k
zN_CHb4`k?h4pqIj`}#I6SU0|fK9ZU8`t}2ee@pWS-Y)qmbSNEtd*$f0PWTWKDSB?#
z8JI5fbc<IItC{G&9UD4y{IK@PiwXei@Z#e?mw+!;qV|790iGd$b)o<M@Lg+73R4m2
z`fMV0cp6oky8BSNysLmp=G*I{o@VmeCmf-@uF%0j!<p&V4~sGfUNb7?;2&(xbXsPu
zybHHpDZ@M9w0mbhWB2K9X06g;9`A8UWo7B`<O+S~<t=hkSJxe*nfxpAPfvL_Tq-Hi
ztRKLfsJ(F;v_=seF#klOJnHq|ABQ#1{<iQ&mT$LH)0+-#a_lniD6Mqv&G_9?bMZd3
zf?%6W#7>=3f3D*>e58-)iM+7whqa$uOzOR(2g?v{L+Y<*i`ccoh75cYTjdVl^Rn*w
zgvleY_foffV_Emd%yDDbJwKSd!}}fQ88;mt-(&4^=fyS?w%!%|-y0b7RRH>UxQKEt
zY^M|mJ41g4R8a+EsLl`#g#p|2zpOx5@wkR(mSdzVT(EV~>F4a+n)|J*#Gr24yj@w%
zqFvbqGV?K_f_J7p`w;eWPw?mf1`L6FU*Ay#egtDq&q7-E<b-OAV`@ws>D1=n_}!yy
z-kC}TeQdv7=*9)&YUf@G>HWOx=|GICl#?WJknNkh`k|K+37@7P&>3^u1B{-MsH4K!
zv@K#cH3ls_b2lIKPTRRQyHy?2?%O1!nd@ciPsVoL`(q1C!5A=Dm-&h}Mv?d#Q=486
z36Y^{@WScs-nxGQsVI>PJiycKOlgK~q(2P(Q6T0kR9_OX{V@CA)1mTP>$~rO!u_5(
zNSVpXZa+&I-m{Be-WL3^=vlrEUF%``m?#-kh#HR*>MYPt-1;1=`1FYPN25uU#$(jf
z8S>WEAIjUT;>_pF3^|V%Au%m(H@eww^ufdu>{0o~&0C=ENht)tEIP@Sew?YU$kW*w
zIi&w6Y8)D>lQ?;>cX7a@1DNRIIA+61sMjkqf&RkC)owZcygDDcDm+TNYM4WX-Bvhc
zoNUj3ev9V^$e!JZ?MdnYQV5mtm<1{xI2ON5Vo@Oci%i}9JDyq3m2{$Vt=Q@3Tw#`8
zbD_bN1d*c#o0-8OJEOh1`4H~)^I^_aEp8d)y)85{8D$;NPah8pDO%tNDV;}Z3pzHZ
zpvin}M+GRtv|jf+c9h6NakpZ+@1>A@RDelCUx<b7Yb`m*jtHFCis0~Zl9t|T7Ke#o
zjdSKDaDjvq2C~o*7cN9Mg)RxujEbrLOkk0>B&l;_=HkAm8%u=C2pGe2!M*;5#oIi7
zK%>&Rv4yXI;?I;igX58&A7MfVa|7xeG5%aPX5101_d?8Wo3kKr5$C>w7p|=6RP^sS
z?{2*wToIC90FP-Hk#5JHIHVp}Ik4o$Mr<_4eOs|ISZ*!}UGamEQA-w4Z&vDDnDgpu
zzxyMsEU3u!v>5lZ>dXJu56U6Ji*E_Oi0j*GlzK6y&eZoDl{3sC8t3R+NAtC~^qBi~
z{{QBR<E*y*+E5qv5b-icRNa=l++`P7gLR7to5nmWbLWl%-%JFm*>u&7uDjTDQ~rzU
zUd7)nuCEQr6GK1KVu|cOK#?R0l(j6YRGc8z=pe9{R___FM3<$zK3~;6>qfQ#;!IXT
zx=ca?Abw62lY(n&`qOBEd!mNo0mK_w3s+vetKM1wLT0qC83bZIV8XJV1|vu(aMgtM
z_ipZzAZYe|p`lgf34lu9-#K6!Y<PC|V%4IuPQx|QhqQWd#wvhcl5)f#UBon8XVhI=
zV2)t;C$533wJw)D>aj{C9so@*+^6wx$a}HrOw;Pw6Yq8$G+vDbd7ov=;t^ZECqf;7
z;I_s(eNxX@Y)cOh7<eD3{h^ngS7+s7t_zeJ%gDFO82i8(fH+Wsuq3s_rtof9hie$1
zK9myY`op!Brj?i#+vmOfxlnyeD+aZiceM8PfQXN{*T+MaH&FhL1B(@GFG2kG#_h>t
z{KE~%fQ5zs|0LES^~Bo^Fw0E%<j3y78j=>q-JeeI>%r{ifk9{+jci+`e@X(As6y71
zRG})}xr^4sm;n$2U<k=n<j6L2CM085``!7>Sazt)AI$iBcC38dt6Ai5xq##YbKQ*Q
z9Z_BALo##JZ*6kL2L`?34ELsbmW#2(722wC@Rg*sI1%fear=sU0gTxu_yXD@9`t2=
zB6jUa3Mds`l8zhsQOj&_JeQ@i)r{f9Xa%nykDah{v%&&SR9rB3#bepmkG&+VFI{5t
z>wlDfVxbJYzO?nco1iN6lK79n8=c0nF}t~(RBvhEGAfT&hOB)UF*1YoR0jmaGT7O>
zGMc75!pPjA^s~7x9v`}*7-E=H^ngH(arvJtBXm#qjl05u^9hAq;ytg1RzS4I1_v%Y
zKjtOL7bMf9ZR`^;s6$xOIieBnywW!QbyVKCR9zS>gjrRE$fxBAx-Om>o~(x{D5Q1s
zE($}3?kICISBITR$i_G;8&~QxFED!iFnMs)S{oO^Ao9AiUfE}-90dD~y%h21AVEem
zyckv>D*&%`kMT(SlG}#k%6Oyw+U_UiLq>OIdE{cI9%}zv%D%!N+%29)%{7fHvGu57
z04gffW|0(C|KVl!PBO{Pn2}w8+Zo=$^A|jaCOSHv2uaDSfah1C((T7}d{BzNp_7t!
znt-<6CBTHL;WKPmj^uh8wn3Q40B`1Z53yV+R^;_uV8sd6;#Uh|2t7W-c}zlFQeR`s
zy0OJ^GpD3nPt5P84N9BrWhw3+OGiNF8;o%y8fIF)O2qoKP$vF-B}Yu`YTRA#1Z>u`
zB8mkDX>Bv*&grMGuhMg<-2#|?&hcSmVB`=()@Ia8g<0%XJ&bjx{o@%yFfF`4!h=et
zW7B(}LSZ6}4^e>+!kpwKN)rosI4iln?8s+5UlAI6&U>?O%ue=OJUZR%OE#wJLz+En
zN;i|rHMYF&WC=~Ba5aG@$p-G<mz<95B$;05^-1`ugKZ71Uj9<+kxN_^U<M&kmv?v6
zODk{sy$%)zb&$3%Op>4mo1BUAaXj}r?wY=E&gamZ@b(5%>Yb-*y{;IRLIWQ+<%}?q
zvzMNo^?Z&aajz}ep)Z*^t^L?Ue%>u6W}M9wR54B|A-;v_pn&HQB)j0ax~Z-O?GU6;
zj!Jle?0|}qJ|#ii+j(?ejm+-|efl-17ziIu@3;n&jFREL4F7DNI%Lths5P=ckc2dV
z8(M_3b_OK^8|Xd@RWbO7XbVrvdu}DPalJ?`zmNa*vh*gUCQeC(AZ8Gwk>WL5NA&m7
zW}|v*1DM?YvG(qI&V2-%VM9wrP5CD0Kk|m+a2L^Tau3s(>7EZ_av3N87OoyUa%lKG
zV2YmjoirpWQV80?V?&ZZsw_9s7MXS|#KkMd;n#JA0!O12oMSKeuQQ1a<8&Bi!`0mF
z!G)brTqhNLm@B=NlbHpmdy5j32CoaU^;v!&!2u#o0-7+1Zf7(U8UX|u*63_#@*M$#
z7tgyo`$tmx-dyHr5Jbni#S|zpuEC)>_6HWOP%t^;!fZ0tgXUZ{(koxcq?5|mB!~+8
zjl|o#w)h;zE6Oa~e>hMd`W{O(jZ$nrUmhkSQqZ~^AaCYR(Qvy$`jLm#yrfLB^C-i8
zo2+-+v%Yk!RQ_{oJsT=t4rWdfpS`iE=@$?46bUyxMibNqp!Q$yT=dlU?C@+q&Upd=
zaap<2-!$pFS%g2!y79y<GUr8CtIOdeZ^fDEx3B$zoDqsgWx9KpwOidThJTp_6XsXJ
zjGnUDt?6wmglWQ}&f95dfkPfm`y1JPiibcJ$6#hS6ynl7q2L4sIV0Zl`3+k~C5tSP
zAyM9KZGQ8Up83!ef^Ll>B=>K%jjE{S9e0oK*s&};WQYOlcywI5W1j@l{{Gz6y=3HM
zZT$PLK=v;01(S{h;JX-nZAmp)AkBAT$b;>gdvUNHLV6I{g#GFNb+fZ#Q9-ESi(ds_
zRxn<Xs4N!}<MA94A{7t9>yiF^!RGF0q2b?=i1F~-g;L0}-M}k3;)O+n9|GLQDiXzt
z(<AX@lJTC?DR+E;woD{;rC$V=qwP7Dkp_uQ+sz3~1bno7;qYFN@Z7lIKu^GU?m<93
z1~r&_?<j(ub{~-yC|zwAq?2EArJ*h1MwX;ut9&iJ+G@}P`VHVvSh!2!$Z#J7&S$n2
z6t<*_M^)Td&M<eKkR(LsF<DO^a{fp7P!T-KYRK;xS+Hcb+-f=82|s7uwl!PPD&<9*
z^e6qh`1z8*kS<2e9YG!}i0=@cupN|Aqg5V#`%~(E&t7id%s@fTx1saN`Ud1E&#%Q;
z>ZusHGFTKbt3k;e3h`r>RhyE{QjrMrbIj{~w)U1;o~w$gjpqCG`qA-M_Aht;4L@%J
zIravn%P0S0fut39NXqhC2D0C&@$gOHdG#kVi0%}*2q%P3<yP-y>OBtk97o2xpu-9X
z&zPjxGoQ}=a!Gta5Sf{`#(X>)iz*1Z@*|&MEb?t~gsA3r3DJt`n8axt3*Y6peIF|#
zU2!*!j@mS5>J;LaFQk!l1+GC-(05p7VbfN{U%f;lt>^CJGWxPMvKa=c7J!~&t48G;
zuEu|U%0+VzY<KM7za8Yyw{S(;)q*ONaL~AgfWN_MgQXvQg;0}|C|?IHG-%W%w7Wq`
zWdSoT;xa1y0p2?arPdn^TU}hY@l?wRV0^tR(gOJ}51K!NHYz#8M)vZ#zh>maX*>UZ
z{d1c~qX`B1f*lT0&DtHmi!D?Br08*ZN=Lknf_)_F1%$05k-78=V@kev!J<I_CpEQB
zh(@-jKyAXaz)WB5yf#G_4R1H2y_)V2(G+G*!xCS+-?h|dP{+jv5qb-imDLVrHqho{
ze6s$Q+34uig<q|++k*SqBDyqd3lgrmK2IgkFNM{vAvn#qJRK)MY`>u9ic$C+L{q3H
z71|b<3ay?vl}rg_Y(}_rhp>;#B2t2`=N5kZ`8jE6tQIG}U!)y&HStFjn^4H4mP|t4
zje%BbPznHbRT}?SjYKfrWWx(&q%NJdyxkg192sf97G66)^t0XVj(jQ&?7|{gcuif7
zJ1*eooZ|!2xgYRZ-sfB3xp}D~&@Wwxf7<?x)c&$oMMOoUyHL|zxy~NfGl6G>`PbAI
zOI0RAU8xMU?WES~tW?5PwrIHLv7DX1`O2mVo5{3c@r1FeW5WIz`^WN^KUmw!i*mis
zPYn>?Yu>m1s>;6qotJ!;X*R;VkR!zp{hJ5Ril-YxS1Xwjrsi~ETGbLUEJ1|sXDI*%
zpj%SZxL3$k1a1xuoP$y^aXL`&9-{fWIFH@z^hgvyhSH^$e}3=#jK<<ln+Y|8qy8gD
zyn#8c@04C#&No{-Y*gOrjf7c0Fj!!JA>3Df#qujTb-&?1<HyL-dJ)!(oTb_&DOb%S
zzgXw}5Inc<iA;-6u3xq*9Ky#oMB(0B7*c;rvDhVO&CC<OMG=@TJ+lgVrL8NN?yi|!
zC`kPJxAo)D94{_^T=z~@Y0lit5&9H)-mk~DeZR#|_!B&5V^-zZ1vz0q(*l_`%0D_O
zFq}!j^)pgcL2ZmG|3FM=^nmC_?E#Bp!V*tQU%jWc#~+VHl1ODoS^t;%4~jGx2p5OI
zm1{NyF6Ol<NpWVL!(pL!UG!pM1@({NU+lE66Mz5Me-{t`Pr51RKAURIiK10^VTuD}
zpD?P>;TD!A0TgY>sw%A4pP?1{0_yYlQ{cF%#=r|e-BDZiNcpt@zU*A4pYobUVAojF
zeL#ofhV}&ae|Dm-a5M8+{Te6{*B!2xaM4~R(RWk)tJG_M!mGZ8sxhN&ukMl7NF@e~
zu!~;JLAZ7cJZ3T{y@{G|d=sd<s|oG~pvV0N$i8b&!>Due@ji|l7=-S<j6T(F9sj|S
zGFkz0pA^B0(cPWEF|#xRiitS}uGTwXK8#JquX=&wW1i$B<0~3-IVTfJFk&EW;h-&e
zt3&^|c7p~y3-z{|AB_mFKK6nn8+ULwg2vnnd3?3H%kqV*PhL4Md*z5?WyvK~^NO_W
zzKtcb{Qbh4fCBB))$Wwis^GZ~#A#$pVK7Ox3THPr_Si7YMu+ywH{^sK2m6`CVmvn#
zUQo5vE|T8?(&W@y(P?c?2YQ>`J@cE(<;ySRN<`}`gwPD26c*GDpyzrQxm=pE(7J+}
zU3Vz<0Y`3Z;_(t=WF;-g8ci@?lBn)c%Gyrr))bUxG#U0s0j?o)v$wZ?D%OB9;Oj(<
z2C+3?Be3-Xi+n<0@%bd-9BcYw6y0k1sHOK2$x*veE<udGi3h4EK~!OqBGA0;YS4G7
zbw~Yg9GHBVDjGhFkHSvPj!8Y0>eS0&oVi66_&AdAlXc^iTD4*#@)(cC<cm};Vq2S<
z^m-``If$wc@Nx?)YLgn@x)$anKV44XvK!^$CGfEi?@A=uvAkupS<{SKsKz}zd1)(3
z3f-bYQCnpVU8o`wF)$of_fAqQpI1XD6<4@_Aj6*(Pg+_P_p|Y*_)A8)1uZ%KCvPeH
z_q#FpB|5QLLdYdkZ?SZ?E)Q20IJ}11j}n<n;A%@o5dS&VjecTe&~fq`!d1%?^-r2Q
z0wo@;q9AD4pg~@bLlJmeAdtW|?7+7e=PIKiWtLy2tjxy$YzgXOU||aRetsIjJcuGB
zzdAtpb~l`rWE1U!WxQDi-9mmeY`Ty*CJxGIt(X@C%(#X<**<?m!`KhZ4-Kv)6=p6O
zv`2i#qg4<{5g4q*D@Sl+L?bW;ZCY4$&WTQqTz10Amv;9XKn<3)noF0^We4C~Gg|zy
zmJ-GND7QW>Ou(~H`0_dG)e<H6OvK|1(pBNROFx5|(!%n-4Tsog%rTWfC!q_;Vd0_)
zw}->#`r9aC#~m!@UfbU+n3&AJN0Oq=VO){ekb{hFJ7^ciuk=*IgP7;8g+9YVA3H>8
z_><h#LI5RORB1#hVsB58zR><(F84@a9H~vKwE`%=u@yaISWCWg{M~HK@xk*a=K;v9
z&^lG@%*;E4NX_26oi>DQ9Y0H2ZnpqLiZvnPr!YE$!(%0CBy^(~@hO>Stt@SI{oVe|
zMe2k)eeTn@%11&qA^~utIAH$L6Wx7Wn~#2E#rOm)bnKd7t8NjCAg!a+m-d!;-;v_m
zLa>`!g&doBHbqenK7b(;DWIsC2}xMy1p${C8UY#D6R-UcmRl%tbU#2xw67W8+EzK%
zEC*43gxhG{BbBmSvdT3$`EcaGDVR@T@o%wf{^rqA5mV~_R1rlR6z}d6A^(JX8{T_w
z5qK@~8!R}c_J-E*(nK7D;`n0tW*|E(G6U}+kBIFTUz1m@8^7K^iA0x!hs$nABJ}b-
zVgOuc<@0gu!d1%AB5cxUZ(!<Ec8xvj&`%oJm&+yuKkP7#EdLb0eP!_Ln{>kkcN6{}
zD_%vpNst#nKHES4vVUfYGff{#F+KPMU`W7c?UuPCLIQr(x=(BRhU2%v)|<Sd;JAih
zmF}ugb4r_wQ)Zj2opm;^*tZq*s!X>c<kr!({Hj9tMKj>}wX?$@6E>PGDkVUZtfeo<
zyzIIx#oip?Bh$g|?EM_L8VZ@d!Wv_fis}t(9`M0WjlVnl=wU*j_owxDgFtzg_pQGj
zSX#s7_U~5z2p212@!@vsH`%@xx@R~&_5Gqf3gG!N?(4j+N80a~Q-17qQ_SxbIF_%K
z@EPgWXj^vtNH}>fqIKPo-}*|p)caresw%lVO+oFhM{StxYs)G0%gu5-Jj(6-&9gqE
zin<R$<-Mu0*tb(YO?(G!jm9oyi!uJDjXMfgSv$y$nZpSuc>@M?X{Z0D33*oP?Ej~;
hQm=PV--S~#V@Ssl9}NGJ@i&Px7N*uF7$a!H{{yKCe^dYf

diff --git a/examples/benchmarks/TCTS/workflow_config_tcts_Alpha360.yaml b/examples/benchmarks/TCTS/workflow_config_tcts_Alpha360.yaml
index cd3bbf59ca5..484ed45b17a 100644
--- a/examples/benchmarks/TCTS/workflow_config_tcts_Alpha360.yaml
+++ b/examples/benchmarks/TCTS/workflow_config_tcts_Alpha360.yaml
@@ -22,9 +22,9 @@ data_handler_config: &data_handler_config
         - class: CSRankNorm
           kwargs:
               fields_group: label
-    label: ["Ref($close, -1) / $close - 1",
-            "Ref($close, -2) / Ref($close, -1) - 1",
-            "Ref($close, -3) / Ref($close, -2) - 1"]
+    label: ["Ref($close, -2) / Ref($close, -1) - 1", 
+            "Ref($close, -3) / Ref($close, -1) - 1", 
+            "Ref($close, -4) / Ref($close, -1) - 1"]
 port_analysis_config: &port_analysis_config
     strategy:
         class: TopkDropoutStrategy
@@ -53,9 +53,8 @@ task:
             d_feat: 6
             hidden_size: 64
             num_layers: 2
-            dropout: 0.0
+            dropout: 0.3
             n_epochs: 200
-            lr: 1e-3
             early_stop: 20
             batch_size: 800
             metric: loss
@@ -64,12 +63,11 @@ task:
             fore_optimizer: adam
             weight_optimizer: adam
             output_dim: 3
-            fore_lr: 5e-4
-            weight_lr: 5e-4
+            fore_lr: 2e-3
+            weight_lr: 2e-3
             steps: 3
-            target_label: 1
+            target_label: 0
             lowest_valid_performance: 0.993
-            seed: 0
     dataset:
         class: DatasetH
         module_path: qlib.data.dataset
@@ -93,8 +91,7 @@ task:
           kwargs: 
             ana_long_short: False
             ann_scaler: 252
-            label_col: 1
         - class: PortAnaRecord
           module_path: qlib.workflow.record_temp
           kwargs: 
-            config: *port_analysis_config
+            config: *port_analysis_config
\ No newline at end of file
diff --git a/qlib/contrib/model/pytorch_tcts.py b/qlib/contrib/model/pytorch_tcts.py
index da7fda5f5ed..598a357a2e4 100644
--- a/qlib/contrib/model/pytorch_tcts.py
+++ b/qlib/contrib/model/pytorch_tcts.py
@@ -61,8 +61,9 @@ def __init__(
         weight_lr=5e-7,
         steps=3,
         GPU=0,
-        seed=None,
         target_label=0,
+        mode="soft",
+        seed=None,
         lowest_valid_performance=0.993,
         **kwargs
     ):
@@ -87,6 +88,7 @@ def __init__(
         self.weight_lr = weight_lr
         self.steps = steps
         self.target_label = target_label
+        self.mode = mode
         self.lowest_valid_performance = lowest_valid_performance
         self._fore_optimizer = fore_optimizer
         self._weight_optimizer = weight_optimizer
@@ -100,6 +102,8 @@ def __init__(
             "\nn_epochs : {}"
             "\nbatch_size : {}"
             "\nearly_stop : {}"
+            "\ntarget_label : {}"
+            "\nmode : {}"
             "\nloss_type : {}"
             "\nvisible_GPU : {}"
             "\nuse_GPU : {}"
@@ -111,6 +115,8 @@ def __init__(
                 n_epochs,
                 batch_size,
                 early_stop,
+                target_label,
+                mode,
                 loss,
                 GPU,
                 self.use_gpu,
@@ -120,9 +126,17 @@ def __init__(
 
     def loss_fn(self, pred, label, weight):
 
-        loc = torch.argmax(weight, 1)
-        loss = (pred - label[np.arange(weight.shape[0]), loc]) ** 2
-        return torch.mean(loss)
+        if self.mode == "hard":
+            loc = torch.argmax(weight, 1)
+            loss = (pred - label[np.arange(weight.shape[0]), loc]) ** 2
+            return torch.mean(loss)
+
+        elif self.mode == "soft":
+            loss = (pred - label.transpose(0, 1)) ** 2
+            return torch.mean(loss * weight.transpose(0, 1))
+
+        else:
+            raise NotImplementedError("mode {} is not supported!".format(self.mode))
 
     def train_epoch(self, x_train, y_train, x_valid, y_valid):
 
@@ -132,6 +146,10 @@ def train_epoch(self, x_train, y_train, x_valid, y_valid):
         indices = np.arange(len(x_train_values))
         np.random.shuffle(indices)
 
+        task_embedding = torch.zeros([self.batch_size, self.output_dim])
+        task_embedding[:, self.target_label] = 1
+        task_embedding = task_embedding.to(self.device)
+
         init_fore_model = copy.deepcopy(self.fore_model)
         for p in init_fore_model.parameters():
             p.init_fore_model = False
@@ -155,12 +173,13 @@ def train_epoch(self, x_train, y_train, x_valid, y_valid):
 
                 init_pred = init_fore_model(feature)
                 pred = self.fore_model(feature)
-
                 dis = init_pred - label.transpose(0, 1)
-                weight_feature = torch.cat((feature, dis.transpose(0, 1), label, init_pred.view(-1, 1)), 1)
+                weight_feature = torch.cat(
+                    (feature, dis.transpose(0, 1), label, init_pred.view(-1, 1), task_embedding), 1
+                )
                 weight = self.weight_model(weight_feature)
 
-                loss = self.loss_fn(pred, label, weight)  # hard
+                loss = self.loss_fn(pred, label, weight)
 
                 self.fore_optimizer.zero_grad()
                 loss.backward()
@@ -188,11 +207,11 @@ def train_epoch(self, x_train, y_train, x_valid, y_valid):
 
             pred = self.fore_model(feature)
             dis = pred - label.transpose(0, 1)
-            weight_feature = torch.cat((feature, dis.transpose(0, 1), label, pred.view(-1, 1)), 1)
+            weight_feature = torch.cat((feature, dis.transpose(0, 1), label, pred.view(-1, 1), task_embedding), 1)
             weight = self.weight_model(weight_feature)
             loc = torch.argmax(weight, 1)
-            valid_loss = torch.mean((pred - label[:, 0]) ** 2)
-            loss = torch.mean(-valid_loss * torch.log(weight[np.arange(weight.shape[0]), loc]))
+            valid_loss = torch.mean((pred - label[:, abs(self.target_label)]) ** 2)
+            loss = torch.mean(valid_loss * torch.log(weight[np.arange(weight.shape[0]), loc]))
 
             self.weight_optimizer.zero_grad()
             loss.backward()
@@ -207,7 +226,6 @@ def test_epoch(self, data_x, data_y):
 
         self.fore_model.eval()
 
-        scores = []
         losses = []
 
         indices = np.arange(len(x_values))
@@ -242,6 +260,7 @@ def fit(
         x_valid, y_valid = df_valid["feature"], df_valid["label"]
         x_test, y_test = df_test["feature"], df_test["label"]
 
+
         if save_path == None:
             save_path = get_or_create_path(save_path)
         best_loss = np.inf
@@ -277,7 +296,7 @@ def training(
             dropout=self.dropout,
         )
         self.weight_model = MLPModel(
-            d_feat=360 + 2 * self.output_dim + 1,
+            d_feat=360 + 3 * self.output_dim + 1,
             hidden_size=self.hidden_size,
             num_layers=self.num_layers,
             dropout=self.dropout,
@@ -303,8 +322,6 @@ def training(
         best_loss = np.inf
         best_epoch = 0
         stop_round = 0
-        fore_best_param = copy.deepcopy(self.fore_optimizer.state_dict())
-        weight_best_param = copy.deepcopy(self.weight_optimizer.state_dict())
 
         for epoch in range(self.n_epochs):
             print("Epoch:", epoch)

From caeacd09c53ee9fce378e3d439be41411d22525c Mon Sep 17 00:00:00 2001
From: lewwang <lwwang@microsoft.com>
Date: Tue, 12 Oct 2021 00:01:27 +0800
Subject: [PATCH 2/8] Update TCTS README.

---
 examples/benchmarks/TCTS/README.md | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/examples/benchmarks/TCTS/README.md b/examples/benchmarks/TCTS/README.md
index 8a26ff91533..ca53afa17c3 100644
--- a/examples/benchmarks/TCTS/README.md
+++ b/examples/benchmarks/TCTS/README.md
@@ -12,10 +12,9 @@ Given that there are usually multiple temporally correlated tasks, the key chall
 At step <img src="https://latex.codecogs.com/png.latex?s" title="s" />, with training data <img src="https://latex.codecogs.com/png.latex?x_s,y_s" title="x_s,y_s" />, the scheduler <img src="https://latex.codecogs.com/png.latex?\varphi" title="\varphi" /> chooses a suitable task <img src="https://latex.codecogs.com/png.latex?T_{i_s}" title="T_{i_s}" /> (green solid lines) to update the model <img src="https://latex.codecogs.com/png.latex?f" title="f" /> (blue solid lines). After <img src="https://latex.codecogs.com/png.latex?S" title="S" /> steps, we evaluate the model <img src="https://latex.codecogs.com/png.latex?f" title="f" /> on the validation set and update the scheduler <img src="https://latex.codecogs.com/png.latex?\varphi" title="\varphi" /> (green dashed lines).
 
 ### Experiments
-Due to different data versions and different Qlib versions, the original data and data preprocessing methods of the experimental settings in the paper are different from those experimental settings in the existing Qlib version. Therefore, we provide two versions of the code according to the two kind of settings, 1) the code[https://github.com/lwwang1995/tcts] that can be used to reproduce the experimental results and 2) the code[https://github.com/microsoft/qlib/blob/main/qlib/contrib/model/pytorch_tcts.py] in the current Qlib baseline.
+Due to different data versions and different Qlib versions, the original data and data preprocessing methods of the experimental settings in the paper are different from those experimental settings in the existing Qlib version. Therefore, we provide two versions of the code according to the two kind of settings, 1) the [code](https://github.com/lwwang1995/tcts) that can be used to reproduce the experimental results and 2) the [code](https://github.com/microsoft/qlib/blob/main/qlib/contrib/model/pytorch_tcts.py) in the current Qlib baseline.
 
-#### Task Description
-Setting1
+#### Setting1
 * Dataset: We use the historical transaction data for 300 stocks on [CSI300](http://www.csindex.com.cn/en/indices/index-detail/000300) from 01/01/2008 to 08/01/2020. We split the data into training (01/01/2008-12/31/2013), validation (01/01/2014-12/31/2015), and test sets (01/01/2016-08/01/2020) based on the transaction time. 
 
 * The main tasks <img src="https://latex.codecogs.com/png.latex?T_k" title="T_k" /> refers to forecasting return of stock <img src="https://latex.codecogs.com/png.latex?i" title="i" /> as following,
@@ -25,7 +24,7 @@ Setting1
 
 * Temporally correlated task sets <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_k&space;=&space;\{T_1,&space;T_2,&space;...&space;,&space;T_k\}" title="\mathcal{T}_k = \{T_1, T_2, ... , T_k\}" />, in this paper, <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_3" title="\mathcal{T}_3" />, <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_5" title="\mathcal{T}_5" /> and <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_{10}" title="\mathcal{T}_{10}" /> are used in <img src="https://latex.codecogs.com/png.latex?T_1" title="T_1" />, <img src="https://latex.codecogs.com/png.latex?T_2" title="T_2" />, and <img src="https://latex.codecogs.com/png.latex?T_3" title="T_3" />.
 
-Setting2
+#### Setting2
 * Dataset: We use the historical transaction data for 300 stocks on [CSI300](http://www.csindex.com.cn/en/indices/index-detail/000300) from 01/01/2008 to 08/01/2020. We split the data into training (01/01/2008-12/31/2014), validation (01/01/2015-12/31/2016), and test sets (01/01/2017-08/01/2020) based on the transaction time. 
 
 * The main tasks <img src="https://latex.codecogs.com/png.latex?T_k" title="T_k" /> refers to forecasting return of stock <img src="https://latex.codecogs.com/png.latex?i" title="i" /> as following,
@@ -35,5 +34,5 @@ Setting2
 
 * In Qlib baseline, <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_3" title="\mathcal{T}_3" />, is used in  <img src="https://latex.codecogs.com/png.latex?T_1" title="T_1" />.
 
-Experimental Result
-You can find the experimental result of setting1 in paper[http://proceedings.mlr.press/v139/wu21e/wu21e.pdf] and the experimental result of setting2 in this page[https://github.com/microsoft/qlib/tree/main/examples/benchmarks].
\ No newline at end of file
+### Experimental Result
+You can find the experimental result of setting1 in [paper](http://proceedings.mlr.press/v139/wu21e/wu21e.pdf) and the experimental result of setting2 in this [page](https://github.com/microsoft/qlib/tree/main/examples/benchmarks).
\ No newline at end of file

From 5c6e0ada0d950759dd1353032e6088db3f418270 Mon Sep 17 00:00:00 2001
From: lewwang <lwwang@microsoft.com>
Date: Tue, 12 Oct 2021 00:04:19 +0800
Subject: [PATCH 3/8] Update TCTS README.

---
 examples/benchmarks/TCTS/README.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/examples/benchmarks/TCTS/README.md b/examples/benchmarks/TCTS/README.md
index ca53afa17c3..0b405c6bee3 100644
--- a/examples/benchmarks/TCTS/README.md
+++ b/examples/benchmarks/TCTS/README.md
@@ -3,7 +3,7 @@
 Sequence learning has attracted much research attention from the machine learning community in recent years. In many applications, a sequence learning task is usually associated with multiple temporally correlated auxiliary tasks, which are different in terms of how much input information to use or which future step to predict. In stock trend forecasting, as demonstrated in Figure1, one can predict the price of a stock in different future days (e.g., tomorrow, the day after tomorrow). In this paper, we propose a framework to make use of those temporally correlated tasks to help each other. 
 
 ### Method
-Given that there are usually multiple temporally correlated tasks, the key challenge lies in which tasks to use and when to use them in the training process. In this work, we introduce a learnable task scheduler for sequence learning, which adaptively selects temporally correlated tasks during the training process. The scheduler accesses the model status and the current training data (e.g., in current minibatch), and selects the best auxiliary task to help the training of the main task. The scheduler and the model for the main task are jointly trained through bi-level optimization: the scheduler is trained to maximize the validation performance of the model, and the model is trained to minimize the training loss guided by the scheduler. The process is demonstrated in Figure2.
+Given that there are usually multiple temporally correlated tasks, the key challenge lies in which tasks to use and when to use them in the training process. This work introduces a learnable task scheduler for sequence learning, which adaptively selects temporally correlated tasks during the training process. The scheduler accesses the model status and the current training data (e.g., in the current minibatch) and selects the best auxiliary task to help the training of the main task. The scheduler and the model for the main task are jointly trained through bi-level optimization: the scheduler is trained to maximize the validation performance of the model, and the model is trained to minimize the training loss guided by the scheduler. The process is demonstrated in Figure2.
 
 <p align="center"> 
 <img src="workflow.png"/>
@@ -12,7 +12,7 @@ Given that there are usually multiple temporally correlated tasks, the key chall
 At step <img src="https://latex.codecogs.com/png.latex?s" title="s" />, with training data <img src="https://latex.codecogs.com/png.latex?x_s,y_s" title="x_s,y_s" />, the scheduler <img src="https://latex.codecogs.com/png.latex?\varphi" title="\varphi" /> chooses a suitable task <img src="https://latex.codecogs.com/png.latex?T_{i_s}" title="T_{i_s}" /> (green solid lines) to update the model <img src="https://latex.codecogs.com/png.latex?f" title="f" /> (blue solid lines). After <img src="https://latex.codecogs.com/png.latex?S" title="S" /> steps, we evaluate the model <img src="https://latex.codecogs.com/png.latex?f" title="f" /> on the validation set and update the scheduler <img src="https://latex.codecogs.com/png.latex?\varphi" title="\varphi" /> (green dashed lines).
 
 ### Experiments
-Due to different data versions and different Qlib versions, the original data and data preprocessing methods of the experimental settings in the paper are different from those experimental settings in the existing Qlib version. Therefore, we provide two versions of the code according to the two kind of settings, 1) the [code](https://github.com/lwwang1995/tcts) that can be used to reproduce the experimental results and 2) the [code](https://github.com/microsoft/qlib/blob/main/qlib/contrib/model/pytorch_tcts.py) in the current Qlib baseline.
+Due to different data versions and different Qlib versions, the original data and data preprocessing methods of the experimental settings in the paper are different from those experimental settings in the existing Qlib version. Therefore, we provide two versions of the code according to the two kinds of settings, 1) the [code](https://github.com/lwwang1995/tcts) that can be used to reproduce the experimental results and 2) the [code](https://github.com/microsoft/qlib/blob/main/qlib/contrib/model/pytorch_tcts.py) in the current Qlib baseline.
 
 #### Setting1
 * Dataset: We use the historical transaction data for 300 stocks on [CSI300](http://www.csindex.com.cn/en/indices/index-detail/000300) from 01/01/2008 to 08/01/2020. We split the data into training (01/01/2008-12/31/2013), validation (01/01/2014-12/31/2015), and test sets (01/01/2016-08/01/2020) based on the transaction time. 
@@ -35,4 +35,4 @@ Due to different data versions and different Qlib versions, the original data an
 * In Qlib baseline, <img src="https://latex.codecogs.com/png.latex?\mathcal{T}_3" title="\mathcal{T}_3" />, is used in  <img src="https://latex.codecogs.com/png.latex?T_1" title="T_1" />.
 
 ### Experimental Result
-You can find the experimental result of setting1 in [paper](http://proceedings.mlr.press/v139/wu21e/wu21e.pdf) and the experimental result of setting2 in this [page](https://github.com/microsoft/qlib/tree/main/examples/benchmarks).
\ No newline at end of file
+You can find the experimental result of setting1 in the [paper](http://proceedings.mlr.press/v139/wu21e/wu21e.pdf) and the experimental result of setting2 in this [page](https://github.com/microsoft/qlib/tree/main/examples/benchmarks).
\ No newline at end of file

From ac1153849ac81266a704f2c77485a430523544ed Mon Sep 17 00:00:00 2001
From: lewwang <lwwang@microsoft.com>
Date: Tue, 12 Oct 2021 00:33:47 +0800
Subject: [PATCH 4/8] Update TCTS.

---
 qlib/contrib/model/pytorch_tcts.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/qlib/contrib/model/pytorch_tcts.py b/qlib/contrib/model/pytorch_tcts.py
index 598a357a2e4..c0dae98e418 100644
--- a/qlib/contrib/model/pytorch_tcts.py
+++ b/qlib/contrib/model/pytorch_tcts.py
@@ -260,7 +260,6 @@ def fit(
         x_valid, y_valid = df_valid["feature"], df_valid["label"]
         x_test, y_test = df_test["feature"], df_test["label"]
 
-
         if save_path == None:
             save_path = get_or_create_path(save_path)
         best_loss = np.inf

From 44bb149503a2bf26f4704364065da7538bd8fc13 Mon Sep 17 00:00:00 2001
From: lwwang1995 <lewen.wang@microsoft.com>
Date: Sun, 14 Nov 2021 22:25:17 +0800
Subject: [PATCH 5/8] Add ADARNN.

---
 README.md                                     |   2 +
 examples/benchmarks/ADARNN/requirements.txt   |   4 +
 .../workflow_config_adarnn_Alpha360.yaml      |  88 ++
 examples/benchmarks/README.md                 |   3 +-
 qlib/contrib/model/pytorch_adarnn.py          | 804 ++++++++++++++++++
 5 files changed, 900 insertions(+), 1 deletion(-)
 create mode 100644 examples/benchmarks/ADARNN/requirements.txt
 create mode 100644 examples/benchmarks/ADARNN/workflow_config_adarnn_Alpha360.yaml
 create mode 100644 qlib/contrib/model/pytorch_adarnn.py

diff --git a/README.md b/README.md
index 261d6cf75b7..7125ee8fe69 100644
--- a/README.md
+++ b/README.md
@@ -11,6 +11,7 @@
 Recent released features
 | Feature | Status |
 | --                      | ------    |
+| ADARNN  model | [Released](https://github.com/microsoft/qlib/pull/689) on Nov 14, 2021 |
 | TCN  model | [Released](https://github.com/microsoft/qlib/pull/668) on Nov 4, 2021 |
 |Temporal Routing Adaptor (TRA) | [Released](https://github.com/microsoft/qlib/pull/531) on July 30, 2021 |
 | Transformer & Localformer | [Released](https://github.com/microsoft/qlib/pull/508) on July 22, 2021 |
@@ -296,6 +297,7 @@ Here is a list of models built on `Qlib`.
 - [Localformer based on pytorch (Juyong Jiang, et al.)](examples/benchmarks/Localformer/)
 - [TRA based on pytorch (Hengxu, Dong, et al. KDD 2021)](examples/benchmarks/TRA/)
 - [TCN based on pytorch (Shaojie Bai, et al. 2018)](examples/benchmarks/TCN/)
+- [ADARNN based on pytorch (YunTao Du, et al. 2021)](examples/benchmarks/ADARNN/)
 
 Your PR of new Quant models is highly welcomed.
 
diff --git a/examples/benchmarks/ADARNN/requirements.txt b/examples/benchmarks/ADARNN/requirements.txt
new file mode 100644
index 00000000000..16de0a43842
--- /dev/null
+++ b/examples/benchmarks/ADARNN/requirements.txt
@@ -0,0 +1,4 @@
+pandas==1.1.2
+numpy==1.17.4
+scikit_learn==0.23.2
+torch==1.7.0
diff --git a/examples/benchmarks/ADARNN/workflow_config_adarnn_Alpha360.yaml b/examples/benchmarks/ADARNN/workflow_config_adarnn_Alpha360.yaml
new file mode 100644
index 00000000000..ac49d014570
--- /dev/null
+++ b/examples/benchmarks/ADARNN/workflow_config_adarnn_Alpha360.yaml
@@ -0,0 +1,88 @@
+qlib_init:
+    provider_uri: "~/.qlib/qlib_data/cn_data"
+    region: cn
+market: &market csi300
+benchmark: &benchmark SH000300
+data_handler_config: &data_handler_config
+    start_time: 2008-01-01
+    end_time: 2020-08-01
+    fit_start_time: 2008-01-01
+    fit_end_time: 2014-12-31
+    instruments: *market
+    infer_processors:
+        - class: RobustZScoreNorm
+          kwargs:
+              fields_group: feature
+              clip_outlier: true
+        - class: Fillna
+          kwargs:
+              fields_group: feature
+    learn_processors:
+        - class: DropnaLabel
+        - class: CSRankNorm
+          kwargs:
+              fields_group: label
+    label: ["Ref($close, -2) / Ref($close, -1) - 1"]
+port_analysis_config: &port_analysis_config
+    strategy:
+        class: TopkDropoutStrategy
+        module_path: qlib.contrib.strategy
+        kwargs:
+            model: <MODEL>
+            dataset: <DATASET>
+            topk: 50
+            n_drop: 5
+    backtest:
+        start_time: 2017-01-01
+        end_time: 2020-08-01
+        account: 100000000
+        benchmark: *benchmark
+        exchange_kwargs:
+            limit_threshold: 0.095
+            deal_price: close
+            open_cost: 0.0005
+            close_cost: 0.0015
+            min_cost: 5
+task:
+    model:
+        class: ADARNN
+        module_path: qlib.contrib.model.pytorch_adarnn
+        kwargs:
+            d_feat: 6
+            hidden_size: 64
+            num_layers: 2
+            dropout: 0.0
+            n_epochs: 200
+            lr: 1e-3
+            early_stop: 20
+            batch_size: 800
+            metric: loss
+            loss: mse
+            GPU: 0
+    dataset:
+        class: DatasetH
+        module_path: qlib.data.dataset
+        kwargs:
+            handler:
+                class: Alpha360
+                module_path: qlib.contrib.data.handler
+                kwargs: *data_handler_config
+            segments:
+                train: [2008-01-01, 2014-12-31]
+                valid: [2015-01-01, 2016-12-31]
+                test: [2017-01-01, 2020-08-01]
+    record: 
+        - class: SignalRecord
+          module_path: qlib.workflow.record_temp
+          kwargs: 
+            model: <MODEL>
+            dataset: <DATASET>
+        - class: SigAnaRecord
+          module_path: qlib.workflow.record_temp
+          kwargs: 
+            ana_long_short: False
+            ann_scaler: 252
+        - class: PortAnaRecord
+          module_path: qlib.workflow.record_temp
+          kwargs: 
+            config: *port_analysis_config
diff --git a/examples/benchmarks/README.md b/examples/benchmarks/README.md
index 1a7d2fc269c..85456ca4be8 100644
--- a/examples/benchmarks/README.md
+++ b/examples/benchmarks/README.md
@@ -54,13 +54,14 @@ The numbers shown below demonstrate the performance of the entire `workflow` of
 | XGBoost(Tianqi Chen, et al.)              | Alpha360 | 0.0394±0.00 | 0.2909±0.00 | 0.0448±0.00 | 0.3679±0.00 | 0.0344±0.00       | 0.4527±0.02       | -0.1004±0.00 |
 | DoubleEnsemble(Chuheng Zhang, et al.)     | Alpha360 | 0.0404±0.00 | 0.3023±0.00 | 0.0495±0.00 | 0.3898±0.00 | 0.0468±0.01       | 0.6302±0.20       | -0.0860±0.01 |
 | LightGBM(Guolin Ke, et al.)               | Alpha360 | 0.0400±0.00 | 0.3037±0.00 | 0.0499±0.00 | 0.4042±0.00 | 0.0558±0.00       | 0.7632±0.00       | -0.0659±0.00 |
+| TCN(Shaojie Bai, et al.)                  | Alpha360 | 0.0441±0.00 | 0.3301±0.02 | 0.0519±0.00 | 0.4130±0.01 | 0.0604±0.02       | 0.8295±0.34       | -0.1018±0.03 |
 | ALSTM (Yao Qin, et al.)                   | Alpha360 | 0.0497±0.00 | 0.3829±0.04 | 0.0599±0.00 | 0.4736±0.03 | 0.0626±0.02       | 0.8651±0.31       | -0.0994±0.03 |
 | LSTM(Sepp Hochreiter, et al.)             | Alpha360 | 0.0448±0.00 | 0.3474±0.04 | 0.0549±0.00 | 0.4366±0.03 | 0.0647±0.03       | 0.8963±0.39       | -0.0875±0.02 |
 | GRU(Kyunghyun Cho, et al.)                | Alpha360 | 0.0493±0.00 | 0.3772±0.04 | 0.0584±0.00 | 0.4638±0.03 | 0.0720±0.02       | 0.9730±0.33       | -0.0821±0.02 |
+| AdaRNN(Yuntao Du, et al.)                 | Alpha360 | 0.0464±0.01 | 0.3619±0.08 | 0.0539±0.01 | 0.4287±0.06 | 0.0753±0.03       | 1.0200±0.40       | -0.0936±0.03 |
 | GATs (Petar Velickovic, et al.)           | Alpha360 | 0.0476±0.00 | 0.3508±0.02 | 0.0598±0.00 | 0.4604±0.01 | 0.0824±0.02       | 1.1079±0.26       | -0.0894±0.03 |
 | TCTS(Xueqing Wu, et al.)                  | Alpha360 | 0.0508±0.00 | 0.3931±0.04 | 0.0599±0.00 | 0.4756±0.03 | 0.0893±0.03       | 1.2256±0.36       | -0.0857±0.02 |
 | TRA(Hengxu Lin, et al.)                   | Alpha360 | 0.0485±0.00 | 0.3787±0.03 | 0.0587±0.00 | 0.4756±0.03 | 0.0920±0.03       | 1.2789±0.42       | -0.0834±0.02 |
-| TCN(Shaojie Bai, et al.)                  | Alpha360 | 0.0441±0.00 | 0.3301±0.02 | 0.0519±0.00 | 0.4130±0.01 | 0.0604±0.02       | 0.8295±0.34       | -0.1018±0.03 |
 
 - The selected 20 features are based on the feature importance of a lightgbm-based model.
 - The base model of DoubleEnsemble is LGBM.
diff --git a/qlib/contrib/model/pytorch_adarnn.py b/qlib/contrib/model/pytorch_adarnn.py
new file mode 100644
index 00000000000..398b4617819
--- /dev/null
+++ b/qlib/contrib/model/pytorch_adarnn.py
@@ -0,0 +1,804 @@
+# Copyright (c) Microsoft Corporation.
+import os
+from pdb import set_trace
+from torch.utils.data import Dataset, DataLoader
+
+import copy
+from typing import Text, Union
+
+import math
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+from torch.autograd import Function
+from qlib.contrib.model.pytorch_utils import count_parameters
+from qlib.data.dataset import DatasetH
+from qlib.data.dataset.handler import DataHandlerLP
+from qlib.log import get_module_logger
+from qlib.model.base import Model
+from qlib.utils import get_or_create_path
+
+
+
+
+class ADARNN(Model):
+    """ADARNN Model
+
+    Parameters
+    ----------
+    d_feat : int
+        input dimension for each time step
+    metric: str
+        the evaluate metric used in early stop
+    optimizer : str
+        optimizer name
+    GPU : str
+        the GPU ID(s) used for training
+    """
+
+    def __init__(
+        self,
+        d_feat=6,
+        hidden_size=64,
+        num_layers=2,
+        dropout=0.0,
+        n_epochs=200,
+        pre_epoch=40,
+        dw=0.5,
+        loss_type='cosine',
+        len_seq=60,
+        len_win=0,
+        lr=0.001,
+        metric="mse",
+        batch_size=2000,
+        early_stop=20,
+        loss="mse",
+        optimizer="adam",
+        n_splits=2,
+        GPU=0,
+        seed=None,
+        **kwargs
+    ):
+        # Set logger.
+        self.logger = get_module_logger("ADARNN")
+        self.logger.info("ADARNN pytorch version...")
+        os.environ["CUDA_VISIBLE_DEVICES"] = str(GPU)
+
+        # set hyper-parameters.
+        self.d_feat = d_feat
+        self.hidden_size = hidden_size
+        self.num_layers = num_layers
+        self.dropout = dropout
+        self.n_epochs = n_epochs
+        self.pre_epoch = pre_epoch
+        self.dw = dw
+        self.loss_type = loss_type
+        self.len_seq = len_seq
+        self.len_win = len_win
+        self.lr = lr
+        self.metric = metric
+        self.batch_size = batch_size
+        self.early_stop = early_stop
+        self.optimizer = optimizer.lower()
+        self.loss = loss
+        self.n_splits = n_splits
+        self.device = torch.device("cuda:%d" % (
+            GPU) if torch.cuda.is_available() and GPU >= 0 else "cpu")
+        self.seed = seed
+
+        self.logger.info(
+            "ADARNN parameters setting:"
+            "\nd_feat : {}"
+            "\nhidden_size : {}"
+            "\nnum_layers : {}"
+            "\ndropout : {}"
+            "\nn_epochs : {}"
+            "\nlr : {}"
+            "\nmetric : {}"
+            "\nbatch_size : {}"
+            "\nearly_stop : {}"
+            "\noptimizer : {}"
+            "\nloss_type : {}"
+            "\nvisible_GPU : {}"
+            "\nuse_GPU : {}"
+            "\nseed : {}".format(
+                d_feat,
+                hidden_size,
+                num_layers,
+                dropout,
+                n_epochs,
+                lr,
+                metric,
+                batch_size,
+                early_stop,
+                optimizer.lower(),
+                loss,
+                GPU,
+                self.use_gpu,
+                seed,
+            )
+        )
+
+        if self.seed is not None:
+            np.random.seed(self.seed)
+            torch.manual_seed(self.seed)
+
+        n_hiddens = [hidden_size for _ in range(num_layers)]
+        self.model = AdaRNN(
+            use_bottleneck=False,
+            bottleneck_width=64,
+            n_input=d_feat,
+            n_hiddens=n_hiddens,
+            n_output=1,
+            dropout=dropout,
+            model_type='AdaRNN',
+            len_seq=len_seq,
+            trans_loss=loss_type
+        )
+        self.logger.info("model:\n{:}".format(self.model))
+        self.logger.info("model size: {:.4f} MB".format(
+            count_parameters(self.model)))
+
+        if optimizer.lower() == "adam":
+            self.train_optimizer = optim.Adam(
+                self.model.parameters(), lr=self.lr)
+        elif optimizer.lower() == "gd":
+            self.train_optimizer = optim.SGD(
+                self.model.parameters(), lr=self.lr)
+        else:
+            raise NotImplementedError(
+                "optimizer {} is not supported!".format(optimizer))
+
+        self.fitted = False
+        self.model.cuda()
+
+    @property
+    def use_gpu(self):
+        return self.device != torch.device("cpu")
+
+    def train_AdaRNN(self, train_loader_list, epoch, dist_old=None, weight_mat=None):
+        self.model.train()
+        criterion = nn.MSELoss()
+        dist_mat = torch.zeros(self.num_layers, self.len_seq).cuda()
+        len_loader = np.inf
+        for loader in train_loader_list:
+            if len(loader) < len_loader:
+                len_loader = len(loader)
+        for data_all in zip(*train_loader_list):
+            #  for data_all in zip(*train_loader_list):
+            self.train_optimizer.zero_grad()
+            list_feat = []
+            list_label = []
+            for data in data_all:
+                # feature :[36, 24, 6]
+                feature, label_reg = data[0].cuda(
+                ).float(), data[1].cuda().float()
+                list_feat.append(feature)
+                list_label.append(label_reg)
+            flag = False
+            index = get_index(len(data_all) - 1)
+            for temp_index in index:
+                s1 = temp_index[0]
+                s2 = temp_index[1]
+                if list_feat[s1].shape[0] != list_feat[s2].shape[0]:
+                    flag = True
+                    break
+            if flag:
+                continue
+
+            total_loss = torch.zeros(1).cuda()
+            for i in range(len(index)):
+                feature_s = list_feat[index[i][0]]
+                feature_t = list_feat[index[i][1]]
+                label_reg_s = list_label[index[i][0]]
+                label_reg_t = list_label[index[i][1]]
+                feature_all = torch.cat((feature_s, feature_t), 0)
+
+                if epoch < self.pre_epoch:
+                    pred_all, loss_transfer, out_weight_list = self.model.forward_pre_train(
+                        feature_all, len_win=self.len_win)
+                else:
+                    pred_all, loss_transfer, dist, weight_mat = self.model.forward_Boosting(
+                        feature_all, weight_mat)
+                    dist_mat = dist_mat + dist
+                pred_s = pred_all[0:feature_s.size(0)]
+                pred_t = pred_all[feature_s.size(0):]
+
+                loss_s = criterion(pred_s, label_reg_s)
+                loss_t = criterion(pred_t, label_reg_t)
+
+                total_loss = total_loss + loss_s + loss_t + self.dw * loss_transfer
+            self.train_optimizer.zero_grad()
+            total_loss.backward()
+            torch.nn.utils.clip_grad_value_(self.model.parameters(), 3.)
+            self.train_optimizer.step()
+        if epoch >= self.pre_epoch:
+            if epoch > self.pre_epoch:
+                weight_mat = self.model.update_weight_Boosting(
+                    weight_mat, dist_old, dist_mat)
+            return weight_mat, dist_mat
+        else:
+            weight_mat = self.transform_type(out_weight_list)
+            return weight_mat, None
+
+    def calc_all_metrics(self, pred):
+        """pred is a pandas dataframe that has two attributes: score (pred) and label (real)"""
+        res = {}
+        ic = pred.groupby(level='datetime').apply(
+            lambda x: x.label.corr(x.score))
+        rank_ic = pred.groupby(level='datetime').apply(
+            lambda x: x.label.corr(x.score, method='spearman'))
+        res['ic'] = ic.mean()
+        res['icir'] = ic.mean() / ic.std()
+        res['ric'] = rank_ic.mean()
+        res['ricir'] = rank_ic.mean() / rank_ic.std()
+        res['mse'] = -(pred['label']-pred['score']).mean()
+        res['loss'] = res['mse']
+        return res
+
+    def test_epoch(self, df):
+        self.model.eval()
+        preds = self.infer(df['feature'])
+        label = df['label'].squeeze()
+        preds = pd.DataFrame({'label': label, 'score': preds}, index=df.index)
+        metrics = self.calc_all_metrics(preds)
+        return metrics
+
+    def log_metrics(self, mode, metrics):
+        metrics = ['{}/{}: {:.6f}'.format(k, mode, v)
+                   for k, v in metrics.items()]
+        metrics = ', '.join(metrics)
+        self.logger.info(metrics)
+
+    def fit(
+        self,
+        dataset: DatasetH,
+        evals_result=dict(),
+        save_path=None,
+    ):
+
+        df_train, df_valid = dataset.prepare(
+            ["train", "valid"],
+            col_set=["feature", "label"],
+            data_key=DataHandlerLP.DK_L,
+        )
+        #  splits = ['2011-06-30']
+        days = df_train.index.get_level_values(level=0).unique()
+        train_splits = np.array_split(days, self.n_splits)
+        train_splits = [df_train[s[0]:s[-1]] for s in train_splits]
+        train_loader_list = [get_stock_loader(
+            df, self.batch_size) for df in train_splits]
+
+        save_path = get_or_create_path(save_path)
+        stop_steps = 0
+        best_score = -np.inf
+        best_epoch = 0
+        evals_result["train"] = []
+        evals_result["valid"] = []
+
+        # train
+        self.logger.info("training...")
+        self.fitted = True
+        best_score = -np.inf
+        best_epoch = 0
+        weight_mat, dist_mat = None, None
+
+        for step in range(self.n_epochs):
+            self.logger.info("Epoch%d:", step)
+            self.logger.info("training...")
+            weight_mat, dist_mat = self.train_AdaRNN(
+                train_loader_list, step, dist_mat, weight_mat)
+            self.logger.info("evaluating...")
+            train_metrics = self.test_epoch(df_train)
+            valid_metrics = self.test_epoch(df_valid)
+            self.log_metrics('train: ', train_metrics)
+            self.log_metrics('valid: ', valid_metrics)
+
+            valid_score = valid_metrics[self.metric]
+            train_score = train_metrics[self.metric]
+            evals_result["train"].append(train_score)
+            evals_result["valid"].append(valid_score)
+            if valid_score > best_score:
+                best_score = valid_score
+                stop_steps = 0
+                best_epoch = step
+                best_param = copy.deepcopy(self.model.state_dict())
+            else:
+                stop_steps += 1
+                if stop_steps >= self.early_stop:
+                    self.logger.info("early stop")
+                    break
+
+        self.logger.info("best score: %.6lf @ %d" % (best_score, best_epoch))
+        self.model.load_state_dict(best_param)
+        torch.save(best_param, save_path)
+
+        if self.use_gpu:
+            torch.cuda.empty_cache()
+        return best_score
+
+    def predict(self, dataset: DatasetH, segment: Union[Text, slice] = "test"):
+        if not self.fitted:
+            raise ValueError("model is not fitted yet!")
+        x_test = dataset.prepare(
+            segment, col_set="feature", data_key=DataHandlerLP.DK_I)
+        return self.infer(x_test)
+
+    def infer(self, x_test):
+        index = x_test.index
+        self.model.eval()
+        x_values = x_test.values
+        sample_num = x_values.shape[0]
+        x_values = x_values.reshape(
+            sample_num, self.d_feat, -1).transpose(0, 2, 1)
+        preds = []
+
+        for begin in range(sample_num)[:: self.batch_size]:
+
+            if sample_num - begin < self.batch_size:
+                end = sample_num
+            else:
+                end = begin + self.batch_size
+
+            x_batch = torch.from_numpy(
+                x_values[begin:end]).float().cuda()
+
+            with torch.no_grad():
+                pred = self.model.predict(x_batch).detach().cpu().numpy()
+
+            preds.append(pred)
+
+        return pd.Series(np.concatenate(preds), index=index)
+
+    def transform_type(self, init_weight):
+        weight = torch.ones(self.num_layers, self.len_seq).cuda()
+        for i in range(self.num_layers):
+            for j in range(self.len_seq):
+                weight[i, j] = init_weight[i][j].item()
+        return weight
+
+
+class data_loader(Dataset):
+    def __init__(self, df):
+        self.df_feature = df['feature']
+        self.df_label_reg = df['label']
+        self.df_index = df.index
+        self.df_feature = torch.tensor(
+            self.df_feature.values.reshape(-1, 6, 60).transpose(0, 2, 1), dtype=torch.float32)
+        self.df_label_reg = torch.tensor(
+            self.df_label_reg.values.reshape(-1), dtype=torch.float32)
+
+    def __getitem__(self, index):
+        sample, label_reg = self.df_feature[index], self.df_label_reg[index]
+        return sample, label_reg
+
+    def __len__(self):
+        return len(self.df_feature)
+
+
+def get_stock_loader(df, batch_size, shuffle=True):
+    train_loader = DataLoader(data_loader(
+        df), batch_size=batch_size, shuffle=shuffle)
+    return train_loader
+
+
+def get_index(num_domain=2):
+    index = []
+    for i in range(num_domain):
+        for j in range(i+1, num_domain+1):
+            index.append((i, j))
+    return index
+
+
+class AdaRNN(nn.Module):
+    """
+    model_type:  'Boosting', 'AdaRNN'
+    """
+
+    def __init__(self, use_bottleneck=False, bottleneck_width=256, n_input=128, n_hiddens=[64, 64], n_output=6, dropout=0.0, len_seq=9, model_type='AdaRNN', trans_loss='mmd'):
+        super(AdaRNN, self).__init__()
+        self.use_bottleneck = use_bottleneck
+        self.n_input = n_input
+        self.num_layers = len(n_hiddens)
+        self.hiddens = n_hiddens
+        self.n_output = n_output
+        self.model_type = model_type
+        self.trans_loss = trans_loss
+        self.len_seq = len_seq
+        in_size = self.n_input
+
+        features = nn.ModuleList()
+        for hidden in n_hiddens:
+            rnn = nn.GRU(
+                input_size=in_size,
+                num_layers=1,
+                hidden_size=hidden,
+                batch_first=True,
+                dropout=dropout
+            )
+            features.append(rnn)
+            in_size = hidden
+        self.features = nn.Sequential(*features)
+
+        if use_bottleneck == True:  # finance
+            self.bottleneck = nn.Sequential(
+                nn.Linear(n_hiddens[-1], bottleneck_width),
+                nn.Linear(bottleneck_width, bottleneck_width),
+                nn.BatchNorm1d(bottleneck_width),
+                nn.ReLU(),
+                nn.Dropout(),
+            )
+            self.bottleneck[0].weight.data.normal_(0, 0.005)
+            self.bottleneck[0].bias.data.fill_(0.1)
+            self.bottleneck[1].weight.data.normal_(0, 0.005)
+            self.bottleneck[1].bias.data.fill_(0.1)
+            self.fc = nn.Linear(bottleneck_width, n_output)
+            torch.nn.init.xavier_normal_(self.fc.weight)
+        else:
+            self.fc_out = nn.Linear(n_hiddens[-1], self.n_output)
+
+        if self.model_type == 'AdaRNN':
+            gate = nn.ModuleList()
+            for i in range(len(n_hiddens)):
+                gate_weight = nn.Linear(
+                    len_seq * self.hiddens[i]*2, len_seq)
+                gate.append(gate_weight)
+            self.gate = gate
+
+            bnlst = nn.ModuleList()
+            for i in range(len(n_hiddens)):
+                bnlst.append(nn.BatchNorm1d(len_seq))
+            self.bn_lst = bnlst
+            self.softmax = torch.nn.Softmax(dim=0)
+            self.init_layers()
+
+    def init_layers(self):
+        for i in range(len(self.hiddens)):
+            self.gate[i].weight.data.normal_(0, 0.05)
+            self.gate[i].bias.data.fill_(0.0)
+
+    def forward_pre_train(self, x, len_win=0):
+        out = self.gru_features(x)
+        fea = out[0] # [2N,L,H]
+        if self.use_bottleneck == True:
+            fea_bottleneck = self.bottleneck(fea[:, -1, :])
+            fc_out = self.fc(fea_bottleneck).squeeze()
+        else:
+            fc_out = self.fc_out(fea[:, -1, :]).squeeze() #[N,]
+
+        out_list_all, out_weight_list = out[1], out[2]
+        out_list_s, out_list_t = self.get_features(out_list_all)
+        loss_transfer = torch.zeros((1,)).cuda()
+        for i in range(len(out_list_s)):
+            criterion_transder = TransferLoss(
+                loss_type=self.trans_loss, input_dim=out_list_s[i].shape[2])
+            h_start = 0 
+            for j in range(h_start, self.len_seq, 1):
+                i_start = j - len_win if j - len_win >= 0 else 0
+                i_end = j + len_win if j + len_win < self.len_seq else self.len_seq - 1
+                for k in range(i_start, i_end + 1):
+                    weight = out_weight_list[i][j] if self.model_type == 'AdaRNN' else 1 / (
+                        self.len_seq - h_start) * (2 * len_win + 1)
+                    loss_transfer = loss_transfer + weight * criterion_transder.compute(
+                        out_list_s[i][:, j, :], out_list_t[i][:, k, :])
+        return fc_out, loss_transfer, out_weight_list
+
+    def gru_features(self, x, predict=False):
+        x_input = x
+        out = None
+        out_lis = []
+        out_weight_list = [] if (
+             self.model_type == 'AdaRNN') else None
+        for i in range(self.num_layers):
+            out, _ = self.features[i](x_input.float())
+            x_input = out
+            out_lis.append(out)
+            if self.model_type == 'AdaRNN' and predict == False:
+                out_gate = self.process_gate_weight(x_input, i)
+                out_weight_list.append(out_gate)
+        return out, out_lis, out_weight_list
+
+    def process_gate_weight(self, out, index):
+        x_s = out[0: int(out.shape[0]//2)]
+        x_t = out[out.shape[0]//2: out.shape[0]]
+        x_all = torch.cat((x_s, x_t), 2)
+        x_all = x_all.view(x_all.shape[0], -1)
+        weight = torch.sigmoid(self.bn_lst[index](
+            self.gate[index](x_all.float())))
+        weight = torch.mean(weight, dim=0)
+        res = self.softmax(weight).squeeze()
+        return res
+
+    def get_features(self, output_list):
+        fea_list_src, fea_list_tar = [], []
+        for fea in output_list:
+            fea_list_src.append(fea[0: fea.size(0) // 2])
+            fea_list_tar.append(fea[fea.size(0) // 2:])
+        return fea_list_src, fea_list_tar
+
+    # For Boosting-based
+    def forward_Boosting(self, x, weight_mat=None):
+        out = self.gru_features(x)
+        fea = out[0]
+        if self.use_bottleneck:
+            fea_bottleneck = self.bottleneck(fea[:, -1, :])
+            fc_out = self.fc(fea_bottleneck).squeeze()
+        else:
+            fc_out = self.fc_out(fea[:, -1, :]).squeeze()
+
+        out_list_all = out[1]
+        out_list_s, out_list_t = self.get_features(out_list_all)
+        loss_transfer = torch.zeros((1,)).cuda()
+        if weight_mat is None:
+            weight = (1.0 / self.len_seq *
+                      torch.ones(self.num_layers, self.len_seq)).cuda()
+        else:
+            weight = weight_mat
+        dist_mat = torch.zeros(self.num_layers, self.len_seq).cuda()
+        for i in range(len(out_list_s)):
+            criterion_transder = TransferLoss(
+                loss_type=self.trans_loss, input_dim=out_list_s[i].shape[2])
+            for j in range(self.len_seq):
+                loss_trans = criterion_transder.compute(
+                    out_list_s[i][:, j, :], out_list_t[i][:, j, :])
+                loss_transfer = loss_transfer + weight[i, j] * loss_trans
+                dist_mat[i, j] = loss_trans
+        return fc_out, loss_transfer, dist_mat, weight
+
+    # For Boosting-based
+    def update_weight_Boosting(self, weight_mat, dist_old, dist_new):
+        epsilon = 1e-5
+        dist_old = dist_old.detach()
+        dist_new = dist_new.detach()
+        ind = dist_new > dist_old + epsilon
+        weight_mat[ind] = weight_mat[ind] * \
+            (1 + torch.sigmoid(dist_new[ind] - dist_old[ind]))
+        weight_norm = torch.norm(weight_mat, dim=1, p=1)
+        weight_mat = weight_mat / weight_norm.t().unsqueeze(1).repeat(1, self.len_seq)
+        return weight_mat
+
+    def predict(self, x):
+        out = self.gru_features(x, predict=True)
+        fea = out[0]
+        if self.use_bottleneck == True:
+            fea_bottleneck = self.bottleneck(fea[:, -1, :])
+            fc_out = self.fc(fea_bottleneck).squeeze()
+        else:
+            fc_out = self.fc_out(fea[:, -1, :]).squeeze()
+        return fc_out
+
+
+class TransferLoss(object):
+    def __init__(self, loss_type='cosine', input_dim=512):
+        """
+        Supported loss_type: mmd(mmd_lin), mmd_rbf, coral, cosine, kl, js, mine, adv
+        """
+        self.loss_type = loss_type
+        self.input_dim = input_dim
+
+    def compute(self, X, Y):
+        """Compute adaptation loss
+
+        Arguments:
+            X {tensor} -- source matrix
+            Y {tensor} -- target matrix
+
+        Returns:
+            [tensor] -- transfer loss
+        """
+        if self.loss_type == 'mmd_lin' or self.loss_type =='mmd':
+            mmdloss = MMD_loss(kernel_type='linear')
+            loss = mmdloss(X, Y)
+        elif self.loss_type == 'coral':
+            loss = CORAL(X, Y)
+        elif self.loss_type == 'cosine' or self.loss_type == 'cos':
+            loss = 1 - cosine(X, Y)
+        elif self.loss_type == 'kl':
+            loss = kl_div(X, Y)
+        elif self.loss_type == 'js':
+            loss = js(X, Y)
+        elif self.loss_type == 'mine':
+            mine_model = Mine_estimator(
+                input_dim=self.input_dim, hidden_dim=60).cuda()
+            loss = mine_model(X, Y)
+        elif self.loss_type == 'adv':
+            loss = adv(X, Y, input_dim=self.input_dim, hidden_dim=32)
+        elif self.loss_type == 'mmd_rbf':
+            mmdloss = MMD_loss(kernel_type='rbf')
+            loss = mmdloss(X, Y)
+        elif self.loss_type == 'pairwise':
+            pair_mat = pairwise_dist(X, Y)
+            loss = torch.norm(pair_mat)
+
+        return loss
+
+def cosine(source, target):
+    source, target = source.mean(), target.mean()
+    cos = nn.CosineSimilarity(dim=0)
+    loss = cos(source, target)
+    return loss.mean()
+
+class ReverseLayerF(Function):
+
+    @staticmethod
+    def forward(ctx, x, alpha):
+        ctx.alpha = alpha
+        return x.view_as(x)
+
+    @staticmethod
+    def backward(ctx, grad_output):
+        output = grad_output.neg() * ctx.alpha
+        return output, None
+
+
+class Discriminator(nn.Module):
+    def __init__(self, input_dim=256, hidden_dim=256):
+        super(Discriminator, self).__init__()
+        self.input_dim = input_dim
+        self.hidden_dim = hidden_dim
+        self.dis1 = nn.Linear(input_dim, hidden_dim)
+        self.dis2 = nn.Linear(hidden_dim, 1)
+
+    def forward(self, x):
+        x = F.relu(self.dis1(x))
+        x = self.dis2(x)
+        x = torch.sigmoid(x)
+        return x
+
+
+def adv(source, target, input_dim=256, hidden_dim=512):
+    domain_loss = nn.BCELoss()
+    # !!! Pay attention to .cuda !!!
+    adv_net = Discriminator(input_dim, hidden_dim).cuda()
+    domain_src = torch.ones(len(source)).cuda()
+    domain_tar = torch.zeros(len(target)).cuda()
+    domain_src, domain_tar = domain_src.view(domain_src.shape[0], 1), domain_tar.view(domain_tar.shape[0], 1)
+    reverse_src = ReverseLayerF.apply(source, 1)
+    reverse_tar = ReverseLayerF.apply(target, 1)
+    pred_src = adv_net(reverse_src)
+    pred_tar = adv_net(reverse_tar)
+    loss_s, loss_t = domain_loss(pred_src, domain_src), domain_loss(pred_tar, domain_tar)
+    loss = loss_s + loss_t
+    return loss
+
+def CORAL(source, target):
+    d = source.size(1)
+    ns, nt = source.size(0), target.size(0)
+
+    # source covariance
+    tmp_s = torch.ones((1, ns)).cuda() @ source
+    cs = (source.t() @ source - (tmp_s.t() @ tmp_s) / ns) / (ns - 1)
+
+    # target covariance
+    tmp_t = torch.ones((1, nt)).cuda() @ target
+    ct = (target.t() @ target - (tmp_t.t() @ tmp_t) / nt) / (nt - 1)
+
+    # frobenius norm
+    loss = (cs - ct).pow(2).sum()
+    loss = loss / (4 * d * d)
+
+    return loss
+
+class MMD_loss(nn.Module):
+    def __init__(self, kernel_type='linear', kernel_mul=2.0, kernel_num=5):
+        super(MMD_loss, self).__init__()
+        self.kernel_num = kernel_num
+        self.kernel_mul = kernel_mul
+        self.fix_sigma = None
+        self.kernel_type = kernel_type
+
+    def guassian_kernel(self, source, target, kernel_mul=2.0, kernel_num=5, fix_sigma=None):
+        n_samples = int(source.size()[0]) + int(target.size()[0])
+        total = torch.cat([source, target], dim=0)
+        total0 = total.unsqueeze(0).expand(
+            int(total.size(0)), int(total.size(0)), int(total.size(1)))
+        total1 = total.unsqueeze(1).expand(
+            int(total.size(0)), int(total.size(0)), int(total.size(1)))
+        L2_distance = ((total0-total1)**2).sum(2)
+        if fix_sigma:
+            bandwidth = fix_sigma
+        else:
+            bandwidth = torch.sum(L2_distance.data) / (n_samples**2-n_samples)
+        bandwidth /= kernel_mul ** (kernel_num // 2)
+        bandwidth_list = [bandwidth * (kernel_mul**i)
+                          for i in range(kernel_num)]
+        kernel_val = [torch.exp(-L2_distance / bandwidth_temp)
+                      for bandwidth_temp in bandwidth_list]
+        return sum(kernel_val)
+
+    def linear_mmd(self, X, Y):
+        delta = X.mean(axis=0) - Y.mean(axis=0)
+        loss = delta.dot(delta.T)
+        return loss
+
+    def forward(self, source, target):
+        if self.kernel_type == 'linear':
+            return self.linear_mmd(source, target)
+        elif self.kernel_type == 'rbf':
+            batch_size = int(source.size()[0])
+            kernels = self.guassian_kernel(
+                source, target, kernel_mul=self.kernel_mul, kernel_num=self.kernel_num, fix_sigma=self.fix_sigma)
+            with torch.no_grad():
+                XX = torch.mean(kernels[:batch_size, :batch_size])
+                YY = torch.mean(kernels[batch_size:, batch_size:])
+                XY = torch.mean(kernels[:batch_size, batch_size:])
+                YX = torch.mean(kernels[batch_size:, :batch_size])
+                loss = torch.mean(XX + YY - XY - YX)
+            return loss
+
+class Mine_estimator(nn.Module):
+    def __init__(self, input_dim=2048, hidden_dim=512):
+        super(Mine_estimator, self).__init__()
+        self.mine_model = Mine(input_dim, hidden_dim)
+
+    def forward(self, X, Y):
+        Y_shffle = Y[torch.randperm(len(Y))]
+        loss_joint = self.mine_model(X, Y)
+        loss_marginal = self.mine_model(X, Y_shffle)
+        ret = torch.mean(loss_joint) - \
+            torch.log(torch.mean(torch.exp(loss_marginal)))
+        loss = -ret
+        return loss
+
+
+class Mine(nn.Module):
+    def __init__(self, input_dim=2048, hidden_dim=512):
+        super(Mine, self).__init__()
+        self.fc1_x = nn.Linear(input_dim, hidden_dim)
+        self.fc1_y = nn.Linear(input_dim, hidden_dim)
+        self.fc2 = nn.Linear(hidden_dim, 1)
+
+    def forward(self, x, y):
+        h1 = F.leaky_relu(self.fc1_x(x)+self.fc1_y(y))
+        h2 = self.fc2(h1)
+        return h2
+
+def pairwise_dist(X, Y):
+    n, d = X.shape
+    m, _ = Y.shape
+    assert d == Y.shape[1]
+    a = X.unsqueeze(1).expand(n, m, d)
+    b = Y.unsqueeze(0).expand(n, m, d)
+    return torch.pow(a - b, 2).sum(2)
+
+
+def pairwise_dist_np(X, Y):
+    n, d = X.shape
+    m, _ = Y.shape
+    assert d == Y.shape[1]
+    a = np.expand_dims(X, 1)
+    b = np.expand_dims(Y, 0)
+    a = np.tile(a, (1, m, 1))
+    b = np.tile(b, (n, 1, 1))
+    return np.power(a - b, 2).sum(2)
+
+def pa(X, Y):
+    XY = np.dot(X, Y.T)
+    XX = np.sum(np.square(X), axis=1)
+    XX = np.transpose([XX])
+    YY = np.sum(np.square(Y), axis=1)
+    dist = XX + YY - 2 * XY
+
+    return dist
+
+def kl_div(source, target):
+    if len(source) < len(target):
+        target = target[:len(source)]
+    elif len(source) > len(target):
+        source = source[:len(target)]
+    criterion = nn.KLDivLoss(reduction='batchmean')
+    loss = criterion(source.log(), target)
+    return loss
+
+
+def js(source, target):
+    if len(source) < len(target):
+        target = target[:len(source)]
+    elif len(source) > len(target):
+        source = source[:len(target)]
+    M = .5 * (source + target)
+    loss_1, loss_2 = kl_div(source, M), kl_div(target, M)
+    return .5 * (loss_1 + loss_2)
\ No newline at end of file

From a6772d33e83a4f564b8f0f9fee681257a963108f Mon Sep 17 00:00:00 2001
From: lwwang1995 <lewen.wang@microsoft.com>
Date: Sun, 14 Nov 2021 22:35:24 +0800
Subject: [PATCH 6/8] Update README.

---
 examples/benchmarks/README.md | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/examples/benchmarks/README.md b/examples/benchmarks/README.md
index 85456ca4be8..779a2bc127a 100644
--- a/examples/benchmarks/README.md
+++ b/examples/benchmarks/README.md
@@ -21,6 +21,7 @@ The numbers shown below demonstrate the performance of the entire `workflow` of
 
 | Model Name                               | Dataset                             | IC          | ICIR        | Rank IC     | Rank ICIR   | Annualized Return | Information Ratio | Max Drawdown |
 |------------------------------------------|-------------------------------------|-------------|-------------|-------------|-------------|-------------------|-------------------|--------------|
+| TCN(Shaojie Bai, et al.)                 | Alpha158                            | 0.0275±0.00 | 0.2157±0.01 | 0.0411±0.00 | 0.3379±0.01 | 0.0190±0.02       | 0.2887±0.27       | -0.1202±0.03 |
 | TabNet(Sercan O. Arik, et al.)           | Alpha158                            | 0.0204±0.01 | 0.1554±0.07 | 0.0333±0.00 | 0.2552±0.05 | 0.0227±0.04       | 0.3676±0.54       | -0.1089±0.08 |
 | Transformer(Ashish Vaswani, et al.)      | Alpha158                            | 0.0264±0.00 | 0.2053±0.02 | 0.0407±0.00 | 0.3273±0.02 | 0.0273±0.02       | 0.3970±0.26       | -0.1101±0.02 |
 | GRU(Kyunghyun Cho, et al.)               | Alpha158(with selected 20 features) | 0.0315±0.00 | 0.2450±0.04 | 0.0428±0.00 | 0.3440±0.03 | 0.0344±0.02       | 0.5160±0.25       | -0.1017±0.02 |
@@ -38,8 +39,6 @@ The numbers shown below demonstrate the performance of the entire `workflow` of
 | MLP                                      | Alpha158                            | 0.0376±0.00 | 0.2846±0.02 | 0.0429±0.00 | 0.3220±0.01 | 0.0895±0.02       | 1.1408±0.23       | -0.1103±0.02 |
 | LightGBM(Guolin Ke, et al.)              | Alpha158                            | 0.0448±0.00 | 0.3660±0.00 | 0.0469±0.00 | 0.3877±0.00 | 0.0901±0.00       | 1.0164±0.00       | -0.1038±0.00 |
 | DoubleEnsemble(Chuheng Zhang, et al.)    | Alpha158                            | 0.0544±0.00 | 0.4340±0.00 | 0.0523±0.00 | 0.4284±0.01 | 0.1168±0.01       | 1.3384±0.12       | -0.1036±0.01 |
-| TCN                                      | Alpha158                            | 0.0275±0.00 | 0.2157±0.01 | 0.0411±0.00 | 0.3379±0.01 | 0.0190±0.02       | 0.2887±0.27       | -0.1202±0.03 |
-
 
 
 ## Alpha360 dataset

From 66094ec8ef6ac824c2fbbf967781bf6626da8848 Mon Sep 17 00:00:00 2001
From: lwwang1995 <lewen.wang@microsoft.com>
Date: Sun, 14 Nov 2021 22:38:14 +0800
Subject: [PATCH 7/8] Reformat ADARNN.

---
 qlib/contrib/model/pytorch_adarnn.py | 259 +++++++++++++--------------
 1 file changed, 122 insertions(+), 137 deletions(-)

diff --git a/qlib/contrib/model/pytorch_adarnn.py b/qlib/contrib/model/pytorch_adarnn.py
index 398b4617819..aad01011cc7 100644
--- a/qlib/contrib/model/pytorch_adarnn.py
+++ b/qlib/contrib/model/pytorch_adarnn.py
@@ -22,8 +22,6 @@
 from qlib.utils import get_or_create_path
 
 
-
-
 class ADARNN(Model):
     """ADARNN Model
 
@@ -48,7 +46,7 @@ def __init__(
         n_epochs=200,
         pre_epoch=40,
         dw=0.5,
-        loss_type='cosine',
+        loss_type="cosine",
         len_seq=60,
         len_win=0,
         lr=0.001,
@@ -85,8 +83,7 @@ def __init__(
         self.optimizer = optimizer.lower()
         self.loss = loss
         self.n_splits = n_splits
-        self.device = torch.device("cuda:%d" % (
-            GPU) if torch.cuda.is_available() and GPU >= 0 else "cpu")
+        self.device = torch.device("cuda:%d" % (GPU) if torch.cuda.is_available() and GPU >= 0 else "cpu")
         self.seed = seed
 
         self.logger.info(
@@ -134,23 +131,19 @@ def __init__(
             n_hiddens=n_hiddens,
             n_output=1,
             dropout=dropout,
-            model_type='AdaRNN',
+            model_type="AdaRNN",
             len_seq=len_seq,
-            trans_loss=loss_type
+            trans_loss=loss_type,
         )
         self.logger.info("model:\n{:}".format(self.model))
-        self.logger.info("model size: {:.4f} MB".format(
-            count_parameters(self.model)))
+        self.logger.info("model size: {:.4f} MB".format(count_parameters(self.model)))
 
         if optimizer.lower() == "adam":
-            self.train_optimizer = optim.Adam(
-                self.model.parameters(), lr=self.lr)
+            self.train_optimizer = optim.Adam(self.model.parameters(), lr=self.lr)
         elif optimizer.lower() == "gd":
-            self.train_optimizer = optim.SGD(
-                self.model.parameters(), lr=self.lr)
+            self.train_optimizer = optim.SGD(self.model.parameters(), lr=self.lr)
         else:
-            raise NotImplementedError(
-                "optimizer {} is not supported!".format(optimizer))
+            raise NotImplementedError("optimizer {} is not supported!".format(optimizer))
 
         self.fitted = False
         self.model.cuda()
@@ -174,8 +167,7 @@ def train_AdaRNN(self, train_loader_list, epoch, dist_old=None, weight_mat=None)
             list_label = []
             for data in data_all:
                 # feature :[36, 24, 6]
-                feature, label_reg = data[0].cuda(
-                ).float(), data[1].cuda().float()
+                feature, label_reg = data[0].cuda().float(), data[1].cuda().float()
                 list_feat.append(feature)
                 list_label.append(label_reg)
             flag = False
@@ -199,13 +191,13 @@ def train_AdaRNN(self, train_loader_list, epoch, dist_old=None, weight_mat=None)
 
                 if epoch < self.pre_epoch:
                     pred_all, loss_transfer, out_weight_list = self.model.forward_pre_train(
-                        feature_all, len_win=self.len_win)
+                        feature_all, len_win=self.len_win
+                    )
                 else:
-                    pred_all, loss_transfer, dist, weight_mat = self.model.forward_Boosting(
-                        feature_all, weight_mat)
+                    pred_all, loss_transfer, dist, weight_mat = self.model.forward_Boosting(feature_all, weight_mat)
                     dist_mat = dist_mat + dist
-                pred_s = pred_all[0:feature_s.size(0)]
-                pred_t = pred_all[feature_s.size(0):]
+                pred_s = pred_all[0 : feature_s.size(0)]
+                pred_t = pred_all[feature_s.size(0) :]
 
                 loss_s = criterion(pred_s, label_reg_s)
                 loss_t = criterion(pred_t, label_reg_t)
@@ -213,12 +205,11 @@ def train_AdaRNN(self, train_loader_list, epoch, dist_old=None, weight_mat=None)
                 total_loss = total_loss + loss_s + loss_t + self.dw * loss_transfer
             self.train_optimizer.zero_grad()
             total_loss.backward()
-            torch.nn.utils.clip_grad_value_(self.model.parameters(), 3.)
+            torch.nn.utils.clip_grad_value_(self.model.parameters(), 3.0)
             self.train_optimizer.step()
         if epoch >= self.pre_epoch:
             if epoch > self.pre_epoch:
-                weight_mat = self.model.update_weight_Boosting(
-                    weight_mat, dist_old, dist_mat)
+                weight_mat = self.model.update_weight_Boosting(weight_mat, dist_old, dist_mat)
             return weight_mat, dist_mat
         else:
             weight_mat = self.transform_type(out_weight_list)
@@ -227,30 +218,27 @@ def train_AdaRNN(self, train_loader_list, epoch, dist_old=None, weight_mat=None)
     def calc_all_metrics(self, pred):
         """pred is a pandas dataframe that has two attributes: score (pred) and label (real)"""
         res = {}
-        ic = pred.groupby(level='datetime').apply(
-            lambda x: x.label.corr(x.score))
-        rank_ic = pred.groupby(level='datetime').apply(
-            lambda x: x.label.corr(x.score, method='spearman'))
-        res['ic'] = ic.mean()
-        res['icir'] = ic.mean() / ic.std()
-        res['ric'] = rank_ic.mean()
-        res['ricir'] = rank_ic.mean() / rank_ic.std()
-        res['mse'] = -(pred['label']-pred['score']).mean()
-        res['loss'] = res['mse']
+        ic = pred.groupby(level="datetime").apply(lambda x: x.label.corr(x.score))
+        rank_ic = pred.groupby(level="datetime").apply(lambda x: x.label.corr(x.score, method="spearman"))
+        res["ic"] = ic.mean()
+        res["icir"] = ic.mean() / ic.std()
+        res["ric"] = rank_ic.mean()
+        res["ricir"] = rank_ic.mean() / rank_ic.std()
+        res["mse"] = -(pred["label"] - pred["score"]).mean()
+        res["loss"] = res["mse"]
         return res
 
     def test_epoch(self, df):
         self.model.eval()
-        preds = self.infer(df['feature'])
-        label = df['label'].squeeze()
-        preds = pd.DataFrame({'label': label, 'score': preds}, index=df.index)
+        preds = self.infer(df["feature"])
+        label = df["label"].squeeze()
+        preds = pd.DataFrame({"label": label, "score": preds}, index=df.index)
         metrics = self.calc_all_metrics(preds)
         return metrics
 
     def log_metrics(self, mode, metrics):
-        metrics = ['{}/{}: {:.6f}'.format(k, mode, v)
-                   for k, v in metrics.items()]
-        metrics = ', '.join(metrics)
+        metrics = ["{}/{}: {:.6f}".format(k, mode, v) for k, v in metrics.items()]
+        metrics = ", ".join(metrics)
         self.logger.info(metrics)
 
     def fit(
@@ -268,9 +256,8 @@ def fit(
         #  splits = ['2011-06-30']
         days = df_train.index.get_level_values(level=0).unique()
         train_splits = np.array_split(days, self.n_splits)
-        train_splits = [df_train[s[0]:s[-1]] for s in train_splits]
-        train_loader_list = [get_stock_loader(
-            df, self.batch_size) for df in train_splits]
+        train_splits = [df_train[s[0] : s[-1]] for s in train_splits]
+        train_loader_list = [get_stock_loader(df, self.batch_size) for df in train_splits]
 
         save_path = get_or_create_path(save_path)
         stop_steps = 0
@@ -289,13 +276,12 @@ def fit(
         for step in range(self.n_epochs):
             self.logger.info("Epoch%d:", step)
             self.logger.info("training...")
-            weight_mat, dist_mat = self.train_AdaRNN(
-                train_loader_list, step, dist_mat, weight_mat)
+            weight_mat, dist_mat = self.train_AdaRNN(train_loader_list, step, dist_mat, weight_mat)
             self.logger.info("evaluating...")
             train_metrics = self.test_epoch(df_train)
             valid_metrics = self.test_epoch(df_valid)
-            self.log_metrics('train: ', train_metrics)
-            self.log_metrics('valid: ', valid_metrics)
+            self.log_metrics("train: ", train_metrics)
+            self.log_metrics("valid: ", valid_metrics)
 
             valid_score = valid_metrics[self.metric]
             train_score = train_metrics[self.metric]
@@ -323,8 +309,7 @@ def fit(
     def predict(self, dataset: DatasetH, segment: Union[Text, slice] = "test"):
         if not self.fitted:
             raise ValueError("model is not fitted yet!")
-        x_test = dataset.prepare(
-            segment, col_set="feature", data_key=DataHandlerLP.DK_I)
+        x_test = dataset.prepare(segment, col_set="feature", data_key=DataHandlerLP.DK_I)
         return self.infer(x_test)
 
     def infer(self, x_test):
@@ -332,8 +317,7 @@ def infer(self, x_test):
         self.model.eval()
         x_values = x_test.values
         sample_num = x_values.shape[0]
-        x_values = x_values.reshape(
-            sample_num, self.d_feat, -1).transpose(0, 2, 1)
+        x_values = x_values.reshape(sample_num, self.d_feat, -1).transpose(0, 2, 1)
         preds = []
 
         for begin in range(sample_num)[:: self.batch_size]:
@@ -343,8 +327,7 @@ def infer(self, x_test):
             else:
                 end = begin + self.batch_size
 
-            x_batch = torch.from_numpy(
-                x_values[begin:end]).float().cuda()
+            x_batch = torch.from_numpy(x_values[begin:end]).float().cuda()
 
             with torch.no_grad():
                 pred = self.model.predict(x_batch).detach().cpu().numpy()
@@ -363,13 +346,13 @@ def transform_type(self, init_weight):
 
 class data_loader(Dataset):
     def __init__(self, df):
-        self.df_feature = df['feature']
-        self.df_label_reg = df['label']
+        self.df_feature = df["feature"]
+        self.df_label_reg = df["label"]
         self.df_index = df.index
         self.df_feature = torch.tensor(
-            self.df_feature.values.reshape(-1, 6, 60).transpose(0, 2, 1), dtype=torch.float32)
-        self.df_label_reg = torch.tensor(
-            self.df_label_reg.values.reshape(-1), dtype=torch.float32)
+            self.df_feature.values.reshape(-1, 6, 60).transpose(0, 2, 1), dtype=torch.float32
+        )
+        self.df_label_reg = torch.tensor(self.df_label_reg.values.reshape(-1), dtype=torch.float32)
 
     def __getitem__(self, index):
         sample, label_reg = self.df_feature[index], self.df_label_reg[index]
@@ -380,15 +363,14 @@ def __len__(self):
 
 
 def get_stock_loader(df, batch_size, shuffle=True):
-    train_loader = DataLoader(data_loader(
-        df), batch_size=batch_size, shuffle=shuffle)
+    train_loader = DataLoader(data_loader(df), batch_size=batch_size, shuffle=shuffle)
     return train_loader
 
 
 def get_index(num_domain=2):
     index = []
     for i in range(num_domain):
-        for j in range(i+1, num_domain+1):
+        for j in range(i + 1, num_domain + 1):
             index.append((i, j))
     return index
 
@@ -398,7 +380,18 @@ class AdaRNN(nn.Module):
     model_type:  'Boosting', 'AdaRNN'
     """
 
-    def __init__(self, use_bottleneck=False, bottleneck_width=256, n_input=128, n_hiddens=[64, 64], n_output=6, dropout=0.0, len_seq=9, model_type='AdaRNN', trans_loss='mmd'):
+    def __init__(
+        self,
+        use_bottleneck=False,
+        bottleneck_width=256,
+        n_input=128,
+        n_hiddens=[64, 64],
+        n_output=6,
+        dropout=0.0,
+        len_seq=9,
+        model_type="AdaRNN",
+        trans_loss="mmd",
+    ):
         super(AdaRNN, self).__init__()
         self.use_bottleneck = use_bottleneck
         self.n_input = n_input
@@ -412,13 +405,7 @@ def __init__(self, use_bottleneck=False, bottleneck_width=256, n_input=128, n_hi
 
         features = nn.ModuleList()
         for hidden in n_hiddens:
-            rnn = nn.GRU(
-                input_size=in_size,
-                num_layers=1,
-                hidden_size=hidden,
-                batch_first=True,
-                dropout=dropout
-            )
+            rnn = nn.GRU(input_size=in_size, num_layers=1, hidden_size=hidden, batch_first=True, dropout=dropout)
             features.append(rnn)
             in_size = hidden
         self.features = nn.Sequential(*features)
@@ -440,11 +427,10 @@ def __init__(self, use_bottleneck=False, bottleneck_width=256, n_input=128, n_hi
         else:
             self.fc_out = nn.Linear(n_hiddens[-1], self.n_output)
 
-        if self.model_type == 'AdaRNN':
+        if self.model_type == "AdaRNN":
             gate = nn.ModuleList()
             for i in range(len(n_hiddens)):
-                gate_weight = nn.Linear(
-                    len_seq * self.hiddens[i]*2, len_seq)
+                gate_weight = nn.Linear(len_seq * self.hiddens[i] * 2, len_seq)
                 gate.append(gate_weight)
             self.gate = gate
 
@@ -462,52 +448,53 @@ def init_layers(self):
 
     def forward_pre_train(self, x, len_win=0):
         out = self.gru_features(x)
-        fea = out[0] # [2N,L,H]
+        fea = out[0]  # [2N,L,H]
         if self.use_bottleneck == True:
             fea_bottleneck = self.bottleneck(fea[:, -1, :])
             fc_out = self.fc(fea_bottleneck).squeeze()
         else:
-            fc_out = self.fc_out(fea[:, -1, :]).squeeze() #[N,]
+            fc_out = self.fc_out(fea[:, -1, :]).squeeze()  # [N,]
 
         out_list_all, out_weight_list = out[1], out[2]
         out_list_s, out_list_t = self.get_features(out_list_all)
         loss_transfer = torch.zeros((1,)).cuda()
         for i in range(len(out_list_s)):
-            criterion_transder = TransferLoss(
-                loss_type=self.trans_loss, input_dim=out_list_s[i].shape[2])
-            h_start = 0 
+            criterion_transder = TransferLoss(loss_type=self.trans_loss, input_dim=out_list_s[i].shape[2])
+            h_start = 0
             for j in range(h_start, self.len_seq, 1):
                 i_start = j - len_win if j - len_win >= 0 else 0
                 i_end = j + len_win if j + len_win < self.len_seq else self.len_seq - 1
                 for k in range(i_start, i_end + 1):
-                    weight = out_weight_list[i][j] if self.model_type == 'AdaRNN' else 1 / (
-                        self.len_seq - h_start) * (2 * len_win + 1)
+                    weight = (
+                        out_weight_list[i][j]
+                        if self.model_type == "AdaRNN"
+                        else 1 / (self.len_seq - h_start) * (2 * len_win + 1)
+                    )
                     loss_transfer = loss_transfer + weight * criterion_transder.compute(
-                        out_list_s[i][:, j, :], out_list_t[i][:, k, :])
+                        out_list_s[i][:, j, :], out_list_t[i][:, k, :]
+                    )
         return fc_out, loss_transfer, out_weight_list
 
     def gru_features(self, x, predict=False):
         x_input = x
         out = None
         out_lis = []
-        out_weight_list = [] if (
-             self.model_type == 'AdaRNN') else None
+        out_weight_list = [] if (self.model_type == "AdaRNN") else None
         for i in range(self.num_layers):
             out, _ = self.features[i](x_input.float())
             x_input = out
             out_lis.append(out)
-            if self.model_type == 'AdaRNN' and predict == False:
+            if self.model_type == "AdaRNN" and predict == False:
                 out_gate = self.process_gate_weight(x_input, i)
                 out_weight_list.append(out_gate)
         return out, out_lis, out_weight_list
 
     def process_gate_weight(self, out, index):
-        x_s = out[0: int(out.shape[0]//2)]
-        x_t = out[out.shape[0]//2: out.shape[0]]
+        x_s = out[0 : int(out.shape[0] // 2)]
+        x_t = out[out.shape[0] // 2 : out.shape[0]]
         x_all = torch.cat((x_s, x_t), 2)
         x_all = x_all.view(x_all.shape[0], -1)
-        weight = torch.sigmoid(self.bn_lst[index](
-            self.gate[index](x_all.float())))
+        weight = torch.sigmoid(self.bn_lst[index](self.gate[index](x_all.float())))
         weight = torch.mean(weight, dim=0)
         res = self.softmax(weight).squeeze()
         return res
@@ -515,8 +502,8 @@ def process_gate_weight(self, out, index):
     def get_features(self, output_list):
         fea_list_src, fea_list_tar = [], []
         for fea in output_list:
-            fea_list_src.append(fea[0: fea.size(0) // 2])
-            fea_list_tar.append(fea[fea.size(0) // 2:])
+            fea_list_src.append(fea[0 : fea.size(0) // 2])
+            fea_list_tar.append(fea[fea.size(0) // 2 :])
         return fea_list_src, fea_list_tar
 
     # For Boosting-based
@@ -533,17 +520,14 @@ def forward_Boosting(self, x, weight_mat=None):
         out_list_s, out_list_t = self.get_features(out_list_all)
         loss_transfer = torch.zeros((1,)).cuda()
         if weight_mat is None:
-            weight = (1.0 / self.len_seq *
-                      torch.ones(self.num_layers, self.len_seq)).cuda()
+            weight = (1.0 / self.len_seq * torch.ones(self.num_layers, self.len_seq)).cuda()
         else:
             weight = weight_mat
         dist_mat = torch.zeros(self.num_layers, self.len_seq).cuda()
         for i in range(len(out_list_s)):
-            criterion_transder = TransferLoss(
-                loss_type=self.trans_loss, input_dim=out_list_s[i].shape[2])
+            criterion_transder = TransferLoss(loss_type=self.trans_loss, input_dim=out_list_s[i].shape[2])
             for j in range(self.len_seq):
-                loss_trans = criterion_transder.compute(
-                    out_list_s[i][:, j, :], out_list_t[i][:, j, :])
+                loss_trans = criterion_transder.compute(out_list_s[i][:, j, :], out_list_t[i][:, j, :])
                 loss_transfer = loss_transfer + weight[i, j] * loss_trans
                 dist_mat[i, j] = loss_trans
         return fc_out, loss_transfer, dist_mat, weight
@@ -554,8 +538,7 @@ def update_weight_Boosting(self, weight_mat, dist_old, dist_new):
         dist_old = dist_old.detach()
         dist_new = dist_new.detach()
         ind = dist_new > dist_old + epsilon
-        weight_mat[ind] = weight_mat[ind] * \
-            (1 + torch.sigmoid(dist_new[ind] - dist_old[ind]))
+        weight_mat[ind] = weight_mat[ind] * (1 + torch.sigmoid(dist_new[ind] - dist_old[ind]))
         weight_norm = torch.norm(weight_mat, dim=1, p=1)
         weight_mat = weight_mat / weight_norm.t().unsqueeze(1).repeat(1, self.len_seq)
         return weight_mat
@@ -572,7 +555,7 @@ def predict(self, x):
 
 
 class TransferLoss(object):
-    def __init__(self, loss_type='cosine', input_dim=512):
+    def __init__(self, loss_type="cosine", input_dim=512):
         """
         Supported loss_type: mmd(mmd_lin), mmd_rbf, coral, cosine, kl, js, mine, adv
         """
@@ -589,40 +572,40 @@ def compute(self, X, Y):
         Returns:
             [tensor] -- transfer loss
         """
-        if self.loss_type == 'mmd_lin' or self.loss_type =='mmd':
-            mmdloss = MMD_loss(kernel_type='linear')
+        if self.loss_type == "mmd_lin" or self.loss_type == "mmd":
+            mmdloss = MMD_loss(kernel_type="linear")
             loss = mmdloss(X, Y)
-        elif self.loss_type == 'coral':
+        elif self.loss_type == "coral":
             loss = CORAL(X, Y)
-        elif self.loss_type == 'cosine' or self.loss_type == 'cos':
+        elif self.loss_type == "cosine" or self.loss_type == "cos":
             loss = 1 - cosine(X, Y)
-        elif self.loss_type == 'kl':
+        elif self.loss_type == "kl":
             loss = kl_div(X, Y)
-        elif self.loss_type == 'js':
+        elif self.loss_type == "js":
             loss = js(X, Y)
-        elif self.loss_type == 'mine':
-            mine_model = Mine_estimator(
-                input_dim=self.input_dim, hidden_dim=60).cuda()
+        elif self.loss_type == "mine":
+            mine_model = Mine_estimator(input_dim=self.input_dim, hidden_dim=60).cuda()
             loss = mine_model(X, Y)
-        elif self.loss_type == 'adv':
+        elif self.loss_type == "adv":
             loss = adv(X, Y, input_dim=self.input_dim, hidden_dim=32)
-        elif self.loss_type == 'mmd_rbf':
-            mmdloss = MMD_loss(kernel_type='rbf')
+        elif self.loss_type == "mmd_rbf":
+            mmdloss = MMD_loss(kernel_type="rbf")
             loss = mmdloss(X, Y)
-        elif self.loss_type == 'pairwise':
+        elif self.loss_type == "pairwise":
             pair_mat = pairwise_dist(X, Y)
             loss = torch.norm(pair_mat)
 
         return loss
 
+
 def cosine(source, target):
     source, target = source.mean(), target.mean()
     cos = nn.CosineSimilarity(dim=0)
     loss = cos(source, target)
     return loss.mean()
 
-class ReverseLayerF(Function):
 
+class ReverseLayerF(Function):
     @staticmethod
     def forward(ctx, x, alpha):
         ctx.alpha = alpha
@@ -664,6 +647,7 @@ def adv(source, target, input_dim=256, hidden_dim=512):
     loss = loss_s + loss_t
     return loss
 
+
 def CORAL(source, target):
     d = source.size(1)
     ns, nt = source.size(0), target.size(0)
@@ -682,8 +666,9 @@ def CORAL(source, target):
 
     return loss
 
+
 class MMD_loss(nn.Module):
-    def __init__(self, kernel_type='linear', kernel_mul=2.0, kernel_num=5):
+    def __init__(self, kernel_type="linear", kernel_mul=2.0, kernel_num=5):
         super(MMD_loss, self).__init__()
         self.kernel_num = kernel_num
         self.kernel_mul = kernel_mul
@@ -693,20 +678,16 @@ def __init__(self, kernel_type='linear', kernel_mul=2.0, kernel_num=5):
     def guassian_kernel(self, source, target, kernel_mul=2.0, kernel_num=5, fix_sigma=None):
         n_samples = int(source.size()[0]) + int(target.size()[0])
         total = torch.cat([source, target], dim=0)
-        total0 = total.unsqueeze(0).expand(
-            int(total.size(0)), int(total.size(0)), int(total.size(1)))
-        total1 = total.unsqueeze(1).expand(
-            int(total.size(0)), int(total.size(0)), int(total.size(1)))
-        L2_distance = ((total0-total1)**2).sum(2)
+        total0 = total.unsqueeze(0).expand(int(total.size(0)), int(total.size(0)), int(total.size(1)))
+        total1 = total.unsqueeze(1).expand(int(total.size(0)), int(total.size(0)), int(total.size(1)))
+        L2_distance = ((total0 - total1) ** 2).sum(2)
         if fix_sigma:
             bandwidth = fix_sigma
         else:
-            bandwidth = torch.sum(L2_distance.data) / (n_samples**2-n_samples)
+            bandwidth = torch.sum(L2_distance.data) / (n_samples ** 2 - n_samples)
         bandwidth /= kernel_mul ** (kernel_num // 2)
-        bandwidth_list = [bandwidth * (kernel_mul**i)
-                          for i in range(kernel_num)]
-        kernel_val = [torch.exp(-L2_distance / bandwidth_temp)
-                      for bandwidth_temp in bandwidth_list]
+        bandwidth_list = [bandwidth * (kernel_mul ** i) for i in range(kernel_num)]
+        kernel_val = [torch.exp(-L2_distance / bandwidth_temp) for bandwidth_temp in bandwidth_list]
         return sum(kernel_val)
 
     def linear_mmd(self, X, Y):
@@ -715,12 +696,13 @@ def linear_mmd(self, X, Y):
         return loss
 
     def forward(self, source, target):
-        if self.kernel_type == 'linear':
+        if self.kernel_type == "linear":
             return self.linear_mmd(source, target)
-        elif self.kernel_type == 'rbf':
+        elif self.kernel_type == "rbf":
             batch_size = int(source.size()[0])
             kernels = self.guassian_kernel(
-                source, target, kernel_mul=self.kernel_mul, kernel_num=self.kernel_num, fix_sigma=self.fix_sigma)
+                source, target, kernel_mul=self.kernel_mul, kernel_num=self.kernel_num, fix_sigma=self.fix_sigma
+            )
             with torch.no_grad():
                 XX = torch.mean(kernels[:batch_size, :batch_size])
                 YY = torch.mean(kernels[batch_size:, batch_size:])
@@ -729,6 +711,7 @@ def forward(self, source, target):
                 loss = torch.mean(XX + YY - XY - YX)
             return loss
 
+
 class Mine_estimator(nn.Module):
     def __init__(self, input_dim=2048, hidden_dim=512):
         super(Mine_estimator, self).__init__()
@@ -738,8 +721,7 @@ def forward(self, X, Y):
         Y_shffle = Y[torch.randperm(len(Y))]
         loss_joint = self.mine_model(X, Y)
         loss_marginal = self.mine_model(X, Y_shffle)
-        ret = torch.mean(loss_joint) - \
-            torch.log(torch.mean(torch.exp(loss_marginal)))
+        ret = torch.mean(loss_joint) - torch.log(torch.mean(torch.exp(loss_marginal)))
         loss = -ret
         return loss
 
@@ -752,10 +734,11 @@ def __init__(self, input_dim=2048, hidden_dim=512):
         self.fc2 = nn.Linear(hidden_dim, 1)
 
     def forward(self, x, y):
-        h1 = F.leaky_relu(self.fc1_x(x)+self.fc1_y(y))
+        h1 = F.leaky_relu(self.fc1_x(x) + self.fc1_y(y))
         h2 = self.fc2(h1)
         return h2
 
+
 def pairwise_dist(X, Y):
     n, d = X.shape
     m, _ = Y.shape
@@ -775,6 +758,7 @@ def pairwise_dist_np(X, Y):
     b = np.tile(b, (n, 1, 1))
     return np.power(a - b, 2).sum(2)
 
+
 def pa(X, Y):
     XY = np.dot(X, Y.T)
     XX = np.sum(np.square(X), axis=1)
@@ -784,21 +768,22 @@ def pa(X, Y):
 
     return dist
 
+
 def kl_div(source, target):
     if len(source) < len(target):
-        target = target[:len(source)]
+        target = target[: len(source)]
     elif len(source) > len(target):
-        source = source[:len(target)]
-    criterion = nn.KLDivLoss(reduction='batchmean')
+        source = source[: len(target)]
+    criterion = nn.KLDivLoss(reduction="batchmean")
     loss = criterion(source.log(), target)
     return loss
 
 
 def js(source, target):
     if len(source) < len(target):
-        target = target[:len(source)]
+        target = target[: len(source)]
     elif len(source) > len(target):
-        source = source[:len(target)]
-    M = .5 * (source + target)
+        source = source[: len(target)]
+    M = 0.5 * (source + target)
     loss_1, loss_2 = kl_div(source, M), kl_div(target, M)
-    return .5 * (loss_1 + loss_2)
\ No newline at end of file
+    return 0.5 * (loss_1 + loss_2)

From 55184da96ce093471fc31edf04c689891e298366 Mon Sep 17 00:00:00 2001
From: lwwang1995 <lewen.wang@microsoft.com>
Date: Tue, 16 Nov 2021 11:18:08 +0800
Subject: [PATCH 8/8] Add README for adarnn.

---
 examples/benchmarks/ADARNN/README.md | 4 ++++
 1 file changed, 4 insertions(+)
 create mode 100644 examples/benchmarks/ADARNN/README.md

diff --git a/examples/benchmarks/ADARNN/README.md b/examples/benchmarks/ADARNN/README.md
new file mode 100644
index 00000000000..2296af92388
--- /dev/null
+++ b/examples/benchmarks/ADARNN/README.md
@@ -0,0 +1,4 @@
+# AdaRNN
+* Code: [https://github.com/jindongwang/transferlearning/tree/master/code/deep/adarnn](https://github.com/jindongwang/transferlearning/tree/master/code/deep/adarnn)
+* Paper: [AdaRNN: Adaptive Learning and Forecasting for Time Series](https://arxiv.org/pdf/2108.04443.pdf).
+