refactoring data modules and implementing prompt generator

[niansong1996]

Working on #25 and potentially will also work on #36 in this PR.

[niansong1996]

TODOs:

• fix the chat format
• MBPP promptify function
• text-to-sql promptify function refactoring

[niansong1996]

@troyfeng116 Can you help me figure something out here: I was able to run the inference pipeline test locally (on lada) but it gives a weird error when running CI. Do you have an idea about why this is happening?

Also, it might be helpful to see if you can run the test locally on your end.

[troyfeng116]

@troyfeng116 Can you help me figure something out here: I was able to run the inference pipeline test locally (on lada) but it gives a weird error when running CI. Do you have an idea about why this is happening?

Also, it might be helpful to see if you can run the test locally on your end.

I'm getting the same error locally, what is your local transformers version?

[niansong1996]

@troyfeng116 This is my pip freeze

absl-py==1.3.0
aiohttp==3.8.3
aiosignal==1.3.1
astunparse==1.6.3
async-timeout==4.0.2
attrs==22.2.0
beautifulsoup4==4.12.0
cachetools==5.2.1
certifi @ file:///croot/certifi_1671487769961/work/certifi
charset-normalizer==2.1.1
click==8.1.3
deepspeed==0.6.7
docker-pycreds==0.4.0
docstring-parser==0.15
filelock==3.9.0
flit_core @ file:///opt/conda/conda-bld/flit-core_1644941570762/work/source/flit_core
frozenlist==1.3.3
fsspec==2022.11.0
func-timeout==4.3.5
gdown==4.7.1
gitdb==4.0.10
GitPython==3.1.30
google-auth==2.15.0
google-auth-oauthlib==0.4.6
grpcio==1.51.1
hjson==3.1.0
huggingface-hub==0.12.1
idna==3.4
importlib-metadata==6.0.0
importlib-resources==5.10.2
joblib==1.2.0
jsonargparse==4.15.0
Markdown==3.4.1
MarkupSafe==2.1.1
multidict==6.0.4
ninja==1.11.1
nltk==3.8.1
numpy==1.24.1
oauthlib==3.2.2
openai==0.27.2
overrides==7.3.1
packaging==23.0
pandas==1.5.2
pathtools==0.1.2
promise==2.3
protobuf==3.20.3
psutil==5.9.4
py-cpuinfo==9.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pydantic==1.10.4
pyDeprecate==0.3.2
PySocks==1.7.1
python-dateutil==2.8.2
pytorch-lightning==1.7.4
pytz==2022.7
PyYAML==6.0
regex==2022.10.31
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.9
scipy==1.10.0
sentencepiece==0.1.97
sentry-sdk==1.12.1
setproctitle==1.3.2
shortuuid==1.0.11
six==1.16.0
smmap==5.0.0
soupsieve==2.4
sqlparse==0.4.3
tensorboard==2.11.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tokenizers==0.12.1
torch==1.12.1
torchmetrics==0.9.3
tqdm==4.64.1
transformers==4.26.1
tree-sitter==0.19.0
typeshed-client==2.1.0
typing_extensions @ file:///croot/typing_extensions_1669924550328/work
urllib3==1.26.13
wandb==0.13.7
Werkzeug==2.2.2
yarl==1.8.2
zipp==3.11.0

[niansong1996]

I'm using Python 3.8.15 btw

[troyfeng116]

@niansong1996 Just pushed a fix, ran into something like this before and it was a jsonargparse version issue

[troyfeng116]

@niansong1996 Just pushed a fix, ran into something like this before and it was a jsonargparse version issue

Forgot to upgrade transformers version as well

[niansong1996]

@troyfeng116 Hmm, I tried to vary the version of jsonargparse but didn't think that the version of transformers would also break it.

Thanks a lot for the fix!