[Question]: RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1)

[LiweiPE]

Describe the issue

The bug is produced when running in multi gpus, since I only can test benchmark_e2e.py with --context_window 512000. Then, I used dual card for Qwen2-7B for context 1M and this error is produced. How to fix?

[iofu728]

Fixed in #30, please try to use pip install minference -U.