./Data/Array/Accelerate/LLVM/PTX/State.hs:50:9: (unhandled): CUDA Exception: invalid argument

[alang9]

I'm getting this weird error:

*** Exception: 
*** Internal error in package accelerate ***
*** Please submit a bug report at https://github.com/AccelerateHS/accelerate/issues
./Data/Array/Accelerate/LLVM/PTX/State.hs:50:9: (unhandled): CUDA Exception: invalid argument
CallStack (from HasCallStack):
  error, called at ./Data/Array/Accelerate/LLVM/PTX/State.hs:50:9 in accelerate-llvm-ptx-1.0.0.0-NmZValj0z98sMHA20PqUN:Data.Array.Accelerate.LLVM.PTX.State

On this code. I'm using accelerate, accelerate-llvm, accelerate-llvm-ptx 1.0.0.0. I'm also using master of linear-accelerate: https://github.com/ekmett/linear-accelerate

Running foo here causes the error:

multiplyMatrix4Vector ::
   forall a f. (A.Elt a, P.Num (Exp a), Functor f, Box f a, Additive f, A.Elt (f a)) =>
   Acc (Matrix (f a)) ->
   Acc (Vector a) ->
   Acc (Vector (f a))
multiplyMatrix4Vector m v =
    A.fold1 (^+^) $
    (A.zipWith (\v' s' -> s' *^ v') m
       (A.replicate (A.lift $ Any :. rows :. All) v) :: Acc (Matrix (f a)))
  where
    Z :. (rows :: Exp Int) :. (_cols :: Exp Int) = unlift (shape m)

foo :: ([Double], [Double], [Double])
foo = 
    (A.toList dCArr, map (view L._x) (A.toList outMat :: [V4 Double]), A.toList outVec)
  where
    n = 1
    xs = [27000]
    (dCArr, outMat :: Matrix (V4 Double), outVec) =
        run1 code ( A.fromList (Z :. P.length xs) xs, singleton n)
    code :: Acc ( A.Vector Double, A.Scalar Int)
         -> Acc ( A.Vector Double, Matrix (V4 Double), A.Vector Double)
    code (unlift -> (xs, (the -> n))) = out
      where
        out :: Acc (A.Vector Double, Matrix (V4 Double), A.Vector Double)
        out = A.lift
            ( dC -- A.fromList (Z :. 0) []
            , phiMat -- A.fromList (Z :. 0 :. 0) []
            , uPut -- A.fromList (Z :. 0) []
            )
        dC = A.map (view _x) $
            multiplyMatrix4Vector phiMat uPut
        phiMat = A.generate (A.index2 (A.length xs) n) $
            \_ -> A.lift (V4 0 0 0 0 :: V4 (Exp Double))
        uPut = A.generate (lift $ Z :. n) $ \_ -> 0

• replacing dC by A.fromList (Z :. 0) [] makes the error go away
• replacing phiMat by A.fromList (Z :. 0 :. 0) [] makes the error go away
• replacing uPut by A.fromList (Z :. 0) [] makes the error go away
• replacing all the V4s by V3s or V2s makes the error go away

[tmcdonell]

I added the imports etc. to make a runnable example here: https://gist.github.com/tmcdonell/e4ef72ae43d8695df852febc3b01e819. I trust that that gives the same error on your end?

This works for me. I suspect that it is configuring for your GPU incorrectly. Can you paste the output of stack exec nvidia-device-query, and then run the following (using the above gist)...

bash $ stack exec ghci -- Issue375
ghci > setFlags [dump_cc, dump_exec]
ghci > foo

[alang9]

Sorry I should have written a more complete example.

The stack command above doesn't work for me. stack doesn't seem to install the dependencies before loading the file in ghci. If I run stack build, I get the following error:

$ stack build
Error parsing targets: The project contains no local packages (packages not marked with 'extra-dep')

If I run it in my own ghci, I get the following debug output:

*Main> foo
([   2.028] llvm: generated 22 instructions in 4 blocks
[   2.028] llvm: optimisation did work? True
[   2.032] cc: kernel function 'generate' used 8 registers, 0 bytes smem, 0 bytes lmem, 0 bytes cmem
  ... multiprocessor occupancy 100.0 % : 2048 threads over 64 warps in 32 blocks
[   2.032] llvm: generated 30 instructions in 4 blocks
[   2.032] llvm: optimisation did work? True
[   2.036] cc: kernel function 'generate' used 12 registers, 0 bytes smem, 0 bytes lmem, 0 bytes cmem
  ... multiprocessor occupancy 100.0 % : 2048 threads over 64 warps in 32 blocks
[   2.036] llvm: generated 1484 instructions in 191 blocks
[   2.064] llvm: optimisation did work? True
[   2.084] cc: kernel function 'fold' used 62 registers, 50176 bytes smem, 0 bytes lmem, 0 bytes cmem
  ... multiprocessor occupancy 50.0 % : 1024 threads over 32 warps in 1 blocks
[   2.088] exec: generate <<< 1, 64, 0 >>> 17.870 µs (wall), 0.000 s (cpu), 5.088 µs (gpu)
[   2.088] exec: generate <<< 1, 64, 0 >>> 22.339 µs (wall), 0.000 s (cpu), 16.288 µs (gpu)
*** Exception: 
*** Internal error in package accelerate ***
*** Please submit a bug report at https://github.com/AccelerateHS/accelerate/issues
./Data/Array/Accelerate/LLVM/PTX/State.hs:50:9: (unhandled): CUDA Exception: invalid argument
CallStack (from HasCallStack):
  error, called at ./Data/Array/Accelerate/LLVM/PTX/State.hs:50:9 in accelerate-llvm-ptx-1.0.0.0-NmZValj0z98sMHA20PqUN:Data.Array.Accelerate.LLVM.PTX.State

[alang9]

I forgot to include the output of nvidia-device-query too:

$ nvidia-device-query 
CUDA device query (Driver API, statically linked)
CUDA driver version 8.0
Detected 1 CUDA capable device

Device 0: GeForce GTX 960
  CUDA capability:                    5.2
  CUDA cores:                         1024 cores in 8 multiprocessors (128 cores/MP)
  Global memory:                      2 GB
  Constant memory:                    64 kB
  Shared memory per block:            48 kB
  Registers per block:                65536
  Warp size:                          32
  Maximum threads per multiprocessor: 2048
  Maximum threads per block:          1024
  Maximum grid dimensions:            2147483647 x 65535 x 65535
  Maximum block dimensions:           1024 x 1024 x 64
  GPU clock rate:                     1.228 GHz
  Memory clock rate:                  3.505 GHz
  Memory bus width:                   128-bit
  L2 cache size:                      1 MB
  Maximum texture dimensions         
    1D:                               65536
    2D:                               65536 x 65536
    3D:                               4096 x 4096 x 4096
  Texture alignment:                  512 B
  Maximum memory pitch:               2 GB
  Concurrent kernel execution:        Yes
  Concurrent copy and execution:      Yes, with 2 copy engines
  Runtime limit on kernel execution:  Yes
  Integrated GPU sharing host memory: No
  Host page-locked memory mapping:    Yes
  ECC memory support:                 No
  Unified addressing (UVA):           Yes
  PCI bus/location:                   2/0
  Compute mode:                       Default
    Multiple contexts are allowed on the device simultaneously

[maoe]

I've made the gist self-contained: https://gist.github.com/maoe/c96a6b6dca10418412ba56a58b395da5

You can set it up like:

stack setup
stack build

Then you can stack repl to get the REPL. foo just worked for me on my machine.

% stack repl
Configuring GHCi with the following packages: accelerate-issue375                                                               
GHCi, version 8.0.2: http://www.haskell.org/ghc/  :? for help   
Loaded GHCi configuration from /home/maoe/dotfiles/haskell/ghci.symlink                                                         
[1 of 1] Compiling Issue375         ( Issue375.hs, interpreted )                                                                
Ok, modules loaded: Issue375.   
Loaded GHCi configuration from /tmp/ghci29728/ghci-script       
λ> foo                          
([0.0],[0.0],[0.0])             
it :: ([Double], [Double], [Double])                            
λ>                              
Leaving GHCi.  

[tmcdonell]

Thanks @maoe!

Since it works for you, could you also post your nvidia-device-query output?

[maoe]

@tmcdonell I'm afraid I don't have such command.

My environment is as follows:

• Ubuntu Trusty (16.04)
• GeForce GTX 760
• CUDA from Ubuntu official repository: 5.5.22-3ubuntu1

[maoe]

Sorry I didn't know that it's available in the Haskell cuda package:

% nvidia-device-query
CUDA device query (Driver API, statically linked)               
CUDA driver version 8.0         
Detected 1 CUDA capable device  

Device 0: GeForce GTX 760       
  CUDA capability:                    3.0                       
  CUDA cores:                         1152 cores in 6 multiprocessors (192 cores/MP)                                            
  Global memory:                      2 GB                      
  Constant memory:                    64 kB                     
  Shared memory per block:            48 kB                     
  Registers per block:                65536                     
  Warp size:                          32                        
  Maximum threads per multiprocessor: 2048                      
  Maximum threads per block:          1024                      
  Maximum grid dimensions:            2147483647 x 65535 x 65535                                                                
  Maximum block dimensions:           1024 x 1024 x 64          
  GPU clock rate:                     1.0585 GHz                
  Memory clock rate:                  3.004 GHz                 
  Memory bus width:                   256-bit                   
  L2 cache size:                      512 kB                    
  Maximum texture dimensions                                    
    1D:                               65536                     
    2D:                               65536 x 65536             
    3D:                               4096 x 4096 x 4096        
  Texture alignment:                  512 B                     
  Maximum memory pitch:               2 GB                      
  Concurrent kernel execution:        Yes                       
  Concurrent copy and execution:      Yes, with 1 copy engine   
  Runtime limit on kernel execution:  Yes                       
  Integrated GPU sharing host memory: No                        
  Host page-locked memory mapping:    Yes                       
  ECC memory support:                 No                        
  Unified addressing (UVA):           Yes                       
  PCI bus/location:                   2/0                       
  Compute mode:                       Default                   
    Multiple contexts are allowed on the device simultaneously 

[maoe]

Just in case this is with setFlags [dump_cc, dump_exec]:

λ> foo                          
([   1.259] llvm: generated 22 instructions in 4 blocks         
[   1.264] llvm: optimisation did work? True                    
[   1.268] cc: kernel function 'generate' used 7 registers, 0 bytes smem, 0 bytes lmem, 0 bytes cmem                            
  ... multiprocessor occupancy 100.0 % : 2048 threads over 64 warps in 16 blocks                                                
[   1.268] llvm: generated 30 instructions in 4 blocks          
[   1.270] llvm: optimisation did work? True                    
[   1.273] cc: kernel function 'generate' used 13 registers, 0 bytes smem, 0 bytes lmem, 0 bytes cmem                           
  ... multiprocessor occupancy 100.0 % : 2048 threads over 64 warps in 16 blocks                                                
[   1.274] llvm: generated 1484 instructions in 191 blocks      
[   1.315] llvm: optimisation did work? True                    
[   1.345] cc: kernel function 'fold' used 59 registers, 48608 bytes smem, 0 bytes lmem, 0 bytes cmem                           
  ... multiprocessor occupancy 48.4 % : 992 threads over 31 warps in 1 blocks                                                   
[   1.350] exec: generate <<< 1, 128, 0 >>> 40.000 µs (wall), 0.000 s (cpu), 6.304 µs (gpu)                                     
[   1.350] exec: generate <<< 1, 128, 0 >>> 24.000 µs (wall), 0.000 s (cpu), 5.856 µs (gpu)                                     
[[   1.350] exec: fold <<< 1, 992, 48608 >>> 21.000 µs (wall), 0.000 s (cpu), 301.248 µs (gpu)                                  
0.0],[0.0],[0.0])               
it :: ([Double], [Double], [Double])  

[tmcdonell]

@alang9 could you try with cuda-0.7.5.2 ?

[alang9]

The version of cuda i'm currently using is 0.7.5.2

[tmcdonell]

And 0.7.5.3 also doesn't work?

The 0.7.5.3 version was a bug fix to the occupancy calculator which caused a similar problem on pascal. I was checking to see if I had broken the magic numbers for maxwell in the transition (which would have been easier to fix). I'll see if I can find a maxwell card to debug this on.

[alang9]

@tmcdonell: Ahh it seems to work with 0.7.5.3. I should have tried that earlier

[tmcdonell]

Ah, excellent! So, it was the same occupancy calculator bug after all. Strange that that lurked undiscovered for so long, but glad that it is working for you now (: