Serious performance drop in OpenCL backend after upgrading from 3.2.2 to 3.3

[unbornchikken]

There are the results:

3.2.2 - OpenCL

> fft_opencl.exe
ArrayFire v3.2.2 (OpenCL, 64-bit Windows, build 7507b61)
[0] AMD     : Pitcairn
-1- INTEL   : Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
-2- AMD     : Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
Benchmark N-by-N 2D fft
 128 x  128:    29 Gflops
 256 x  256:    54 Gflops
 512 x  512:    76 Gflops
1024 x 1024:   115 Gflops
2048 x 2048:   140 Gflops
4096 x 4096:   127 Gflops

3.3 - OpenCL

> fft_opencl.exe
ArrayFire v3.3.0 (OpenCL, 64-bit Windows, build 5842ed2)
[0] AMD     : Pitcairn, 2048 MB
-1- INTEL   : Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz, 8075 MB
-2- AMD     : Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz, 8075 MB
Benchmark N-by-N 2D fft
 128 x  128:    30 Gflops
 256 x  256:    46 Gflops
 512 x  512:    80 Gflops
1024 x 1024:    94 Gflops
2048 x 2048:    49 Gflops
4096 x 4096:    20 Gflops

I'm on Windows 10. On the other hand I can see around a +10% CPU performance increase in 3.3 though, so it gotta be an OpenCL only problem.

Do you aware of this?

[pavanky]

Hmm. We just use upstream clFFT. We assumed the performance was OK. I will report a bug with a stand alone test case. Thanks for doing this btw.

Also internally we are trying to benchmark some key functions to check for performance regressions. It is not automated yet and but we are in the process of doing so. This should help us catch issues like this sooner than latter in the future.

[pavanky]

The performance regression seems to have occurred between clFFT 2.8 and clFFT 2.10.

@unbornchikken if you can build arrayfire from source on your machine you can simply change the clFFT tag and rebuild it with v2.8

[unbornchikken]

The other benchmarks are affected as well, so I really doubt that is a clFFT only issue though.

[pavanky]

@unbornchikken What other benchmarks ? I just rebuilt arrayfire 3.3.0 with clFFT 2.8 and I get the old performance. Atleast in this case it is clFFT.

[unbornchikken]

In arrayfire-js the neural network example's performance has been dropped by roughly 50%. I cannot build AF and examples right now, but there should be another issues other than FFT because the NN example doesn't use that for sure.

[unbornchikken]

Could you try the nn ex at your side?

[pavanky]

If you meant the neural network example in arrayfire, I just did and the performance is identical.

If you need me to benchmark the example from arrayfire-js, I'll need some time.

[pavanky]

@unbornchikken I do see a performance hit when I am using Tahiti instead of Hawaii GPU. Here are some numbers:

GPU	3.2.2	3.3.0
Tahiti	23s	71s
Hawaii	61s	61s

We do not do anything special based on architecture inside arrayfire. This could be coming from clBLAS..

[unbornchikken]

Yeah, that can sucks. Maybe it's worth to report it to upstream.

[pavanky]

Yeah we are in the process of reproducing the issues in stand alone code so we can report it upstream.

[shehzan10]

Cross referencing upstream issue: clMathLibraries/clFFT#134

[unbornchikken]

I still have performance degradation with NN at my side. Did you see that: clMathLibraries/clBLAS#239 ? This might be the cause.

[pavanky]

@unbornchikken The fix was on for clFFT. As for clBLAS, the issue you are pointing to is for square, complex matrices. In NN example we use heavily rectangular real matrices. Those are two different issues.

We are still narrowing down the whats causing the slowdown for NN example.

[unbornchikken]

Hi. Does 3.3.1 resolve the issue with the NN example or that's just for the FFT?

[pavanky]

Just fft.