Speeding up GPU neural network - neural-network

Question

I'm trying to implement a neural network to run on a GPU using the Thrust and CUBLAS libraries, but I'm having a lot of trouble getting it to run faster than our current multithreaded and vectorized CPU implementation. The network has a single hidden layer with logistic units and an output layer with linear units, and here is the code for that:
// Functor to add bias before computing logistic
template <typename T>
struct bias_logistic_f {
__host__ __device__
T operator()(const T& x, const T& y) const {
return 1/(1+exp(-(x+y)));
}
};
bias_logistic_f bias_logistic();
// Thrust vectors for input/hidden/output units
thrust::device_vector<FLT> batch(batch_rows*ndim);
thrust::device_vector<FLT> hid(batch_rows*nhid);
thrust::device_vector<FLT> gpu_code(ndata*ncode);
// ...Load data and network weights...
// Multiply input (batch) by weights (vis2hid)
// Our matrices are stored row-major, but BLAS wants column-major,
// so pretend they're transposed and compute hid' = vis2hid' * batch'
cublasDgemm(handle, CUBLAS_OP_N, CUBLAS_OP_N, nhid, batch_rows, ndim,
&alpha, thrust::raw_pointer_cast(&vis2hid[0]), nhid,
thrust::raw_pointer_cast(&batch[0]), ndim,
&beta, thrust::raw_pointer_cast(&hid[0]), nhid);
// Add hidbiases to hid and compute logistic
thrust::transform(hid.begin(), hid.end(), hidbiases.begin(), hid.begin(),
bias_logistic);
// Multiply hid by weights (hid2code)
cublasDgemm(handle, CUBLAS_OP_N, CUBLAS_OP_N, ncode, batch_rows, nhid,
&alpha, thrust::raw_pointer_cast(&hid2code[0]), ncode,
thrust::raw_pointer_cast(&hid[0]), nhid,
&beta, thrust::raw_pointer_cast(&gpu_code[b*batch_rows*ncode]), ncode);
// Add codebiases
thrust::transform(gpu_code.begin() + b*batch_rows*ncode, gpu_code.begin() + (b+1)*batch_rows*ncode,
codebiases.begin(), gpu_code.begin() + b*batch_rows*ncode,
thrust::plus<FLT>());
Our input data is a sparse matrix with about 150,000 rows and 6,500 columns, with about 100 non-zero elements per row on average. This is too large to store the full matrix as a dense matrix on the GPU, so what I do is loop through the sparse matrix expanding batches of 1,000 rows each to input into the neural network:
for(int b=0; b<nbatch; ++b) {
// Zero out batch b
thrust::fill(batch.begin(), batch.end(), 0.0f);
// batch_val contains the non-zero values for the current batch, batch_idx the indices within the batch,
// and batch_ptr indexes into batch_val/batch_idx
// This is like CSR format except instead of compressing rows, it's compressing submatrices of 1,000 rows
thrust::scatter(batch_val.begin() + batch_ptr[b],
batch_val.begin() + batch_ptr[b+1],
batch_idx.begin() + batch_ptr[b],
batch.begin());
// ...Input batch to network (shown above)...
}
Our CPU implementation does the same thing, using STL vectors. When I ran both and compared their run times, I was surprised to find that the GPU code takes about 38 seconds on average to process our data, while the CPU code only takes about 27 seconds. It could be that some of this difference is due to the GPU being a few years old (a Tesla C1060) while the server is a newer 24-core machine. But still I would've thought with thousands of threads available, it wouldn't end up being 50% slower.
Any ideas how I can make this code run faster? I'm new to GPU programming so I'm at a loss as to what I could be doing wrong. Is there a more efficient way to deal with sparse matrices than what I'm doing here, such as using the CUSPARSE library? Or would it be a better idea to forget about the high-level libraries altogether and just write my own kernels in CUDA to combine the matrix multiplication/logistic/addition steps?