TensorFlow GPU build for Windows for TensorFlowSharp

After two days of struggling, I was finally able to build libtensorflow.dll (v1.2.1) for Windows with GPU (CUDA 8 + CUDNN 6) support. TL;DR; here's the file (~160MB). It seems (so far) usable with TensorFlowSharp from Nuget, but you have to manually swap DLL in the package cache.

UPD 2019-03-29: instead of using TensorFlowSharp, I am now using Gradient - it provides access to the full Python API. And you don't have to manually build TensorFlow for GPU - just install Python 3.6, and follow the official TensorFlow instructions to install tensorflow 1.10 or tensorflow-gpu 1.10, or tensorflow-rocm for ATI. Gradient picks it up automatically or via GradientSetup class.

Some notes on the build (in case you want to reproduce it):

I used Visual Studio 2017

Despite mentioning only VS 2015 C++ compiler as compatible, I was able to build with VS 2017 compiler.

Build with CMake

CMake files are in ./tensorflow/contrib/cmake

Do not forget SWIG

Download and install SWIG.

Use 64-bit compiler with 64-bit target

I had to explicitly pick 64-bit compiler binaries for CMAKE_CXX_COMPILER, CMAKE_C_COMPILER (C:/Program Files (x86)/Visual Studio/2017/VC/Tools/MSVC/14.10.25017/bin/HostX64/x64/cl.exe) and 64-bit CMAKE_LINKER (C:/Program Files (x86)/Visual Studio/2017/VC/Tools/MSVC/14.10.25017/bin/HostX64/x64/link.exe).

Explicitly specify CUDA_HOST_COMPILER

It probably must be VS 2015 C++ compiler, but I did not try otherwise. VS 2017 comes with it: C:/Program Files (x86)/Microsoft Visual Studio/Shared/14.0/VC/bin/amd64/cl.exe

Disable all unnecessary bits

I had to disable Python building build in CMake to avoid dealing with Python dev packages. You still need to configure Python 3.4+ binary though (I used the one coming with VS 2017 - C:/Program Files/Anaconda3/python.exe).

Explicitly specify CUDNN path

CUDNN_HOME (in my case it was C:/Program Files/Cudnn/6.0)

Installed CUDA 8.0 was autodetected.

MSBuild

Here's my command line:

msbuild /p:Configuration=Release /p:Platform=x64 /m:6 tensorflow.sln /p:PreferredToolArchitecture=x64

No debug info

In all my attempts I've failed to build RelWithDebInfo. Compiler would fail to write PDB files (debug info). I suspect it is caused by PDB file limit,

Restrict number of build processes if you don't have TONS of RAM

Initially I tried /m:10 on my 24GiB machine. But that build would choke on RAM and start swapping. 6-10 link.exe processes each taking 2GiB+. I had to reduce it to /m:6 to make it manageable.

Basically, either have more RAM, or at some point your machine is going to be completely unusable.

Set /p:PreferredToolArchitecture=x64 for MSBuild

Otherwise it might still use 32-bit compiler, which would fail due to insufficiency of 32-bit address space.

I am not sure why CMake does not do this.