At the moment, I’m working on generating installers for Optizelle on Windows. Overall, it’s going well, but the process has reminded me of my adventures in compiling scientific codes on Windows. Certainly, it’s not impossible, but there are several gotchas. In any case, I just finished up an experimental installer, which is a good opportunity to discuss some of the nuances of compiling Optizelle and other scientific codes on Windows.
Microsoft distributes Visual Studio Express, which is a free set of compilers for Windows. Unfortunately, it doesn’t contain a Fortran compiler, which is our first roadblock. In fact, the full version of Visual Studio doesn’t contain a Fortran compiler either. It must be purchased separately. Now, most codes that I see nowadays aren’t being written in Fortran, so why is this a big deal? In short, BLAS and LAPACK. Basically, for dense linear algebra kernels on shared memory machines, BLAS and LAPACK are still king. They’re robust, well written, reliable, and standard. Certainly, projects like Eigen contain some fantastic code, but BLAS and LAPACK still dominate.
Now, most people don’t compile BLAS and LAPACK from scratch. Rather, we get optimized builds such as Intel’s MKL, AMD’s ACML, or Apple’s vecLib. As a minor annoyance, these libraries tend to trail the official LAPACK releases, but they’re still pretty good. Personally, I prefer ATLAS BLAS, which requires compilation and tuning, but it’s easier for me to keep it up to date with the newest versions of LAPACK. In any case, if we already have a precompiled version of BLAS and LAPACK, why do we need a Fortran compiler? Basically, we need a Fortran compiler to determine the name mangling scheme for the current platform. In general Fortran mangles the symbol names of its functions in one of four ways: upper case, lower case, upper case trailing underscore, lower case trailing underscore. For example, the function dgemm could be: DGEMM, dgemm, DGEMM, or dgemm_. Due to its Fortran roots, these precompiled libraries all follow this naming scheme. In order to determine it, we use our build environment to query the Fortran compiler. Both CMake as well as Autoconf have scripts to make this work.
Certainly, projects such as CBLAS and CLAPACK provide wrappers to BLAS and LAPACK where knowledge of the name mangling scheme are not required. Unfortunately, these libraries don’t do the right thing on 64-bit systems. While Fortran BLAS and LAPACK routines use an architecture specific integer for their sizes, CBLAS and CLAPACK hard code this to be a 32-bit int. In my opinion, they should use ptrdiff_t, which correctly selects the architecture specific integer. Unfortunately, that’s not the case and it doesn’t look like it’s going to change. As such, we’re stuck with Fortran BLAS and LAPACK, which means we need a Fortran compiler.
Thankfully, GCC has been ported to 64-bit Windows under the Mingw-w64 project. However, the exact compiler suite required becomes a little bit confusing. MinGW (Minimalist GNU for Windows) has been around for some time, but focused on compilers for 32-bit Windows. Really, we want a 64-bit compiler, which is where Mingw-w64 comes in. However, they have their own website and downloads, which is a little bit confusing. In fact, the one setup that I found to be most reliable is the Mingw-builds project, which is hosted in a third location. I strongly recommend using Mingw-builds as opposed to Cygwin. It works well and doesn’t include a dependency on cygwin1.dll.
Now, Mingw-builds contains the compilers, but it doesn’t contain the necessary build environment such as GNU Make. In order to get these, we can use Cygwin, but I’ve had better luck with MSYS. Unlike the compilers, MSYS remains 32-bit. That’s not a problem as long as we link to the correct compilers. As far as getting MSYS, the attached link contains instructions for building it, but that’s not what we need. Frankly, we need precompiled binaries, which are thankfully provided by the Mingw-builds project here. Use these precompiled files; it will save a ton of grief.
As far as the build environment, I prefer CMake. It works well for cross-platform builds and contains the necessary scripts to interact with Fortran with the FortranCInterface. Certainly, Autoconf can do the job, but it’s better suited for POSIX systems, which Windows is not.
In Optizelle, we offer the ability to compile BLAS, LAPACK, and jsoncpp along with Optizelle. For the most part, this works well, but, again, we have Windows specific issues. Specifically, CMake installs all dynamically linked libraries into the bin directory, not lib. This complicates our CMake scripts because it makes it difficult to automatically locate our libraries. Generally, we want to assume that all libraries can be found under lib. In order to simplify the issue, we use CMake to forcefully move all of our dlls from bin to lib.
Next, the current trendy thing in scientific programming is to wrap our codes in Python. Thankfully, Python has a native 64-bit Windows version, which mostly works well. I say mostly because using it with Optizelle will generate the error
error: '::hypot' has not been declared
This is bug in Python, which has been documented here. In order to combat this bug, we add -include math.h to CMAKE_CXX_FLAGS. We use this instead of -include cmath because the new version of jsoncpp includes math.h instead of cmath and including both causes compilation to fail.
As for linking Python, we need to link python27.dll, which CMake does not find. The problem is that CMake is 32-bit and python27.dll sits in the Windows\System32 directory. On 64-bit Windows, 64-bit libraries sit in System32 and 32-bit libraries sit in SysWOW64. Now, when a program wants to access a library, it refers to System32, but Windows automatically changes the path to SysWOW64 for 32-bit programs. This brings us back to CMake. Since python27.dll is 64-bit, CMake can not see the library. In addition, since MSYS is 32-bit, it can not see the library either. However, it shows up perfectly fine under Windows Explorer. In order to rectify this issue, we simply have to copy the python27.dll into the Python libs (not Lib) directory in the Python installation. Then, we can direct CMake to link directly to it.
This gets us closer, but we’re still not quite there. At this point, we generate the linking error
undefined reference to `_imp__Py_InitModule4'
In order to fix this error, we must add -DMS_WIN64 to CMAKE_CXX_FLAGS. This enables the correct bits of code in pyconfig.h.
Outside of Python, we frequently need to link our code to MATLAB. Thankfully, MathWorks has done their homework and there aren’t too many gotchas when creating mex files. For Optizelle, we note that the include directory can be found in extern\include. As far as the libraries, we require both libmex.dll and libmx.dll, which are found in bin\win64.
Finally, in order to package our code, we use CPack, which comes with CMake. It has a nice generator to the Nullsoft Scriptable Install System (NSIS). In short, NSIS creates nice Windows install and uninstall programs. Nevertheless, since CMake is 32-bit, it will incorrectly name the resulting install files as win32 instead of win64. Perhaps there’s a fancy way to fix this, but simply renaming the executable works. As another side affect of this, the default install directory will be Program Files (x86) and not Program Files. This is irritating, but doesn’t technically hurt anything.
In any case, hopefully the above helps others who work in scientific computing and want to develop on Windows. In anything, hopefully this will help others who want to compile Optizelle on Windows. At this point, I acknowledge that compiling Optizelle on Windows is way too complicated for most users, so it’s my intention to distribute Windows installers for common platforms. Stay tuned for that development!