December 15, 2015.

# Building a Mingw-w64 Toolchain that Links with a Specific Visual Studio Runtime Library

Compiling native Python modules from source packages using a Un*x compatible toolchain can sometimes be tricky. Both the Cygwin and Mingw-w64 toolchains end up linking with msvcrt.dll while the Python interpreter, built with Visual Studio 2008 links with a more recent msvcr90.dll C runtime library. This can cause instability if resources allocated in one runtime (file, or memory) is used or freed using the other runtime. In this post we will explain how to modify an open-source compiler to build Python modules linking to a specific C runtime library.

Compiling native Python modules from source packages using a Un*x compatible toolchain can sometimes be tricky. Both the Cygwin and Mingw-w64 toolchains end up linking with msvcrt.dll while the Python interpreter, built with Visual Studio 2008 links with a more recent msvcr90.dll C runtime library. This can cause instability if resources allocated in one runtime (file, or memory) is used or freed using the other runtime. In this post we will explain how to modify an open-source compiler to build Python modules linking to a specific C runtime library.

Before we begin, let's mention that building modules containing a working Microsoft Visual C project file can be done using a custom Python interpreter built either with Visual Studio 2010 or Visual Studio 2012. You can find a nice step by step tutorial on the p-nand-q website. The modules we are targetting here build only on U*ix compatible systems and can be compiled on Windows using environments like those provided by Cygwin and MSys2.

This entry will describe how to modify a Mingw-w64 toolchain to make sure all build products are linking to the same runtime library. In this example, we will target Visual Studio 2012, but if your target is the standard Python interpreter, you only need to substitute 90 everywhere you see 110.

## Setting Up the Build Environment

We will use MSys2 as the build environment. Fetch the 64-bit base installer from the main website and install it somewhere on your development machine. Then open an MSys2 shell from the Windows Start Menu. You will need to bring the MSys2 platform up to date with the following commands:

$pacman -Sy$ pacman --needed -S bash pacman pacman-mirrors msys2-runtime

Open a new MSys2 shell and then install all the dependencies needed to build a Mingw-w64 toolchain:

$pacman -S git make texinfo pkgfile diffutils tar flex coreutils cvs subversion wget patch man zip p7zip automake autoconf libtool bison gettext-devel sshpass Do not install any pre packaged compilers, especially the mingw-w64 ones. They will interfere with the build process and the build script we will be using does not tolerate that. ## Building the Toolchain We will use the Mingw-w64 build scripts provided by niXman on GitHub: and use the latest from the development branch. $ cd ~
$git clone https://github.com/niXman/mingw-builds.git$ cd mingw-builds/
$git checkout develop Let's launch the build and see if it first completes without any tampering on our side: $ cd ~/mingw-builds/
$./build --mode=gcc-5.1.0 --exceptions=seh --jobs=10 --rt-version=trunk --threads=win32 --arch=x86_64 --enable-languages=c,c++,fortran --bootstrap Let's now start modifying the build to link with msvcr110.dll: $ ./toolchains/mingw64/bin/gcc.exe -dumpspecs > toolchains/mingw64/lib/gcc/x86_64-w64-mingw32/4.9.3/specs

The freshly generated specfile now controls how the new compiler will be built. For this to work, the path must be the one from the compiler downloaded by the build script (gcc 4.9.3 in this case). Open the specs file in a U*ix compatible text editor that undertands cross-platform line endings and modify the following sections by adding the text in blue:

*cpp:
%{posix:-D_POSIX_SOURCE} %{mthreads:-D_MT} %{municode:-DUNICODE} %{!no-pthread:-D_REENTRANT} %{pthread:-U_REENTRANT} -D__MSVCRT_VERSION__=0x1100 -D__USE_MINGW_ACCESS
*cc1plus:
-D__MSVCRT_VERSION__=0x1100 -D__USE_MINGW_ACCESS
*libgcc:
%{mthreads:-lmingwthrd} -lmingw32 %{static|static-libgcc:-lgcc -lgcc_eh} %{!static: %{!static-libgcc: %{!shared: %{!shared-libgcc:-lgcc -lgcc_eh} %{shared-libgcc:-lgcc_s -lgcc} } %{shared:-lgcc_s -lgcc} } } -lmoldname110 -lmingwex -lmsvcr110

Then you need to build the libmoldname110 library that links to msvcr110.dll (since any symbol linked via the regular libmoldname will pull symbols from msvcrt.dll):

$cd ~/mingw-builds/toolchains/mingw64/x86_64-w64-mingw32/lib/$ ../../bin/dlltool.exe -d ~/mingw-gcc-5.1.0/src/mingw-w64/mingw-w64-crt/lib64/moldname-msvcrt.def -U --dllname msvcr110.dll -l libmoldname110.a -k --as=/home/jgamache/mingw-builds/toolchains/mingw64/bin/as.exe --as-flags=--64 -m i386:x86-64

With these changes you are now ready to start the build. First, if you have artefacts from previous builds, you should clean the build areas:

$rm -rf ~/mingw-gcc-5.1.0/x86_64-510-win32-seh-rt_v4/ ~/mingw-gcc-5.1.0/prerequisites-build/ And launch the build: $ cd ~/mingw-builds/
$find -name libmoldname.a -execdir cp ~/mingw-builds/toolchains/mingw64/x86_64-w64-mingw32/lib/libmoldname110.a . \; ### 3- Newly built libraries not linking to the right runtime: You will probably want to check as early as possible that the newly built libraries are correctly linked. The best tool for the job is Dependency Walker. Use it to check one of the early build products (like ~/mingw-gcc-5.1.0/x86_64-510-win32-seh-rt_v4/build/bzip2-1.0.6/bzip2.exe) and make sure msvcr110.dll is the only runtime library referenced by this executable, either directly or indirectly. ## Post Build Fixups: These changes are not necessary for the build to complete. They will help when compiling some Python packages. One of the errors you can encounter while building Python modules is the preprocessor complaining that INT32 is being redefined, or is already defined. To fix, open the file ~/mingw-gcc-5.1.0/x86_64-510-win32-seh-rt_v4/mingw64/x86_64-w64-mingw32/include/basetsd.h and update the typedef section with proper guards to detect symbols already defined: #if !defined(INT8) typedef signed char INT8,*PINT8; #endif #if !defined(INT16) typedef signed short INT16,*PINT16; #endif #if !defined(INT32) typedef signed int INT32,*PINT32; #endif #if !defined(INT64) __MINGW_EXTENSION typedef signed __int64 INT64,*PINT64; #endif #if !defined(UINT8) typedef unsigned char UINT8,*PUINT8; #endif #if !defined(UINT16) typedef unsigned short UINT16,*PUINT16; #endif #if !defined(UINT32) typedef unsigned int UINT32,*PUINT32; #endif #if !defined(UINT64) __MINGW_EXTENSION typedef unsigned __int64 UINT64,*PUINT64; #endif #if !defined(LONG32) typedef signed int LONG32,*PLONG32; #endif #if !defined(ULONG32) typedef unsigned int ULONG32,*PULONG32; #endif #if !defined(DWORD32) typedef unsigned int DWORD32,*PDWORD32; #endif You are now ready to use the toolchain. You can wrap it into a tarball to install it later on any virgin MSys2 installation. $ cd ~/mingw-gcc-5.1.0/x86_64-510-win32-seh-rt_v4/
$tar -cJvf ~/x86_64-510-win32-seh-rt_v4_VS2012.tar.xz$ cd /
\$ tar -xf ~/x86_64-510-win32-seh-rt_v4_VS2012.tar.xz

If you close the MSys2 shell and open a new one using the C:\msys64\mingw64_shell.bat instead, you should be able to compile a test C file that will link with the expected runtime.