17
Homebrew, pyenv, ctypes... oh my!
I usually have a number of Python versions, environments and tools in play. I generally use pyenv to manage Python versions, and Homebrew to manage pyenv.
If you've come across this post because you're having problems installing Python via Homebrew-managed pyenv, you have my sympathies. And in case it "just works" for you, feel free to give something like this a try:
CC="$(brew --prefix gcc)/bin/gcc-11" \
pyenv install --verbose 3.10.0If you're interested in the circuitous road of troubleshooting and exploration that led to that command, you're welcome to keep reading and come along for the ride. Nice to have company :).
After successfully installing Python 3.10 with pyenv, I found that I couldn't pip install packages with C extension modules. They complained about _ctypes being missing. And trying to import ctypes from the Python REPL confirmed the issue:
Python 3.10.0 (default, Dec 31 2021, 16:09:32) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/tmp/python-build.20211231145343.269699/Python-3.10.0/Lib/ctypes/__init__.py", line 8, in <module>
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'Looking at the logs of my pyenv install command, I found this error:
*** WARNING: renaming "_ctypes" since importing it failed: libffi.so.8: cannot open shared object file: No such file or directoryWhich took things a bit further and pointed at an underlying problem. Now, I've seen that issue (or one of its kin) before, and it usually means something like "go muck with LDFLAGS until it works". And after a bit of searching I found a number of relevant resources with plenty of suggestions:
Along with the always-helpful suggested build environment information from the pyenv wiki.
That was plenty of information to fuel a few cycles of "try stuff and see what happens".
Exploring those linked issues was productive, but my Python environment was still broken. I also wasn't sure why my previous pyenv-installed versions of Python could import ctypes just fine.
But I noticed that many of the suggested fixes worked because they took homebrew out of the equation. What I really wanted was to use the build tooling Homebrew already had installed. And so I thought "Why am I passing Homebrew paths in for CFLAGS and LDFLAGS but still letting pyenv install use my system gcc?". So I tried:
CC="$(brew --prefix gcc)/bin/gcc-11" \
CFLAGS="$(pkg-config --cflags libffi)" \
LDFLAGS="$(pkg-config --libs libffi)" \
pyenv install --verbose 3.10.0And it worked! I commented about it in this issue because I suspected other folks with a similar problem might benefit from the same solution. Then I went back to whatever the heck I was trying to do with Python in the first place :).
The "solution" got my Python 3.10 environments working perfectly, but it also felt unsatisfying. I had some hunches about why it worked, but I wasn't really sure. And then I saw...
- A few other folks hit the same issue, and note that my suggestion worked for them
- A really great post from Julia Evans debugging a very similar issue
Which made me think, "This is a great problem to explore during the last week of December".
I had been thinking that since adding a custom CC effectively fixed my Python build, the CFLAGS and LDFLAGS bits might be unnecessary. So I tried without them:
CC="$(brew --prefix gcc)/bin/gcc-11" \
pyenv install --verbose 3.10.0And yes, that still worked. But why?
The next step was to drop down a level and use python-build to set up side-by-side build directories: one working, one broken. To get a broken one, I used:
python-build --keep --verbose 3.10.0 ./py310And for a working one:
CC="$(brew --prefix gcc)/bin/gcc-11" \
python-build --keep --verbose 3.10.0 ./py310-2From the broken directory, I could run make and see the ctypes warning. I could also see the object file renamed to ..._failed.so:
❯ fd "ctypes.*so"
build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu_failed.so
build/lib.linux-x86_64-3.10/_ctypes_test.cpython-310-x86_64-linux-gnu.soCompared to the working directory which showed the expected file names:
❯ fd "ctypes.*so"
build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.10/_ctypes_test.cpython-310-x86_64-linux-gnu.soThe earlier error messages mentioned an error finding libffi, and ldd helped highlight and reproduce that. From my broken build, I see "not found":
> ldd build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu_failed.so
linux-vdso.so.1 (0x00007ffd97fbf000)
libffi.so.8 => not found
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f478258f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f478256c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f478237a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f47825d8000)While the working file references homebrew's libffi:
> ldd build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffcd35fb000)
libffi.so.8 => /home/linuxbrew/.linuxbrew/lib/libffi.so.8 (0x00007f306da2d000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f306da08000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f306d9e5000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f306d7f3000)
/lib64/ld-linux-x86-64.so.2 (0x00007f306dc5f000)I was providing custom CFLAGS and LDFLAGS options that I thought would avoid this issue, but obviously that didn't do it. What was that custom CC doing that the CFLAGS/LDFLAGS change didn't do?
I looked in the python build log to get the gcc command that built _ctypes.cpython-310-x86_64-linux-gnu.so. It looked something like this (noise alert):
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/home/linuxbrew/.linuxbrew/opt/zlib -I/home/linuxbrew/.linuxbrew/opt/zlib -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I/home/linuxbrew/.linuxbrew/Cellar/libffi/3.4.2/include -I./Include -I. -I/home/linuxbrew/.linuxbrew/opt/readline/include -I/home/aj/code/scratch/pyenv-builds/./py310/include -I/home/linuxbrew/.linuxbrew/include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/tmp/python-build.20211231172024.425603/Python-3.10.0/Include -I/tmp/python-build.20211231172024.425603/Python-3.10.0 -c /tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/stgdict.c -o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/stgdict.o -DPy_BUILD_CORE_MODULE -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1
gcc -pthread -shared -L/home/linuxbrew/.linuxbrew/opt/readline/lib -L/home/aj/code/scratch/pyenv-builds/./py310/lib -L/home/linuxbrew/.linuxbrew/lib build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/_ctypes.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/callbacks.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/callproc.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/cfield.o build/temp.linux-x86_64-3.10/tmp/python-build.20211231172024.425603/Python-3.10.0/Modules/_ctypes/stgdict.o -L/home/linuxbrew/.linuxbrew/opt/readline/lib -L/home/aj/code/scratch/pyenv-builds/./py310/lib -L/home/linuxbrew/.linuxbrew/lib -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -lffi -ldl -o build/lib.linux-x86_64-3.10/_ctypes.cpython-310-x86_64-linux-gnu.soThere's a lot going on there, but as a test I ran the same command with all the same command line options, but using Homebrew's gcc-11 instead of my system gcc. And... that still worked. That was another helpful input.
Once I found that I could fail or succeed to build _ctypes.cpython-310-x86_64-linux-gnu.so by varying only the gcc version, I ran both versions of gcc with the --verbose option to look for more clues. And I found my answer in the form of gcc specs. The verbose logs from my system gcc included this bit:
Using built-in specs.While the Homebrew gcc log included:
Reading specs from /home/linuxbrew/.linuxbrew/Cellar/gcc/11.2.0_3/lib/gcc/11/gcc/x86_64-pc-linux-gnu/11/specsAnd peeking into /home/linuxbrew/.linuxbrew/Cellar/gcc/11.2.0_3/lib/gcc/11/gcc/x86_64-pc-linux-gnu/11/specs revealed some custom options:
*cpp_unique_options:
+ -isysroot /home/linuxbrew/.linuxbrew/nonexistent -idirafter /home/linuxbrew/.linuxbrew/include -idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include
*link_libgcc:
+ -L/home/linuxbrew/.linuxbrew/lib/gcc/11 -L/home/linuxbrew/.linuxbrew/lib
*link:
+ --dynamic-linker /home/linuxbrew/.linuxbrew/lib/ld.so -rpath /home/linuxbrew/.linuxbrew/lib/gcc/11
*homebrew_rpath:
-rpath /home/linuxbrew/.linuxbrew/libI've never messed around with gcc spec files to be honest. But it sure looks like that's what was causing my Homebrew gcc to work where my system gcc didn't.
My condensed takeaway after all of this exploration and troubleshooting is:
I'm trying to install Python using Homebrew-managed build dependencies. Those tools know where their pieces live and how they should work together. I should get out of their way and let them work. By pointing at Homebrew's gcc with pyenv install, I'm letting pyenv operate in the world it knows. The world where it was born.
There are some things I'm still not sure about. If you're reading this, you are a lovely human and a patient angel. And maybe you can help shed additional light on my busted system? Notably...
- My Homebrew-managed pyenv on this system previously installed Python versions from 3.6 through 3.9 without any issues and without requiring any CC-fiddling. Did something change with Homebrew or pyenv, or is this possibly something specific to Python 3.10?
- The options in the gcc spec look similar to the CFLAGS and LDFLAGS I was trying... is there some other magic combination of flags that would have allowed my system gcc to find everything it needed?
I may come back to these questions myself in a later troubleshooting phase. But I suspect I'm missing something obvious that someone else would catch at a glance!
17