Gem Install Fails On Building Native Extensions

Context

  • I was attempting to install the cucumber gem
  • This was the first gem I tried to install that required native extensions since I upgraded to Snow Leopard.
  • I did an in-place upgrade of Snow Leopard (not a fresh install) about two (2) weeks ago.

Symptoms

Primary: Gem Install failed

jonamac:rspec$ gem install cucumber
  Building native extensions.  This could take a while...
  ERROR:  Error installing cucumber:
      ERROR: Failed to build gem native extension.

  /Users/john/.rvm/rubies/ruby-1.9.2-p0/bin/ruby extconf.rb
  checking for ruby/re.h... *** extconf.rb failed ***
  Could not create Makefile due to some reason, probably lack of
  necessary libraries and/or headers.  Check the mkmf.log file for more
  details.  You may need configuration options.

  Provided configuration options:
      --with-opt-dir
      --without-opt-dir
      --with-opt-include
      --without-opt-include=${opt-dir}/include
      --with-opt-lib
      --without-opt-lib=${opt-dir}/lib
      --with-make-prog
      --without-make-prog
      --srcdir=.
      --curdir
      --ruby=/Users/john/.rvm/rubies/ruby-1.9.2-p0/bin/ruby
  /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:368:in `try_do': The complier failed to generate an executable file. (RuntimeError)
  You have to install development tools first.
      from /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:452:in `try_cpp'
      from /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:834:in `block in have_header'
      from /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:693:in `block in checking_for'
      from /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:280:in `block (2 levels) in postpone'
      from /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:254:in `open'
      from /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:280:in `block in postpone'
      from /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:254:in `open'
      from /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:276:in `postpone'
      from /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:692:in `checking_for'
      from /Users/john/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/mkmf.rb:833:in `have_header'
      from extconf.rb:17:in `<main>'

  Gem files will remain installed in /Users/john/.rvm/gems/ruby-1.9.2-p0@learn-rspec/gems/json-1.4.6 for inspection.
  Results logged to /Users/john/.rvm/gems/ruby-1.9.2-p0@learn-rspec/gems/json-1.4.6/ext/json/ext/generator/gem_make.out

  jonamac:generator$ cat /Users/john/.rvm/gems/ruby-1.9.2-p0@learn-rspec/gems/json-1.4.6/ext/json/ext/generator/mkmf.log
  "gcc -o conftest -I/Users/john/.rvm/rubies/ruby-1.9.2-p0/include/ruby-1.9.1/i386-darwin9.8.0 -I/Users/john/.rvm/rubies/ruby-1.9.2-p0/include/ruby-1.9.1/ruby/backward -I/Users/john/.rvm/rubies/ruby-1.9.2-p0/include/ruby-1.9.1 -I. -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE    -O3 -ggdb -Wextra -Wno-unused-parameter -Wno-parentheses -Wpointer-arith -Wwrite-strings -Wno-missing-field-initializers -Wshorten-64-to-32 -Wno-long-long  -fno-common -pipe -O3 -Wall conftest.c  -L. -L/Users/john/.rvm/rubies/ruby-1.9.2-p0/lib -L. -L/usr/local/lib     -lruby.1.9.1-static  -lpthread -ldl -lobjc "
  In file included from /Users/john/.rvm/rubies/ruby-1.9.2-p0/include/ruby-1.9.1/ruby.h:32,
                   from conftest.c:1:
  /Users/john/.rvm/rubies/ruby-1.9.2-p0/include/ruby-1.9.1/ruby/ruby.h:108: error: size of array ‘ruby_check_sizeof_long’ is negative
  /Users/john/.rvm/rubies/ruby-1.9.2-p0/include/ruby-1.9.1/ruby/ruby.h:112: error: size of array ‘ruby_check_sizeof_voidp’ is negative
  In file included from /Users/john/.rvm/rubies/ruby-1.9.2-p0/include/ruby-1.9.1/ruby/intern.h:29,
                   from /Users/john/.rvm/rubies/ruby-1.9.2-p0/include/ruby-1.9.1/ruby/ruby.h:1327,
                   from /Users/john/.rvm/rubies/ruby-1.9.2-p0/include/ruby-1.9.1/ruby.h:32,
                   from conftest.c:1:
  /Users/john/.rvm/rubies/ruby-1.9.2-p0/include/ruby-1.9.1/ruby/st.h:69: error: size of array ‘st_check_for_sizeof_st_index_t’ is negative
  checked program was:
  /* begin */
  1: #include "ruby.h"
  2: 
  3: int main() {return 0;}
  /* end */

Ancillary: MacPorts stopped working

jonamac:generator$ sudo port update
dlopen(/opt/local/share/macports/Tcl/macports1.0/MacPorts.dylib, 10): no suitable image found.  Did find:
    /opt/local/share/macports/Tcl/macports1.0/MacPorts.dylib: mach-o, but wrong architecture
    while executing
"load /opt/local/share/macports/Tcl/macports1.0/MacPorts.dylib"
    ("package ifneeded macports 1.0" script)
    invoked from within
"package require macports"
    (file "/opt/local/bin/port" line 39)
jonamac:generator$ sudo port self update
dlopen(/opt/local/share/macports/Tcl/macports1.0/MacPorts.dylib, 10): no suitable image found.  Did find:
    /opt/local/share/macports/Tcl/macports1.0/MacPorts.dylib: mach-o, but wrong architecture
    while executing
"load /opt/local/share/macports/Tcl/macports1.0/MacPorts.dylib"
    ("package ifneeded macports 1.0" script)
    invoked from within
"package require macports"
    (file "/opt/local/bin/port" line 39)

Observations

Compilation failure was due to architecture mismatch

from the gem install failure, an important signal is this:

ruby.h:108: error: size of array ‘ruby_check_sizeof_long’ is negative

that line reads:

typedef char ruby_check_sizeof_long[SIZEOF_LONG == sizeof(long) ? 1 : -1];

which looks to me like a sanity check that the size of a long integer as known by the installed library being linked matches the size of a long integer as reported by the compiler. In this case, they do not. So, the installed library and the compiler are not from matching architectures (one is 32-bit and the other is 64-bit).

Instructive error message says I need development tools

The line:

  You have to install development tools first.

was confusing because I JUST installed the latest Xcode + iOS SDK. I'm new to MacOS Development and checked to see if perhaps the 10.6 SDK wasn't installed with it, but it was

jonamac:SDKs$ pwd
/Developer/SDKs
jonamac:SDKs$ l
total 0
drwxr-xr-x   4 root  wheel  136 Dec 18 16:04 ./
drwxrwxr-x@ 17 root  admin  578 Dec 21 07:22 ../
drwxr-xr-x   7 root  wheel  238 Jun 24  2010 MacOSX10.5.sdk/
drwxr-xr-x   7 root  wheel  238 Oct  4 17:40 MacOSX10.6.sdk/

Turns out, then that this error message is not so instructive. It's misleading. What it REALLY should say is that an attempt to verify the development tools failed. gcc was on the path and the latest development tools WAS installed. It's just that during the linking test (described above), the link failed.

Researched: Snow Leopard runs binaries in 64-bit mode by default (new from Leopard)

I read in all my searching that Snow Leopard switched to running binaries in 64-bit mode, if at all possible. This is different from Leopard.

Solutions

Reinstalled Ruby

I had installed my MRI's via rvm BEFORE the Snow Leopard upgrade. Eventually (via process of elimination and a pinch of desperation), I uninstalled all my "rubies" and re-installed.

jonamac:~$ rvm install 1.9.2
/Users/john/.rvm/rubies/ruby-1.9.2-p0, this may take a while depending on your cpu(s)...

ruby-1.9.2-p0 - #fetching 
ruby-1.9.2-p0 - #extracting ruby-1.9.2-p0 to /Users/john/.rvm/src/ruby-1.9.2-p0
ruby-1.9.2-p0 - #extracted to /Users/john/.rvm/src/ruby-1.9.2-p0
ruby-1.9.2-p0 - #configuring 
ruby-1.9.2-p0 - #compiling 
ruby-1.9.2-p0 - #installing 
ruby-1.9.2-p0 - updating #rubygems for /Users/john/.rvm/gems/ruby-1.9.2-p0@global
ruby-1.9.2-p0 - updating #rubygems for /Users/john/.rvm/gems/ruby-1.9.2-p0
ruby-1.9.2-p0 - adjusting #shebangs for (gem).
ruby-1.9.2-p0 - #importing default gemsets (/Users/john/.rvm/gemsets/)
Install of ruby-1.9.2-p0 - #complete 
jonamac:~$ rvm use 1.9.2
Using /Users/john/.rvm/gems/ruby-1.9.2-p0
jonamac:~$ gem install cucumber
Building native extensions.  This could take a while...
Building native extensions.  This could take a while...

(::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::)

Thank you for installing cucumber-0.10.0.
Please be sure to read http://wiki.github.com/aslakhellesoy/cucumber/upgrading
for important information about this release. Happy cuking!

(::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::)

Successfully installed json-1.4.6
Successfully installed gherkin-2.3.3
Successfully installed term-ansicolor-1.0.5
Successfully installed builder-3.0.0
Successfully installed diff-lcs-1.1.2
Successfully installed cucumber-0.10.0
6 gems installed
Installing ri documentation for json-1.4.6...
Installing ri documentation for gherkin-2.3.3...
Installing ri documentation for term-ansicolor-1.0.5...
Installing ri documentation for builder-3.0.0...
Installing ri documentation for diff-lcs-1.1.2...
Installing ri documentation for cucumber-0.10.0...
Installing RDoc documentation for json-1.4.6...
Installing RDoc documentation for gherkin-2.3.3...
Installing RDoc documentation for term-ansicolor-1.0.5...
Installing RDoc documentation for builder-3.0.0...
Installing RDoc documentation for diff-lcs-1.1.2...
Installing RDoc documentation for cucumber-0.10.0...
jonamac:~$

Reinstalled MacPorts

(from scratch!)

Yeah, I had to completely uninstall all my MacPorts (most/many are compiled or linked on installed).

1. Install the Snow Leopard version of MacPorts
2. Get a listing of all ports I had installed (I actually didn't do this, but I should next time)

$ sudo port list installed >all_mac_ports.txt

3. Uninstall ALL ports
$ sudo port uninstall -c -f installed

4. Scan the list of installed ports and see which top-level ports I wanted to keep; install each, one at a time.

Lessons Learned

Lesson One: When doing a major upgrade, scan for migration instructions

I performed the Snow Leopard upgrade in a pinch — I was trying to install Oracle and needed to do the upgrade fast. I didn't have the process in my head that I should comb the message boards for hints of upgrade pains.

Before performing any substantial upgrade, I will put together a small research effort to identify possible pain points and mitigation.

Lesson Two: When doing a major upgrade, you avoid a class of problems by reinstalling

I used to think that doing a fresh install was just about having a pristine environment (setting aside the cruft from the years on the old OS). Turns out, there's additional reasons to do a fresh install.

  • This may be the last time I do this, but from Leopard to Snow Leopard, all binaries run in 64-bit mode, by default. So, this was effectively a change in architecture. If you install fresh binaries, they will be guaranteed to be of the right type.
  • Doing in-place upgrades can leave some software semi-broken and you're inadvertently setting land-mines for yourself.
    • in this case, MacPorts were totally broken… but the binaries (e.g. git, curl, etc) were all working. They were happily running in 32-bit mode.

Lesson Three: Even when using interpreted languages, there's almost always native extensions

Whether it's Java or Ruby (or any other interpreted language), there's almost always a need for native extensions. In fact, one way to think about it is that the interpreter is one big native extension. In this way, don't get lulled into a sense of "platform agnosticism" when managing the environment.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License