I recently had an absolute bear of a time trying to install the postgis extension for Postgresql 9.6.3 which I’m running because there is an error on upgrade from 9.6 to 10 around the hstore extension which seems impossible to get over unless I blow away all the data related to hstore or try this very tricky migration that basically massages hstore data.

First I tried to simply install postgis with homebrew. Silly me.
That upgraded my 9.6.3 installation to Postgress 11 and installed the extension. It took me 2 hours to pick through the debris, (dyld errors, had to ln -s the new readline library to the old one that postgres expecting after I ran
brew switch postgres 9.6.3

After that, I was able to install postgis with brew via two commands

brew pin postgres
and
brew install postgis --ignore-dependencies

That looked like it worked, but everytime I then tried to install the extension I’d get and error around this phrase (I lost the stacktrace in my glee of finding a fix)

Symbol not found: _AllocSetContextCreateExtended

I turns out what I had to do was uninstall postgis from brew

brew remove postgis

then go to the official postgis source page, pick a stable version (I had issues with 2.4.7. compile error on Mac OS) but 2.5.3 worked just fine.

I downloaded the tar file, uncompressed it then ran
./configure
make
make install

And VOILA! I was able to create the postgis extension for the db I was working on. phew!