Installing Python 2.6, virtualenv, and virtualenvwrapper on Dreamhost

Late last year, I set out to get some python apps running on my dreamhost account, previously I maintained this stuff on an ec2 ubuntu instance, so I could do whatever I wanted there. But ec2 is expensive and dreamhost is cheap, so I made the move.

Update May 15, 2010: Thanks to Tommaso Lanza, who, has written a shell script to automate this installation, which now includes mercurial, he has also left a few notes in the comments. Use at your own risk!.

Python

I thought all I would need to do is compile Python and get virtualenv running, but there was a little more to it, because as soon as I compiled Python I received the following warning:

    Failed to find the necessary bits to build these modules:
    _bsddb             _tkinter           bsddb185        
    bz2                readline           sunaudiodev     
    To find the necessary bits, look in setup.py in detect_modules() for the 
    module's name.

This doesn’t mean it didn’t compile, but it means those modules were not compiled in. I really wanted readline, and thought it would be a good idea to have bz2. I didn’t expect I’d need _tkinter, but who knows and I wasn’t sure when and where I might need berkeley db (_bsddb, note: you’ll need a 4.5 version of Berkeley DB if you want support). So I set out to compile these modules in. I wasn’t worried about needing sunaudiodev or bsddb185, they’re both very old.

Without documenting my errors along the way, here is how I installed Python 2.6.4 on dreamhost:

First give yourself a place to install the necessary packages, and then a place to compile them. You can name them whatever you want, I used $HOME/opt to install and $HOME/tmp to compile in. Dreamhost recommends $HOME/run to install and $HOME/soft to compile in. Do whatever your comfortable with.

    $ cd ~
    $ mkdir opt tmp

Grab everything you’ll need and put that in tmp:

    $ cd tmp
    $ wget http://www.bzip.org/1.0.5/bzip2-1.0.5.tar.gz
    $ tar -xzf bzip2-1.0.5.tar.gz
    $ wget ftp://ftp.gnu.org/gnu/readline/readline-5.2.tar.gz
    $ tar -xzf readline-5.2.tar.gz
    $ wget http://prdownloads.sourceforge.net/tcl/tcl8.5.8-src.tar.gz
    $ tar -xzf tcl8.5.8-src.tar.gz
    $ wget http://prdownloads.sourceforge.net/tcl/tk8.5.8-src.tar.gz
    $ tar -xzf tk8.5.8-src.tar.gz
    $ wget http://python.org/ftp/python/2.6.4/Python-2.6.4.tgz
    $ tar -xzf Python-2.6.4.tgz
    $ wget http://download.oracle.com/berkeley-db/db-4.5.20.tar.gz
    $ tar -xzf db-4.5.20.tar.gz

Setup your environment:

    $ export LD_LIBRARY_PATH=$HOME/opt/local/lib:$HOME/opt/local/BerkeleyDB.4.5/lib
    $ export LD_RUN_PATH=$LD_LIBRARY_PATH
    $ export LDFLAGS="-L$HOME/opt/local/lib -L$HOME/opt/local/BerkeleyDB.4.5/lib"    
    $ export CPPFLAGS="-I$HOME/opt/local/include -I$HOME/opt/local/BerkeleyDB.4.5/include"
    $ export CXXFLAGS=$CPPFLAGS
    $ export CFLAGS=$CPPFLAGS

Setting all these environment variables may not be necessary it’s what I had at the end of compiling, Python looks at LDFLAGS and CPPFLAGS, the others are pretty standard compile env variables.

Update: Thanks to Christian, who, in the comments, provided additional instructions to grab and install SSL support, if you think you’ll need SSL, here are the steps:

    $ wget http://www.openssl.org/source/openssl-0.9.8m.tar.gz
    $ tar -xzf openssl-0.9.8m.tar.gz 
    $ cd openssl-0.9.8m 
    $ ./config --prefix=$HOME/opt/local --openssldir=$HOME/opt/local/openssl shared 
    $ make 
    $ make install
    $ cd ..

Compile and Install

    $ cd readline-5.2
    $ ./configure --prefix=$HOME/opt/local
    $ make
    $ make install
    $ cd ..
 
    $ cd tcl8.5.8/unix
    $ ./configure --prefix=$HOME/opt/local
    $ make
    $ make install
    $ cd ../..
 
    $ cd tk8.5.8/unix
    $ ./configure --prefix=$HOME/opt/local
    $ make
    $ make install
    $ cd ../..
 
    $ cd db-4.5.20/build_unix
    $ ../dist/configure \
    > --prefix=$HOME/opt/local/BerkeleyDB.4.5 \
    > --enable-tcl \
    > --with-tcl=$HOME/opt/local/lib
    $ make
    $ make install
    $ cd ../..
 
    $ cd bzip2-1.0.5
    $ make
    $ make install PREFIX=$HOME/opt/local
    $ cd ..
 
    $ cd Python-2.6.4
    $ ./configure --prefix=$HOME/opt/Python-2.6.4
    $ make
    $ make install

When you ran make on Python-2.6.4, the only warning should now be:

    Failed to find the necessary bits to build these modules:
    bsddb185           sunaudiodev                        
    To find the necessary bits, look in setup.py in detect_modules() for the 
    module's name.

As I mentioned above, I don’t think I’ll ever need these 2 modules, if i ever do, I’ll worry about it then.

Next I updated my PATH to include the new binaries

    $ cd ~
    $ vi ~/.bashrc
 
    export PATH="$HOME/opt/local/bin:$PATH"
 
    $ source ~/.bashrc

virtualenv and virtualenvwrapper

And finally I installed virtualenv and virtualenvwrapper along with some things in my .bashrc to make life easier.

    $ cd ~/tmp
    $ wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.4.3.tar.gz
    $ tar -xzf virtualenv-1.4.3.tar.gz
    $ ~/opt/Python-2.6.4/bin/python2.6 virtualenv-1.4.3/virtualenv.py $HOME/opt/local
    New python executable in /home/dawatts/opt/local/bin/python2.6
    Also creating executable in /home/dawatts/opt/local/bin/python
    Installing setuptools............done.
    $ easy_install virtualenv
    $ cd ..
 
    $ wget http://www.doughellmann.com/downloads/virtualenvwrapper-1.23.tar.gz
    $ tar -xzf virtualenvwrapper-1.23.tar.gz
    $ mkdir $HOME/bin
    $ cp virtualenvwrapper-1.23/virtualenvwrapper_bashrc ~/bin/
 
    $ cd ~
    $ mdkir $HOME/.virtualenvs
    $ vi ~/.bashrc
 
    export WORKON_HOME=$HOME/.virtualenvs
    source $HOME/bin/virtualenvwrapper_bashrc
 
    $ source ~/.bashrc
    $ workon
    * 
    $ mkvirtualenv testenv
    Using real prefix '/home/dawatts/opt/Python-2.6.4'
    New python executable in testenv/bin/python2.6
    Also creating executable in testenv/bin/python
    Installing setuptools............done.
    (testenv)$ workon
    testenv
    (testenv)$ deactivate
    $ rmvirtualenv testenv

And there you have it, a Python 2.6.4/virtualenv environment setup on dreamhost to do whatever you want.

31 comments

  1. [...] python 2.3.5 is dated, and I’ve written previously about setting up python 2.6.4 alongside virtualenv and virtualenvwrapper on dreamhost. If you’ve followed those instructions, you can now setup a virtualenv running python 2.6 for [...]

  2. @CadetZemm says:

    Wow, you just saved me so much time. This worked flawlessly for me. Thank you so much!

  3. @b3ng0 says:

    Thanks for taking the time to post this. Worked great for me. :)

  4. Landon Noss says:

    Hot damn, worked like a charm. Thanks man!

  5. Christian says:

    thanks for your great instructions!

    I had to install openssl for ssl support in python 2.6.4 (it was still complaining about the bz2 module, but I don't care too much at the moment):

    $ wget http://www.openssl.org/source/openssl-0.9.8m.tar…. $ tar xzf openssl-0.9.8m.tar.gz $ cd openssl-0.9.8m $ ./config –prefix=$HOME/opt/local –openssldir=$HOME/opt/local/openssl shared $ make $ make install

  6. ydaniv says:

    I get more warnings which later on become errors when trying to install virtualencv:

    Failed to find the necessary bits to build these modules: _tkinter bsddb185 dl imageop sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module’s name.

    Failed to build these modules: _hashlib _ssl

    these seem to appear again and again on the web but still unanswered i followed all your steps but still…

    tried again with python 2.6.3, still no good.

    any idea?

    BR, ~Y

    • Andrew Watts says:

      Sorry, not exactly sure, but perhaps your environment is misconfigured, I'd start over once again (log out, then log back in), follow these instructions exactly and make sure your environment looks like the above, which may require unsetting some environment variables, if you've set anything in your .bashrc, .profile, etc…

      I've heard publicly (as noted in these comments) and privately these instructions do work.

  7. Saq says:

    Hi, thank you for taking the time to post this. Worked like a charm for me. I did run into a related issue where the sqlite version remains at the ancient 3.2.1 which doesn't always work so well with Django. Any ideas on how to update sqlite to the latest version as well?

    I tried building sqlite 3.6.23 from source before building Python but still end up with sqlite 3.2.1 in the end if I run: python -c "import sqlite3; print sqlite3.sqlite_version"

    • andrew watts says:

      You need to compile and install sqlite3, then make sure your environment variables are setup as I defined above prior to compiling python. eg:

      $ cd ~/tmp $ wget http://www.sqlite.org/sqlite-amalgamation-3.6.23…. $ tar xzf sqlite-amalgamation-3.6.23.tar.gz $ cd sqlite-3.6.23 $ ./configure –prefix=$HOME/opt/local $ make $ make install $ cd ..

      Make sure your PATH is pointing to the newly installed python and then verify.

      $ ./python Python 2.6.4 (r264:75706, Mar 14 2010, 09:53:15) [GCC 3.3.5 (Debian 1:3.3.5-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sqlite3 >>> sqlite3.sqlite_version '3.6.23' >>>

      HTH

  8. tmslnz says:

    Hi, thanks for one of the clearest explanations I found so far. However, I am still getting the following:

    $ Failed to find the necessary bits to build these modules: $ _bsddb bsddb185 dl
    $ imageop sunaudiodev
    $ To find the necessary bits, look in setup.py in detect_modules() for the module's name. $ $ $ Failed to build these modules: $ _hashlib bz2

    I am on a shared server on Dreamhost. I followed everything step by step and tried the following: – do everything as you outlined: FAIL – clear my .bashrc from any pre-existing stuff: FAIL – redo everything under a newly created user: FAIL

    I keep getting those messages. Any clue? How does your .bashrc file look like before beginning?

    Best, Tommaso

  9. Andrew Watts says:

    I don't think I'm going to be much help, but to answer your questions, there is nothing special in my .bashrc. Here are my .bashrc files before and after.

    <script src="http://gist.github.com/387353.js?file=before.bashrc"></script&gt;

    <script src="http://gist.github.com/387353.js?file=after.bashrc"></script&gt;

    I would guess your paths are not correct, or perhaps getting messed up along the way. Also, keep in mind you only need to resolve these issues if you plan on using them in python code.

    It's not much, but hope it helps. Best of Luck.

  10. @afriza_na says:

    For bsddb version, you can check which latest version is suitable for which version of your Python athttp://www.jcea.es/programacion/pybsddb.htm make sure the first 2 numbers match (e.g. Python 2.6.X with bsddb-4.7.X)

    For bz2, I needed to compile using: $ make -f Makefile-libbz2_so $ make $ make install PREFIX=/the/new $ cp libbz2.so.1.0.4 /the/new/directory/lib $ cd /the/new/directory/lib $ rm -f libbz2.so.1.0 $ ln -s libbz2.so.1.0.4 libbz2.so.1.0

  11. tmslnz says:

    So, thanks to all those who commented and replied. I finally managed to get Python 2.6.5 to compile without complaining. I put together a shell script which you can try out. The script installs Mercurial too (which is why I did all this in the first place). http://bitbucket.org/tmslnz/python-dreamhost-batc… Read it and try at your own risk (it backs up the ~/opt folder for safety). I tried it on my account a seems to work smooth. NOTE: I only ran it on my Dreamhost shared server. If you have a private server I guess all of this doesn't really apply much. If robots scare you, just create a new user and test it there.

  12. srid says:

    Have you tried ActivePython (I assume DreamHost gives you a Linux shell)?

    Install ActivePython 2.6 &lt ;http://www.activestate.com/activepython/downloads&gt; to $HOME/opt/ActivePython-2.6

    $ echo 'export PATH=$HOME/opt/ActivePython-2.6/bin:$PATH' >> ~/.bashrc $ pypm install virtualenvwrapper readline ipython $ echo 'export WORKON_HOME=$HOME/.virtualenvs' >> ~/.bashrc $ echo '. ~/.local/bin/virtualenvwrapper.sh' >> ~/.bashrc

    You don't have compile any software that way.

  13. Gezim Hoxha says:

    I'm soooooo close.

    When I try to setup up ~/opt/local as a virtual env it fails as follows:

    ~/opt/Python-2.6.4/bin/python2.6 soft/virtualenv-1.4.3/virtualenv.py $HOME/opt/local Traceback (most recent call last): File "soft/virtualenv-1.4.3/virtualenv.py", line 1475, in <module> main() File "soft/virtualenv-1.4.3/virtualenv.py", line 528, in main use_distribute=options.use_distribute) File "soft/virtualenv-1.4.3/virtualenv.py", line 611, in create_environment site_packages=site_packages, clear=clear)) File "soft/virtualenv-1.4.3/virtualenv.py", line 674, in install_python fix_lib64(lib_dir) File "soft/virtualenv-1.4.3/virtualenv.py", line 890, in fix_lib64 if [p for p in distutils.sysconfig.get_config_vars().values() File "/usr/lib/python2.5/distutils/sysconfig.py", line 499, in get_config_vars func() File "/usr/lib/python2.5/distutils/sysconfig.py", line 351, in _init_posix filename = get_makefile_filename() File "/usr/lib/python2.5/distutils/sysconfig.py", line 210, in get_makefile_filename return os.path.join(lib_dir, "config" + (sys.pydebug and "_d" or ""), "Makefile") AttributeError: 'module' object has no attribute 'pydebug'

    Any ideas would be greatly appreciated.

    • Gezim Hoxha says:

      I think my issue was the PATH and PYTHONPATHs I had set in a previous attempt.

      Thanks so much everyone!

  14. [...] came across this excellent and very useful installation guide on installing python 2.6, virtualenv, and virtualenwrapper. var a2a_config = a2a_config || {}; [...]

  15. [...] upgrading and it turned out there were many changes that needed to be made. I came across this very helpful guide on intalling python 2.6, virtualenv, and virtualenvwrapper. After a few attempts and tweaks it worked and beautifully too! Not only did I have python 2.6 [...]

  16. [...] be done with it. If you want all the modules that you can get, you are in for some extra building. This post does a good job of explaining the installation of Python 2.6; mine is based on it. Let’s put [...]

  17. [...] be done with it. If you want all the modules that you can get, you are in for some extra building. This post does a good job of explaining the installation of Python 2.6; mine is based on it. Let’s put the [...]

  18. [...] Installing Python 2.6, virtualenv, and virtualenvwrapper on Dreamhost Installing SSL on [...]

  19. Carl says:

    Thank you.

  20. [...] be done with it. If you want all the modules that you can get, you are in for some extra building. This post does a good job of explaining the installation of Python 2.6; mine is based on it. Let’s put the [...]

  21. panjianom says:

    how to uninstall this script from dreamhost? just delete the folder or anything else?

  22. Panjianom says:

    After install the script from Tomma Solanza, I got following error when run bash:

    Traceback (most recent call last): File "/home/user/opt/local/lib/python2.7/site.py", line 67, in <module> import os File "/home/user/opt/local/lib/python2.7/os.py", line 398, in <module> import UserDict File "/home/user/opt/local/lib/python2.7/UserDict.py", line 83, in <module> import _abcoll File "/home/user/opt/local/lib/python2.7/_abcoll.py", line 11, in <module> from abc import ABCMeta, abstractmethod File "/home/user/opt/local/lib/python2.7/abc.py", line 8, in <module> from _weakrefset import WeakSet ImportError: No module named _weakrefset

    How to fix this? And how remove that script from dreamhost? just delete the folder or anything else?

  23. Adan says:

    It’s hard to find your page in google. I found it on 16 spot, you should build quality backlinks , it will help you to increase traffic. I know how to help you, just type in google – k2 seo tricks

Post a comment

Copyright © andrew.io/weblog
andrew watts' weblog

Built on Notes Blog Core
Powered by WordPress