Tag Archives: server

RhodeCode As FastCGI Program With Lighttpd In Debian

So one day I want to have a github-esque push box, somewhere I can push to collect my projects in one place that is NOT GitHub or BitBucket, or whatever. As always, my internet connection sucks and so I prefer to mirror interesting projects with incremental updates through daily/weekly/whenever-I-feels-like pull or fetch instead downloading each release.

My old setup was two separate hgweb and gitweb running as fastcgi applications connected to a lighttpd server through sockets each serving as mercurial and git web gui. In status quo, everything works and I like it.

Why Change A Perfectly Working System?

Because I can. Besides I want to get my hands on this RhodeCode thing I have put on hold for quite some time. It can handle both mercurial and git in one application, so that’s a good selling point that I would like to try.

RhodeCode Installation

My opinion on the overall installation, it was painless. First, I create a special user and group for this program:


    $ sudo adduser --disabled-login --no-create-home --uid 500 rhodecode
    

Next, I set /opt/rhodecode/ so everything about this RhodeCode stay inside that directory. Furthermore to make things easier on installation I chown that directory to my regular user & group, for now:


    $ sudo mkdir -p /opt/rhodecode
    $ sudo chown -R ariel:ariel /opt/rhodecode
    

Next, I create a new virtualenv instance:


    $ virtualenv --no-site-packages /opt/rhodecode/venv/2.7
    $ cd /opt/rhodecode/
    

This will initialize new virtualenv inside venv/2.7, the “2.7” part is because I use Debian Wheezy and its Python is at 2.7.3.

Okay now, I begin using the new virtualenv to download rhodecode and its dependencies:


    $ source venv/2.7/bin/activate
    (2.7)$ pip install rhodecode
    

Oh joy, I get to wait for something. So off to making tea then …

For some strange reason RhodeCode fail when downloading Mercurial (2.6.2). Now my pushbox is on Wheezy Stable, its Mercurial was at 2.2.2. So, while I’m at it anyway why not just use 2.6.2 directly in this virtualenv?

Mercurial Installation

I set to build a fresh Mercurial straight from upstream:


    (2.7)$ cd /tmp/
    (2.7)$ wget http://selenic.com/hg/archive/cceaf7af4c9e.tar.bz2
    (2.7)$ tar xf cceaf7af4c9e.tar.bz
    

The cceaf7af4c9e is Mercurial tag for version 2.6.2, the version that RhodeCode 1.7.1 wants. So I build that:


    (2.7)$ cd Mercurial-cceaf7af4c9e/
    (2.7)$ python setup.py build
    (2.7)$ python setup.py install
    

I’m still in my virtualenv environment as you can see in my prompt string, it says “(2.7)$“. So I know when I do an “install” the new Mercurial version will be installed to /opt/rhodecode/venv/2.7/lib/python/site-packages.

Not surprisingly Mercurial installed successfully. I have no further use for the Mercurial source code, so I remove them and get back to rhodecode directory:


    (2.7)$ cd /opt/rhodecode/
    (2.7)$ rm -rf /tmp/Mercurial-cceaf7af4c9e/
    

RhodeCode Installation Part 2

Mercurial should not be a problem anymore, continue with the installation:


    (2.7)$ pip install rhodecode
    

After waiting a while for downloading and compiling (this push-box is a re-purposed 2005-ish desktop, not exactly what you’ll call as “fast”) we are done with the downloads.

Now, according to RhodeCode documentation I should create a configuration file. I’m new, so I’ll just follow that:


    (2.7)$ paster make-config RhodeCode pushbox.ini
    (2.7)$ paster setup-rhodecode pushbox.ini
    

It then ask for “Do I want to delete database?”. Of course I say yes, I don’t have one yet!

Next, it prompts me for my repository location. I guess for my first try I should create a new directory, so I stop the job (Ctrl + Z) and do a quick mkdir:


    (2.7)$ sudo mkdir -p /srv/repos/
    (2.7)$ sudo chown ariel:ariel /srv/repos/
    (2.7)$ fg
    

Continuing the job, I put “/srv/repos/” here. Of course I don’t have anything in that directory yet, you think I’ll use my REAL repository location?

It then asks for admin user name, password, and email address, you know, the standard stuff.

Maiden Flight Of RhodeCode

Alright, let’s run this thing:


    (2.7)$ paster serve pushbox.ini 
    
RhodeCode first run

RhodeCode first run

Well, that was easy.. so I kill the RhodeCode program by pressing Ctrl + C.

What About My Needs?

Now that the “default configuration” RhodeCode is working, its time for adapting it to my requirements:

  1. First, I do not want to run another web server on port 5000. So it has to work with my current webserver: lighttpd.
  2. Web access, push and pull, must use SSL not standard http.
  3. I want it to start automatically at startup like any other fastcgi apps.

RhodeCode As A FastCGI Program

After a while browsing about Paste, I found out to make it run as fastcgi program is to specify/use flup (surprise! surprise!) on the server section. So I get flup on this virtualenv too:


    (2.7)$ pip install flup
    

To make RhodeCode use flup I put this on my /opt/rhodecode/pushbox.ini file:


    [server:main]

    ** more lines here, I commented as I don't use waitress, gunicorn, paste
       http server, etc. **

    ## BEGIN: USE FLUP FASTCGI FOR LIGHTTPD ##
    use = egg:PasteScript#flup_fcgi_thread
    socket = %(here)s/socket
    umask = 000
    ## END: USE FLUP FASTCGI FOR LIGHTTPD ##

    ** some more irrelevant lines **
    

The main idea is for flup to make RhodeCode running as threaded fastcgi program communicating to outside world using UNIX socket at /opt/rhodecode/socket, and that socket’s permission is 777.

On the web server side, I create new config file at /etc/lighttpd/conf-enabled/45-rhodecode.conf:


    # BEGIN /etc/lighttpd/conf-enabled/45-rhodecode.conf
    $SERVER["socket"] == ":443" {
        $HTTP["url"] =~ "^/repos" {
            fastcgi.server = ("/repos" => (("check-local" => "disable",
                                            "socket"      => "/opt/rhodecode/socket"))
                             )
        }
    }

    $HTTP["url"] =~ "^/repos" {
        $SERVER["socket"] != ":443" {
            $HTTP["host"] =~ "(.*)" {
                url.redirect += ( "^/(.*)" => "https://%1/$1" )
            }
        }

    }
    # EOF /etc/lighttpd/conf-enabled/45-rhodecode.conf
    

Using this configuration, lighttpd will:

To test the new configuration, I restart the lighttpd web server and try to run the RhodeCode program again:


    (2.7)$ sudo service lighttpd restart
    (2.7)$ paster serve pushbox.ini 
    

** Checks http://pushbox.home/repos/ on the browser **

This time it should work as before, using a cute little socket instead of running as http server on some port.

Debian-ish Setup

Now that basically the program is done, it’s time to make it run as daemon.

In my case I wrote my own /etc/init.d/rhodecode script. But I probably didn’t have to, because later I found out that the developer also have a nice init script for Debian:

https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/init.d/rhodecode-daemon2

But for completeness sake I will post my init script too. My script is pretty much the skeleton file modified to execute “paster serve” in daemon mode under the privilege of rhodecode user and group:


    #! /bin/sh
    ### BEGIN INIT INFO
    # Provides:          rhodecode
    # Required-Start:    $remote_fs $syslog
    # Required-Stop:     $remote_fs $syslog
    # Default-Start:     2 3 4 5
    # Default-Stop:      0 1 6
    # Short-Description: rhodecode repository application service
    ### END INIT INFO

    # Author: ariel 

    # Do NOT "set -e"

    # PATH should only include /usr/* if it runs after the mountnfs.sh script
    PATH=/sbin:/usr/sbin:/bin:/usr/bin
    DESC="RhodeCode Repositories Service"
    NAME=rhodecode
    SCRIPTNAME=/etc/init.d/$NAME
    RUNUSER=rhodecode
    RUNGROUP=rhodecode
    HOME=/opt/rhodecode
    PIDFILE=$HOME/paster.pid
    LOGFILE=$HOME/paster.log
    CONFIG=$HOME/pushbox.ini
    PYTHON_PATH=$HOME/venv/2.7/
    PASTER=$PYTHON_PATH/bin/paster
    PASTER_ARGS="serve --daemon --user=$RUNUSER --group=$RUNGROUP --pid-file=$PIDFILE --log-file=$LOGFILE $CONFIG"

    # Exit if the package is not installed
    [ -x "$PASTER" ] || exit 0
    [ -f "$CONFIG" ] || exit 0

    # Load the VERBOSE setting and other rcS variables
    . /lib/init/vars.sh

    # Define LSB log_* functions.
    # Depend on lsb-base (>= 3.2-14) to ensure that this file is present
    # and status_of_proc is working.
    . /lib/lsb/init-functions

    #
    # Function that starts the daemon/service
    #
    do_start()
    {
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        start-stop-daemon --start --quiet --chdir $HOME --chuid $RUNUSER:$RUNGROUP --pidfile $PIDFILE --exec $PASTER -- $PASTER_ARGS start > /dev/null 2>&1 || return 2
    }

    #
    # Function that stops the daemon/service
    #
    do_stop()
    {
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --start --quiet --chdir $HOME --chuid $RUNUSER:$RUNGROUP --pidfile $PIDFILE --exec $PASTER -- $PASTER_ARGS stop > /dev/null 2>&1 || return 1

        if [ -f $PIDFILE ]; then
            rm $PIDFILE
        fi

        return 0
    }

    #
    # Function that sends a SIGHUP to the daemon/service
    #
    do_reload() {
        #
        # If the daemon can reload its configuration without
        # restarting (for example, when it is sent a SIGHUP),
        # then implement that here.
        #
        start-stop-daemon --start --quiet --chdir $HOME --chuid $RUNUSER:$RUNGROUP --pidfile $PIDFILE --exec $PASTER -- $PASTER_ARGS --reload > /dev/null 2>&1 
        return 0
    }

    case "$1" in
      start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
            0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
            2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
      stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
            0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
            2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
      status)
        $PASTER $PASTER_ARGS status > /dev/null
        exit $?
        ;;
      reload)
        #
        # If do_reload() is not implemented then leave this commented out
        # and leave 'force-reload' as an alias for 'restart'.
        #
        log_daemon_msg "Reloading $DESC" "$NAME"
        do_reload
        log_end_msg $?
        ;;
      restart)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop
        case "$?" in
          0|1)
            do_start
            case "$?" in
                0) log_end_msg 0 ;;
                1) log_end_msg 1 ;; # Old process is still running
                *) log_end_msg 1 ;; # Failed to start
            esac
            ;;
          *)
            # Failed to stop
            log_end_msg 1
            ;;
        esac
        ;;
      *)
        echo "Usage: $SCRIPTNAME {start|stop|status|restart|reload}" >&2
        exit 3
        ;;
    esac

    :
    

Using the usual command to make it run automatically on startup:


    (2.7)$ sudo update-rc.d rhodecode defaults
    

Now that I have program set, init script and all, it’s time to clean up.

For starters, everything under /opt/rhodecode/ should be owned by user and group rhodecode:


    (2.7)$ sudo chown -R rhodecode:rhodecode /opt/rhodecode/
    

The same thing should be applied to /srv/repos/, but in my case I own it to user and group repo. Why? My current repositories that is stored elsewhere, also owned by this user/group, so I’m just being consistent here:


    (2.7)$ sudo chown -R repo:repo /srv/repos/
    

Because the rhodecode program executed under user rhodecode I also add user rhodecode to group repo, and make sure anyone in repo group can write to /srv/repos/:


    (2.7)$ sudo adduser rhodecode repo
    (2.7)$ sudo chmod -R g+w /srv/repos/
    

Some Gotchas Along The Way

  1. My first gotcha was about file permission of socket, it seems flup defaults to create socket that is not writable to others. And since my web server runs under user lighttpd (not www-data, rhodecode, or ariel) the web server returns “503 Service Unavailable”. So by adding “umask = 000” in my pushbox.ini flup will create a socket with 777 permission that is accessible to user running the program (user rhodecode) and the user running the web server (user lighttpd).
  2. Second gotcha came when pushing through rhodecode, the git client program says the server “hung up” and by adding GIT_CURL_VERBOSE=1 environment variable I see curl got “413 Requested Entity To Large” from the server.

    In my case my problem basically caused by /var/cache/lighttpd/ being owned by user www-data. You see, as I said earlier, in my setup my lighttpd program runs as user lighttpd and NOT as www-data which is the default in Debian. So if you’re having your push fail with “hung up” message, check the filesystem permission of the directory pointed by lighttpd as server.upload-dirs.

  3. The third gotcha is on the browser, my browser is Mozilla Firefox with No Script on by default on ALL domains. RhodeCode uses AJAX heavily, so if there is no repository showing on RhodeCode even though you know you already push something to it, check your No Script (temporary) whitelist.

Closing

There are some other things that need to be done such as using my real repositories location, setting proper filesystem permissions, and so on; but they are normal administrator daily work so I’ll leave that out from this post.

Some observations:

  • RhodeCode seems to install some hooks to repositories, which you might not want. There are settings somewere when you login as admin to disable this but I dont know if those would remove hook(s) that RhodeCode installed the first time it found a new repository.
  • From the log it seems, in my setup, it fails on file permission when installing some hooks but other than that I haven’t experienced any problem on pulling, cloning, and pushing to repositories.
  • Its appearance is not as sexy as github or bitbucket, sure. But I’ll manage, I rarely use web gui anyway.
Advertisements

2 Comments

Filed under Ilmu, Orang bego punya kegiatan

A Story of Migration from SVN to Mercurial

This post inspired by a few problems with my subversion copy of fpc, fpc documentation, and lazarus when I want to compile them. All of them can be described in three, gorgeous, seemingly-important bullet-points:

  • First of all, I have them on my home directory which is fine until I want to compile. At that moment I just have to copy or export the entire tree because I’m a neat guy and it doesn’t strike me as an acceptable situation when build-generated files sprawled all over the place.
  • Second, I have set up a Mercurial repository on my private server and this thing sits on my home directory adding unnecessary burden when I have to backup stuff. Besides, when I want to have it on different computer I would like to be able to grab it from my server, not my desktop computer —which is only accessible to others when my server is powered on, a “feature” in my network topology.
  • Third, I like Mercurial (or distributed revision control in general) workflow.

I found out about hgsvn and set to try it. After a browse-reading, I think I know how to use this thingie. I ssh’d to my server and browsed to my Mercurial repository dir.

Importing

First lets start with the smallest: fpcdocs. Small enough for an import straight from the fpcdocs svn server.

me@server$ hgimportsvn -v -r 710 http://svn.freepascal.org/svn/fpcdocs/trunk fpcdocs
me@server$ cd fpcdocs
me@server$ hgpullsvn

After the minor task of creating the hgrc and modifying .hgignore file, I’m satisfied that now I can browse the fpcdocs repository from my Mercurial web-interface.

Importing From Working Copy

For the next two project (fpc and lazarus), I decided not to re-download it from the project’s subversion server because my internet connection is slow and it is unbearable to download (again) what I already have. So I decided to try to import from working copy. See --local-only on man hgimportsvn.

Since my working copy of lazarus and fpc is inside my home directory on the desktop computer I have to switch back to my desktop computer to copy it using scp. I browsed into ~/programming/pascal/followed/, a directory appropriately named to contain local copy of followed projects, before invoking the copy command.

me@desktop$ cd ~/programming/pascal/followed/
me@desktop$ scp -qr fpc lazarus me@server://tmp/

The last line is to copy directory named fpc and lazarus, located in ~/programming/pascal/followed/, to /tmp/ on the server computer. Anyway it took a while on my network, a rather slow for the impatient as myself, but then again I was streaming music :D.

For some reason scp unable to copy two MacOS-related files. This is the message:

lazarus/lazarus.app/Contents/Resources/startlazarus.app/Contents/MacOS/startlazarus: No such file or directory
lazarus/lazarus.app/Contents/MacOS/lazarus: No such file or directory

I pondered on it for a while, this looks like a symlink-gone-bad problem and I think an svn update can fix this. So I switched to my server computer, using fg:

me@desktop$ fg
me@server$ cd /tmp/lazarus/
me@server$ svn update
me@server$ cd /tmp/fpc/
me@server$ svn update

The update on lazarus and fpc reverted some file and it fixed copying related problems. Yay, everything looks okay again.

Now, to make this working copy talk and walk as a Mercurial repository I have to run hgimportsvn with --local-only option.

me@server$ cd /tmp/
me@server$ hgimportsvn --local-only /tmp/fpc/
me@server$ hgimportsvn --local-only /tmp/lazarus/

The process of transforming into a Mercurial repository took a while. After all set and done I move the two to the appropriate location, right next to fpcdocs which have been discussed earlier.

me@server$ mv fpc lazarus /media/repodisk/mercurial/FreePascal/

That command will move fpc and lazarus to where my Mercurial repositories is located; which also monitored by hgweb to provide web-interface to Mercurial repository.

After creating two hgrc file one for each repository, modifying .hgignore and doing the obligatory file-ownership changes I ran a test-drive by browsing and downloading the newly-created repositories via browser.

I can see everything is working as expected, with one minor annoyance that I can live with: no history from subversion “days”. Oh well.

Leave a comment

Filed under Orang bego punya kegiatan

Komputer lama jadi server

WinXP Pro SP2 server desktop from VNCOke jadi gw punya komputer tua dengan spesifikasi: Pentium 4 1,7GHz, 256 SDRAM, NVidia GForce4MX, 80 GB HDD, 1 CD-RW, on-board soundcard, dan 4 port USB v1.0. Komputer ini gw setup sebagai komputer server pribadi, untuk melayani beberapa komputer dan sebuah notebook yang sama-sama lebih baru daripada komputer ini.

Di komputer lama ini gw install dua OS (system), yaitu Mandriva Linux 2006 Free Edition (Mandriva) dan Microsoft Windows XP Pro SP2 (WinXP Pro SP2). Sebenarnya pada awalnya hanya ada WinXP Pro SP2, karena gw pernah bekerja menggunakan komputer ini dan pekerjaan gw semua serba Windows. Mandriva baru di-install kira-kira satu tahun sebelum gw memutuskan untuk membeli komputer baru.

Sekedar informasi, sebenarnya WinXP Pro SP2 agak jarang dijalankan karena hampir semuanya sudah bisa dilakukan di Mandriva. WinXP Pro SP2 (masih) ada karena ada beberapa program yang hanya jalan di system Windows (oke, sebenarnya karena gw malas coba-coba dengan Wine :D), seperti FMA untuk ponsel, untuk membaca kamera digital, dsb.
Continue reading

6 Comments

Filed under Orang bego punya kegiatan, Pendapat gak penting