Installing Jenkins on OS X and Ubuntu

The task: setup a simple CI system for my Linode using Subversion, Jenkins, and Ant.

About the challenge

Continuos integration means setting up an automated workflow so that changes to the code base or content are published seamlessly to a production environment, with the possibility of rolling back should there be any problems. The system offers benefits even to a single developer working on their projects, especially if adding to the codebase from different workstations, which is the case for me.

For this weekly challenge I will take the first step towards a continuos integration system for WordPress sites, later to be extended to other projects. This is not a very sophisticated example of CI, there is deployment and roll backs but no testing, no automated documentation generation, etc. All that is left for further weekly challenges. In terms of learning new things, I have done all of this before, I just needed to do it again for this particular server and document it, so it’s not quite a “challenge”, just a job that needed doing.

Feature

Feature: setting up SSH keys
In order to be able to work remotely as safely as possible
As a developer
I need to have SSH keys set up from multiple machines


If you check your server’s access logs, the chances are you’ll find a lot of bots coming knocking and trying to guess passwords. Passwords are often less secure one thinks they are, and SSH keys are a much safer way of logging in, as they are extremely long and thus nearly impossible to guess. They do take some effort in setting up, which is why most ISPs will not do it for you. But I’d rather invest the time managing keys than trying to clean up the server after it’s compromised. And they are not that hard to set up, anyway.

Setting up SSH keys

In my Terminal, on my main local machine (i.e. the one I use the most), I run $ ssh-keygen
Enter file in which to save the key (/Users/me/.ssh/id_rsa): This command creates a pair of public / private keys, id_rsa.pub and id_rsa respectively. The private one stays on my machine, and the public one is copied to the server. I entered a different filename instead of id_rsa, as I have other keys I use for different sites. Everything else, I accepted the defaults.

scp ~/.ssh/id_rsa.pub USERNAME@EXAMPLE.COM:/home/USERNAME/.ssh/uploaded_key.pub
ssh USERNAME@EXAMPLE.COM "echo `cat ~/.ssh/uploaded_key.pub` >> ~/.ssh/authorized_keys"
This uses the scp utility to send the public key up to the server, and add it to the list of keys that can access it.

Disable root access and password authentication via SSH

From now on I set up the server to only allow login using SSH keys. This means I will only be able to login from my main local machine until I set up keys on other machines too. On the server, in the file that controls my SSH settings, /etc/ssh/sshd_config, I change the following two values (using the text editor nano)
$nano /etc/ssh/sshd_config
PermitRootLogin no
#???. other stuff???
PasswordAuthentication no
#on my local machine
ssh USER@EXAMPLE.COMThe first time Keychain Access will ask me for my passphrase, from then on I won’t need to anymore.

Setting up multiple SSH keys for multiple machines

To be able to work from other machines, I run the ssh-keygen command on each machine I need to connect from, generate new public / private key pairs, then email the public ones to myself. Note that you need to generate new keys on each machine – copying private keys from one machine to another is a bad idea. And really, creating a key is a simple command. Back to the main local machine (the only one I can currently login with) then I repeat the scp / ssh steps above to append the new public keys to the server. Now I can login from those other machines too.

Feature

Feature: setting up Subversion
In order to store source code in a centralised location
As a developer
I need to install SVN and create repositories

I have been using Git for a while, but have no experience in managing a repository, so will leave that (and other distributed version control systems) as a separate weekly challenge. Subversion is a known quantity to me, and more than sufficient for my purposes at the moment.

Installing Subversion on Ubuntu

The subversion repository will sit on the server. I connect to it, and run
sudo apt-get install subversionWell, that was easy.

Subversion user management

I created a group for all svn users including admins, and then one for all svn projects. If I were a small company I’d start creating groups for individual clients and projects, perhaps look into creating a readonly group, but since it’s just me that won’t be necessary. I have added myself to all the groups., and created a new svnadmin user. This will have to change when / if I setup WebDAV to access Svn via https, but I’ll cross that bridge if I get to it

#create groups
sudo groupadd svnadmin
sudo groupadd svnuser
sudo groupadd svnprojall
#create new admin user
sudo useradd -G svnuser -g svnadmin -s /bin/false svnadmin
sudo usermod -G svnuser,svnprojall,svnadmin USER
#this is the root of svn repository
sudo mkdir /srv/svn
#make it readable to anyone in svnuser, and only writeable by creator
#this means, potentially any member of svnuser can access anything below
#this will be fine tuned further down
sudo chgrp svnuser /srv/svn/
sudo chmod 750 /srv/svn/

#an intermediate level to control access
#this is where repositories are created
sudo mkdir /srv/svn/repos
#make it writeable by svnadmins, so that only they can create repos
sudo chgrp svnadmin /srv/svn/repos
sudo chmod 770 /srv/svn/repos

Then I logged out an in again (to make the change in groups take effect).

So now I can create repositories, using SSH to connect to the server and following a standard procedure. Not elegant and prone to error, but it will do for now.

svnadmin create /srv/svn/repos/REPO1
#making it writeable by group
sudo chmod -R g+wx /srv/svn/repos/REPO1/db/
svn mkdir file:///srv/svn/repos/REPO1/trunk -m "Creating trunk, branches, tags"
svn mkdir file:///srv/svn/repos/REPO1/tags -m "Creating trunk, branches, tags"
svn mkdir file:///srv/svn/repos/REPO1/branches -m "Creating trunk, branches, tags"
svn import /pth/to/code/already/on/server file:///srv/svn/repos/REPO1/trunk -m "Initial import" 

This is the procedure to follow every time I need to create a new repository. Could obviously be replaced by a script, perhaps adding hooks and so on, but that’s for another weekly challenge.

Now to see whether it works on my local machine.

#just a temp location for testing
cd ~/temp
#check out the code
svn co svn+ssh://USER@EXAMPLE.COM/srv/svn/repos/REPO1/trunk
#open a file and make a small test change
cd trunk
nano some/file/I/know/I/can/change.txt
# ??? add a couple of newlines or something ???
# ??? save CTRL-O
# ??? quit CTRL-X
svn commit -m "testing permissions"

All works as expected. The repository url is quite long and clumsy, but it is only entered once so it is not a big deal.

Feature

Feature: setting up Jenkis
In order to perform continuos integration jobs
As a developer
I need to install Jenkins locally and on the server


Installing Jenkins on an Ubuntu server

To start with, I simply followed the installation instruction on the Jenkins site

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo aptitude update
sudo aptitude install jenkins

That worked well – a little bit too well in fact. Suddenly Jenkins was up and running on port 8080 of all the sites hosted on my server, no password required. Quick – turn it off! Actually no, I may as add some passwords first.

• Clicked on Manage Jenkins, then Configure
• Ticked on Enable Security, Jenkins’s own user database, Matrix-based security
• Allowed ‘read’ permissions for anonymous
• Created a new user and ticked all the permissions
• Saved. Now I was presented with a login screen
• What is the default password for a new Jenkins user? No idea. Strange system. Undo everything.
• sudo nano /var/lib/jenkins/config.xml
• In that file there are a few tags concerned with security, starting with <useSecurity> and ending with </securityRealm>. Replaced them with

  <useSecurity>true</useSecurity>
  <authorizationStrategy class="hudson.security.AuthorizationStrategy$Unsecured"/>
  <securityRealm class="hudson.security.SecurityRealm$None"/>

• Restarted Jenkins sudo /etc/init.d/jenkins restart
• Now it’s back to no-password mode. Start again.
• Folllowed the same sequence from Manage Jenkins, but this time I also ticked on Allow users to sign up
• Now I can register as well as login. I registered with the same username I created earlier, and give myself a password
• Now I can login. Unticked Allow users to sign up and removed read permissions for anonymous user

Configuring Jenkins with an Apache proxy

OK, time to configure Apache now. Basically I want jenkins to run on a subdirectory of a given domain, https://www.example.com/jenkins. I tend to do that for all the tools I need, that way I can share the same certificate for https://www.example.com/ across them, rather than having to install one for each tool. That does make configuration a bit trickier in some cases.

First, install the apache proxy modules, and restart it
sudo a2enmod proxy
sudo a2enmod proxy_http

I add the proxy to the site’s virtual host directive, which in Ubuntu is at /etc/apache2/sites-available/example.com

<VirtualHost example.com>
    # ??? stuff that was already there???
    ProxyRequests Off
    ProxyPass         /JENKINS_DIR  http://127.0.0.1:8080/JENKINS_DIR
    ProxyPassReverse  /JENKINS_DIR  http://127.0.0.1:8080/JENKINS_DIR
    <proxy http://127.0.0.1:8080/JENKINS_DIR>>
       Order deny,allow
       Allow from all
    </Proxy>
</VirtualHost>

Changed the prefix and httpListenAddress in /etc/default/jenkins (making sure the double quotes remain the last character – wasted some time with that silly thing). The former is needed to complete the mapping, the latter to ensure http://example.com:8080 stops working.
JENKINS_ARGS="--webroot=/var/cache/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT --prefix=/jenkins --httpListenAddress=127.0.0.1"
In Jenkins itself, clicked on “Manage Jenkins”, “configure System”, and added http://example.com/JENKINS_DIR under “Jenkins URL”

Finally, stopped and started both jenkins and apache sudo /etc/init.d/jenkins restart
sudo /etc/init.d/apache2 restart That worked – got Jenkins running on http://example.com/JENKINS_DIR but not http://example.com:8080/ nor http://anotherdomain.com/JENKINS_DIR

Configuring Jenkins to run with HTTPS and Apache

I hadn’t set up my self-signed SSL certificate yet, so now was a good chance.

sudo a2enmod ssl
sudo mkdir /etc/apache2/ssl
sudo openssl req -new -x509 -days 1200 -nodes -out /etc/apache2/ssl/apache.pem -keyout /etc/apache2/ssl/apache.key
sudo nano /etc/apache2/sites-available/example.com

In the virtual host conf file, I replicated the virtual host for port 443, added the SSL lines and rebooted apache. That made https:// work. Now for the last piece of the puzzle – added redirect both in the http and https section of the conf file, so that all traffic is automatically redirected to the encrypted version. Note that the two redirects are different on the source (port 80) and destination (port 443)

<VirtualHost exmple.com:80>
    # ...stuff
    # ...the proxy stuff I have added earlier
    #rewrite rules
    RewriteEngine On
    RewriteRule /JENKINS_DIR(.*) https://example.com/JENKINS_DIR$1
</VirtualHost>

<VirtualHost exmple.com:443>
     SSLEngine On
     SSLCertificateFile /etc/apache2/ssl/apache.pem
     SSLCertificateKeyFile /etc/apache2/ssl/apache.key

     # ...stuff
     # ...the proxy stuff I have added earlier
     #
    Header edit Location ^http://example.com/JENKINS_DIR https://example.com/JENKINS_DIR
</VirtualHost>

another reboot, and that was it. Now going to http://example.com/JENKINS_DIR will redirect to the https version, where everything works as expected

Configuring Jenkins to work with Ant, SVN, and mail

Almost there. Jenkins is smart enough to find Ant and SVN on the system, so nothing for me to do there. All that’s left to do then is to set up the mail server to send messages when things go wrong.

I installed Exim, a lightweight mail agent good enough for sending system emails.
sudo apt-get install exim4-daemon-light mailutils
sudo dpkg-reconfigure exim4-config
The first command installs it, the second starts a configuration utility. In it, I picked “Internet” as the type of mail delivery. Next, default values, until it got to the recipient domains, for which I entered fully-qualified.example.com; example; localhost; localhost.localdomain (fully-qualified.example.com I found out by doing hostname -f). Then all defaults again until I get to the screen that asks for an email address. Tested the setup: from the shell echo "This is a test." | mail -s Testing someone@example.com Then in Jenkins, under “Manage Jenkins” / “Configure System”, in the email section I enter localhost as the server, created a sender email address, and left the “Default user e-mail suffix” blank. Clicked on the “test configuration” button – it all worked nicely.

Installing Jenkins on OS X

Turns out the installer and its whacky user / daemon setup was creating all sort of permissions problems. In the end I installed a later version which puts everything in ~/.jenkins. I am now running it manually with sudo java -jar /Applications/Jenkins/jenkins.war

Challenge 100% complete

With everything installed, the week is up and the challenge met. In part 2 I will start creating a new job.

Possible future expansions

• Allow SVN access via WebDAV
• Install a bug tracking system (Redmine?) and integrate with SVN
• Make repository creation simpler
• Pre-commits hooks for disallowing commits without comments
• Hooks for disallowing commits without comments
• Hooks for controlling permissions more granularly