Using Jenkins to replicate WordPress sitesWeekly Challenge

The task: setting Jenkins up to recreate a WordPress site from data stores in a repository.

About the challenge

In the previous weekly challenge I installed the tools needed to set up a continuos integration system for myself. Here I will start using it.

Some artefacts are available on GitHub.

Feature

Feature: setting up CI for WordPress In order to maintain my blog As a developer I need to create CI jobs in apache Ant for my WordPress blog

Requirements for a CI system for WordPress

At this stage I am going to setup Jenkins so that it queries the state of the system (code, assets, and data for a couple of WordPress sites) and then decides what to do with it. I could add hooks to various WP actions, so that so that things happen automatically when the system changes, but I want a generic solution that works for any project.

Here are the kind of jobs I need covered:

1. I check out the sources on a new machine and have a copy of the current production environment up and running
2. I make changes to my theme / plugins on a workstation
3. I upgrade WP itself or a plugin on a workstation
4. I create new content on a workstation
5. I edit content on the production server
6. I update my local version with the latest comments
7. I roll back changes if things go wrong
8. I work on a workstation and want to carry on working on another, without the temporary code being deployed
9. any of these can happen at the same time - e.g. I could start changing some code in my workstation, then make a quick amend to the content directly on the live site while out and about, then carry on working on the code change expecting the content amend not to be overwritten, and so on

2 and 3 are simple subversion jobs - in both cases I push code into svn at one end (a workstation), and pull it out and deploy at the other (the server). 4 and 5 involve db dumps and assets synchronisation in both direction. 6 is a pure db synchronisation job, and 7 is both. 8 and 9 are not really jobs, more a non-functional requirement (i.e., establish a branching strategy). And the first one is basically all of the above in one big job.

So in the end there are only two type of jobs: synching files and dbs with svn, and deploying using ant. That's the theory at least.

Run a copy of the production code from scratch with Jenkins

Scenario: Run a copy of production code from scratch with Jenkins Given that source code and all the tools I need are in subversion And I have created config settings manually as per instructions When I click on a button in Jenkins Then the source code and assets should be checked out And the source code should be copied to a server root of my choice And the database should be created and filled with data from the repository And any previous version with the same parameters should be overwritten And the http server should be started And Jenkins should check that pages are served
This pretty much covers the whole CI flow - all the other scenarios are just subsets of this. Time to create some Jenkins jobs. The Ant and Jenkins files are available on GitHub.

There are two things to bear in mind - Jenkins concepts of jobs and projects are a bit muddled - there is only 'jobs' really. And jobs names don't map to folder names - they are the folder names. What that means is that it makes sense to have a naming convention, such as [PROJECT]-[TYPE OF JOB]-[ENVIRONMENT]-[ETC] to group jobs by project and avoid folder names with spaces. So on the local Jenkins dashboard I select "new job" and call it MY_PROJ-deploy-local-fromScratch, and make it of type "Build a free-style software project". I save it without ticking on anything, then build it once. Now a folder is created at /Users/Shared/Jenkins/Home/workspace/MY_PROJ-deploy-local-fromScratch. That's where I will create the build.xml file for my Ant jobs, and where Svn will checkout the source.

Checking out the sources with Jenkins

To get things going, I connect the Jenkins job to the Svn repository.

• Under "Source Code Management" I select 'subversion'
• Under "Repository URL" I put svn+ssh://USER@EXAMPLE.COM/srv/svn/repos/REPO1/trunk
• Everything else leave as default: Local module directory = .; Check-out Strategy = use svn update as much as possible; Repository Browser = auto
• Save - Jenkins will ask for login details
• enter my passphrase and upload a copy of my private key. That's one thing that's actually quite tricky to do on a Mac, but I was already prepared. To have Finder show all hidden folders I had run the following on Terminaldefaults read com.apple.Finder AppleShowAllFiles Yes so that in Finder I can see the .ssh folder. If you need this to take effect immediately, you need to relaunch Finder by holding alt, right click on the Finder icon in the dock and choose "Relaunch". Now I can see the .ssh folder in Finder, and can drag it into my browser's "file upload" dialog to show it there too.
• Even better, I could have have just clicked on CMD-SHIFT-. to have the hidden folder appear in the file upload dialog itself
• Either way, I save everything then run the build. Jenkins will start downloading the source from version control. Meanwhile I can start working on the next step.

Moving sources with Ant from Jenkins

• Created temp properties file in my home directory, local.properties, with a simple test propertytest.local=[This is set in local.properties]
• Created a simple build file in the Jenkins job home, /Users/Shared/Jenkins/Home/workspace/MY_PROJ/build.xml

  <project name="MY_PROJ" default="fromScratch" basedir=".">
      <description>Build jobs for MY_PROJ local</description>
  
    <!-- set global properties for this build -->
    <property name="test.build" value="[this is set in build.xml]"/>
    <property file="/Users/USER/local.properties"/>
  
    <target name="copySource">
      <echo message="copySource - property ${test.build}" />
    </target>
  
    <target
      name="fromScratch"
      depends="copySource"
      description="copies the checked out source " >
      <echo message="fromScratch - property ${test.local}" />
    </target>
  </project>

• All this does is to display two messages, one including a property from the local property file. To run manuallycd /Users/Shared/Jenkins/Home/workspace/MY_PROJ Ant
• Output is as expected,

  Buildfile: /Users/Shared/Jenkins/Home/workspace/MY_PROJ-deploy-local-fromScratch/build.xml
  
  copySource:
       [echo] copySource - property [this is set in build.xml]
  
  fromScratch:
       [echo] fromScratch - property [This is set in local.properties]
  
  BUILD SUCCESSFUL
  Total time: 0 seconds

• Now I tell Jenkins to run the same job after it's checked the code out from svn. In the job panel, I click on "configure"
• Click on "Add build step" under "Build", selecting an Ant job
• enter "fromScratch" as my target - Jenkins wasn't able to pick it up automaticatilly, so I typed it
• Save, then run the build job. Job is very quick now.
• Under the "build history" panel, I select the job that's just been built, and click on "Console Output"
• It shows the same echo messages as above - all is good
• I changed local.properties so that it now has a directory name for the document root
  local.wwwroot=/Users/ ...
• Changed the Ant job to make it copy the source to the Apache server root, ignoring .svn folders

  <property name="common.src" value="src"/>
  <property file="/Users/USER/local.properties"/>
  
    <target
      name = "fromScratch"
      description = "creates a new instance from scratch"
      depends = "copySource"
      >
    </target>
  
    <target
      name = "copySource"
      description = "copies the checked out source" >
      <mkdir dir = "${local.wwwroot}" />
      <copy todir = "${local.wwwroot}">
        <fileset dir = "${common.src}">
          <exclude name = "**/.*"/>
        </fileset>
      </copy>
    </target>>

• The database access settings are hardcoded in a php file in WordPress, and I'd like to take that out of the repository and use the local properties file instead. So in all the wp-config.php files for my blogs I replaced the dbname, username, and password with tokens - @dbname@, @dbrunuser@, @dbpassword@ - and put the corresponding values in my local.properties file. local.dbname1=.... local.dbrunuser=...
• Added a new Ant target, using the replace task, to replace those tokens with the values in my properties file

  <target name="createWordpressConfig">
      <fail unless="local.wwwroot"/>
      <replace
        file = "${local.wwwroot}/blog/wp-config.php"
        propertyFile = "resources/ant/local.properties">
        <replacefilter
          token = "@dbname@"
          property = "local.dbname2"/>
        <replacefilter
          token = "@dbuser@"
          property = "local.dbrunuser"/>
        <replacefilter
          token = "@dbpwd@"
          property = "local.dbrunpwd"/>
      </replace>
      <!-- repeat for every blog -->
      </target>

Backing up and restoring the database

In order to restore a database, I need something to restore it from. This is where it gets tricky. WordPress uses MySQL, where backups are better performed using replication, but setting that up is too involved for my current purposes. Right now I want eerything to be saved to, and recreated from, text files. Incremental backups are also not MySQL's forte - restoring from the Binary Log on the production server is not the most efficient approach. That leaves only two tools, mysqldump and mysqlhotcopy. The latter is best practice for MyISAM tables, which is what WordPress uses at the time of writing, while the former is more generic and will work with InnoDB too. For that reason I decided to use mysqldump, so that I can reuse the Ant jobs on all my MySQL based projects.

Backing up the database with mysqldump

On the server, I manually create the inital dump and compress it to save space in the repository mysqldump --user=USER --password --hex-blob --dump-date --single-transaction --opt --order-by-primary --databases DB1 DB2 | gzip > /path/to/my/sql/sources/MY_PROJ-all.sql.gz svn add /path/to/my/sql/sources/MY_PROJ-all.sql.gz svn commit /path/to/my/sql/sources/MY_PROJ-all.sql.gz -m "initial sql dump on `date "+%Y-%m-%d %H:%M:%S"`"

mysqldump

calls the mysqldump utility

--user=USER --password

credentials. Note that I don't use --password=PASSWORD as that would leave PASSWORD in the command history. Using just --password will make mysqldump prompt me for one.

--hex-blob

saves binary data as hex (I don't think WP has any, but still)

--dump-date

to have a comment at the end with the date, just in case I need it for trouble shooting

--single-transaction

useless in this case; it is a reminder should I reuse this on another project, as it only works for InnoDB tables

--opt

(optimized) should be on by default anyway, it is a shortcut for a bunch of best practice options: --add-drop-table --add-locks --create-options --disable-keys --extended-insert --lock-tables --quick --set-charset

--order-by-primary

because it is more efficient to restore database if the rows' primary keys are in order

--databases DB1 DB2

pick the two databases I need

| gzip

send the dump directly to gzip, a data compression utility

> MY_PROJ-all.sql.gz

save the compressed sql dump to a file

svn add MY_PROJ-all.sql.gz

make svn aware of it

svn commit MY_PROJ-all.sql.gz -m "initial sql dump on `date "+%Y-%m-%d %H:%M:%S"`"

add it to the svn repository

Restoring the database from mysqldump with Ant

At the other end, my local machine, I add an Ant target for unzipping the sql and creating the database from it

• I add local.dbuser and local.dbpwd to my local properties file
• Add a db-restoring target to my build file. It calls 'mysql' from an exec task
• Because Jenkins run as a daemon, it doesn't know where 'mysql' is. I need to pass the absolute path to the executable. Not too happy with it, but for now I create a new local property for the path, and add it before 'mysql'. That means that property holding the path must always exist, even if it is empty, otherwise Ant will get confused and look for ${local.pthMySQL}mysql
• One thing that mysqldump doesn't take care of is users. So I created a small, tokenized sql file to take care of that

  grant all  on gotofritz_work_02.* to @dbrunuser@ identified by '@dbrunpwd';
  grant all  on gotofritz_blog_02.* to @dbrunuser@ identified by '@dbrunpwd';

• Before running that, another replace task to swap tokens with values from my local.properties file
• Here's the final target. Note that the sql file is passed as input to the exec task

  <target name="restoreDBAll" description="unpacks sql dump and restores it to db">
      <fail unless="local.pthMySQL"/>
      <!-- this is a big file -->
      <gunzip
        src = "${common.pthDB}/${common.dbAll}.sql.gz"
        dest = "${common.pthDB}/${common.dbAll}.sql" />
      <!-- create tables -->
      <exec executable="${local.pthMySQL}mysql" input="${common.pthDB}/${common.dbAll}.sql">
          <arg value="--user=${local.dbuser}" />
          <arg value="--password=${local.dbpwd}" />
          <arg value="--max_allowed_packet=100M" />
      </exec>
      <!-- inject local passwords -->
      <replace
        file = "${common.pthDB}/${common.dbUsers}.sql"
        propertyFile = "resources/ant/local.properties">
        <replacefilter
          token = "gotofritz_work_02"
          property = "local.dbname1"/>
        <replacefilter
          token = "gotofritz_blog_02"
          property = "local.dbname2"/>
        <replacefilter
          token = "@dbrunuser@"
          property = "local.dbrunuser"/>
        <replacefilter
          token = "@dbrunpwd@"
          property = "local.dbrunpwd"/>
      </replace>
      <!-- create users -->
      <exec executable="${local.pthMySQL}mysql" input="${common.pthDB}/${common.dbUsers}.sql">
          <arg value="--user=${local.dbuser}" />
          <arg value="--password=${local.dbpwd}" />
      </exec>
    </target>

• All of a sudden I started getting "Got a packet bigger than 'max_allowed_packet' bytes" error messages. So I added a --max-allowed-packet attribute, and run the following inside mysql to fix it, as per this stackoverflow answer set global net_buffer_length=1000000; set global max_allowed_packet=1000000000;

Tokenizing the database dump with sed

That's not quite it though. WordPress puts absolute paths and URLs all over the database, meaning it won't work on machines with a different setup. There isn't a good workaround for it except going through the sql dump file and substituting some strings with others. Thank heavens for sed, then.

• On the server, I do a mysqldump again.
• I create a file, tokenize.sed, with a list of all the substitutions I need to do to the sql files/http\:\/\/gotofritz.net/\@homeDomain\@/g s/blog.gotofritz.net/\@blogDomain\@/g # ... etc s/\/pth\/to\/wwwroot\//\@wwwroot\@/g
• I run sed -f tokenize.sed < /MY_PROJ-all.sql > temp.sql This uses sed on the sql file to run all the regular expressions I have just created.
• Gzip the sql file again, and commit it.
• In my local build file, between the gunzip and the sql restore tasks, I add a replace task to replace those tokens with values from my local.properties file

      <gunzip
        src = "${common.pthDB}/${common.dbAll}.sql.gz"
        dest = "${common.pthDB}/${common.dbAll}.sql" />
      <replace
        file = "${common.pthDB}/${common.dbAll}.sql"
        propertyFile = "resources/ant/local.properties">
        <replacefilter
          token = "blog.gotofritz.net"
          property = "local.blogDomain"/>
        <replacefilter
          token = "http://gotofritz.net///////"
          property = "local.homeDomain"/>
        <replacefilter
          token = "work.gotofritz.net                        "
          property = "local.workDomain"/>
        <replacefilter
          token = "/srv/www/gotofritz.net/public_html"
          property = "local.wwwroot"/>
      </replace>

• Now the sql dump has all the local paths and URLs I need.

Rebooting Apache and checking whether it restarted correctly

• To restart apache locally from Jenkins, I added another build step after the Ant one. This time a shell script that simply doessudo /usr/local/bin/apache/bin/apachectl restart
• In order to run that, the user Jenkins runs as needs to be in the sudo list, without being asked for a password. On Terminal I start up sudo visudo and adddaemon ALL= NOPASSWD: /usr/local/bin/apache/bin/apachectl
• Tested by shutting down Apache, running the command - pages are being served.
• Next, I want Jenkins to check for me. I create another target, checkApacheIsUp, which fails if a url I define in my local.properties file is not available

  <target name="checkApacheIsUp" description="check URL is serving pages">
    <waitfor
      maxwait = "20"
      maxwaitunit = "second"
      checkevery = "500"
      timeoutproperty = "failed">
      <http url = "${local.urlHome}" />
    </waitfor>
    <fail
      if="${failed}"
      message="${local.urlHome} is not up" />
  </target>

• It all seems to work

Putting config data under Subversion

One of the requirements in the original feature was "And I have created config settings manually as per instructions". I have put these in local.properties so far. I wasn't sure what to do with it, I was considering using a paramterized build, but that doesn't really make anything easier. Instead:

• I created an ant folder, resources/ant and added it to svn
• moved the local.properties file there, renamed it local.properties.sample, and added it to svn
• added local.properties to svn:ignore withsvn propset svn:ignore local.properties .
• made a directory for the Jenkins job called resources/ant/MY_PROJ-deploy-local-fromScratch and created a symbolic link to the Ant build file for that job in there, then added it to svn
• in the Jenkins dashboard, under "configure" for the job, ticked "This build is parameterized", and added a file parameter with location resources/ant/local.properties and descriptionA bunch of local properties needed by ant. There is a sample in subversion, under resources/ant/local.properties.sample
NOTE: if you have provided this once, and it hasn't changed, you don't need to provide it again
• Added a fail tasks at the start of each target to ensure none of them will run unless a local.properties file was provided
• while I was at it, I also backed up the config data for the Jenkins job itself from /Users/Shared/Jenkins/Home/jobs/MY_PROJ-deploy-local-fromScratch/config.xml to /resources/jenkins/MY_PROJ-deploy-local-fromScratch and added that to svn
• It all works fine.

The build and some config files are available on GitHub.

Challenge 100% complete

This is only one step. It was very involved but hopefully a lot of the groundwork will pay off. In part 3 I shall deal with rolling back and deployment strategies.

Possible future expansions

• Refactoring Ant job so that I can have a generic reusable library