This is a followup of my previous Post about Behavior Driven Infrastructure.
Using Behat to Test a Server
Here's an example for a Test for tzdata
Feature: tzdata Configuration
As a Server
I want to have a tzdata installation
So that my calculations for various timezones are always correct
Scenario: The tzdata data Installation and Configuration
Given i have the "tzdata" Package installed
Then the directory "/usr/share/zoneinfo" should exist
And the "tzdata" Package Version should match "2011(d|e)"
And the file "/usr/share/zoneinfo/localtime" should exist
And the file "/etc/localtime" should exist
Scenario: The tzdata checks for correct times
When i execute "date"
Then the output should match "CEST|CET"
When i execute "date --utc -d '2006-08-07 12:34:56-06:00'"
Then the output should match "UTC"
And the output should match "18:34:56"
When i execute "TZ=Europe/London date -d '2006-08-07 12:34:56-06:00'"
Then the output should match "BST|GMT"
And the output should match "19:34:56"
Writing the Unit Tests
I've put up 2 examples, one for ntp and one for tzdata, on my Github here. The Code is very Quick'n'Dirty, just as a "see what is possible" quality.
Advantages
Now when a random Dictator somewhere decides that his Country should change the timezones again, you can setup a quick test, rollout the tests, then rollout a new tzdata and be assured that the Timezone changes reliably hit every Server.
This is a very Simple Test, but it is just here as an example and to explore the viability. For an Apache installation for example you could proceed to check Various configuration settings, check for helper services that need to be there, check your logfile collection is setup properly, check your logrotation, check if all PHP packages are installed and correctly configured.
While I'm busy getting acquainted with Chef I'm starting to wonder why
the topic of "Behavior driven Infrastructure" hasn't picked up more
momentum than what appears to be the case right now. (Or I've just been
living under a Rock, and missed all of it).
Behavior driven Infrastructure
BDD Has been
around in Software Development for a while now, but coverage for it's
use in a Systems Administrators life has been pretty vague from what i
can tell.
I've found a few interesting posts, but not much beyond this.
I've read Test-Driven Infrastructure with
Chef which touched a bit on
the subject.
Behavior Driven Monitoring?
With growing Infrastructure, Monitoring becomes a major Pain. Especially
if you do it the "classic" way, that focuses on monitoring Components
rather then Services. If your system checks thousands of hosts, chances
are on some of them something is broken. Broken disk maybe? MySQL Slave that is causing
high loads because a batch Job is running some statistics? Apache that is running low on Childs?
But really, does it matter? As long as the Service is up and running and performing?
Do we need to monitor every little cog in our infrastructure if there is a way to do "Top Down" Monitoring?
I certainly don't enjoy being woken in the middle of the night by Monitoring that tells me that a Database is chocking somewhere.
And now?
cucumber-nagios seems like the only project touching on that Subject.
As PHP is my language of choice, and I'm not quite convinced that learning Ruby is going to make me a happier person, I'll stick to that.
Behat is Cucumber for PHP, installation is done quickly:
pear channel-discover pear.behat.org
pear channel-discover pear.symfony.com
pear install behat/gherkin
pear install behat/behat
(Remind me to build a Chef Cookbook for this.)
Time to explore...
Motive
I have in the past used OpenVPN to setup a VPN connection between home and
my VPS. Usually everything was good, and worked the way i wanted it to.
The troubles started when i wanted to add a Notebook and various
Development Virtual Machines to this network. I somehow managed to put
this
together, add some more services and keep this thing alive.
It was no fun whatsoever to use and maintain though, so i needed a new
Solution.
Tinc
I started exploring alternatives to OpenVPN and quickly stumbled on
Tinc which looked exactly like what i needed. The
Setup is fairly easy to do, and there is plenty of Documentation available
online already.
At first i used Tinc's "Router" Mode with multiple networks connected
together, like i used to have with the OpenVPN setup. Routing in such a
small network though is painfull, and if you constantly add new devices
and networks you'll pretty quickly grow tired of maintaining your static
routes.
I've since simply setup Tinc to do "Switch" mode where it acts as a simple
Network switch (as the name suggests). All endpoints can now share the
same
network and there is no need to setup routes anymore.
Another advantage is, that you can use the Linux Bridge Utils to put your
Tinc interface into a bridge with a local Lan interface, and immediately
have that entire network added to your Tinc VPN.
My current setup looks like this:
- At home i have a OpenWRT Router with Tinc installed, where the Tinc
interface is simply added to the Bridge that OpenWRT already has
- On my VPS i have a Tinc endpoint
- On my Notebook i also just have a tinc endpoint.
Both my Notebook and my VPS are now always on my Local Lan, no matter
where i am physically.
Adding Zeroconf DNS to the mix
With the Tinc setup above your life will already be much better. With some
tiny shell scripts my Notebook figures out if it's at home or on the road
and connects to my VPN automatically. Everywhere i go i have my Home
Network with me.
Now there's only one Problem left: DNS
When you connect to a foreign network, you will usually be issued a DHCP
IP and a DNS Server along with it. That DNS Server obviously knows nothing
about the Hosts you have at home.
After i started using tinc, i setup a DNS Server at home to serve my
Hostnames, and built a bunch of shell scripts that would make sure all
endpoints used that DNS. That is not a very good solution.
The only real alternative to using DNS is Zeroconf DNS or mDNS: Your
System will announce it's Hostname and IP Address via Multicast.
As i already had a Switched VPN network, all i had to do was to replace
the DNS server with Zeroconf. I simply installed Avahi and the mDNS
Resolver on all my hosts, and started using the .local hostnames for
everything. This works flawlessly on OS-X too. Windows is a bit hit and
miss, as the Bonjour implementation from Apple on Windows seems a bit
lacking.
Benefits
This Setup has a couple of benefits that i don't want to miss anymore.
- Static Network Layout: It doesn't matter if i am at home or on the road,
my Network configuration always looks the same everywhere.
- Dynamic Autoconfiguration: Once you have Tinc and Avahi running, there
is really nothing you need to do anymore configuration wise. You configure
it once, and it just works. I haven't had that experience with OpenVPN
which was a constant struggle to keep running.
- Encryption everywhere: I don't like people spying on me, if i connect my
Notebook to a "hostile" network, chances are my usage will be monitored
somehow. With the VPN Configuration i can just route all my HTTP traffic
to
a Squid Proxy running on my VPS and know that nobody will be able to sniff
my connection.
- Easy to Expand: At home i don't really need to do anything, i can just
add Virtual Machines. The Tinc Tunnel is bridged with my LAN, and all
traffic is automatically forwarded.
I have a MySQL Database running on my VPS. If i wanted to connect to it
from Home without the VPN configuration, i would need to expose the MySQL
server to the public internet. With the VPN i can just let it listen on
the
VPN interface, and don't need to worry about exposing it. With Avahi on
the
VPS i have a "public" (aello.beerta.net) and a "private" (aello.local)
hostname. This way i don't need to remember IP Addresses.
Before i ditched Apple, i also used to have a iTunes running at home to
serve music through Bonjour, as my VPN network was in one broadcast
domain,
i could listen to Music from whereever i was. Something that Apple tries
to
make sure you can't do.
The only problem with this setup is: If your Home Lan Network Range
clashes with the network you connect to physically you are doomed. So
choose a network range at home that is as small as possible and fairly
uncommon.
Aliases with Avahi
For my Development VMs i need a way to have multiple Hostnames for one IP.
Unfortunately the standard Avahi Installation does not yet allow
Aliases. Fortunately though somebody spent some time and build
avahi-alias.py
that does exactly what i need.
On Debian you just need to install python-avahi and python-dbus and
you can run avahi-alias.py development.local to Create the alias in your
lan, which will then instantly broadcast througout the Tinc Network.
More Info
If you'd like a bit more details on the Setup drop me a note, and i'll put
up some more documentation on the entire setup.
What?
Time to revive an ancient project: rssReader
It's been a while since i last worked on my own little Feed Reader project
(7 Years).
Why?
I've recently started using Google Reader as i got myself a fancy Android
Phone with a data flat rate, and i didn't get
TT-RSS running properly on it (probably
totally my fault).
The only thing i didn't want on Google Reader were all my authenticated
and NSFW feeds, i needed something different for those.
I remembered that i had this little project way back when, and decided to
have a look at it. Unfortunetly it has grown quite old. Still based on
PHP4, not very web2.0'ish, and generally not really pretty to look at.
So i just rebuilt the thing from Scratch.
The only thing it has in common with the old version is that it uses
Smarty as Template engine. I commited the old thing, if you want to have a good laugh.
Where?
First: May I suggest you try TT-RSS. Or
if you don't care much about privacy: Google
Reader is quite awesome. It's even more Awesome
with this extension installed.
If you still want to have a look, you can get it on
Github. Installation should be very
straight forward.
The only advantage it has over TT-RSS, is that it doesn't need a database
whatsoever (It's also somewhat easier to configure, but with the
difference
in features that's not much of a surprise)
It has plenty of disadvantages though.
I have Subversion Repository that holds various smaller projects lumped
into one directory.
This is annoying if you want to work on such projects with GIT as
Subversion front-end, or want to put that code on Github.
I Therefore needed a way to extract these directories from one Repository
and dump them into a dedicated repo with a proper directory structure
(trunk, branches, tags).
These are the steps needed for my rssReader example:
First Dump all of /home/svn and extract the php/rssreader directory
svnadmin dump /home/svn | \
svndumpfilter include --drop-empty-revs \
--renumber-revs --skip-missing-merge-sources \
php/rssreader > rssreader.dump
--drop-empty-revs and --renumber-revs make sure that the dumps history
looks clean and doesn't have all commits from the parent directories.
I then needed to edit the dump file and remove a svn-sync revprop which
would make svnadmin load barf.
If you have Copied contents around in the source repository, you may need
to include the sources of these copies as well, and clean them up in the
destination repository. svndumpfilter will complain that it can't copy
the Source directory in such cases.
Create new Repo and the Basic Layout
We now need a new, empty repository
svnadmin create rssreader
svn -m 'Initial Layout' \
mkdir file:///$PWD/rssreader/php \
file:///$PWD/rssreader/branches \
file:///$PWD/rssreader/tags
It is important that you create the directory structure as it was in the
Source Repository.
For this example the code i wanted to export was located at
php/rssreader i thus needed to create the php/ path in my destination
repository, as svnadmin load won't do that.
Import the dump
Now just load the dump file into the repository with:
cat rssreader.dump | svnadmin load rssreader
Cleanup the final repo
svn mv -m 'Move to trunk' \
file:///$PWD/rssreader/php/rssreader \
file:///$PWD/rssreader/trunk
svn rm -m 'Remove junk' file:///$PWD/rssreader/php
Finally clone it with git
Now i can go ahead and create a git clone of it:
# first create the authors file
echo "claus = Claus Beerta " > ~/svnauthors
# Now Clone
git svn clone -A ~/svnauthors \
-s file:///home/claus/rssreader \
rssreader-git
(git doesn't expand $PWD properly, so you need to give it the complete
path. Wierd)
Now i can go ahead and hack at the clean repo, commit it to Git or my
Private SVN Server.
