Insert Wit Here

Archive for the ‘Software’ Category

Route53 Latency Based Routing

AWS Route53 launched a new feature today, Latency Based Routing. While playing around with this I created a few bits.

First I set up a t1.micro instance running in each region which will serve you a cat picture. Using a CloudFormation template I created the EC2 instance, downloaded the content from an S3 bucket, setup Apache, created an A record for the host, and another A record encoding the region. Each instance is now directly available via http://<aws region string> Here’s the direct link to west coast cat in us-west-1.

The next step was create an LBR A record of with appropriate RRs for each region. This was done via the AWS console. When entering the IP address the console automatically detected which region the EC2 instance is associated with. Slick!

And now this image comes directly from an AWS region near you:
your fastest cat

LBR isn’t limited to just A records, any record type should work. You could could send Puppet clients to their fastest Puppet Master using LBR SRV records, for example.
To troubleshoot “where am I routed to” questions I created a handy TXT record, This record simply echos back the AWS region it is associated with. For example:

# dig +short txt

LBR is an awesome feature for directing end users to the fastest available dynamic content. I can’t wait to see what people use this for.

Written by donavan

2012/03/21 at 17:26

Posted in Software

Tagged with , ,

Structured log output from puppet

A post on puppet-users got me thinking about improved output. A simple method is to just JSON encode the log and parse the message objects on the other side. So here’s an easy way to do that.

This works pretty well with puppet master & puppet apply:

tmp donavanm$ puppet apply /tmp/test.pp --logdest json
{"level":"notice","time":"Sat Feb 19 17:01:12 -0800 2011","tags":["notice"],"source":"Puppet","message":"this is a message"}
{"line":1,"level":"notice","time":"Sat Feb 19 17:01:12 -0800 2011","tags":["notice","notify","class"],"file":"/tmp/test.pp","source":"/Stage[main]//Notify[this is a message]/message","message":"defined 'message' as 'this is a message'"}
{"level":"notice","time":"Sat Feb 19 17:01:12 -0800 2011","tags":["notice"],"source":"Puppet","message":"Finished catalog run in 0.06 seconds"}

It works less well with puppet agent, unfortunately. If --verbose or --debug is specified the :console destination is added, so you’ll get duplicate messages. Lame.

Currently there’s no way to load this as a plugin, see #6522. Appending this method to puppet/util/log/destinations.rb works. Meh.

Ideally I think we’d have different options for transaction reports. Currently the agent application always forces reports to the rest terminus. Being able to set up multiple destinations, like nagios or a local cache, would be great.

Update I’d previously mentioned #4248, which is similar but not what I really want. Updated with the correct issue. Also opened #6523 to clean up logdest handling in applications like agent.

Written by donavan

2011/02/19 at 17:28

Posted in Software

Tagged with ,

A simple IP Address reflector

Or, Why the hell do I have to build every tool myself. I find it incredible that no one else has wanted this before.

In the course of another side project I needed to determine a nodes externally visible IP Address and FQDN. After a while I eventually found this forum post. And their “bot” url does work, though it’s a bit limited. So I hacked up some simple PHP to do it in a flavor that I like.

donavanm$  curl

Will also accept source and (reasonable) timeout parameters. A simple TCP connection is attempted back to your IP and specified port. Handy to see if NAT/iptables/etc is functioning.

donavanm$  curl

Get the source or try it yourself.

Written by donavan

2011/02/10 at 19:16

Posted in Software

Tagged with

Managing Amazon Route 53 with Puppet

This has been sitting in a work dir for a month now. Hopefully posting it motivates me polish it up and release it to the internets.

A while back I got new DSL service at my house in Seattle. In the course of moving I had to reconfigure a few nodes, setup  a gateway, etc. And in doing so I discovered that dynamic dns providers totally suck. It’s incredible. $20/year and you can’t even properly do delegations?

Coincidently I also noticed the new hotness from AWS at about the same time. DNS is part of my infrastructure, and puppet manages my infrastructure… So time to make puppet manage my DNS. After an evening hacking this up I present The AWS Route 53 type & provider:

Ensure a record:

tmp donavanm$ sudo puppet apply /tmp/r53.pp 
notice: /Stage[main]//Route53[]/ensure: created

Get a list of my current records:

tmp donavanm$ sudo puppet resource route53
route53 { '':
    ensure => 'present',
    value => [''],
    rtype => 'A',
    zone => '',
    ttl => '360'

Change ensure => 'absent' and remove that record:

tmp donavanm$ sudo puppet apply /tmp/r53.pp 
notice: /Stage[main]//Route53[]/ensure: removed

And yup, it’s really gone:

tmp donavanm$ sudo puppet resource route53
tmp donavanm$

Being a fully functional type and provider it should Just Work in any of the puppet applications. I think the most powerful model would be using something like exported resources with puppet agent and master. The clients would export a resource, like I’ve shown. A trusted master periodically collects and updates all of the entries.

# dynamic clients export their settings
class r53::client::dynamic {
    @@route53 { 
            value => $ipaddress,
            rtype => 'A',
            zone => "${domain}.",
            ttl => '360'
# A puppet master collects and updates
class r53::server::dynamic {
    Route53 <<| tag == 'r53::client::dynamic' |>>

I could see this being a great tool for people with cloudy puppet deployments. Or when you just really want your laptops dns record to be current.

At a dollar a month its half the cost of those dynamic dns guys, totally automated, and a thousand times cooler.

Written by donavan

2011/02/04 at 23:17

Posted in Software

Tagged with , , ,

Array of Hashes in Puppet DSL

A while back I was writing some custom server side functions to interrogate an inventory system for live information about other nodes. The returned data can be used to dynamically configure all sorts of settings that rely on other nodes: DNS resolvers, LDAP servers, Ganglia forwarders, etc.

Returning an array of nodes I could iterate a Define multiple times to create a set of resources. Here’s an example creating a set of Host resources for all of my DNS resolvers:

$mynameservers = find_nearest_nodes("puppetclass == dns::resolver")
# => ['ns1.domain.tld','ns2.domain.tld']
define hostentry() {
    $entry_ip = find_node_fact($name, "ipaddress")
    $entry_hostname = find_node_fact($name, "hostname")
    host{ $name: ip => $entry_ip, host_aliases => $entry_hostname }
hostentry { $mynameservers }

The two lookups in the define bothered me a bit. Assuming each lookup had some overhead latency that could add up. Why not simply return a hash with all the information i needed?

$myhashes = find_nearest_nodes("puppetclass == dns::resolver", ['ipaddress', 'hostname'])
# => [ {"ns2.domain.tld"=>{"ipaddress"=>"", "hostname"=>"ns2"}}, {"ns1.domain.tld"=>{"ipaddress"=>"", "hostname"=>"ns1"}} ]

But this array of hashes isn’t very useful. Each array entry is passed as $name to the target resource. Native resource types like File or Host will expect a string as the $name. Passing them the hash object won’t do what you want, I’m guessing it’d be collapsed in to a string with #to_s.

We can solve that though by wrapping the native type with a defined type . Inside the define you’ll be able to access the subkeys of $name like a normal hash. This allows you to pass your hash values as parameters to the actual resource:

$myhashes = find_nearest_nodes("puppetclass == dns::resolver", ['ipaddress', 'hostname'])
# =>  [ {"ns2.domain.tld"=>{"ipaddress"=>"", "hostname"=>"ns2"}}, {"ns1.domain.tld"=>{"ipaddress"=>"", "hostname"=>"ns1"}} ]
define hostentry() {
    # $name looks like {"ns2.domain.tld"=>{"ipaddress"=>"", "hostname"=>"ns2"}}
    host{ $name: ip => $name[ipaddress], host_aliases => $name[hostname] }
hostentry{ $myhashes: }

EDIT: 4/7/11
Sorry #puppet, I’ve updated that last example. I’d written this post from memory and obviously didnt try the examples. $myhashes is an Array of Hashes, not HoH, and I missed the : when calling the hostentry define.
And if you think you want to do this, don’t. Drop in to a template or Ruby DSL. Usage is non obvious and relies on undefined behavior with namevar. I wouldn’t be surprised if it breaks horribly in a point release (though it works in 2.6.7).

Written by donavan

2011/01/19 at 17:36

Posted in Software

Tagged with ,