Elasticsearch registration agent for MCollective
tl;dr : mcollective-es
I have mentioned before that I’ve found integrating mcollective & elasticsearch to be interesting, so here is another way of expressing that, in the form of an elasticsearch registration agent for mcollective. Registration in mcollective allows you to develop inventory style apps/data from your infrastructure. There are existing agents which do this with mongodb & plain text files.
Some neat things about using elasticsearch include : awesome search (obviously!) and automatic document expiry for mcollective - since documents can come with a ttl of their own, they’ll expire after that, and your monitoring tool can alert you if a node drops using a query probably like this or any other query really for your needs. You can disable the ttl setting if you want.
The code is pretty basic :
def handlemsg(msg, connection)
req = msg[:body]
begin
Tire.index(@esindex) do |index|
index.create :mappings => {
:document => {
:_ttl => { :enabled => true, :default => @docttl }
}
} unless @ttldisabled == "false"
index.store :data => req,
:id => msg[:senderid],
:type => @estype
end
rescue Exception => e
Log.instance.debug("Couldn't index : #{e.backtrace.inspect}")
end
nil
end
And then a doc might look like this:
alcy@ub1104:~$ curl 'localhost:9200/hosts/_search?pretty=true'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "hosts",
"_type" : "document",
"_id" : "ub1104",
"_score" : 1.0, "_source" : {"type":"document","id":"ub1104","data":{"agentlist":["rpcutil","registration","discovery"],"facts":{"processorcount":"1","kernel":"Linux","netmask":"255.255.255.0","swapfree":"470.57 MB","physicalprocessorcount":"0","fqdn":"ub1104.foo.com","lsbmajdistrelease":"11","operatingsystemrelease":"11.04","uniqueid":"007f0101","memorysize":"243.28 MB","virtual":"physical","ipaddress":"192.168.1.102","is_virtual":"false","kernelrelease":"2.6.38-8-generic-pae","hardwaremodel":"i686","rubysitedir":"/usr/local/lib/site_ruby/1.8","ps":"ps -ef","netmask_eth0":"255.255.255.0","domain":"foo.com","macaddress_eth0":"08:00:27:a3:27:b7","id":"alcy","macaddress_eth1":"08:00:27:0e:39:ea","timezone":"IST","uptime_days":"0","memoryfree":"32.32 MB","interfaces":"eth0,eth1","uptime_hours":"7","lsbdistrelease":"11.04","hardwareisa":"i686","selinux":"false","processor0":"Intel(R) Core(TM) i3 CPU M 380 @ 2.53GHz","lsbdistdescription":"Ubuntu 11.04","path":"/home/alcy/perl5/perlbrew/bin:/home/alcy/perl5/perlbrew/perls/perl-5.14.2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/alcy/.rvm/bin","lsbdistcodename":"natty","kernelversion":"2.6.38","puppetversion":"2.7.17","uptime":"7:04 hours","hostname":"ub1104","facterversion":"1.5.8","kernelmajversion":"2.6","macaddress":"08:00:27:a3:27:b7","operatingsystem":"Ubuntu","swapsize":"510.00 MB","ipaddress_eth0":"192.168.1.102","uptime_seconds":"25491","network_eth0":"192.168.1.0","rubyversion":"1.8.7","architecture":"i386","lsbdistid":"Ubuntu"},"classes":[]}}
} ]
}
}
So one thing to note is we are using the :senderid
of the mcollective node from the the main message hash as the _id
so as to allow updates of existing data without creating new docs. Apart from that, the default _ttl
mapping helps us in detecting failing nodes. We should probably allow this to be a per node parameter - the ttl, that is. Also, would be nice to have the corresponding puppet functions to query elasticsearch for the node manifests too. Will try wiring in these features in a later version. For now, have fun with the plugin !