Thursday 19 May 2016

Removing unnecessary Puppet reports

A Puppet server i'm managing was running out of disk space and the culprit turned out to be Puppet's rather verbose report files. I had a whole bunch of reports which simply informed that the following umpteen files were not changed at all. This is both useless and wasteful, at 38 megs a report, per server, twice an hour. Even though the environment is small, i ended up with 22 gigs of reports...

After much googling and stackoverflowing, i came up with the following script:

#!/bin/bash grep -Pzl "status: unchanged(\n)metrics" /opt/puppetlabs/server/data/puppetserver/reports/*/*.yaml > $(dirname $0)/unchanged-reports # this is one long line, not four while read p; do sed '/metrics:/,$d' $p > ${p}.0 rm $p mv ${p}.0 $p done < $(dirname 0)/unchanged-reports

Run as root. Comment out the rm and mv bits if you're nervous or you just want to experiment.

The command line switches for grep (only work on GNU Grep, ie on Linux):

P turns on experimental Perl regexp mode and can potentially break things
z will effectively allow for multiline regexp patterns
l will return the file name where the pattern was found rather than the pattern itself

And then you can automate this, say, with cron.

In addition to this script, i use logrotate to compress and eventually remove old report files.

Sunday 15 May 2016

Mac tip: what's eating your net

Mac command line tip of the day:

nettop

...and then press c and d.

This will show you which programs (or more to the point, processes) use your network connection and how much. c collapsed the rows so you don't get one line per connection (you can get back to the expanded view by pressing e). “Delta mode” d would show you how much network capacity each program (or connection, if you're in expanded view) is using right now and pressing d again will take you to showing the total amount of traffic transported.

If you want to get geeky, you can toggle p to see the number of bytes transported rather than the more human readable so and so many megs or gigs.

Use the arrow keys to scroll to the sides for more statistics (use j to select which stats to display) and up-and-down if you have a really large number of programs on the list, or are watching the expanded view.

Finally, h will bring you the help screen so you don't have to remember all the keys i just wrote about :D