Blog

31
January 2014

Gavin Pickin

Dev Ops - From Mail Logs to DB Stats in a CFML Dashboard - Part 3

Server Admin

This is our third post in this mini series. You can read the first and second posts here. We know what we want out of the log files… so lets setup the process of getting the logs moved, and get the "goods" out of them… and in a position for our CFML Agent to do its job. We want to keep separation of duties on the servers, so we have Server A with Mail running, and the log files… and we want to move them to Server B… massage them, and prep them for CFML. So lets setup our cron jobs on Server A.

Server A

First we want to setup a job to move our log files that are archived. By default, the normal mail log file is maillog, and as they are archived, they get a dash then datestamp added to it. So knowing this, we can use the following command to move them to our MaillogAgent Home Directory

cp -f /var/log/maillog-2* /home/maillogagent/

 

Next, we need to change the permissions, as by default, the maillogs are very locked down, so lets change the owner, and then the permissions.

chown maillogagent. maillogagent /home/maillogagent/maillo*
chmod 775 /home/maillogagent/maillo*

 

Once we have processed one of these files, we want to make sure we don't process it again, so we'll move the log into a subfolder for future use.

mv /var/log/maillog-2* /var/log/maillogs/

 

So put it altogether, and we have our cronDailyMaillog.sh file.

#!/bin/bash

cp -f /var/log/maillog-2* /home/maillogagent/
chown maillogagent.maillogagent /home/maillogagent/maillo*
chmod 775 /home/maillogagent/maillo*
mv /var/log/maillog-2* /var/log/maillogs/

 

Our cronHourlyMaillog.sh is a little simpler, it merely copies the current maillog to the same folder, so we can import the newest update hourly, to keep up on those trying to cause trouble.

#!/bin/bash

cp -f /var/log/maillog /home/maillogagent/
chown maillogagent.maillogagent maillog

 

Make sure you give the scripts execution permissions with chmod +x cron*
Lets add them both to our crontab. To edit your crontab use the following command

crontab -e

This opens a vi editor of your cron tasks. 
Press "i" to go into edit mode, and add the following lines

00 */1 * * * /pathtoyourshells/cronHourlyMaillog.sh
00 2 */1 * * /pathtoyourshells/cronDailyMaillog.sh

The first line states we want our Hourly job to run every hour on the hour.
The second line states we want our Daily job to run at 2am every day.

To leave edit mode, hit esc. Then you can navigate, or run commands in vi. 
To quit without saving, type :q! which without warning
To quit type :q if you have changes, it will warn you that you cannot quit without saving, or forcing quit with :q!
To write the changes, type :w 
Once you have written your changes, you can use :q to quit safely.

Cron is a great tool, and obviously I have barely touched anything here. Maybe I will write more sometime, but there are plenty of great resources out there to learn how to configure cron.

Cron does not need to be restarted for the changes to take effect, as it is always looking for crontab changes.

 

Server B

Again, we're working on two separate processes, an hourly process, to pull the latest and greatest logs, so we can crunch the newest data possible, and we want to pull in the weekly archives, checking daily (I could check weekly if I figure out what day its archived on) and checking that weekly archive for any leftover logs, and then saving the log file for longer term keeping and analysis if we want it.

So lets build our cron shells.

cronDailyMaillog.sh

First, we need to use scp to copy the file from Server A to Server B. With scp I always pull the file, thats why this task is on Server B. Since our maillogagent has the files in its home directory, it makes permissions pretty simple, we just setup SSH Keys for each login (no password required for the scp) and use the following command

scp maillogagent@servera:maillog-2* /home/maillogagent/

 

  • scp is the command, secure copy over ssh.
  • maillogagent@ is what user are we connecting as, our ssh user is maillogagent in this case
  • @servera is the server we're connecting to just like normal ssh
  • : the colon separates the ssh authentication, with the file. 
  • maillog-2* gets us anything that matches maillog-2* in the home directory of the user we're ssh-ing with, in this case, its giving us all the maillog-2* files in /home/maillogagent/ on Server A
  • /home/maillogagent/ is the destination on Server B we want the file copied to.

1 command, a lot of explanation, but pretty easy. Using SSH keys for server access is very handy, I barely covered SSH Keys in the Source Control Series for using SSH Keys with Bit Bucket, but not with SSH Server Access, I will try to post a how to on that shortly, because if you are not using them, you probably should be.

Next, we need to make sure our permissions on the file are right.
Since we'll be sharing it with our cfml engines, I'm going to change the group ownership to the group that has that access.

chown maillogagent.webserver /home/maillogagent/*
chmod -R 775 /home/maillogagent/*

 

Once we have set permissions, we're going to use an SSH command, to move the file on Server A into a different folder on Server A so we don't process the same file later. When we look for files, we never look recursively, so moving the file into a subfolder is perfect in this case.

ssh maillogagent@servera 'mv /home/maillogagent/maillog-2* /home/maillogagent/movedMaillogs'

 

We use ssh command, connect as maillogagent@servera and then run the command in quotes… just like we were on that machine. It simply moves any maillog-2* file in the maillogagent home dir into the movedMaillogs folder, so we know we have already moved it from Server A to Server B and stops us from repeating this in the future.

Next, we want to grep the file, and process the log file, that our cfml engine (ColdFusion or Railo) will process.

grep 'CHKUSER' maillog-2* | grep -v 'sender:' > clean_maillog.log

 

I assume you know grep, but quickly, we're gripping the maillog-2* files for 'CHKUSER' then we grep the result of the first grep and we ask for anything that DOES NOT contain 'sender:' and then save the results into clean_maillog.log

Once we have grep'd the file, then we want to move the maillog-2* log file into another subdirectory, so we don't re-process this file over and over again either. 

mv /home/maillogagent/maillog-2* /home/maillogagent/processedLogs

 

So our final cronDailyMaillog.sh file should look like this.

#!/bin/bash

scp maillogagent@servera:maillog-2* /home/maillogagent/
chown maillogagent.webserver /home/maillogagent/*
chmod -R 775 /home/maillogagent/*
ssh maillogagent@servera 'mv /home/maillogagent/maillog-2* /home/maillogagent/movedMaillogs'
grep 'CHKUSER' maillog-2* | grep -v 'sender:' > clean_maillog.log
mv /home/maillogagent/maillog-2* /home/maillogagent/processedLogs
chown maillogagent.webserver /home/maillogagent/*
chmod -R 775 /home/maillogagent/*

 

Save the file, chmod +x cronDailyMaillog.sh and that file is ready to go.

Next, the hourly cron. This one is pretty similar, except, we don't move the file on Server A, and we don't move the file on Server B. The files keep being overwritten, and just keep on getting processed over and over. The only difference is, when we grep the file, we want to grep it to a different filename, just so we don't have conflicts with the daily cron process, so we'll add an H to the end of the file name. So it looks something like this.

#!/bin/bash

scp maillogagent@servera:maillog /home/maillogagent/
chown maillogagent.webserver /home/maillogagent/*
chmod -R 775 /home/maillogagent/*
grep 'CHKUSER' maillog | grep -v 'sender:' > clean_maillogH.log
chown maillogagent.webserver /home/maillogagent/*
chmod -R 775 /home/maillogagent/*

Now, chmod +x cronHourlyMailllog.sh and then we can add them to our cron service.

crontab -e

 

Click i to edit mode, and we'll add the following 2 lines.

05 */1 * * * /pathtoyourshells/cronHourlyMaillog.sh
00 4 * * * /pathtoyourshells/cronDailyMaillog.sh

 

Press esc, then :w enter and then :q enter to save and exit.

The hourly cron job will run hourly, at 5 minutes after the hour. The Server A task runs on the hour, so 5 minutes should give Server A plenty of time to complete its job, and then Server B hourly will start at 5 on the hour.

The daily cron job will run daily, at 4 hours past midnight, or 4am every morning. The daily cron job runs at 2am on Server A, giving it more than enough time. I plan to see when the archiving actually takes place, and then edit the cron jobs to run only 1 time a week, right after the archive is complete, but for the time being, this works.

Now, we are all set. Server A moves the maillog hourly, and daily it looks for new archived (old archived are moved to a subfolder so they're not accidentally caught with the wildcard * file name) maillogs, and moves them. Server B secure copies them over, moves the original file on Server A into a subfolder, and then greps the files, and moved the archived on Server B into a subfolder 2. Everything is ready for our cfml engine to schedule a task to look for those clean log files, and then do something with them.

I think that is enough for this post, check back, when we get into MySql, ColdFusion / Railo Processing, and we're getting closer and closer to those pretty graphs in our Web Management Dashboard.

Thanks for following along, 

Gavin

Blog Search