Our project was inching closer and closer to going live. \u00a0On the evening before the Easter break some clod started a massive report which tried to process all the data since the beginning of time.\u00a0My colleague’s inefficient program gathered up all of the data from the database and wrote it to the data directory in a temp file.<\/p>\n
The data directory was more than large enough for a test environment but it wasn’t a big enough for such a test. \u00a0We could have easily recovered if it had occurred while we were at work but instead the system attempted to process nightly batches over those four days without enough space and made a pretty big mess.<\/p>\n
My boss Theodore was more upset than he should have been for a test environment and kept yammering on about what if this has been production. \u00a0He was right of course but one of the preconditions of the system is that enough resources are available. \u00a0It is our groups responsibility to write the programs but it is someone else’s to ensure that the system doesn’t run out of resources.<\/p>\n
Anyway, we implemented the boss’ warning email feature. \u00a0Each time the program is run it checks for enough disk space when not enough space exists then send out a warning email and quits. To be on the safe side, my boss asked that I have my email address as one of the recipients.<\/p>\n
If I receive one more warning email from one of the test systems I am afraid I am going to kill someone.<\/p>\n
Von: Automatically.Generated.Message@acme.com [mailto:Automatically.Generated.Message@acme.com] \r\nGesendet: Dienstag, 7. Februar 2017 11:54\r\nAn: process_monitoring@acme.com; \r\nBetreff: Warning ... the end is near on acme-app1\r\n\r\nAn error has occurred, the copy app has not been launched \r\nbecause of insufficient disk space on following partition.\r\n\r\nFilesystem size used avail capacity Mounted on\r\nacme-app1_dpool\/app 85G 85G 0K 100% \/appdir\r\n\r\nCorrective action is required immediately.\r\n\r\n\r\nThe status of rest of machine is as follows.\r\n\r\nFilesystem size used avail capacity Mounted on\r\n\/ 10G 7.7G 2.3G 77% \/\r\n\/dev 10G 7.7G 2.3G 77% \/dev\r\nproc 0K 0K 0K 0% \/proc\r\nctfs 0K 0K 0K 0% \/system\/contract\r\nmnttab 0K 0K 0K 0% \/etc\/mnttab\r\nobjfs 0K 0K 0K 0% \/system\/object\r\nswap 140G 400K 140G 1% \/etc\/svc\/volatile\r\nfd 0K 0K 0K 0% \/dev\/fd\r\nswap 8.0G 700M 7.3G 9% \/tmp\r\nswap 140G 40K 140G 1% \/var\/run\r\nacme-app1_dpool\/app 85G 85G 0K 100% \/appdir\r\nacme-app1_dpool\/acme_home 1.0G 353K 1.0G 1% \/appdir\/home\/gast\r\nacme-app1_dpool\/acme_samba 2.0G 36K 2.0G 1% \/appdir\/samba\r\nacme-app1_dpool\/acme_scripts 2.0G 249M 1.8G 13% \/appdir\/scripts\r\n\r\nThis is an automatically generated message for informational purposes.<\/pre>\nThe idea seemed ok; when no disk space exists then send out an email. \u00a0The underlying assumption was someone in IT would deal with the problem.<\/p>\n
Apparently the
idiot<\/del> users turn off half of the system about a week back but not every process. \u00a0I came to work and found hundreds of emails clogging up my inbox. \u00a0Looking through them you could literally see the space filling up over time.<\/p>\nWell, hundreds of files are annoying but the general functionality is awesome. \u00a0A combination of the bash script\u00a0and sendmail allows me to capture the important facts about our system and send it to someone.<\/p>\n
Just look at the script.<\/p>\n
#!\/usr\/bin\/bash\r\nSUBJECT=\"warning the end is near\"\r\nDF_Command=`df -h ${FILESYSTEM}`\r\nFULL_DF=`df -h`\r\nTO=process_monitoring@acme.com\r\nFROM=\"automatically generated message\"\r\n\r\nHOSTMACHINE=acme-app1\r\n\r\n( cat << !\r\nTo: ${TO}\r\nFrom: ${FROM}\r\nSubject: warning\u00a0... the end is near on $HOSTMACHINE\r\n\r\nAn error has occurred, the copy app has not been launched because of insufficient disk space on following partition.\r\n\r\n${DF_Command}\r\n\r\nCorrective action is required immediately.\r\n\r\nThe status of rest of machine is as follows.\r\n\r\n${FULL_DF}\r\n\r\nThis is automatically generated message for informational purposes.\r\n!\r\n) | \/usr\/sbin\/sendmail -t<\/pre>\nFill the variables with information ranging from a single word up to a lot of lines of text and then substitute them into your mail. \u00a0The bash shell will expand them before sending the mail out.<\/p>\n
I guess that the moral of the story should be that more logic should be used because some idiot will inevitably trigger it on a non-production environment. \u00a0Well, that or just get rid of the idiots ….<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
(flashback) Our project was inching closer and closer to going live. \u00a0On the evening before the Easter break some clod started a massive report which tried to process all the data since the beginning of time.\u00a0My colleague’s inefficient program gathered … Continue reading