The uptime metrics for my group are based on the times OEM has recorded for each target.  Timely blackouts, of course, are essential to keeping the numbers up and (more importantly) preventing Target Down notifications from being sent out.

Initially I’d develpoed EM CLI scripts to create and remove the blackouts.  That solution was crisp, well-characterized, and scalable with one huge problem:  for convenience the blackout script should execute from host containing the target and not all of my hosts have EM CLI client installed.  The OEM agent is deployable and patchable from your management server, but EM CLI client is not.

Let’s think about how OEM blackouts work.  When you create a blackout through the console, your management server contacts the EM Agent and tells it to ignore metrics for the target.  The process is identical when you create a blackout with EM CLI  (CLI uses the same code base as the OMS, so this shouldn’t be a surprise).

Fortunately you can also create and stop blackouts using EMCTL.

This solution consists of two shell scripts, one to create and start the blackout and anther to stop the blackout  when you’re ready.

#!/bin/ksh
#set -x
######################################################################
# File: blackout_database.ksh
# Purpose: Create an ad hoc database system blackout using emctl 
######################################################################
# Variables
# ====================================================================
if [ `uname` = Linux ]; then
   export ECHO="/bin/echo -e"
else
   export ECHO="echo"
fi
if [ -f /opt/oracle/agent12c/agent_inst/bin/emctl ]; then
  EMCTL=/opt/oracle/agent12c/agent_inst/bin/emctl
else
   $ECHO "\nCould not find the emctl utility on this box\n\n"
   exit 0
fi

HOSTNAME=`hostname | cut -d. -f1`
WORKFILE=/tmp/database_blackout_work01.lst
function CleanUpFiles {
[ $WORKFILE ] && rm -f ${WORKFILE}
}
# ====================================================================
# Run-time procedure
# ====================================================================
CleanUpFiles
$ECHO "\n\nThese databases are currently registered with the local OEM agent:\n"
$EMCTL config agent listtargets | grep oracle_database | cut -d, -f1 | sed -e 's/\[//' >${WORKFILE}
echo "Never_mind" >>${WORKFILE}
PS3="Select the database target to blackout for the next four hours > "
select thisSID in `cat ${WORKFILE}`; do
   if [ ${thisSID} == "Never_mind" ]; then
      exit 0
   else
      $ECHO "\nCreating the blackout for ${thisSID} ...\n" 
      BO_NAME=Blackout_${thisSID}
      $EMCTL start blackout ${BO_NAME} ${thisSID}:oracle_database -d 04:00
      sleep 3
      $ECHO "\nBlackout status:\n" 
      $EMCTL status blackout ${BO_NAME}
      CleanUpFiles
      exit 0
   fi
done

Tasks:

  1. Determine which oracle_database targets are known to the local agent.  Databases that aren’t registered in OEM don’t need a blackout.
  2. Dump that list of databases into a work file and then add an escape option (Never_mind) to be used in the menu that is created next.
  3. Change the prompt, temporarily to solicit a selection from the list generated from the work file.  Notice that I arbitrarily assigned a  four hour duration (the -d flag)
  4. Give the blackout a standard name, create it, and then ensure that it’s working.

Run-time example of that script will make those steps easier to visualize:

> blackout_database.ksh
These databases are currently registered with the local OEM agent:
1) orcl2t
2) orcl3k
3) Never_mind
Select the database target to blackout for the next four hours > 1
Creating the blackout for orcl2t ...
Oracle Enterprise Manager Cloud Control 12c Release 3 
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Blackout Blackout_orcl2t added successfully
EMD reload completed successfully
Blackout status:
Oracle Enterprise Manager Cloud Control 12c Release 3 
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Blackoutname = Blackout_orcl2t
Targets = (orcl2t:oracle_database,)
Time = ({2014-07-23|15:14:23|240 Min,|} )
Expired = False

The flipside is even less complicated.

#!/bin/ksh
#set -x
######################################################################
# File: stop_database_blackout.ksh
# Purpose: Stop blackouts for a particular database
######################################################################
# Variables
# ====================================================================
if [ `uname` = Linux ]; then
 export ECHO="/bin/echo -e"
else
 export ECHO="echo"
fi
if [ -f /opt/oracle/agent12c/agent_inst/bin/emctl ]; then
 EMCTL=/opt/oracle/agent12c/agent_inst/bin/emctl
else
 $ECHO "\nCould not find the emctl utility on this box\n\n"
 echo 0
fi
HOSTNAME=`hostname | cut -d. -f1`
WORKFILE=/tmp/database_blackout_work01.lst
function CleanUpFiles {
[ $WORKFILE ] && rm -f ${WORKFILE}
}
# ====================================================================
# Run-time procedure
# ====================================================================
CleanUpFiles
$ECHO "\n\nThese databases are currently registered with the local OEM agent:\n"
$EMCTL config agent listtargets | grep oracle_database | cut -d, -f1 | sed -e 's/\[//' >${WORKFILE}
echo "Never_mind" >>${WORKFILE}
PS3="Select the OEM database target to take out of blackout mode > "
select thisSID in `cat ${WORKFILE}`; do
 if [ ${thisSID} == "Never_mind" ]; then
    exit 0
 else
    BO_NAME=Blackout_${thisSID}
    $ECHO "\nChecking for OEM blackout ${BO_NAME} ...\n" 
    if [ `$EMCTL status blackout | grep ${BO_NAME} | wc -l` -eq 0 ]; then
      $ECHO "There is no OEM blackout named ${BO_NAME}\n\n"
      CleanUpFiles
      exit 0
    else
       $ECHO "\nStopping blackout ${BO_NAME} ..." 
       $EMCTL stop blackout ${BO_NAME} 
       sleep 3
       $ECHO "\nVerifying status of all blackouts on ${HOSTNAME}\n" 
       $EMCTL status blackout
       CleanUpFiles
       exit 0
    fi
  fi 
done

 

Here’s what that one looks like at runtime:

> stop_database_blackout.ksh
These databases are currently registered with the local OEM agent:
1) orcl2t
2) orcl3k
3) Never_mind
Select the OEM database target to take out of blackout mode > 1
Checking for OEM blackout Blackout_orcl2t ...
Stopping blackout Blackout_orcl2t ...
Oracle Enterprise Manager Cloud Control 12c Release 3 
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Blackout Blackout_orcl2t stopped successfully
EMD reload completed successfully
Verifying status of all blackouts on mydemohost.edu
Oracle Enterprise Manager Cloud Control 12c Release 3 
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
No Blackout registered.

Dang, that’s handy.  Let’s see if we can get those numbers up!

One last thing:  EMCTL is not able to delete blackouts in 12.1.0.3.  You still need to visit the console or run a cleanup through EM CLI.  Perhaps this feature has been added in 12.1.0.4 …