How to Remove an Exadata Server from OEM

Just like everything else in Enterprise Manager, an Exadata is a target and all of its components are targets.  All of these pieces can be removed through the 12c console.

Select Exadata from the bottom of the main Targets drop-down to display a list of Database Machines in your environment.  Highlight the row containing the subject machine and press the Big Red X to take you into the detail page.


The left panel shows you the individual components effected by this change. Leave that panel highlighted at the DB Machine level to remove everything.


The right-hand panel is quite straight-forward.  The default action is to Remove all members of the Oracle Exadata Machine.  You can also select individual components for removal from this screen.

Check the box to Unsubscribe SNMP subscriptions if you have any configured.


Set the credentials for both the Storage Server and Infiniband Switch.  Both of them will require root.

remove_exadata_04 remove_exadata_05

That’s all there is to it.   Remember that metrics are gathered and stored in your OMR repository database for each of those targets, so the target removal transactions can take a while to process.  For instance, the machines I removed had been in production for over four years and target removal on a hefty OMR database took about thirty minutes each.

This is kind of a rare event, so I thought I’d share it with you.

How to Document Upload Destinations

We’re in the process of implementing a load balancing server for our OMS cluster.  The migration itself consists of changing the REPOSITORY_URL and emdWalletSrcUrl addresses inside the $ORACLE_HOME/agent_inst/sysman/config/ file and then resecuring the agent — on each host in our environment.

How do I know when I’ve changed the upload destination on every host?  I need a scorecard.

The SYSMAN schema doesn’t contain a view containing this information but you can find the data in the table sysman.mgmt_emd_ping.  This SQL query maps the agents to their upload destination using that table.

SUBSTR ( b.heartbeat_recorder_url, 0,
INSTR ( b.heartbeat_recorder_url, ‘.’ ) – 1 ),  ‘https://’ ) upload_dest,
SUBSTR ( a.target_name, 0, instr(a.target_name,’.’)-1 ) agent,
a.target_name agent_url
FROM sysman.mgmt$target a, sysman.mgmt_emd_ping b
WHERE a.target_type = ‘oracle_emd’
AND a.target_guid = b.target_guid
ORDER BY upload_dest, target_name;

BTW: You can dispense with the first two values if you like (upload_dest and agent) that I’ve used to visually simplify the result set.

I’ve pasted this query into an Information Publisher view to avoid logins to the OMR repository database and scheduled the report to update daily.  You must grant select on the table to the mgmt_view user for use in the report.


demopomsserver01 abcprod01
demopomsserver01 abcprod02
demopomsserver01 abctest02
demopomsserver01 defprod01
demopomsserver02 defprod02
demopomsserver02 deftest01
demopomsserver03 demotdb01b
demopomsserver03 demotdb04b
demopomsserver04 demotdb07b
demopomsserver04 demotdb08a
demopomsserver04 demotdb09a
demopomsserver04 demotdb09b
demopomsserver05 demotdb10a

Clean Up Your Old OEM Blackouts



All those limited duration or one-time blackouts need to be manually cleaned after they end or are stopped.  This combination of database query and EM CLI command does the job quickly and easily.

$ECHO "\n\nGathering current-state information from OEM repository\n"
sqlplus -s ${SYSMAN_CONNECT} <<EOF
SET echo OFF
SET feedback OFF verify OFF heading OFF 
SET lines 150 pages 999

SELECT DISTINCT 'delete_blackout -name="'
     ||'" -createdby="'
FROM mgmt\$blackout_history a, 
     mgmt\$blackouts b
WHERE a.blackout_guid = b.blackout_guid
AND b.status IN ('Ended', 'Stopped')
AND schedule_type = 'One Time'
AND last_end_time < SYSDATE - 1;

spool off

if [ `cat ${BLACKOUT_LIST} | wc -l` -gt 0 ]; then
 emcli login -username="SYSMAN" -password="${SYSMAN_PWD}" 2>/dev/null
 emcli sync 2>/dev/null 
 emcli argfile ${BLACKOUT_LIST}
 emcli logout


Change Ownership of Named Credentials


Changing ownership of Named Credentials is quick and easy.  I’d created a Named Credential under my own OEM account and decided to share it with SYSMAN so we could set it as the Preferred Credential for a certain class of targets across the enterprise.

While logged in as the current owner,  traverse down through Setup | Security | Named Credentials to this screen:


Highlight the Named Credential and press the Manage Access button to bring up this screen:


Press the Change button to bring a Search and Select page:


Press the Change button and you’re done!


Revoking OEM12c License Packs


Disabling a management pack is just as easy as enabling the pack in the first place.

Go to the bottom of the Setup menu and select Management Packs | Management Pack Access.


To completely disable the pack, click radio buttons

  • All Targets (Licensable targets and dependent targets)
  • Pack based Batch Update

Select the management pack from the list and move it to the right-hand panel

Press the Apply button


Verify your completion by checking the Auto Licensing Disabled List and then check the target list


Verify that targets no longer have access to the pack by clicking on

  • Licensable Targets (Licensable targets and dependent targets)
  • Target Based Pack Access

at the top of the same page.


Bounce Counter Mismatch


EM agents can end up in several different error states and each of them have their own error image.  This one popped up in my environment this morning on one agent and all of its targets:


A bounce counter mismatch (represented by the ‘N’ in the icon?) occurs when the agent has been restored from a backup on the remote host.  The agent is trying to upload metric data from timestampX but the repository contains metric data from timestampX+10.


The fix is quite simple from the EM administrator’s point of view: Resynchronize the agent.



The detailed log from the resynchronization job has a list of all the tasks involved:


The Bottom Line:  the agent is completely reset then repopulated from the repository!

Ordinarily the agent uploads its knowledge to the repository database.  This time the repository tells the agent what it should be looking for — targets, plugins, and metric extensions, and pushes that knowledge to the agent.

Resynchronization ensures that you don’t lost any metric data and aligns the collection state between the agent and the repository.  Very nice work Oracle!

In researching this issue I found a great presentation on OTN.  It covers lots of other EM management techniques but the section on agent errors is the best I found:

Listing OEM Administrators

We still use database authentication for our OEM Administrator accounts.  Every once a while (like today) that list needs to be reviewed to evaluate contract employees and others that may have changed jobs.  We use the USER_DESCRIPTION field to store people’s names in OEM so we poll the user_name and its description.

Here’s a simple query, run as SYSMAN user, that creates a list for you in SQL+

COL user_name FORMAT a20
COL user_description FORMAT a35 WRAPPED

SELECT user_name,
FROM sysman.gc_users
WHERE user_name LIKE (‘P%’)
AND user_name NOT LIKE ‘PAM%’
AND user_name NOT LIKE ‘%PAGER’
ORDER BY user_name;

Post Agent Upgrade Tasks



The typical EM agent upgrade leaves the old Oracle Homes in place on your hosts.  That includes the old home for the agent binaries and also the homes for each upgraded plug-in.

You can clean all of them up through the same Agent Upgrade Console that you used to push new agent out.  Select the same Upgrade Agents from the drop-down.


Select the Post Agent Upgrade Tasks tab to expose the Clean-up Agents functionality


Search for the Agents available for Clean-up then make your choices.  Of course, you can filter the search as you see fit.


After the selections are highlighted, hit the Submit button to create the EM Job.



Like all EM Jobs you can click through to watch the progress and to check the run-time logs (and you should)



Get every new post delivered to your Inbox.

Join 289 other followers