How to Document Upload Destinations

We’re in the process of implementing a load balancing server for our OMS cluster.  The migration itself consists of changing the REPOSITORY_URL and emdWalletSrcUrl addresses inside the $ORACLE_HOME/agent_inst/sysman/config/ file and then resecuring the agent — on each host in our environment.

How do I know when I’ve changed the upload destination on every host?  I need a scorecard.

The SYSMAN schema doesn’t contain a view containing this information but you can find the data in the table sysman.mgmt_emd_ping.  This SQL query maps the agents to their upload destination using that table.

SUBSTR ( b.heartbeat_recorder_url, 0,
INSTR ( b.heartbeat_recorder_url, ‘.’ ) – 1 ),  ‘https://’ ) upload_dest,
SUBSTR ( a.target_name, 0, instr(a.target_name,’.’)-1 ) agent,
a.target_name agent_url
FROM sysman.mgmt$target a, sysman.mgmt_emd_ping b
WHERE a.target_type = ‘oracle_emd’
AND a.target_guid = b.target_guid
ORDER BY upload_dest, target_name;

BTW: You can dispense with the first two values if you like (upload_dest and agent) that I’ve used to visually simplify the result set.

I’ve pasted this query into an Information Publisher view to avoid logins to the OMR repository database and scheduled the report to update daily.  You must grant select on the table to the mgmt_view user for use in the report.


demopomsserver01 abcprod01
demopomsserver01 abcprod02
demopomsserver01 abctest02
demopomsserver01 defprod01
demopomsserver02 defprod02
demopomsserver02 deftest01
demopomsserver03 demotdb01b
demopomsserver03 demotdb04b
demopomsserver04 demotdb07b
demopomsserver04 demotdb08a
demopomsserver04 demotdb09a
demopomsserver04 demotdb09b
demopomsserver05 demotdb10a

Clean Up Your Old OEM Blackouts



All those limited duration or one-time blackouts need to be manually cleaned after they end or are stopped.  This combination of database query and EM CLI command does the job quickly and easily.

$ECHO "\n\nGathering current-state information from OEM repository\n"
sqlplus -s ${SYSMAN_CONNECT} <<EOF
SET echo OFF
SET feedback OFF verify OFF heading OFF 
SET lines 150 pages 999

SELECT DISTINCT 'delete_blackout -name="'
     ||'" -createdby="'
FROM mgmt\$blackout_history a, 
     mgmt\$blackouts b
WHERE a.blackout_guid = b.blackout_guid
AND b.status IN ('Ended', 'Stopped')
AND schedule_type = 'One Time'
AND last_end_time < SYSDATE - 1;

spool off

if [ `cat ${BLACKOUT_LIST} | wc -l` -gt 0 ]; then
 emcli login -username="SYSMAN" -password="${SYSMAN_PWD}" 2>/dev/null
 emcli sync 2>/dev/null 
 emcli argfile ${BLACKOUT_LIST}
 emcli logout


Change Ownership of Named Credentials


Changing ownership of Named Credentials is quick and easy.  I’d created a Named Credential under my own OEM account and decided to share it with SYSMAN so we could set it as the Preferred Credential for a certain class of targets across the enterprise.

While logged in as the current owner,  traverse down through Setup | Security | Named Credentials to this screen:


Highlight the Named Credential and press the Manage Access button to bring up this screen:


Press the Change button to bring a Search and Select page:


Press the Change button and you’re done!


Revoking OEM12c License Packs


Disabling a management pack is just as easy as enabling the pack in the first place.

Go to the bottom of the Setup menu and select Management Packs | Management Pack Access.


To completely disable the pack, click radio buttons

  • All Targets (Licensable targets and dependent targets)
  • Pack based Batch Update

Select the management pack from the list and move it to the right-hand panel

Press the Apply button


Verify your completion by checking the Auto Licensing Disabled List and then check the target list


Verify that targets no longer have access to the pack by clicking on

  • Licensable Targets (Licensable targets and dependent targets)
  • Target Based Pack Access

at the top of the same page.


Bounce Counter Mismatch


EM agents can end up in several different error states and each of them have their own error image.  This one popped up in my environment this morning on one agent and all of its targets:


A bounce counter mismatch (represented by the ‘N’ in the icon?) occurs when the agent has been restored from a backup on the remote host.  The agent is trying to upload metric data from timestampX but the repository contains metric data from timestampX+10.


The fix is quite simple from the EM administrator’s point of view: Resynchronize the agent.



The detailed log from the resynchronization job has a list of all the tasks involved:


The Bottom Line:  the agent is completely reset then repopulated from the repository!

Ordinarily the agent uploads its knowledge to the repository database.  This time the repository tells the agent what it should be looking for — targets, plugins, and metric extensions, and pushes that knowledge to the agent.

Resynchronization ensures that you don’t lost any metric data and aligns the collection state between the agent and the repository.  Very nice work Oracle!

In researching this issue I found a great presentation on OTN.  It covers lots of other EM management techniques but the section on agent errors is the best I found:

Listing OEM Administrators

We still use database authentication for our OEM Administrator accounts.  Every once a while (like today) that list needs to be reviewed to evaluate contract employees and others that may have changed jobs.  We use the USER_DESCRIPTION field to store people’s names in OEM so we poll the user_name and its description.

Here’s a simple query, run as SYSMAN user, that creates a list for you in SQL+

COL user_name FORMAT a20
COL user_description FORMAT a35 WRAPPED

SELECT user_name,
FROM sysman.gc_users
WHERE user_name LIKE (‘P%’)
AND user_name NOT LIKE ‘PAM%’
AND user_name NOT LIKE ‘%PAGER’
ORDER BY user_name;

Post Agent Upgrade Tasks



The typical EM agent upgrade leaves the old Oracle Homes in place on your hosts.  That includes the old home for the agent binaries and also the homes for each upgraded plug-in.

You can clean all of them up through the same Agent Upgrade Console that you used to push new agent out.  Select the same Upgrade Agents from the drop-down.


Select the Post Agent Upgrade Tasks tab to expose the Clean-up Agents functionality


Search for the Agents available for Clean-up then make your choices.  Of course, you can filter the search as you see fit.


After the selections are highlighted, hit the Submit button to create the EM Job.



Like all EM Jobs you can click through to watch the progress and to check the run-time logs (and you should)


Targets in Perpetual Pending State


, ,

After my issues with the patch bundle some of my database targets ended up in an odd ‘Status Pending’ state.  I received this guidance from RachelB on the OTN site for OEM Community at

“The database system target is one of those targets that has it’s status evaluated in the repository.  Normally if the status is not changing, it’s a good idea to check the internal repostory (dba scheduler job) which is responsible for this.

“I have seen this kind of behaviour when the repository job responsible for status evaluation has a problem – you can check like this:-

  • Go to setup/manage cloud control/repository
  • Look at the repository scheduler job status – remember to scroll down this to view all the jobs
  • Check the “Composite Target Availability Computation” job – in particular check the date/time – to check it’s running on time
  • If this job has any problems,
    • Drill down to the repository database
    • Go to Administration/Oracle Scheduler/jobs
    • Look for job EM_REPOS_SEV_EVAL
    • If it’s not running on time stop it
  • After getting  a message that it’s stopped, click on ‘run now’ to re-submit it.”

It worked like a charm!

Monsters Ate My Repository Database



I applied the December Bundle Patch for OEM earlier this week and kicked off a series of agent upgrades.  Then monsters at my repository database.

Bug 19458672 caused the newly upgraded agents to hang and to consume all the CPU cycles on the repository database host.  It happened during the final step of registering each agent with their new release number when executing:

em_policy.handle_type_meta_ver_change ( :1);

The one-off patch  “Patch 19458672: High contention from metaver upgrade callback if default metric keys don’t exist” described the problem and solved the issue when applied.

But it begs the question:  If the monthly Bundle Patches are cumulative, why wasn’t this tremendously disruptive patch included?

AdminServer shows Down after IP Change


, , ,

We recently changed IP addresses on hosts serving up LDAP/OID.  After the IP change the OID targets properly showed Up status in my OEM console but the WLS Admin Servers were bright RED.

The local 11g OEM running on those web servers showed the correct status and I could log into those Admin Servers directly, so something had to be wrong in OEM 12c.

Each EM agent home contains an XML file named targets.xml in ../agent_inst/sysman/emd directory.  Target definitions are stored there, including IP addresses for selected properties.

The fix was rather simple once I knew where to look.

  1. Move into the ..sysman/emd directory
  2. Make a backup copy of targets.xml
  3. Open targets.xml with vim
  4. Replace the old IP address with the new one
  5. Bounce your EM agent and run an emctl upload agent

The properties that needed changing on my servers were:

<Property NAME=”ListenAddress” VALUE=”99.999.9.99″/>
<Property NAME=”serviceURL” VALUE=”various services:t3://99.999.9.99:7001/…

Sed makes quick work of it inside vim:


For instance:


There’s one more thing to verify – the Java Keystore for agent must contain the certificate from your WLS.

Refer to

“When using Enterprise Manager version 12.1 and a Secure Socket Layer (SSL) protocol to discover and monitor WebLogic servers, the Management Agent must be able to trust the server before it can establish a secure communication link. The Agent maintains a Java Keystore (JKS) truststore containing certificates of Certification Authorities (CAs) that it can trust when establishing a secure connection. The Agent comes with nine well-known CA certificates.”

You do it like this (the default password is ‘welcome’):

> emctl secure add_trust_cert_to_jks -password welcome

Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

Message : keytool error: java.lang.Exception: Certificate not imported, alias <wlscertgenca> already exists
ExitStatus: SUCCESS
Message : keytool error: java.lang.Exception: Certificate not imported, alias <wlscertgencab> already exists
ExitStatus: SUCCESS

I’d already done this server, as you can tell.

Happy New Year!


Get every new post delivered to your Inbox.

Join 270 other followers