Change Ownership of Named Credentials


Changing ownership of Named Credentials is quick and easy.  I’d created a Named Credential under my own OEM account and decided to share it with SYSMAN so we could set it as the Preferred Credential for a certain class of targets across the enterprise.

While logged in as the current owner,  traverse down through Setup | Security | Named Credentials to this screen:


Highlight the Named Credential and press the Manage Access button to bring up this screen:


Press the Change button to bring a Search and Select page:


Press the Change button and you’re done!


Revoking OEM12c License Packs


Disabling a management pack is just as easy as enabling the pack in the first place.

Go to the bottom of the Setup menu and select Management Packs | Management Pack Access.


To completely disable the pack, click radio buttons

  • All Targets (Licensable targets and dependent targets)
  • Pack based Batch Update

Select the management pack from the list and move it to the right-hand panel

Press the Apply button


Verify your completion by checking the Auto Licensing Disabled List and then check the target list


Verify that targets no longer have access to the pack by clicking on

  • Licensable Targets (Licensable targets and dependent targets)
  • Target Based Pack Access

at the top of the same page.


Bounce Counter Mismatch


EM agents can end up in several different error states and each of them have their own error image.  This one popped up in my environment this morning on one agent and all of its targets:


A bounce counter mismatch (represented by the ‘N’ in the icon?) occurs when the agent has been restored from a backup on the remote host.  The agent is trying to upload metric data from timestampX but the repository contains metric data from timestampX+10.


The fix is quite simple from the EM administrator’s point of view: Resynchronize the agent.



The detailed log from the resynchronization job has a list of all the tasks involved:


The Bottom Line:  the agent is completely reset then repopulated from the repository!

Ordinarily the agent uploads its knowledge to the repository database.  This time the repository tells the agent what it should be looking for — targets, plugins, and metric extensions, and pushes that knowledge to the agent.

Resynchronization ensures that you don’t lost any metric data and aligns the collection state between the agent and the repository.  Very nice work Oracle!

In researching this issue I found a great presentation on OTN.  It covers lots of other EM management techniques but the section on agent errors is the best I found:

Listing OEM Administrators

We still use database authentication for our OEM Administrator accounts.  Every once a while (like today) that list needs to be reviewed to evaluate contract employees and others that may have changed jobs.  We use the USER_DESCRIPTION field to store people’s names in OEM so we poll the user_name and its description.

Here’s a simple query, run as SYSMAN user, that creates a list for you in SQL+

COL user_name FORMAT a20
COL user_description FORMAT a35 WRAPPED

SELECT user_name,
FROM sysman.gc_users
WHERE user_name LIKE (‘P%’)
AND user_name NOT LIKE ‘PAM%’
AND user_name NOT LIKE ‘%PAGER’
ORDER BY user_name;

Post Agent Upgrade Tasks



The typical EM agent upgrade leaves the old Oracle Homes in place on your hosts.  That includes the old home for the agent binaries and also the homes for each upgraded plug-in.

You can clean all of them up through the same Agent Upgrade Console that you used to push new agent out.  Select the same Upgrade Agents from the drop-down.


Select the Post Agent Upgrade Tasks tab to expose the Clean-up Agents functionality


Search for the Agents available for Clean-up then make your choices.  Of course, you can filter the search as you see fit.


After the selections are highlighted, hit the Submit button to create the EM Job.



Like all EM Jobs you can click through to watch the progress and to check the run-time logs (and you should)


Targets in Perpetual Pending State


, ,

After my issues with the patch bundle some of my database targets ended up in an odd ‘Status Pending’ state.  I received this guidance from RachelB on the OTN site for OEM Community at

“The database system target is one of those targets that has it’s status evaluated in the repository.  Normally if the status is not changing, it’s a good idea to check the internal repostory (dba scheduler job) which is responsible for this.

“I have seen this kind of behaviour when the repository job responsible for status evaluation has a problem – you can check like this:-

  • Go to setup/manage cloud control/repository
  • Look at the repository scheduler job status – remember to scroll down this to view all the jobs
  • Check the “Composite Target Availability Computation” job – in particular check the date/time – to check it’s running on time
  • If this job has any problems,
    • Drill down to the repository database
    • Go to Administration/Oracle Scheduler/jobs
    • Look for job EM_REPOS_SEV_EVAL
    • If it’s not running on time stop it
  • After getting  a message that it’s stopped, click on ‘run now’ to re-submit it.”

It worked like a charm!

Monsters Ate My Repository Database



I applied the December Bundle Patch for OEM earlier this week and kicked off a series of agent upgrades.  Then monsters at my repository database.

Bug 19458672 caused the newly upgraded agents to hang and to consume all the CPU cycles on the repository database host.  It happened during the final step of registering each agent with their new release number when executing:

em_policy.handle_type_meta_ver_change ( :1);

The one-off patch  “Patch 19458672: High contention from metaver upgrade callback if default metric keys don’t exist” described the problem and solved the issue when applied.

But it begs the question:  If the monthly Bundle Patches are cumulative, why wasn’t this tremendously disruptive patch included?

AdminServer shows Down after IP Change


, , ,

We recently changed IP addresses on hosts serving up LDAP/OID.  After the IP change the OID targets properly showed Up status in my OEM console but the WLS Admin Servers were bright RED.

The local 11g OEM running on those web servers showed the correct status and I could log into those Admin Servers directly, so something had to be wrong in OEM 12c.

Each EM agent home contains an XML file named targets.xml in ../agent_inst/sysman/emd directory.  Target definitions are stored there, including IP addresses for selected properties.

The fix was rather simple once I knew where to look.

  1. Move into the ..sysman/emd directory
  2. Make a backup copy of targets.xml
  3. Open targets.xml with vim
  4. Replace the old IP address with the new one
  5. Bounce your EM agent and run an emctl upload agent

The properties that needed changing on my servers were:

<Property NAME=”ListenAddress” VALUE=”99.999.9.99″/>
<Property NAME=”serviceURL” VALUE=”various services:t3://99.999.9.99:7001/…

Sed makes quick work of it inside vim:


For instance:


There’s one more thing to verify – the Java Keystore for agent must contain the certificate from your WLS.

Refer to

“When using Enterprise Manager version 12.1 and a Secure Socket Layer (SSL) protocol to discover and monitor WebLogic servers, the Management Agent must be able to trust the server before it can establish a secure communication link. The Agent maintains a Java Keystore (JKS) truststore containing certificates of Certification Authorities (CAs) that it can trust when establishing a secure connection. The Agent comes with nine well-known CA certificates.”

You do it like this (the default password is ‘welcome’):

> emctl secure add_trust_cert_to_jks -password welcome

Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

Message : keytool error: java.lang.Exception: Certificate not imported, alias <wlscertgenca> already exists
ExitStatus: SUCCESS
Message : keytool error: java.lang.Exception: Certificate not imported, alias <wlscertgencab> already exists
ExitStatus: SUCCESS

I’d already done this server, as you can tell.

Happy New Year!

emcli manage_agent_partnership



OEM 12c Release 4 has several new EM CLI verbs, including manage_agent_partnership.

From the on-line help:

“A partner agent is an agent that, in addition to its other functions, is assigned to another agent as its ‘partner’ in order to remotely monitor the availability of that agent and its host.”

Today I noticed that a totally unrelated agent was listed as the partner agent for one of my hosts.  Since partner agents pick up metric collection when the agent on its partner is unavailable, it was something that needed fixing.

The sequence is straightforward:

  1. Log into EM CLI and run emcli sync (like you always do, right?)
  2. Remove the old partnership
  3. Add a new partnership
  4. Verify on the Agent’s console page

> emcli manage_agent_partnership


> is already partnered with

> emcli manage_agent_partnership


> Manage Agent Partnership operation completed successfully

> emcli manage_agent_partnership


> Manage Agent Partnership operation completed successfully

NWOUG Women in Technology Scholarship



This year the Northwest Oracle User Group created a scholarship for an outstanding female student in the Portland State University Computer Engineering  Department.  The criteria was straight-forward.  In addition to being an outstanding student we asked all the applicants to answer this question:

“How does employee participation in professional communities add value for their employers?”

The arguments for promoting WIT are well-documented elsewhere, and it had a bearing on our decision.   But we also did it with an eye toward our future:   Through the scholarship we’re developing a relationship with a potential decision-maker when the student enters the workforce.  By associating Oracle and user groups with IT success the scholarship recipients may become advocates at their companies.

Here’s the process, so you can do with your user group:

  • The board needs to capture the criteria and implementation plan in a formal policy.  We decided to leave the scholarship amount to the discretion of the board inside the policy.
  • Set the scholarship amount for the current school year and select a college to approach.
  • It only took a couple of emails to get introduced the development department for the PSU Engineering Department.  After that it was smooth sailing.
  • The development office deals with this sort of request all the time.  In our case they did all paperwork, performed all of the interviews, and gave us a list of qualified candidates and attached their essays.  The choice was then up to us.

It was almost that simple.

It is illegal for a college to discriminate on the basis of gender, so while they acknowledged our preference they were also required to open the scholarship application to men.  The final decision was ours to make based on the school’s recommendation and the content/quality of the student essays.   You could tell who’d done some research.

Final piece of advice:  Make the decision early in the calendar year so the college has time to work your scholarship into the regular application work flow.  It’s easier on the college staff and your scholarship will get better visibility among the students.

Let me know if you have any questions or comments.


Get every new post delivered to your Inbox.

Join 247 other followers