How to Document Upload Destinations

We’re in the process of implementing a load balancing server for our OMS cluster.  The migration itself consists of changing the REPOSITORY_URL and emdWalletSrcUrl addresses inside the $ORACLE_HOME/agent_inst/sysman/config/emd.properties file and then resecuring the agent — on each host in our environment.

How do I know when I’ve changed the upload destination on every host?  I need a scorecard.

The SYSMAN schema doesn’t contain a view containing this information but you can find the data in the table sysman.mgmt_emd_ping.  This SQL query maps the agents to their upload destination using that table.

SELECT LTRIM (
SUBSTR ( b.heartbeat_recorder_url, 0,
INSTR ( b.heartbeat_recorder_url, ‘.’ ) – 1 ),  ‘https://’ ) upload_dest,
SUBSTR ( a.target_name, 0, instr(a.target_name,’.’)-1 ) agent,
b.heartbeat_recorder_url,
a.target_name agent_url
FROM sysman.mgmt$target a, sysman.mgmt_emd_ping b
WHERE a.target_type = ‘oracle_emd’
AND a.target_guid = b.target_guid
ORDER BY upload_dest, target_name;

BTW: You can dispense with the first two values if you like (upload_dest and agent) that I’ve used to visually simplify the result set.

I’ve pasted this query into an Information Publisher view to avoid logins to the OMR repository database and scheduled the report to update daily.  You must grant select on the table to the mgmt_view user for use in the report.

GRANT SELECT ON SYSMAN.MGMT_EMD_PING TO mgmt_view;

UPLOAD_DEST AGENT HEARTBEAT_RECORDER_URL AGENT_URL
demopomsserver01 abcprod01 https://demopomsserver01.em.org:4903/empbs/upload abcprod01.em.org:3872
demopomsserver01 abcprod02 https://demopomsserver01.em.org:4903/empbs/upload abcprod02.em.org:3872
demopomsserver01 abctest02 https://demopomsserver01.em.org:4903/empbs/upload abctest02.em.org:3872
demopomsserver01 defprod01 https://demopomsserver01.em.org:4903/empbs/upload defprod01.em.org:3872
demopomsserver02 defprod02 https://demopomsserver01.em.org:4903/empbs/upload defprod02.em.org:3872
demopomsserver02 deftest01 https://demopomsserver01.em.org:4903/empbs/upload deftest01.em.org:3872
demopomsserver03 demotdb01b https://demopomsserver01.em.org:4903/empbs/upload demotdb01b.em.org:3872
demopomsserver03 demotdb04b https://demopomsserver01.em.org:4903/empbs/upload demotdb04b.em.org:3872
demopomsserver04 demotdb07b https://demopomsserver01.em.org:4903/empbs/upload demotdb07b.em.org:3872
demopomsserver04 demotdb08a https://demopomsserver01.em.org:4903/empbs/upload demotdb08a.em.org:3872
demopomsserver04 demotdb09a https://demopomsserver01.em.org:4903/empbs/upload demotdb09a.em.org:3872
demopomsserver04 demotdb09b https://demopomsserver01.em.org:4903/empbs/upload demotdb09b.em.org:3872
demopomsserver05 demotdb10a https://demopomsserver01.em.org:4903/empbs/upload demotdb10a.em.org:3872

Clean Up Your Old OEM Blackouts

Tags

,

All those limited duration or one-time blackouts need to be manually cleaned after they end or are stopped.  This combination of database query and EM CLI command does the job quickly and easily.

$ECHO "\n\nGathering current-state information from OEM repository\n"
sqlplus -s ${SYSMAN_CONNECT} <<EOF
SET echo OFF
SET feedback OFF verify OFF heading OFF 
SET lines 150 pages 999

spool ${BLACKOUT_LIST}
SELECT DISTINCT 'delete_blackout -name="'
     ||a.blackout_name
     ||'" -createdby="'
     ||b.created_by
     ||'"'
FROM mgmt\$blackout_history a, 
     mgmt\$blackouts b
WHERE a.blackout_guid = b.blackout_guid
AND b.status IN ('Ended', 'Stopped')
AND schedule_type = 'One Time'
AND last_end_time < SYSDATE - 1;

spool off
exit
EOF

if [ `cat ${BLACKOUT_LIST} | wc -l` -gt 0 ]; then
 emcli login -username="SYSMAN" -password="${SYSMAN_PWD}" 2>/dev/null
 emcli sync 2>/dev/null 
 emcli argfile ${BLACKOUT_LIST}
 emcli logout
fi

[ $BLACKOUT_LIST ] && rm -f ${BLACKOUT_LIST}
unset SYSMAN_CONNECT
unset SYSMAN_PWD

Change Ownership of Named Credentials

Tags

Changing ownership of Named Credentials is quick and easy.  I’d created a Named Credential under my own OEM account and decided to share it with SYSMAN so we could set it as the Preferred Credential for a certain class of targets across the enterprise.

While logged in as the current owner,  traverse down through Setup | Security | Named Credentials to this screen:

change_named_credential_owner01

Highlight the Named Credential and press the Manage Access button to bring up this screen:

change_named_credential_owner02

Press the Change button to bring a Search and Select page:

change_named_credential_owner04

Press the Change button and you’re done!

change_named_credential_owner05

Revoking OEM12c License Packs

Tags

Disabling a management pack is just as easy as enabling the pack in the first place.

Go to the bottom of the Setup menu and select Management Packs | Management Pack Access.

revoke_mgmtpack_access00

To completely disable the pack, click radio buttons

  • All Targets (Licensable targets and dependent targets)
  • Pack based Batch Update

Select the management pack from the list and move it to the right-hand panel

Press the Apply button

revoke_mgmtpack_access01

Verify your completion by checking the Auto Licensing Disabled List and then check the target list

revoke_mgmtpack_access02

Verify that targets no longer have access to the pack by clicking on

  • Licensable Targets (Licensable targets and dependent targets)
  • Target Based Pack Access

at the top of the same page.

revoke_mgmtpack_access04

Bounce Counter Mismatch

Tags

EM agents can end up in several different error states and each of them have their own error image.  This one popped up in my environment this morning on one agent and all of its targets:

bounce_counter_mismatch00

A bounce counter mismatch (represented by the ‘N’ in the icon?) occurs when the agent has been restored from a backup on the remote host.  The agent is trying to upload metric data from timestampX but the repository contains metric data from timestampX+10.

bounce_counter_mismatch01

The fix is quite simple from the EM administrator’s point of view: Resynchronize the agent.

bounce_counter_mismatch02

bounce_counter_mismatch03

The detailed log from the resynchronization job has a list of all the tasks involved:

bounce_counter_mismatch04

The Bottom Line:  the agent is completely reset then repopulated from the repository!

Ordinarily the agent uploads its knowledge to the repository database.  This time the repository tells the agent what it should be looking for — targets, plugins, and metric extensions, and pushes that knowledge to the agent.

Resynchronization ensures that you don’t lost any metric data and aligns the collection state between the agent and the repository.  Very nice work Oracle!

In researching this issue I found a great presentation on OTN.  It covers lots of other EM management techniques but the section on agent errors is the best I found:

http://www.oracle.com/technetwork/oem/enterprise-manager/con8244-manage-the-manager-2332598.pdf

Listing OEM Administrators

We still use database authentication for our OEM Administrator accounts.  Every once a while (like today) that list needs to be reviewed to evaluate contract employees and others that may have changed jobs.  We use the USER_DESCRIPTION field to store people’s names in OEM so we poll the user_name and its description.

Here’s a simple query, run as SYSMAN user, that creates a list for you in SQL+

COL user_name FORMAT a20
COL user_description FORMAT a35 WRAPPED
SET FEEDBACK OFF LINES 150 PAGES 999

SELECT user_name,
user_description
FROM sysman.gc_users
WHERE user_name LIKE (‘P%’)
AND user_name NOT LIKE ‘PAM%’
AND user_name NOT LIKE ‘%PAGER’
ORDER BY user_name;

Post Agent Upgrade Tasks

Tags

,

The typical EM agent upgrade leaves the old Oracle Homes in place on your hosts.  That includes the old home for the agent binaries and also the homes for each upgraded plug-in.

You can clean all of them up through the same Agent Upgrade Console that you used to push new agent out.  Select the same Upgrade Agents from the drop-down.

post_upgrade01

Select the Post Agent Upgrade Tasks tab to expose the Clean-up Agents functionality

post_upgrade02

Search for the Agents available for Clean-up then make your choices.  Of course, you can filter the search as you see fit.

post_upgrade03

After the selections are highlighted, hit the Submit button to create the EM Job.

post_upgrade04

post_upgrade05

Like all EM Jobs you can click through to watch the progress and to check the run-time logs (and you should)

post_upgrade06

Targets in Perpetual Pending State

Tags

, ,

After my issues with the patch bundle some of my database targets ended up in an odd ‘Status Pending’ state.  I received this guidance from RachelB on the OTN site for OEM Community at

https://community.oracle.com/community/support/enterprise_manager/enterprise_manager_generic

“The database system target is one of those targets that has it’s status evaluated in the repository.  Normally if the status is not changing, it’s a good idea to check the internal repostory (dba scheduler job) which is responsible for this.

“I have seen this kind of behaviour when the repository job responsible for status evaluation has a problem – you can check like this:-

  • Go to setup/manage cloud control/repository
  • Look at the repository scheduler job status – remember to scroll down this to view all the jobs
  • Check the “Composite Target Availability Computation” job – in particular check the date/time – to check it’s running on time
  • If this job has any problems,
    • Drill down to the repository database
    • Go to Administration/Oracle Scheduler/jobs
    • Look for job EM_REPOS_SEV_EVAL
    • If it’s not running on time stop it
  • After getting  a message that it’s stopped, click on ‘run now’ to re-submit it.”

It worked like a charm!

Monsters Ate My Repository Database

Tags

,

I applied the December Bundle Patch for OEM 12.1.0.4 earlier this week and kicked off a series of agent upgrades.  Then monsters at my repository database.

Bug 19458672 caused the newly upgraded agents to hang and to consume all the CPU cycles on the repository database host.  It happened during the final step of registering each agent with their new release number when executing:

BEGIN
em_policy.handle_type_meta_ver_change ( :1);
END;

The one-off patch  “Patch 19458672: High contention from metaver upgrade callback if default metric keys don’t exist” described the problem and solved the issue when applied.

But it begs the question:  If the monthly Bundle Patches are cumulative, why wasn’t this tremendously disruptive patch included?

AdminServer shows Down after IP Change

Tags

, , ,

We recently changed IP addresses on hosts serving up LDAP/OID.  After the IP change the OID targets properly showed Up status in my OEM console but the WLS Admin Servers were bright RED.

The local 11g OEM running on those web servers showed the correct status and I could log into those Admin Servers directly, so something had to be wrong in OEM 12c.

Each EM agent home contains an XML file named targets.xml in ../agent_inst/sysman/emd directory.  Target definitions are stored there, including IP addresses for selected properties.

The fix was rather simple once I knew where to look.

  1. Move into the ..sysman/emd directory
  2. Make a backup copy of targets.xml
  3. Open targets.xml with vim
  4. Replace the old IP address with the new one
  5. Bounce your EM agent and run an emctl upload agent

The properties that needed changing on my servers were:

<Property NAME=”ListenAddress” VALUE=”99.999.9.99″/>
<Property NAME=”serviceURL” VALUE=”various services:t3://99.999.9.99:7001/…

Sed makes quick work of it inside vim:

:%s/old_IP/new_IP/g

For instance:

:%s/10.123.4.56/10.130.4.56/g

There’s one more thing to verify – the Java Keystore for agent must contain the certificate from your WLS.

Refer to http://docs.oracle.com/cd/E24628_01/install.121/e24215/weblogic.htm#GSSOA9597

“When using Enterprise Manager version 12.1 and a Secure Socket Layer (SSL) protocol to discover and monitor WebLogic servers, the Management Agent must be able to trust the server before it can establish a secure communication link. The Agent maintains a Java Keystore (JKS) truststore containing certificates of Certification Authorities (CAs) that it can trust when establishing a secure connection. The Agent comes with nine well-known CA certificates.”

You do it like this (the default password is ‘welcome’):

> emctl secure add_trust_cert_to_jks -password welcome

Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

Message : keytool error: java.lang.Exception: Certificate not imported, alias <wlscertgenca> already exists
ExitStatus: SUCCESS
Message : keytool error: java.lang.Exception: Certificate not imported, alias <wlscertgencab> already exists
ExitStatus: SUCCESS

I’d already done this server, as you can tell.

Happy New Year!

Follow

Get every new post delivered to your Inbox.

Join 270 other followers