Tag Archives: stuck

R12 e-Business Suite and OEM Monitoring – Oracle Spins Freezes

Every so often, system load on an e-Business Suite instance ramps up and response time to users starts climbing, often resulting in user observed errors such as:

  • FRM-92100 Your connect to the server has been interrupted
  • FRM-92102 A network error has occurred

    FRM-92102 Forms Error R12 EBS
    That dreaded FRM-91201 / FRM-91200 error causing you to restart your session.

Or sometimes, the screen just freezes (aka spins, stops, is broken, stuck, motionless, looks like a screen saver,can’t do anything, won’t work, froze-up, etc.) and the person has to close their browser, or even shut-down their workstation and restart.

It's simply not doing anything - Nothing to see here, just move your cursor around. And wait... and wait.
It’s simply not doing anything – Nothing to see here, just move your cursor around. And wait… and wait.

Old technology often barks with unrelated error messages to the actual cause.  If there’s a lot going on with concurrent requests, or interfaces, or analytic extracts running, the front-end response-time slows down, sometimes sufficiently to trigger these kinds of Form errors, even though technically there was no interruption to the network connectivity, either between the hosts, nor the workstation and the middle-tier application server.

However, on the database, the user-experience can be seen, although not necessarily in the place you might expect.  OEM  had introduced it’s Adaptive Metric Thresholds technology back in OEM 11g (in a slightly different place than in 12c (in Oracle Management Server/OMS 12.1.0.4.0).  In OEM 11g, they were a link under the AWR Baseline Reports page.

OEM 11g AWR Baselines Page
See the Baseline Metric Thresholds link at the bottom.

In OEM 12c, you’ll find them under

Targets -> Database -> Peformance -> Adaptive Thresholds -> Baseline Metric Thresholds -> Edit Thresholds:

OEM 12c Baseline Metric Thresholds
Where those adaptive metric thresholds moved in 12c.

 

 

On this page and in the list of Baseline Metrics, when you click into them, you can access the trending statistics being gathered for each metric.  Many times this will provide direct insight into what a user experiences as the “the system is frozen” translates into “the back-end database response time is incredibly bad.”

OEM 12c Baseline Metric Response Time per Transaction vs. Baseline
See the spikes around 7AM and 11:30AM? Those are being associated with “System Froze” reports.

 

In the example here, the database experienced a dramatic slow-down in response almost 5 to 10 times slower than usual, which only lasted a few seconds. But that can be enough to show up in many users’ sessions who might have just kicked off a query, or were trying to save something.  Based upon the information gathered, we set the Warning and Critical thresholds to 1500ms and 2000ms respectively to start sending e-mail alert notifications upon breach of the levels. If the settings are left at “None”, no incident would be raised, and thus, no notification would be sent.

If you’re experiencing odd transient outages or sluggish behavior that defies the normal AWR and ADDM snapshot analysis, go take a look at what OEM has been gathering in the background over time and see if the statistics correlate to any of your issues.  There’s value in that data. Just mine it.

Advertisements

2013-March Shinnyo-en Buddhism Introductory Podcast – Going With The Flow

2013-March Shinnyo-en Buddhism Introductory Podcast – Going With The Flow

Subscribe to this Podcast (RSS) or iTunes

Watching How Traffic Gridlock Forms
Harmonious Paths in Disaster
The Bodhisattva Vow

Traffic patterns, much like check-out lines at a store are fascinating to observe – people always jostling for a better position, racing ahead only to get stuck as that lane eventually slows to a crawl again. Having a higher-view, similar to what truckers can see, you start to see the larger picture of traffic. On some freeways, for whatever reason, sometimes the right-side slow lanes go faster than the left-side passing ones. Motorcyclists tend to be more aware of their surroundings, so we notice when the little red sports car that sped up quickly to race ahead, gets blithely stuck in slower traffic and fades to the rear. We notice when there’s an unusually slow driver in the far left lane causing traffic from behind to have to move right to pass.

We also notice how the large truck drivers, sentenced by regulation to an existence imprisoned within the right two lanes, also learn to drive with consistency of both throttle and path. They don’t tend to change lanes, and try to maintain a steady pace of forward momentum. Having so much mass to move, does not allow for rapid fluctuations in speed and direction, so those drivers have to plan well ahead in order to stay safe, and keep from endangering others. These professional drivers make decisions to change paths because of safety and overall-efficiency. Opening wide the throttle to jump into a temporary opening might decrease a few seconds of the trip time, but the sacrifice is an above-average burn of precious (and these days, costly) fuel. Too many snap decisions and there might not be enough in the tank to complete the trip without stopping for more fuel – adding additional delays to the overall arrival time.

Every time someone unexpectedly changes lanes, whether without signalling, or abruptly changing from one lane to another, those people displaced usually end up slowing down slightly (to keep themselves safe), and those in-front glance in their rear mirrors wondering why they are being suddenly tailgated. The people to the sides spend a few thought moments thinking to themselves “how rude!” or other distractions.  Collectively the single decision of one person to try and jump a little farther ahead of the others, ends up having collateral energy expenditures on all those individuals around them, and inevitably, that lack of focus and concentration also results in slightly slower progress.

Buddhism observes many such paths and interactions: water flowing along a stream, winds flowing through and around forests, birds migrating from place to place – each having a starting point and a destination. Each kind of energy makes its best effort to get to its ending point but deviates its path only for natural drives: danger, hunger, thirst, and fatigue. Ultimately the path of least resistance is always sought, to conserve energy.  Water gains flow and volume when it moves as a whole, widening its path and straightening its course. Air becomes powerful as it goes faster and forms streams of continuous flow. Birds in flocks expend less energy to get to their destination when they take turns leading and following each other, staying within the leader’s wake, taking the least amount of energy to maintain formation flight.

March 11 marks the second anniversary of the Great East Japan Earthquake, and the third memorial service for its victims (In Japan, a funeral is counted as the first of the yearly memorial services, so the third memorial service is conducted on the second anniversary.) The torrential water that flooded most of the Tohoku region followed its own path of least resistance. Residents noted the risks associated with building directly in the path of the ocean’s own tides, and recommended higher altitude and raised platform construction be incorporated during the reconstruction. Living in connection with the ever-present tides and constructing flexible buildings that can sway and move with the quaking earth are similar examples of harmonious paths.

Two thousand, five hundred years ago, at the Buddha’s final moment of nirvana, Chunda, the lay follower, showed us in his purity how to truly accomplish dana paramita, or the perfection of giving. Shakyamuni accepted Chunda’s sincere offering and reassured him that he’d be there whenever Chunda needed him. “There is no need to grieve over my death,” he said, “I have entered a state of nirvana. I am in a place of eternal bliss. Listen well, with a sincere attitude. I will explain the bodhisattva vow [to endeavor for the sake of all sentient beings] to you, so that serene and tranquil bliss can be attained equally by all sentient beings. You have now listened to the truth of my ever present tathagata nature; etch this truth and what you have heard into your mind, and train accordingly.”

And so, returning to our traffic analogy, imagine what traffic would be like if the very front person on every lane were concerned about holding up traffic flow for everyone behind them. And each person in-turn was focused on the safety and efficacy of each other driver’s capacity to reach their destination safely and quickly.  And even the last person was conscientious about lagging too far behind, lest someone else follow behind them.  Races go fast because each individual has the same objective – the finish line. A careless and untrained move by any individual can lead to disastrous results for everyone participating, so they each strive to maximize their potential, which results in a collective increase in effort and effectiveness for everyone.