IT Blog

Tag: UCSM

UCS: UCS Manager not accessible, (SWITCHOVER IN PROGRESS) (mgmt services state: INVALID) (HA NOT READY)

by on Jun.14, 2013, under Cisco, UCS, Vblock

UCSM is not accessible when login into UCS cli and running show cluster extended-state command you receive the following:

ucs01-B# show cluster extended-state
Cluster Id: 0xe123456789123456-0xac12345678987456

Start time: Mon Apr 8 00:00:16 2013
Last election time: Tue Apr 22 18:50:00 2013

B: UP, SUBORDINATE, (Management services: SWITCHOVER IN PROGRESS)
A: UP, PRIMARY, (Management services: SWITCHOVER IN PROGRESS)

B: memb state UP, lead state SUBORDINATE, mgmt services state: INVALID
A: memb state UP, lead state PRIMARY, mgmt services state: INVALID
heartbeat state PRIMARY_OK

INTERNAL NETWORK INTERFACES:
eth1, UP
eth2, UP

HA NOT READY
Management services: switchover in progress on local Fabric Interconnect
Detailed state of the device selected for HA storage:
Chassis 1, serial: FOX12345678, state: active
Chassis 2, serial: FOX12345678, state: active

As we can see from the error the switchover is in progress but management services are not running on any of the FI’s so switchover cannot complete and this is the reason why you cannot access UCS Manager.
The blades running on this UCS infrastructure are not affected and should be running fine.

To fix the problem you need to reboot both Fabric Interconnects, one Fabric Interconnect at a time.

  1. SSH to one of the Fabric Interconnects and type:
    connect local-mgmt
    reboot
  2. Wait until the fabric interconnect reboots it can take 20-30 minutes.
  3. Once it rebooted you should be able to open UCSM
  4. In UCSM verify that the fabric that was rebooted has fully came up
  5. SSH to the second Fabric Interconnect and reboot it
  6. After Fabric Interconnect is up, SSH to UCSM IP and run:
    show cluster extended-state
  7. Verify that the cluster is in good state.
2 Comments :, , , , more...

UCS: F0382 thermal-problem

by on Feb.06, 2013, under Cisco, UCS

This is a quick post for simple error that you might see in Cisco UCSM.
t1
Affected object: sys/chassis-2/fan-module-1-2
Description: Fan module 1-2 in chassis 2 temperature: upper-critical
Cause: thermal-problem
Code: F0382

This may indicate that there is hardware problem with the fan and it needs to be replaced but quite often it indicates logical problem. There are a few bugs depending on the version that UCS is running that might cause this behaviour.
As we can see from the error the problem is with the fan in chassis 2.
If you click on the blue text it will open a new window and you’ll be able to see which fan it is.
t2

The first thing you can try is to reseat the fan and see if the error goes away. If it does not then it most likely hardware issue.
In this case the error has cleared.
This error might reappear in couple day/weeks or months or it could happen on the other fan. There is a known I2C bug in UCS. To clear i2c bus it is advised to reseat all fans, PSUs and IOmodules(one at a time).
The bug was fixed in late 1.4(3*) version but was reintroduced again in 2.0 version. It should be fixed in latest version.
I UCS system affected by this bug than you still need to reseat components as the upgrade alone will not clear the I2C bus.
IMO it would be best to reseat them before upgrading as at least you’ll be upgrading a healthy system.

1 Comment :, , , , , more...

UCS: configuration-failed; Code: F0170; connection-placement; There are not enough resources overall

by on Dec.02, 2012, under Cisco, UCS

Here is an interesting issue that I ran into with Cisco UCS blade.
I needed to move service profile from one blade to another. This is a process that should not give any problems but it did. Dissociation worked fine, but when I tried to associate the same profile with diferent blade I ran into problems.

The first thing I noticed is Config Failure error in Status:

The Configuration error was:
connection-placement
There are not enough resources overall

Not enough vHBAs available
Not enough cNICs available (continue reading…)

1 Comment :, , , , , , , , more...

UCS: Blade is stuck on discovery after UCS firmware upgrade (unidentified FRU)

by on Nov.12, 2012, under Cisco, UCS

Here is pretty common problem in UCS 2.0 release.
At any stage of UCS upgrade  one or more blades go into discovery mode and never finishes it. Depending on the version they can get stuck at any percentage but usually between 4% and 40%.
Most of the time a corruption occurs in SEEPROM of  M81kr CNA card because of this corruption checksum fails and UCS cannot recognize the mezzanine card any longer and this prevent Discovery from finishing.
You can see the following errors when this happens:
Configuration Error: adaptor-inoperable. Discovery State: Insufficiently Equipped.
Adapter 1 in server 1/1 has unidentified FRU 

There are multiple Cisco bugs for this issue CSCub16754, CSCty34034, CSCub48862, CSCub99354 and I’ve seen it happening on 2.0(1q), 2.0(2r), 2.0(3a) releases.
Unfortunately the issue is not fixed and there is no workaround. The good thing is that if this occurs the fix is pretty simple and quick and no hardware replacement is needed but only Cisco TAC can fix this or whoever has access to their internal resources.

To verify if corruption occurred you can do the following:

  1. SSH to UCSM IP
  2. Enter connect cimc x/y (Chassis/Blade)
  3. Enter mezz1fru on the versions starting from 2.0(3a) you need to enter fru
    If corruption has occurred the last line of the output will show something like
    ‘Checksum Failed For: Board Area!’

The other method to check is to look at the logs. (continue reading…)

Leave a Comment :, , , , , , , , , more...

UCS: How to update Capability Catalog in UCS Manager

by on Nov.09, 2012, under Cisco, UCS

Here is a guide how to update the Capability Catalog in UCS Manager. Capability Catalog is updated every time you upgrade UCS firmware but you might need to update it separately when a new hardware is added to UCS infrastructure and upgrading the whole UCS is not possible.

1. Login into UCS manager
2. Select Admin tab and change the Filter to Capability Catalog

3. Verify the version of Capability Catalog that is currently installed

(continue reading…)

Leave a Comment :, , , more...