IT Blog

UCS

UCS: UCS Manager not accessible, (SWITCHOVER IN PROGRESS) (mgmt services state: INVALID) (HA NOT READY)

by on Jun.14, 2013, under Cisco, UCS, Vblock

UCSM is not accessible when login into UCS cli and running show cluster extended-state command you receive the following:

ucs01-B# show cluster extended-state
Cluster Id: 0xe123456789123456-0xac12345678987456

Start time: Mon Apr 8 00:00:16 2013
Last election time: Tue Apr 22 18:50:00 2013

B: UP, SUBORDINATE, (Management services: SWITCHOVER IN PROGRESS)
A: UP, PRIMARY, (Management services: SWITCHOVER IN PROGRESS)

B: memb state UP, lead state SUBORDINATE, mgmt services state: INVALID
A: memb state UP, lead state PRIMARY, mgmt services state: INVALID
heartbeat state PRIMARY_OK

INTERNAL NETWORK INTERFACES:
eth1, UP
eth2, UP

HA NOT READY
Management services: switchover in progress on local Fabric Interconnect
Detailed state of the device selected for HA storage:
Chassis 1, serial: FOX12345678, state: active
Chassis 2, serial: FOX12345678, state: active

As we can see from the error the switchover is in progress but management services are not running on any of the FI’s so switchover cannot complete and this is the reason why you cannot access UCS Manager.
The blades running on this UCS infrastructure are not affected and should be running fine.

To fix the problem you need to reboot both Fabric Interconnects, one Fabric Interconnect at a time.

  1. SSH to one of the Fabric Interconnects and type:
    connect local-mgmt
    reboot
  2. Wait until the fabric interconnect reboots it can take 20-30 minutes.
  3. Once it rebooted you should be able to open UCSM
  4. In UCSM verify that the fabric that was rebooted has fully came up
  5. SSH to the second Fabric Interconnect and reboot it
  6. After Fabric Interconnect is up, SSH to UCSM IP and run:
    show cluster extended-state
  7. Verify that the cluster is in good state.
2 Comments :, , , , more...

UCS: How to change ring buffer size and queues for adapters in UCS

by on Jun.14, 2013, under Cisco, UCS, Vblock

Here is the thing that you’ll probably never do but just in case you need to change ring buffer size you’ll know how to do this in Cisco UCS. :)
Before you make changes you should really know what and why you doing this as this is the last thing I would change as the default setting should work fine.

  1. Login to UCSM
  2. Select Servers tab an change Filter to Policies
  3. Expand Adapter Policies you should see all the default adapter policies for Fibre, Ethernet and iSCSI
    1
  4. Do not change these as they might be used by multiple servers. Create new policy.
  5. To create new policy right click on Adapter Policies and select the policy type Fibre, Ethernet or iSCSI
    2
  6. Enter policy Name and if you need you can also enter Description.
    Enter the required Queue and Ring Size values and click OK.
    3
  7. To apply the new policy to the adapter change the Filter to Service Profiles.
  8. Select service profile and the adapter the you want this policy to apply to.
    4
  9. On the right side change the Adapter Policy to a new one.
    5
  10. Click Save Changes
Leave a Comment :, , , , , more...

UCS: Renaming service profile in Cisco UCS 2.1

by on Jun.13, 2013, under Cisco, UCS, Vblock

Starting from UCS version 2.1 there is an option to rename service-profiles and templates in UCS. If you’re running 2.0 or lower then the only way to rename is to clone service profile make some changes and delete the original, you can read about this here.

Rename service profile in UCSM GUI

  1. Start UCSM and select Servers tab.
  2. Change filter to Service Profiles or All so that service profiles are visible
  3. Right click on the service profile you want to rename ans choose Rename Service Profile
    spr1
  4. Enter new name and click OK
    spr2
  5. You’ll get a dialog box that the profile was renamed. Click OK
    spr3

Rename service profile in CLI

  1. Use SSH client to connect to UCS CLI
  2. You have to know where the service profile you want to rename is located. In my case it is in the root. Type:
    scope org /
    ‘/’ is a location of service profile. If it is in sub-organization then you have to provide then name for it
  3. To select service profile type:
    scope service-profile name
    ‘name’ is a name of the service profile you selecting
  4. to rename type:
    rename-to newname
    ‘newname’ is the name you renaming profile to
  5. you will be asked to confirm. Type ‘yes’
  6. to commit changes type
    commit-buffer
    spr4
1 Comment :, , , more...

UCS: How to move or rename service profile in Cisco UCS

by on Jun.13, 2013, under Cisco, UCS, Vblock

So UCS version 2.1 has added feature to rename service profile but what to do if you are running 2.0 or if you need to move service profile to different sub-organization?
Here are the steps you need to perform.
Before performing rename/move you’ll need to schedule maintenance window as during the procedure the blade will go down.

  1. Open UCSM GUI and select Servers tab
  2. Change Filter to Service Profiles or All so that service profiles are visible
    From this point all actions are performed on the original service profile(the one you want to rename/move)
  3. Right click on the service profile you want to rename/move and select Create a Clone
    1
  4. Enter the new name that you want to rename service profile to or if you are moving then enter the same name and change organization. Click OK. A clone should be created.
    2

(continue reading…)

Leave a Comment :, , , , more...

UCS: F0382 thermal-problem

by on Feb.06, 2013, under Cisco, UCS

This is a quick post for simple error that you might see in Cisco UCSM.
t1
Affected object: sys/chassis-2/fan-module-1-2
Description: Fan module 1-2 in chassis 2 temperature: upper-critical
Cause: thermal-problem
Code: F0382

This may indicate that there is hardware problem with the fan and it needs to be replaced but quite often it indicates logical problem. There are a few bugs depending on the version that UCS is running that might cause this behaviour.
As we can see from the error the problem is with the fan in chassis 2.
If you click on the blue text it will open a new window and you’ll be able to see which fan it is.
t2

The first thing you can try is to reseat the fan and see if the error goes away. If it does not then it most likely hardware issue.
In this case the error has cleared.
This error might reappear in couple day/weeks or months or it could happen on the other fan. There is a known I2C bug in UCS. To clear i2c bus it is advised to reseat all fans, PSUs and IOmodules(one at a time).
The bug was fixed in late 1.4(3*) version but was reintroduced again in 2.0 version. It should be fixed in latest version.
I UCS system affected by this bug than you still need to reseat components as the upgrade alone will not clear the I2C bus.
IMO it would be best to reseat them before upgrading as at least you’ll be upgrading a healthy system.

1 Comment :, , , , , more...

UCS: Update firmware on Cisco UCS C200 server using Host Update Utility

by on Feb.06, 2013, under Cisco, UCS

Here is a simple procedure how to update firmware on UCS C200 M2 server. The procedure should be no different on the other models of C series servers as they use Host Update Utility(HUU).
1. First you need to download the firmware for the upgrade. In this instance the server will be upgraded to 1.4(3p)5

  • Go to Cisco.com, select Support and All Dowloads 
  • Choose Servers- Unified Computing and Cisco UCS C-Series Rack-Mount Standalone Server Software. Select your server model(in this case it is C200 M2)
  • From select Software type choose Unified Computing System (UCS) Server Firmware
  • Find the required version and download it(you’ll need to login). It will be iso file:

    1

2. Mount HUU iso in CIMC (continue reading…)

Leave a Comment :, , , , more...

UCS: FSM: Unable to find Storage Controller Device for sys/chassis-4/blade-2/board/storage-SAS-1 to perform Image update

by on Jan.30, 2013, under Cisco, UCS

Here is a strange error that popped up recently.
After server reboot the blade got stuck on 47% of associating service profile.
Here is the error:

Remote Result: Service Unavailable
Remote Error Code: 4106
Remote Error Description: Unable to find Storage Controller Device for sys/chassis-4/blade-2/storage-SAS-1 to perform Image update

1

 

When looking in the KVM Console for the blade I can see that it was sitting in the screen below:
2

Here are the things that I’ve tried to fix this and none of these worked:
1. As it was saying about the image update for Storage adapter I though I’ll remove Host firmware policy from the service profile.
2. Next I created a new Host firmware policy but only added firmware SAS storage controller and chose different version than it had.
3. Remove and add service profile back

Here is what worked though:
Removed service profile from blade, re-acknowledged the blade added service profile back. After this the error was gone and the blade was booting normally.
The last step was to add old Hast firmware policy to the service profile and check if that brakes it again, but all worked as expected.
So I’m happy that the blade is back but cannot explain how and why this has happened.

1 Comment :, , , , , more...

UCS: F999616 [FSM:FAILED] communication service configuration(FSM:sam:dme:CommSvcEpUpdateSvcEp)

by on Jan.12, 2013, under Cisco, UCS

In UCSM you see critical error:
Affected object: sys/svc-ext
Description: [FSM:FAILED] communication service configuration(FSM:sam:dme:CommSvcEpUpdateSvcEp)
Cause: fsm-failed
Code: F999616

ntp

The error itself is not very descriptive, so in order to find out what the problem is best it to look at UCSM show-tech support logs.To do that select Admin tab, select All and click on Create and Download Tech support.

ntp1
(continue reading…)

1 Comment :, , , , , more...

UCS: flashing cursor after installing Linux (RedHat, Suse), server would not boot

by on Jan.04, 2013, under Cisco, UCS

Here is an interesting thing that happens during Linux installation on Cisco UCS blade.
After installing Suse on Cisco UCS blade, the server would not boot and only flashing cursor is seen. Tried other version of Suse and even RHEL, still the same. This looked strange because I had installed Linux on UCS without any problems before.
There is a small thing that you need to be aware when doing installation. Make sure to add CD-rom in boot order in the the service profile and after you mount your ISO let the sever to pick it up itself. Do not press F6 and choose for it as this is where interesting thing happens.
When you let the server to pick the boot device it makes Linux installer to think that this server is running in UEFI mode.
When you press F6 and choose the boot device the installer thinks it is BIOS mode and when partitions are created, it messes things up there for after reboot you see flashing cursor.

Leave a Comment :, , , , , , , more...

UCS: waiting for flogi

by on Dec.04, 2012, under Cisco, Storage, UCS

Here is one very common error that you can see in  UCS Manager. I’ve observed it in multiple UCS firmware versions and all times it was cosmetic and had no impact.

On the blade you’ll see a major error similar to this:
Description: fc VIF 6 /R A-1095 down, reason: waiting for flogi
Cause: link-down
Code: F0283 

If you go to VIF paths on the blade you’ll see  error ‘waiting for flogi’ on vHBA that has the problem.

Now as I mentioned already this is most likely cosmetic issue. So first you need to verify that this is really the case. (continue reading…)

3 Comments :, , , , , , more...