IT Blog

UCS

UCS: Chassis Decommission is stuck

by on Dec.27, 2013, under Cisco, UCS

After trying to decommission chassis the task never finishes. It gets stuck at Erasing chassis identity task:
2

FSM is stuck at 8% with error 403, Error Leaving fabric port-channel for port 1/16 (the port can be different)
3

Trying to Acknowledge or Remove Chassis is not successful.
1

To fix the problem identify all Server ports for that chassis and unconfigure them. You need to do that for both sides of the the fabric.
4

Go back to FSM and monitor the progress. Be patient as it will take some time but you’ll see that erasing chassis identity task will be skipped on both fabrics and then the chassis becomes decommissioned.
5

If you need to bring the chassis back then simply configure the unconfigured ports as server ports and recommission the chassis.

Leave a Comment :, , more...

UCS: Running UCS Server Configuration Utility (SCU) Diagnostic on C series servers

by on Dec.18, 2013, under Cisco, UCS

UCS Server Configuration Utility (SCU) allows user to configure the server and also run diagnostics tests.

Do the following to run SCU diagnostics tests:

1. Download SCU from Cisco.com
2. Login to CIMC of the server and map the SCU ISO image
1
3. Restart the server and wait until it boots to SCU
2
4. Accept the License Agreement
3
5. In SCU expand Diagnostic Tools, select Quick Tasks and choose the test you want to run Quick or Comprehensive.
4
6. Confirm
5
7. You can see the status of the tests under Tests Summary and the results under Test log Summary.
6
8. Once finished click on Reboot button
7

Leave a Comment :, , more...

UCS: After firmware upgrade B230 M1 blade is failing discovery at 4%

by on Dec.15, 2013, under Cisco, UCS

After upgrading B230 M1 blade it gets stuck in discovery at 4% and is showing following errors:

no connection to MC endpoint
1

Error retrieving Server Params-MC Error(-6): Connection is closing

2

No DIMMs present

3

The errors can be rotating but it is stuck at 4%.

1. Check Firmware versions and see if all are as expected.
3_5

2. Reset CIMC and after this is done Reacknowledge the server.
3. Check the firmware versions of the blade again. The Running Board Controller version is set to 00000000
4

4. Activate the Board Controller with correct version and wait until activation is finished.
5

5. Reacknowledge the server

Leave a Comment :, , , more...

UCSPE 2.2(1bPE1) is out

by on Dec.15, 2013, under Cisco, UCS

UCSPE 2.2(1bPE1) is out. Get it here.

Leave a Comment :, more...

UCS: Blade is stuck in EFI Shell, service profile association stuck

by on Dec.11, 2013, under Cisco, UCS

After associating service profile to a blade it gets stuck in EFI Shell and association can not complete.
The KVM is showing EFI Shell
1

FSM status is stuck on Configure Apply Config
2

The problem could be because the service profile does not have all the required configuration. Have a look at Status under General tab to see what is the error.
3

In this case there were no MACs or WWPNs defined. CNA cards must have these defined as they do not have them burned in.

 

Leave a Comment :, , more...

Using Cisco UCS Blade Server Diagnostics

by on Nov.15, 2013, under Cisco, UCS

Cisco has released Cisco UCS Blade Server Diagnostics.
It’s an ISO file that you boot your blade from and there you can run a series of test to find out if there are any problems with hardware.

To run UCS diagnostics do the following:

  • Download ISO file from cisco.com you’ll need login first. At the time of writing the only version available is 1.0.1a
  • Login to UCS Manager and open the KVM console on the blade that you want to test(the blade must be associated with service profile and CD-ROM must be present in Boot Order at the top of it)
  • Select Virtual Media tab, click on Add Image button, locate and select Cisco UCS Blade Server Diagnostic ISO file that you downloaded. Make sure that Mapped box is ticket next to ISO file.
    1
  • Restart the server. The server should start booting form ISO file
  • You’ll be prompted to choose from 3 options(GUI, CLI, memtest86+) if you do not choose in 10sec the blade will into GUI
    2
    GUI and CLI are self explanatory. memtest86+ is RAM testing utility.

(continue reading…)

Leave a Comment :, , , , more...

Configuring NTP and Timezones on Cisco devices(UCS, MDS, Nexus, Catalyst) in Vblock

by on Jun.21, 2013, under Cisco, MDS, Nexus, UCS, Vblock

Cisco UCS

  1. Open UCS manager.
  2. Select Admin tab and change filter to Timezone Management
  3. Select your timezone and click on green + to add NTP server details
    1
  4. Enter IP address of your NTP server and click OK. You can add FQDN of the server but then make sure that DNS is configured in UCS Manager.
    2

Cisco Nexus 5000 and 7000

  1. Login using CLI to your Nexus switch
  2. To configure your timezone and NTP server details:
    conf t
    clock timezone UTC 0 0
       <== Change your name of timezone from UTC. First 0 is for Hours offset and the second is for minutes offset
    ntp server 192.168.101.81 use-vrf management
  3. Verify that the configuration was set and packet are being received:sh ntp statistics peer ipaddr 192.168.101.81
    7
    sh clock

Cisco Nexus 1000v and MDS switchesLogin to CLI to your Nexus switch

  1. Login using CLI to your MDS or 1000v switch
  2. Configure timezone and NTP server details:
    conf t
    clock timezone UTC 0 0    
    <== Change your name of timezone from UTC. First 0 is for Hours offset and the second is for minutes offset
    ntp server 192.168.101.81
  3. Verify that the configuration was set and packet are being received:sh ntp statistics peer ipaddr 192.168.101.81
    5
    sh clock

Cisco Catalyst switches

  1. Login using CLI to your Catalyst switch
  2. Configure timezone and NTP server details:
    conf t
    clock timezone UTC 0 0    
    <== Change your name of timezone from UTC. First 0 is for Hours offset and the second is for minutes offset
    ntp server 192.168.101.81
  3. Verify that the configuration was set and packet are being received:
    sh ntp status
    4
    sh ntp associations detail
    6
    sh clock

Beware that it may take some time for clock to synchronize on all devices.
If you are using Microsoft Windows w32tm service for NTP then on your NTP server you need to change LocalClockDispersion to 0 seconds or Cisco devices will not sync.
To check the current setting:

  1. open command prompt on NTP server
  2. type:w32tm /query /status
    8
    By default Dispersion is set to 10 seconds. It cannot be more than 1 second for Cisco devices to sync
  3. Open registry and find the following key
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Config
  4. Change the value of LocalClockDispersion from a to 0
  5. Restart Windows Time service
  6. The devices should start syncing.
Leave a Comment :, , , , , , more...

UCS: LDAP nested groups are not working with UCS Manager

by on Jun.19, 2013, under Cisco, UCS, Vblock

So you have your LDAP autentication configured in Cisco UCS manager.
You map group in UCS to LDAP group and add user to this LDAP group
2
Try to login with this user account to UCSM. Everything is working as expected.
You remove the user from LDAP group add it to another LDAP group and add this Group to LDAP group that is mapped to UCS group(in other words you nest groups).
1
When trying to login again, authentication fails.

This is a known behaviour of UCS and is explained in more detail under bug id CSCtt44185. With UCS, nested groups should not be used, each group in LDAP must be mapped to a group in UCS.

Leave a Comment :, , , , , more...

UCS: UCS Manager not accessible, (SWITCHOVER IN PROGRESS) (mgmt services state: INVALID) (HA NOT READY)

by on Jun.14, 2013, under Cisco, UCS, Vblock

UCSM is not accessible when login into UCS cli and running show cluster extended-state command you receive the following:

ucs01-B# show cluster extended-state
Cluster Id: 0xe123456789123456-0xac12345678987456

Start time: Mon Apr 8 00:00:16 2013
Last election time: Tue Apr 22 18:50:00 2013

B: UP, SUBORDINATE, (Management services: SWITCHOVER IN PROGRESS)
A: UP, PRIMARY, (Management services: SWITCHOVER IN PROGRESS)

B: memb state UP, lead state SUBORDINATE, mgmt services state: INVALID
A: memb state UP, lead state PRIMARY, mgmt services state: INVALID
heartbeat state PRIMARY_OK

INTERNAL NETWORK INTERFACES:
eth1, UP
eth2, UP

HA NOT READY
Management services: switchover in progress on local Fabric Interconnect
Detailed state of the device selected for HA storage:
Chassis 1, serial: FOX12345678, state: active
Chassis 2, serial: FOX12345678, state: active

As we can see from the error the switchover is in progress but management services are not running on any of the FI’s so switchover cannot complete and this is the reason why you cannot access UCS Manager.
The blades running on this UCS infrastructure are not affected and should be running fine.

To fix the problem you need to reboot both Fabric Interconnects, one Fabric Interconnect at a time.

  1. SSH to one of the Fabric Interconnects and type:
    connect local-mgmt
    reboot
  2. Wait until the fabric interconnect reboots it can take 20-30 minutes.
  3. Once it rebooted you should be able to open UCSM
  4. In UCSM verify that the fabric that was rebooted has fully came up
  5. SSH to the second Fabric Interconnect and reboot it
  6. After Fabric Interconnect is up, SSH to UCSM IP and run:
    show cluster extended-state
  7. Verify that the cluster is in good state.
2 Comments :, , , , more...

UCS: How to change ring buffer size and queues for adapters in UCS

by on Jun.14, 2013, under Cisco, UCS, Vblock

Here is the thing that you’ll probably never do but just in case you need to change ring buffer size you’ll know how to do this in Cisco UCS. :)
Before you make changes you should really know what and why you doing this as this is the last thing I would change as the default setting should work fine.

  1. Login to UCSM
  2. Select Servers tab an change Filter to Policies
  3. Expand Adapter Policies you should see all the default adapter policies for Fibre, Ethernet and iSCSI
    1
  4. Do not change these as they might be used by multiple servers. Create new policy.
  5. To create new policy right click on Adapter Policies and select the policy type Fibre, Ethernet or iSCSI
    2
  6. Enter policy Name and if you need you can also enter Description.
    Enter the required Queue and Ring Size values and click OK.
    3
  7. To apply the new policy to the adapter change the Filter to Service Profiles.
  8. Select service profile and the adapter the you want this policy to apply to.
    4
  9. On the right side change the Adapter Policy to a new one.
    5
  10. Click Save Changes
Leave a Comment :, , , , , more...