Using Cisco UCS Blade Server Diagnostics

15 Nov

Andrius | November 15th,2013 | Cisco, UCS | No Comments »

Cisco has released Cisco UCS Blade Server Diagnostics.
It’s an ISO file that you boot your blade from and there you can run a series of test to find out if there are any problems with hardware.

To run UCS diagnostics do the following:

Download ISO file from cisco.com you’ll need login first. At the time of writing the only version available is 1.0.1a
Login to UCS Manager and open the KVM console on the blade that you want to test(the blade must be associated with service profile and CD-ROM must be present in Boot Order at the top of it)
Select Virtual Media tab, click on Add Image button, locate and select Cisco UCS Blade Server Diagnostic ISO file that you downloaded. Make sure that Mapped box is ticket next to ISO file.
Restart the server. The server should start booting form ISO file
You’ll be prompted to choose from 3 options(GUI, CLI, memtest86+) if you do not choose in 10sec the blade will into GUI

GUI and CLI are self explanatory. memtest86+ is RAM testing utility.

Read the rest of this entry »

Cisco Live 2013 in Orlando

30 Jun

Andrius | June 30th,2013 | Cisco, Vblock | No Comments »

So I had a great opportunity to visit Cisco Live 2013 in Orlando.
This was my first(I hope not last) to Cisco Live and I can say it was really impressive.
Everything was big and american style but that is not important. What is important is the things that you can get from this event.
Seeing new products, getting some hands on in the labs and meeting people in the show and talking to techies was really beneficial. The amount of sessions was just huge and I tried to attend as many as possible. This was intensive brain training for 4 days on steroids.
The amount of information that I was receiving each day was mind blowing and I had problems sleeping(probably jet lag also played here)
Here are some pictures:

Geeks gathering 🙂

20130625_140831
VCE booth

Vblocks

One of hundred sessions

Configuring NTP and Timezones on Cisco devices(UCS, MDS, Nexus, Catalyst) in Vblock

21 Jun

Andrius | June 21st,2013 | Cisco, MDS, Nexus, UCS, Vblock | 2 comments

Cisco UCS

Open UCS manager.
Select Admin tab and change filter to Timezone Management
Select your timezone and click on green + to add NTP server details
Enter IP address of your NTP server and click OK. You can add FQDN of the server but then make sure that DNS is configured in UCS Manager.

Cisco Nexus 5000 and 7000

Login using CLI to your Nexus switch
To configure your timezone and NTP server details:
conf t
clock timezone UTC 0 0 <== Change your name of timezone from UTC. First 0 is for Hours offset and the second is for minutes offset
ntp server 192.168.101.81 use-vrf management
Verify that the configuration was set and packet are being received:sh ntp statistics peer ipaddr 192.168.101.81

sh clock

Cisco Nexus 1000v and MDS switchesLogin to CLI to your Nexus switch

Login using CLI to your MDS or 1000v switch
Configure timezone and NTP server details:
conf t
clock timezone UTC 0 0 <== Change your name of timezone from UTC. First 0 is for Hours offset and the second is for minutes offset
ntp server 192.168.101.81
Verify that the configuration was set and packet are being received:sh ntp statistics peer ipaddr 192.168.101.81

sh clock

Cisco Catalyst switches

Login using CLI to your Catalyst switch
Configure timezone and NTP server details:
conf t
clock timezone UTC 0 0 <== Change your name of timezone from UTC. First 0 is for Hours offset and the second is for minutes offset
ntp server 192.168.101.81
Verify that the configuration was set and packet are being received:
sh ntp status

sh ntp associations detail

sh clock

Beware that it may take some time for clock to synchronize on all devices.
If you are using Microsoft Windows w32tm service for NTP then on your NTP server you need to change LocalClockDispersion to 0 seconds or Cisco devices will not sync.
To check the current setting:

open command prompt on NTP server
type:w32tm /query /status

By default Dispersion is set to 10 seconds. It cannot be more than 1 second for Cisco devices to sync
Open registry and find the following key
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Config
Change the value of LocalClockDispersion from a to 0
Restart Windows Time service
The devices should start syncing.

MDS: How to identify what is connected to MDS on the other side of the cable

21 Jun

Andrius | June 21st,2013 | Cisco, MDS | No Comments »

In ideal world each port configured on any device would have description on it stating what it is connected to, but that it is not always the case and sometime the cables migh have been moved but descriptions left the same.
To identify what is at the other side of the cable of Cisco MDS switch you can run the following command:

show fcns database detail vsan <vsan id>
This will show all the ports for the vsan. Also you can put a range of vsans. Here are couple examples of the output.

------------------------
 VSAN:11 FCID:0x020001
 ------------------------
 port-wwn (vendor) :20:43:00:0d:ec:b4:dc:80 (Cisco)
 node-wwn :20:0b:00:0d:ec:b4:dc:81
 class :3
 node-ip-addr :192.168.104.32
 ipa :ff ff ff ff ff ff ff ff
 fc4-types:fc4_features :npv
 symbolic-port-name :fi6120-B:fc2/3   <== connected to UCS Fabric interconnect port fc2/3
 symbolic-node-name :fi6120-B
 port-type :N
 port-ip-addr :0.0.0.0
 fabric-port-wwn :20:0b:00:0d:ec:c2:62:c0
 hard-addr :0x000000
 permanent-port-wwn (vendor) :20:43:00:0d:ec:b4:dc:80 (Cisco)
 Connected Interface :fc1/11 <==MDS fc port
 Switch Name (IP address) :mds9509B (192.168.101.21)

------------------------
 VSAN:11 FCID:0x020006
 ------------------------
 port-wwn (vendor) :21:00:00:24:ff:32:bd:5f
 node-wwn :20:00:00:24:ff:32:bd:5f
 class :3
 node-ip-addr :0.0.0.0
 ipa :ff ff ff ff ff ff ff ff
 fc4-types:fc4_features :scsi-fcp:init
 symbolic-port-name :
 symbolic-node-name :QLE2462 FW:v5.06.02 DVR:v911.k1.1-19vmw  <== connected directly to Qlogic HBA that is running VMware drivers v911.k1.1-19vmw
 port-type :N
 port-ip-addr :0.0.0.0
 fabric-port-wwn :20:4d:00:0d:ec:c2:62:c0
 hard-addr :0x000000
 permanent-port-wwn (vendor) :21:00:00:24:ff:32:bd:5f
 Connected Interface :fc2/13 <==MDS fc port
 Switch Name (IP address) :mds9509B (192.168.101.21)

------------------------
 VSAN:11 FCID:0x02000b
 ------------------------
 port-wwn (vendor) :50:00:09:74:08:0e:79:5c (EMC)
 node-wwn :50:00:09:74:08:0e:78:00
 class :3
 node-ip-addr :0.0.0.0
 ipa :ff ff ff ff ff ff ff ff
 fc4-types:fc4_features :scsi-fcp:target 253
 symbolic-port-name :SYMMETRIX::000292600926::SAF- 8fA::FC::5876_159+::EMUL B80F0000 3F89624E 914B24 02.28.13 13:42 <==Connected to EMC VMax array(code level 5876_159) port 8fA  
 symbolic-node-name :SYMMETRIX::000292600926::FC::5876_159+
 port-type :N
 port-ip-addr :0.0.0.0
 fabric-port-wwn :20:42:00:0d:ec:c2:62:c0
 hard-addr :0x000000
 permanent-port-wwn (vendor) :50:00:09:74:08:0e:79:5c (EMC)
 Connected Interface :fc2/2  <==MDS fc port
 Switch Name (IP address) :mds9509B (192.168.101.21)

If you wan to look at specific port only then first you need to get FCID for this port:
show interface fc2/2

mds9509B# sh interface fc2/2
 fc2/2 is up
 Port description is Vmax_D9/F0
 Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
 Port WWN is 20:42:00:0d:ec:c2:62:c0
 Admin port mode is F, trunk mode is off
 snmp link state traps are enabled
 Port mode is F, FCID is 0x02000b <== FCID
 Port vsan is 11
 Speed is 4 Gbps
 Rate mode is shared
 Transmit B2B Credit is 20
 Receive B2B Credit is 16
 Receive data field Size is 2112
 Beacon is turned off
 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
 77 frames input, 5008 bytes
 0 discards, 0 errors
 0 CRC, 0 unknown class
 0 too long, 0 too short
 61 frames output, 3828 bytes
 0 discards, 0 errors
 9 input OLS, 9 LRR, 0 NOS, 4 loop inits
 9 output OLS, 9 LRR, 8 NOS, 7 loop inits
 16 receive B2B credit remaining
 20 transmit B2B credit remaining
 20 low priority transmit B2B credit remaining
 Interface last changed at Wed May 1 19:10:57 2013

Now type:
sh fcns database fcid <fcid> detail vsan <vsan id>

This will give output for this specific port.

UCS: LDAP nested groups are not working with UCS Manager

19 Jun

Andrius | June 19th,2013 | Cisco, UCS, Vblock | 1 Comment »

So you have your LDAP autentication configured in Cisco UCS manager.
You map group in UCS to LDAP group and add user to this LDAP group

Try to login with this user account to UCSM. Everything is working as expected.
You remove the user from LDAP group add it to another LDAP group and add this Group to LDAP group that is mapped to UCS group(in other words you nest groups).

When trying to login again, authentication fails.

This is a known behaviour of UCS and is explained in more detail under bug id CSCtt44185. With UCS, nested groups should not be used, each group in LDAP must be mapped to a group in UCS.

VMware: Cannot create Resource Pool, the option is greyed out

14 Jun

Andrius | June 14th,2013 | Vblock, Virtualization, VMware | No Comments »

OK this is a now brainer but still get asked sometimes.

In the vCenter you right click on Cluster and New Resource Pool option is grey out.

To create resource pool you need to have Distributed Resources Scheduler (DRS) enabled on the Cluster.

Right Click on the Cluster and select Edit Settings…
Tick Turn On vSphere DRS click OK.
Now you can create Resource Pools for this Cluster

UCS: UCS Manager not accessible, (SWITCHOVER IN PROGRESS) (mgmt services state: INVALID) (HA NOT READY)

14 Jun

Andrius | June 14th,2013 | Cisco, UCS, Vblock | 2 comments

UCSM is not accessible when login into UCS cli and running show cluster extended-state command you receive the following:

ucs01-B# show cluster extended-state
Cluster Id: 0xe123456789123456-0xac12345678987456

Start time: Mon Apr 8 00:00:16 2013
Last election time: Tue Apr 22 18:50:00 2013

B: UP, SUBORDINATE, (Management services: SWITCHOVER IN PROGRESS)
A: UP, PRIMARY, (Management services: SWITCHOVER IN PROGRESS)

B: memb state UP, lead state SUBORDINATE, mgmt services state: INVALID
A: memb state UP, lead state PRIMARY, mgmt services state: INVALID
heartbeat state PRIMARY_OK

INTERNAL NETWORK INTERFACES:
eth1, UP
eth2, UP

HA NOT READY
Management services: switchover in progress on local Fabric Interconnect
Detailed state of the device selected for HA storage:
Chassis 1, serial: FOX12345678, state: active
Chassis 2, serial: FOX12345678, state: active

As we can see from the error the switchover is in progress but management services are not running on any of the FI’s so switchover cannot complete and this is the reason why you cannot access UCS Manager.
The blades running on this UCS infrastructure are not affected and should be running fine.

To fix the problem you need to reboot both Fabric Interconnects, one Fabric Interconnect at a time.

SSH to one of the Fabric Interconnects and type:
connect local-mgmt
reboot
Wait until the fabric interconnect reboots it can take 20-30 minutes.
Once it rebooted you should be able to open UCSM
In UCSM verify that the fabric that was rebooted has fully came up
SSH to the second Fabric Interconnect and reboot it
After Fabric Interconnect is up, SSH to UCSM IP and run:
show cluster extended-state
Verify that the cluster is in good state.

UCS: How to change ring buffer size and queues for adapters in UCS

14 Jun

Andrius | June 14th,2013 | Cisco, UCS, Vblock | No Comments »

Here is the thing that you’ll probably never do but just in case you need to change ring buffer size you’ll know how to do this in Cisco UCS. 🙂
Before you make changes you should really know what and why you doing this as this is the last thing I would change as the default setting should work fine.

Login to UCSM
Select Servers tab an change Filter to Policies
Expand Adapter Policies you should see all the default adapter policies for Fibre, Ethernet and iSCSI
Do not change these as they might be used by multiple servers. Create new policy.
To create new policy right click on Adapter Policies and select the policy type Fibre, Ethernet or iSCSI
Enter policy Name and if you need you can also enter Description.
Enter the required Queue and Ring Size values and click OK.
To apply the new policy to the adapter change the Filter to Service Profiles.
Select service profile and the adapter the you want this policy to apply to.
On the right side change the Adapter Policy to a new one.
Click Save Changes

UCS: Renaming service profile in Cisco UCS 2.1

13 Jun

Andrius | June 13th,2013 | Cisco, UCS, Vblock | 1 Comment »

Starting from UCS version 2.1 there is an option to rename service-profiles and templates in UCS. If you’re running 2.0 or lower then the only way to rename is to clone service profile make some changes and delete the original, you can read about this here.

Rename service profile in UCSM GUI

Start UCSM and select Servers tab.
Change filter to Service Profiles or All so that service profiles are visible
Right click on the service profile you want to rename ans choose Rename Service Profile
Enter new name and click OK
You’ll get a dialog box that the profile was renamed. Click OK

Rename service profile in CLI

Use SSH client to connect to UCS CLI
You have to know where the service profile you want to rename is located. In my case it is in the root. Type:
scope org /
‘/’ is a location of service profile. If it is in sub-organization then you have to provide then name for it
To select service profile type:
scope service-profile name
‘name’ is a name of the service profile you selecting
to rename type:
rename-to newname
‘newname’ is the name you renaming profile to
you will be asked to confirm. Type ‘yes’
to commit changes type
commit-buffer

UCS: How to move or rename service profile in Cisco UCS

13 Jun

Andrius | June 13th,2013 | Cisco, UCS, Vblock | 1 Comment »

So UCS version 2.1 has added feature to rename service profile but what to do if you are running 2.0 or if you need to move service profile to different sub-organization?
Here are the steps you need to perform.
Before performing rename/move you’ll need to schedule maintenance window as during the procedure the blade will go down.

Open UCSM GUI and select Servers tab
Change Filter to Service Profiles or All so that service profiles are visible
From this point all actions are performed on the original service profile(the one you want to rename/move)
Right click on the service profile you want to rename/move and select Create a Clone
Enter the new name that you want to rename service profile to or if you are moving then enter the same name and change organization. Click OK. A clone should be created.