IT Blog

Tag: Cisco

Nexus: %VMS-1-CONN_SSL_NOAUTH: SSL AUTHENTICATION after upgrading Nexus 1000v

by on Jun.16, 2014, under Cisco, Nexus, Vblock

After upgrading Nexus 1000V VSM to version 4.2(1)SV2(2.1a) you start recieving ‘%VMS-1-CONN_SSL_NOAUTH1: SSL AUTHENTICATION failure.‘ message in the console every couple minutes:

N1Kv# 2014 Jun 1 18:34:48 N1Kv %VMS-1-CONN_SSL_NOAUTH: SSL AUTHENTICATION failure.
2014 Jun 1 18:37:48 N1Kv %VMS-1-CONN_SSL_NOAUTH1: SSL AUTHENTICATION failure.
2014 Jun 1 18:40:48 N1Kv %VMS-1-CONN_SSL_NOAUTH: SSL AUTHENTICATION failure.
2014 Jun 1 18:43:47 N1Kv %VMS-1-CONN_SSL_NOAUTH1: SSL AUTHENTICATION failure.
2014 Jun 1 18:46:47 N1Kv %VMS-1-CONN_SSL_NOAUTH: SSL AUTHENTICATION failure.

This is expected behavior because of a new feature that was added in Nexus 1000v 4.2(1)SV2(2.1a) version:
vCenter Server Certificate Validation
The Cisco Nexus 1000V VSM can validate the certificate presented by vCenter Server to authenticate it. The certificate may be self-signed or signed by a Certificate Authority (CA). The validation is done each time the VSM connects to the vCenter Server. If the certificate authentication fails, a warning is generated but the connection is not impaired. This is an optional feature.

To get rid of the warning please generate a valid SSL certificate for vCenter server.

Leave a Comment :, , more...

Nexus: Pre-upgrade check failed. Return code 0x40930062 (free space in the filesystem is below threshold).

by on Jun.16, 2014, under Cisco, Nexus, Vblock

While trying to upgrade Nexus 5000 series switch I ran into following issue:

Pre-upgrade check failed. Return code 0x40930062 (free space in the filesystem is below threshold).

Pre-upgrade check failed. Return code 0x40930062 (free space in the filesystem is below threshold).

Switch has enough free:

Enough free space

Enough free space

Look at install logs to identify where the problem is by typing:
show system internal log install details | include space

show system internal log install details | include space

show system internal log install details | include space

As you can see the problem is because /var/tmp is bellow the threshold.
To identify what is  taking space in /var/tmp type:
show system internal dir /var/tmp (continue reading…)

Leave a Comment :, , , more...

UCS: Cisco FN – 63812

by on Jun.13, 2014, under Cisco, UCS, Vblock

Cisco has release a field notice FN – 63812 on 4th of June for their servers that are using Seagate hard drives.
According to Cisco some Seagate drives with specific level of firmware might not respond to requests.
There is a list of drives and instructions how to identify these drives. Unfortunately if you have C220 M2 server with integrated controller like LSI 1064E the instruction are not going to work as the drives are not visible in CIMC.

In CIMC there is nothing listed under Storage tab

CIMC/Storage

CIMC/Storage

In CLI you cannot scope to storage adapter as non exists

show storageadapter

show storageadapter

(continue reading…)

Leave a Comment :, , , , more...

UCS: Unable to communicate with Flexible Flash controller

by on May.28, 2014, under UCS, Vblock, VMware

For all of you who are using Cisco UCS C220 M3 server and Flexible Flash controller with SD card beware that there is a bug in firmware prior to 1.5(3a)

The problem shows up as timeout error for FlexFlash controller. The interesting part is even though there is a problem with the FlexFlash, CIMC still is showing that overall Server Status as good:
Once you log-in into CIMC you will not see that there is a problem:

2

No Errors

but once you click on Storage tab you’ll get error message:

Error: Unable to communicate with Flexible Flash controller: operation ffCardsGet, status ERROR_TIMEOUT

In the logs you’ll see: (continue reading…)

Leave a Comment :, , , , more...

UCSPE 2.2(1bPE1) is out

by on Dec.15, 2013, under Cisco, UCS

UCSPE 2.2(1bPE1) is out. Get it here.

Leave a Comment :, more...

Using Cisco UCS Blade Server Diagnostics

by on Nov.15, 2013, under Cisco, UCS

Cisco has released Cisco UCS Blade Server Diagnostics.
It’s an ISO file that you boot your blade from and there you can run a series of test to find out if there are any problems with hardware.

To run UCS diagnostics do the following:

  • Download ISO file from cisco.com you’ll need login first. At the time of writing the only version available is 1.0.1a
  • Login to UCS Manager and open the KVM console on the blade that you want to test(the blade must be associated with service profile and CD-ROM must be present in Boot Order at the top of it)
  • Select Virtual Media tab, click on Add Image button, locate and select Cisco UCS Blade Server Diagnostic ISO file that you downloaded. Make sure that Mapped box is ticket next to ISO file.
    1
  • Restart the server. The server should start booting form ISO file
  • You’ll be prompted to choose from 3 options(GUI, CLI, memtest86+) if you do not choose in 10sec the blade will into GUI
    2
    GUI and CLI are self explanatory. memtest86+ is RAM testing utility.

(continue reading…)

Leave a Comment :, , , , more...

Configuring NTP and Timezones on Cisco devices(UCS, MDS, Nexus, Catalyst) in Vblock

by on Jun.21, 2013, under Cisco, MDS, Nexus, UCS, Vblock

Cisco UCS

  1. Open UCS manager.
  2. Select Admin tab and change filter to Timezone Management
  3. Select your timezone and click on green + to add NTP server details
    1
  4. Enter IP address of your NTP server and click OK. You can add FQDN of the server but then make sure that DNS is configured in UCS Manager.
    2

Cisco Nexus 5000 and 7000

  1. Login using CLI to your Nexus switch
  2. To configure your timezone and NTP server details:
    conf t
    clock timezone UTC 0 0
       <== Change your name of timezone from UTC. First 0 is for Hours offset and the second is for minutes offset
    ntp server 192.168.101.81 use-vrf management
  3. Verify that the configuration was set and packet are being received:sh ntp statistics peer ipaddr 192.168.101.81
    7
    sh clock

Cisco Nexus 1000v and MDS switchesLogin to CLI to your Nexus switch

  1. Login using CLI to your MDS or 1000v switch
  2. Configure timezone and NTP server details:
    conf t
    clock timezone UTC 0 0    
    <== Change your name of timezone from UTC. First 0 is for Hours offset and the second is for minutes offset
    ntp server 192.168.101.81
  3. Verify that the configuration was set and packet are being received:sh ntp statistics peer ipaddr 192.168.101.81
    5
    sh clock

Cisco Catalyst switches

  1. Login using CLI to your Catalyst switch
  2. Configure timezone and NTP server details:
    conf t
    clock timezone UTC 0 0    
    <== Change your name of timezone from UTC. First 0 is for Hours offset and the second is for minutes offset
    ntp server 192.168.101.81
  3. Verify that the configuration was set and packet are being received:
    sh ntp status
    4
    sh ntp associations detail
    6
    sh clock

Beware that it may take some time for clock to synchronize on all devices.
If you are using Microsoft Windows w32tm service for NTP then on your NTP server you need to change LocalClockDispersion to 0 seconds or Cisco devices will not sync.
To check the current setting:

  1. open command prompt on NTP server
  2. type:w32tm /query /status
    8
    By default Dispersion is set to 10 seconds. It cannot be more than 1 second for Cisco devices to sync
  3. Open registry and find the following key
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Config
  4. Change the value of LocalClockDispersion from a to 0
  5. Restart Windows Time service
  6. The devices should start syncing.
Leave a Comment :, , , , , , more...

MDS: How to identify what is connected to MDS on the other side of the cable

by on Jun.21, 2013, under Cisco, MDS

In ideal world each port configured on any device would have  description on it stating what it is connected to, but that it is not always the case and sometime the cables migh have been moved but descriptions left the same.
To identify what is at the other side of the cable of Cisco MDS switch you can run the following command:

show fcns database detail vsan <vsan id>
This will show all the ports for the vsan. Also you can put a range of vsans. Here are couple examples of the output.

------------------------
 VSAN:11 FCID:0x020001
 ------------------------
 port-wwn (vendor) :20:43:00:0d:ec:b4:dc:80 (Cisco)
 node-wwn :20:0b:00:0d:ec:b4:dc:81
 class :3
 node-ip-addr :192.168.104.32
 ipa :ff ff ff ff ff ff ff ff
 fc4-types:fc4_features :npv
 symbolic-port-name :fi6120-B:fc2/3   <== connected to UCS Fabric interconnect port fc2/3
 symbolic-node-name :fi6120-B
 port-type :N
 port-ip-addr :0.0.0.0
 fabric-port-wwn :20:0b:00:0d:ec:c2:62:c0
 hard-addr :0x000000
 permanent-port-wwn (vendor) :20:43:00:0d:ec:b4:dc:80 (Cisco)
 Connected Interface :fc1/11 <==MDS fc port
 Switch Name (IP address) :mds9509B (192.168.101.21)
------------------------
 VSAN:11 FCID:0x020006
 ------------------------
 port-wwn (vendor) :21:00:00:24:ff:32:bd:5f
 node-wwn :20:00:00:24:ff:32:bd:5f
 class :3
 node-ip-addr :0.0.0.0
 ipa :ff ff ff ff ff ff ff ff
 fc4-types:fc4_features :scsi-fcp:init
 symbolic-port-name :
 symbolic-node-name :QLE2462 FW:v5.06.02 DVR:v911.k1.1-19vmw  <== connected directly to Qlogic HBA that is running VMware drivers v911.k1.1-19vmw
 port-type :N
 port-ip-addr :0.0.0.0
 fabric-port-wwn :20:4d:00:0d:ec:c2:62:c0
 hard-addr :0x000000
 permanent-port-wwn (vendor) :21:00:00:24:ff:32:bd:5f
 Connected Interface :fc2/13 <==MDS fc port
 Switch Name (IP address) :mds9509B (192.168.101.21)
------------------------
 VSAN:11 FCID:0x02000b
 ------------------------
 port-wwn (vendor) :50:00:09:74:08:0e:79:5c (EMC)
 node-wwn :50:00:09:74:08:0e:78:00
 class :3
 node-ip-addr :0.0.0.0
 ipa :ff ff ff ff ff ff ff ff
 fc4-types:fc4_features :scsi-fcp:target 253
 symbolic-port-name :SYMMETRIX::000292600926::SAF- 8fA::FC::5876_159+::EMUL B80F0000 3F89624E 914B24 02.28.13 13:42 <==Connected to EMC VMax array(code level 5876_159) port 8fA  
 symbolic-node-name :SYMMETRIX::000292600926::FC::5876_159+
 port-type :N
 port-ip-addr :0.0.0.0
 fabric-port-wwn :20:42:00:0d:ec:c2:62:c0
 hard-addr :0x000000
 permanent-port-wwn (vendor) :50:00:09:74:08:0e:79:5c (EMC)
 Connected Interface :fc2/2  <==MDS fc port
 Switch Name (IP address) :mds9509B (192.168.101.21)

If you wan to look at specific port only then first you need to get FCID for this port:
show interface fc2/2

mds9509B# sh interface fc2/2
 fc2/2 is up
 Port description is Vmax_D9/F0
 Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
 Port WWN is 20:42:00:0d:ec:c2:62:c0
 Admin port mode is F, trunk mode is off
 snmp link state traps are enabled
 Port mode is F, FCID is 0x02000b <== FCID
 Port vsan is 11
 Speed is 4 Gbps
 Rate mode is shared
 Transmit B2B Credit is 20
 Receive B2B Credit is 16
 Receive data field Size is 2112
 Beacon is turned off
 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
 77 frames input, 5008 bytes
 0 discards, 0 errors
 0 CRC, 0 unknown class
 0 too long, 0 too short
 61 frames output, 3828 bytes
 0 discards, 0 errors
 9 input OLS, 9 LRR, 0 NOS, 4 loop inits
 9 output OLS, 9 LRR, 8 NOS, 7 loop inits
 16 receive B2B credit remaining
 20 transmit B2B credit remaining
 20 low priority transmit B2B credit remaining
 Interface last changed at Wed May 1 19:10:57 2013

Now type:
sh fcns database fcid <fcid> detail vsan <vsan id>

This will give output for this specific port.

Leave a Comment :, , , more...

UCS: UCS Manager not accessible, (SWITCHOVER IN PROGRESS) (mgmt services state: INVALID) (HA NOT READY)

by on Jun.14, 2013, under Cisco, UCS, Vblock

UCSM is not accessible when login into UCS cli and running show cluster extended-state command you receive the following:

ucs01-B# show cluster extended-state
Cluster Id: 0xe123456789123456-0xac12345678987456

Start time: Mon Apr 8 00:00:16 2013
Last election time: Tue Apr 22 18:50:00 2013

B: UP, SUBORDINATE, (Management services: SWITCHOVER IN PROGRESS)
A: UP, PRIMARY, (Management services: SWITCHOVER IN PROGRESS)

B: memb state UP, lead state SUBORDINATE, mgmt services state: INVALID
A: memb state UP, lead state PRIMARY, mgmt services state: INVALID
heartbeat state PRIMARY_OK

INTERNAL NETWORK INTERFACES:
eth1, UP
eth2, UP

HA NOT READY
Management services: switchover in progress on local Fabric Interconnect
Detailed state of the device selected for HA storage:
Chassis 1, serial: FOX12345678, state: active
Chassis 2, serial: FOX12345678, state: active

As we can see from the error the switchover is in progress but management services are not running on any of the FI’s so switchover cannot complete and this is the reason why you cannot access UCS Manager.
The blades running on this UCS infrastructure are not affected and should be running fine.

To fix the problem you need to reboot both Fabric Interconnects, one Fabric Interconnect at a time.

  1. SSH to one of the Fabric Interconnects and type:
    connect local-mgmt
    reboot
  2. Wait until the fabric interconnect reboots it can take 20-30 minutes.
  3. Once it rebooted you should be able to open UCSM
  4. In UCSM verify that the fabric that was rebooted has fully came up
  5. SSH to the second Fabric Interconnect and reboot it
  6. After Fabric Interconnect is up, SSH to UCSM IP and run:
    show cluster extended-state
  7. Verify that the cluster is in good state.
2 Comments :, , , , more...

UCS: How to change ring buffer size and queues for adapters in UCS

by on Jun.14, 2013, under Cisco, UCS, Vblock

Here is the thing that you’ll probably never do but just in case you need to change ring buffer size you’ll know how to do this in Cisco UCS. :)
Before you make changes you should really know what and why you doing this as this is the last thing I would change as the default setting should work fine.

  1. Login to UCSM
  2. Select Servers tab an change Filter to Policies
  3. Expand Adapter Policies you should see all the default adapter policies for Fibre, Ethernet and iSCSI
    1
  4. Do not change these as they might be used by multiple servers. Create new policy.
  5. To create new policy right click on Adapter Policies and select the policy type Fibre, Ethernet or iSCSI
    2
  6. Enter policy Name and if you need you can also enter Description.
    Enter the required Queue and Ring Size values and click OK.
    3
  7. To apply the new policy to the adapter change the Filter to Service Profiles.
  8. Select service profile and the adapter the you want this policy to apply to.
    4
  9. On the right side change the Adapter Policy to a new one.
    5
  10. Click Save Changes
Leave a Comment :, , , , , more...