{"id":935,"date":"2012-11-12T19:58:55","date_gmt":"2012-11-12T19:58:55","guid":{"rendered":"http:\/\/www.kozeniauskas.com\/itblog\/?p=935"},"modified":"2012-12-10T19:59:44","modified_gmt":"2012-12-10T19:59:44","slug":"ucs-blade-is-stuck-on-discovery-after-ucs-firmware-upgrade-unidentified-fru","status":"publish","type":"post","link":"https:\/\/www.kozeniauskas.com\/itblog\/2012\/11\/12\/ucs-blade-is-stuck-on-discovery-after-ucs-firmware-upgrade-unidentified-fru\/","title":{"rendered":"UCS: Blade is stuck on discovery after UCS firmware upgrade (unidentified FRU)"},"content":{"rendered":"

Here is pretty common problem in UCS 2.0 release.
\nAt any stage of\u00a0UCS upgrade \u00a0one or more blades go into discovery mode and never finishes it. Depending on the version they can get stuck at any percentage but usually between 4% and 40%.
\nMost of the time a corruption\u00a0occurs\u00a0in SEEPROM of \u00a0M81kr CNA card because of this corruption checksum fails and UCS cannot recognize the mezzanine card any longer and this prevent Discovery from finishing.
\nYou can see the following errors when this happens:
\nConfiguration Error: adaptor-inoperable. Discovery State: Insufficiently Equipped.
\nAdapter 1 in server 1\/1 has unidentified FRU\u00a0<\/strong><\/p>\n

There are multiple Cisco bugs for this issue\u00a0CSCub16754, CSCty34034, CSCub48862, CSCub99354 and I’ve seen it happening on 2.0(1q), 2.0(2r), 2.0(3a) releases.
\nUnfortunately\u00a0the issue is not fixed and there is no\u00a0workaround. The good thing is that if this occurs the fix is pretty simple and quick and no hardware replacement is needed but only Cisco TAC can fix this or whoever has access to their internal resources.<\/p>\n

To verify if\u00a0corruption\u00a0occurred\u00a0you can do the following:<\/p>\n

    \n
  1. SSH to UCSM IP<\/li>\n
  2. Enter\u00a0connect cimc x\/y<\/strong>\u00a0(Chassis\/Blade)<\/li>\n
  3. Enter\u00a0mezz1fru\u00a0<\/strong>on the versions starting from 2.0(3a) you need to enter\u00a0fru<\/strong>
    \nIf\u00a0corruption has\u00a0occurred\u00a0the last line of the output will show something like
    \n‘Checksum Failed For: Board Area!’<\/em><\/li>\n<\/ol>\n

    The other method to check is to look at the logs.<\/p>\n

      \n
    1. Generate show-tech support logs for the chassis<\/li>\n
    2. Extract the log file there you\u2019ll see files called MEZZxy_TechSupport.tar where x is blade id and y is adapter id<\/li>\n
    3. Extract this file. There you\u2019ll find file debugdump.<\/li>\n
    4. In debugdump file look for the line starting with \u2018fruprom \u2013s\u2019 after this line you\u2019ll see FRU values(see bellow).
      \nfruprom -s<\/em>
      \nMezz Internal Use Area<\/em>
      \n CARD_TYPE : 3<\/em>
      \n NUMBER OF MACS : 6<\/em>
      \n MAC : 70:81:05:43:55:0D<\/em>Board Info Area (96)<\/em>
      \n MFG DATE : 08\/25\/11<\/em>
      \n MFG INFO : Cisco Systems Inc<\/em>
      \n PRODUCT NAME : N20-AC0002<\/em>
      \n SERIAL NUM : ABC12345678<\/em>
      \n PART NUM : 73-11789-09<\/em>
      \n FRUFILE ID : AC02<\/em>
      \n PART NUM REV : A0<\/em>
      \n FAB VERSION : 07<\/em>
      \n VID : V03<\/em>
      \n CLEI : 0000000000<\/em>The example above shows when the values are fine. When they are not fine you’ll see that some information is missing or there are some odd characters in most cases it was\u00a0\u00ff\u00a0<\/strong>see below:
      \nMFG INFO\u00a0\u00a0\u00a0\u00a0\u00a0 : \u2018Cisco Sy\u00ff<\/strong>tems Inc\u2019 <– as you can see we have \u00ff here
      \nMFG INFO\u00a0\u00a0\u00a0\u00a0\u00a0 : Cisco Sy\u00ff\u00a0<\/strong>\u00a0 <– as you can see we have \u00ff again but also the rest of the line is missing
      \nPART NUM\u00a0\u00a0\u00a0\u00a0\u00a0 : 73-11\u00ff<\/strong>89-09\u00a0\u00a0 <– here part number got corrupted<\/li>\n<\/ol>\n

      I would strongly advise to use one of the methods above to check if corruption has not\u00a0occurred\u00a0and fix it before doing UCS upgrade or you’ll have unexpected outage if you don’t. As there is no fix at the moment it does not mean that it will not happen again but at least it is healthy before you start upgrading it.<\/strong><\/p>\n

       <\/p>\n","protected":false},"excerpt":{"rendered":"

      Here is pretty common problem in UCS 2.0 release. At any stage of\u00a0UCS upgrade \u00a0one or more blades go into discovery mode and never finishes it. Depending on the version they can get stuck at any percentage but usually between 4% and 40%. Most of the time a corruption\u00a0occurs\u00a0in SEEPROM of \u00a0M81kr CNA card because […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[380,381],"tags":[420,419,421,422,426,423,498,401,424,425],"_links":{"self":[{"href":"https:\/\/www.kozeniauskas.com\/itblog\/wp-json\/wp\/v2\/posts\/935"}],"collection":[{"href":"https:\/\/www.kozeniauskas.com\/itblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kozeniauskas.com\/itblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kozeniauskas.com\/itblog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kozeniauskas.com\/itblog\/wp-json\/wp\/v2\/comments?post=935"}],"version-history":[{"count":5,"href":"https:\/\/www.kozeniauskas.com\/itblog\/wp-json\/wp\/v2\/posts\/935\/revisions"}],"predecessor-version":[{"id":945,"href":"https:\/\/www.kozeniauskas.com\/itblog\/wp-json\/wp\/v2\/posts\/935\/revisions\/945"}],"wp:attachment":[{"href":"https:\/\/www.kozeniauskas.com\/itblog\/wp-json\/wp\/v2\/media?parent=935"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kozeniauskas.com\/itblog\/wp-json\/wp\/v2\/categories?post=935"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kozeniauskas.com\/itblog\/wp-json\/wp\/v2\/tags?post=935"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}