Cisco Nexus 7000 VDC Administration

This article decribes how to manage Virtual Device Context (VDC) on Cisco Nexus 7010 series. Cisco’s VDC feature helps enable virtualization of a single physical device on one or more logical devices. In this lab testing, we are going to cover several scenario like below:

  • Configuring Admin VDC
  • Configuring VDC Resource and Templates
  • Managing VDCs

Configuring Admin VDC

Admin VDC without migrate option

You can create an Admin VDC in one of the following ways:

  • After a fresh switch bootup.
  • Enter the system admin-vdc command after bootup. All the nonglobal configuration in the default VDC is lost after you enter this command.
  • Use system admin-vdc migrate new vdc name option commmand to migrate a non global configuration on non default vdc to new vdc.

In this section we are going to focus on creating vdcs after the switch bootup. On this lab environment I am using Nexus 7010 Chassis with SUP2E and NX-OS 6.2(16).

Now let’s verify our default vdc configuration.

N7K-ADMIN# show run vdc
...
version 6.2(16)
no system admin-vdc
vdc N7K-ADMIN id 1
  limit-resource module-type m1 m1xl m2xl f2e 
  cpu-share 5
  allocate interface ethernet1/1-12
  limit-resource vlan minimum 16 maximum 4094
  limit-resource monitor-session minimum 0 maximum 2
  limit-resource monitor-session-erspan-dst minimum 0 maximum 23
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 768
  limit-resource u4route-mem minimum 96 maximum 96
  limit-resource u6route-mem minimum 24 maximum 24
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8
  limit-resource monitor-session-inband-src minimum 0 maximum 1
  limit-resource anycast_bundleid minimum 0 maximum 16
  limit-resource monitor-session-mx-exception-src minimum 0 maximum 1
  limit-resource monitor-session-extended minimum 0 maximum 12
N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                       state               mac                 type        lc      
------  --------                       -----               ----------          ---------   ------  
1       N7K-ADMIN                      active              40:55:39:0e:43:41   Ethernet    m1 f1 m1xl m2xl
N7K-ADMIN# show interface description 

-------------------------------------------------------------------------------
Interface                Description                                            
-------------------------------------------------------------------------------
mgmt0                    ***Management_Interface***

-------------------------------------------------------------------------------
Port          Type   Speed   Description
-------------------------------------------------------------------------------
Eth1/1        eth    1000    --
Eth1/2        eth    1000    --
Eth1/3        eth    1000    --
Eth1/4        eth    1000    --
Eth1/5        eth    1000    --
Eth1/6        eth    1000    --
Eth1/7        eth    1000    --
Eth1/8        eth    1000    --
Eth1/9        eth    1000    --
Eth1/10       eth    1000    --
Eth1/11       eth    1000    --
Eth1/12       eth    1000    --

Now configure your system to use admin vdc.

N7K-ADMIN(config)# system admin-vdc 
All non-global configuration from the default vdc will be removed, Are you sure you want to continue? (yes/no) [no] yes
N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None
N7K-ADMIN# show interface description 

-------------------------------------------------------------------------------
Interface                Description                                            
-------------------------------------------------------------------------------
mgmt0                    ***Management_Interface***

Do notice that right now we don’t have any interfaces from physical module interface. Only management interface is allowed on admin vdc. Now let’s try to allocate some interfaces to it and see how admin vdc respons to it.

N7K-ADMIN(config)# vdc N7K-ADMIN  
N7K-ADMIN(config-vdc)# allocate interface ethernet1/1-12M
Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the ports (y/n)?  [yes] 
ERROR: 1 or more interfaces are from a module of type not supported by this vdc

According to the error message, it was clear that admin vdc only allow management interface on it. As it purpose as an admin vdc it necessary to have only one interface.

Admin VDC With Migrate Option

Another method to create admin vdc is by adding migrate option. This method allow you to keep you default vdc configuration. This option is recommended for existing deployments where the default VDC is used for production traffic whose downtime must be minimized.

Let’s verify our default vdc configuration before we do some changes.

N7K-ADMIN# sh run vdc
...
version 6.2(16)
no system admin-vdc
vdc N7K-ADMIN id 1
  limit-resource module-type m1 m1xl m2xl f2e 
  cpu-share 5
  limit-resource vlan minimum 16 maximum 4094
  limit-resource monitor-session minimum 0 maximum 2
  limit-resource monitor-session-erspan-dst minimum 0 maximum 23
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 768
  limit-resource u4route-mem minimum 96 maximum 96
  limit-resource u6route-mem minimum 24 maximum 24
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8
  limit-resource monitor-session-inband-src minimum 0 maximum 1
  limit-resource anycast_bundleid minimum 0 maximum 16
  limit-resource monitor-session-mx-exception-src minimum 0 maximum 1
  limit-resource monitor-session-extended minimum 0 maximum 12
N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Ethernet    m1 m1xl m2xl f2e
N7K-ADMIN# show interface description 

-------------------------------------------------------------------------------
Interface                Description                                            
-------------------------------------------------------------------------------
mgmt0                    --

-------------------------------------------------------------------------------
Port          Type   Speed   Description
-------------------------------------------------------------------------------
Eth1/1        eth    1000    --
Eth1/2        eth    1000    --
Eth1/3        eth    1000    --
Eth1/4        eth    1000    --
Eth1/5        eth    1000    --
Eth1/6        eth    1000    --
Eth1/7        eth    1000    --
Eth1/8        eth    1000    --
Eth1/9        eth    1000    --
Eth1/10       eth    1000    --
Eth1/11       eth    1000    --
Eth1/12       eth    1000    --

Now we will configure admin vdc on our system and add new vdc (N7K-DEV) that will have default vdc configuration migrated to it.

N7K-ADMIN(config)# system admin-vdc migrate N7K-DEV
All non-global configuration from the default vdc will be removed, Are you sure you want to continue? (yes/no) [no] yes
Note: Interface mgmt0 will not have its ip address migrated to the new vdc
Note: During migration some configuration may not be migrated. Example: VTP will need to be reconfigured in the new vdc if it was enabled. Please refer to configuration guide for details
Please wait, this may take a while
Note: Ctrl-C has been temporarily disabled for the duration of this command
N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None    
2       N7K-DEV                           active              40:55:39:0e:43:42   Ethernet    m1 m1xl m2xl f2e
N7K-ADMIN# show run vdc
...
version 6.2(16)
system admin-vdc
vdc N7K-ADMIN id 1
  cpu-share 5
  limit-resource vlan minimum 16 maximum 4094
  limit-resource monitor-session minimum 0 maximum 2
  limit-resource monitor-session-erspan-dst minimum 0 maximum 23
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 768
  limit-resource u4route-mem minimum 96 maximum 96
  limit-resource u6route-mem minimum 24 maximum 24
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8
  limit-resource monitor-session-inband-src minimum 0 maximum 1
  limit-resource anycast_bundleid minimum 0 maximum 16
  limit-resource monitor-session-mx-exception-src minimum 0 maximum 1
  limit-resource monitor-session-extended minimum 0 maximum 12
vdc N7K-DEV id 2
  limit-resource module-type m1 m1xl m2xl f2e 
  cpu-share 5
  allocate interface Ethernet1/1-12
  boot-order 1
  limit-resource vlan minimum 16 maximum 4094
  limit-resource monitor-session minimum 0 maximum 2
  limit-resource monitor-session-erspan-dst minimum 0 maximum 23
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 768
  limit-resource u4route-mem minimum 96 maximum 96
  limit-resource u6route-mem minimum 24 maximum 24
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8
  limit-resource monitor-session-inband-src minimum 0 maximum 1
  limit-resource anycast_bundleid minimum 0 maximum 16
  limit-resource monitor-session-mx-exception-src minimum 0 maximum 1
  limit-resource monitor-session-extended minimum 0 maximum 12

vdc resource template admin-vdc-migrate-template
  limit-resource vlan minimum 16 maximum 4094
  limit-resource monitor-session minimum 0 maximum 2
  limit-resource monitor-session-erspan-dst minimum 0 maximum 23
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 768
  limit-resource u4route-mem minimum 96 maximum 96
  limit-resource u6route-mem minimum 24 maximum 24
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8
  limit-resource monitor-session-inband-src minimum 0 maximum 1
  limit-resource anycast_bundleid minimum 0 maximum 16
  limit-resource monitor-session-mx-exception-src minimum 0 maximum 1
  limit-resource monitor-session-extended minimum 0 maximum 12

After admin vdc created, you will also have vdc resource template create based on admin vdc. We will cover vdc resource template on next section. Now try to login to new vdc using switchto vdc vdc_name and verify that it has the same membership of the interfaces on the default vdc.

N7K-ADMIN# switchto vdc N7K-DEV 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
N7K-ADMIN-N7K-DEV#
N7K-ADMIN-N7K-DEV# sh interface description 

-------------------------------------------------------------------------------
Port          Type   Speed   Description
-------------------------------------------------------------------------------
Eth1/1        eth    1000    --
Eth1/2        eth    1000    --
Eth1/3        eth    1000    --
Eth1/4        eth    1000    --
Eth1/5        eth    1000    --
Eth1/6        eth    1000    --
Eth1/7        eth    1000    --
Eth1/8        eth    1000    --
Eth1/9        eth    1000    --
Eth1/10       eth    1000    --
Eth1/11       eth    1000    --
Eth1/12       eth    1000    --

Configuring VDC Resource and Templates

VDC resource templates set the minimum and maximum limits for shared physical device resources when you create the VDC. The Cisco NX-OS software reserves the minimum limit for the resource to the VDC. Any resources allocated to the VDC beyond the minimum are based on the maximum limit and availability on the device.

Below is one of the example of the vdc resource template we are using.

N7K-ADMIN(config)# vdc resource template new_vdc_template
N7K-ADMIN(config-vdc-template)# limit-resource vlan minimum 20 maximum 4094
N7K-ADMIN(config-vdc-template)#   limit-resource monitor-session minimum 0 maximum 2
N7K-ADMIN(config-vdc-template)#   limit-resource monitor-session-erspan-dst minimum 0 maximum 23
N7K-ADMIN(config-vdc-template)#   limit-resource vrf minimum 2 maximum 4096
N7K-ADMIN(config-vdc-template)#   limit-resource port-channel minimum 0 maximum 768
N7K-ADMIN(config-vdc-template)#   limit-resource u4route-mem minimum 96 maximum 96
N7K-ADMIN(config-vdc-template)#   limit-resource u6route-mem minimum 24 maximum 24
N7K-ADMIN(config-vdc-template)#   limit-resource m4route-mem minimum 58 maximum 58
N7K-ADMIN(config-vdc-template)#   limit-resource m6route-mem minimum 8 maximum 8
N7K-ADMIN(config-vdc-template)#   limit-resource monitor-session-inband-src minimum 0 maximum 1
N7K-ADMIN(config-vdc-template)#   limit-resource anycast_bundleid minimum 0 maximum 16
N7K-ADMIN(config-vdc-template)#   limit-resource monitor-session-mx-exception-src minimum 0 maximum 1
N7K-ADMIN(config-vdc-template)#   limit-resource monitor-session-extended minimum 0 maximum 12
N7K-ADMIN# show run vdc
...
vdc resource template new_vdc_template
  limit-resource vlan minimum 20 maximum 4094
  limit-resource monitor-session minimum 0 maximum 2
  limit-resource monitor-session-erspan-dst minimum 0 maximum 23
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 768
  limit-resource u4route-mem minimum 96 maximum 96
  limit-resource u6route-mem minimum 24 maximum 24
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8
  limit-resource monitor-session-inband-src minimum 0 maximum 1
  limit-resource anycast_bundleid minimum 0 maximum 16
  limit-resource monitor-session-mx-exception-src minimum 0 maximum 1
  limit-resource monitor-session-extended minimum 0 maximum 12

Now create new vdc based on our vdc resource template.

N7K-ADMIN(config)# vdc N7K-PROD template new_vdc_template 
Note:  Creating VDC, one moment please ...
N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None    
2       N7K-DEV                           active              40:55:39:0e:43:42   Ethernet    m1 m1xl m2xl f2e 
3       N7K-PROD                          active              40:55:39:0e:43:43   Ethernet    m1 m1xl m2xl f2e
N7K-ADMIN# show vdc N7K-PROD detail 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc id: 3
vdc name: N7K-PROD
vdc state: active
vdc mac address: 40:55:39:0e:43:43
vdc ha policy: RESTART
vdc dual-sup ha policy: SWITCHOVER
vdc boot Order: 1
CPU Share: 5
CPU Share Percentage: 33%
vdc create time: Fri Apr  7 09:25:56 2017
vdc reload count: 0
vdc uptime: 0 day(s), 0 hour(s), 1 minute(s), 32 second(s)
vdc restart count: 0
vdc type: Ethernet
vdc supported linecards: m1 m1xl m2xl f2e
N7K-ADMIN# show vdc N7K-PROD resource

     Resource                   Min       Max       Used      Unused    Avail    
     --------                   ---       ---       ----      ------    -----    
     vlan                       20        4094      5         15        4089     
     monitor-session            0         2         0         0         2        
     monitor-session-erspan-dst 0         23        0         0         23       
     vrf                        2         4096      2         0         4090     
     port-channel               0         768       0         0         768      
     u4route-mem                96        96        1         95        95       
     u6route-mem                24        24        1         23        23       
     m4route-mem                58        58        1         57        57       
     m6route-mem                8         8         1         7         7        
     monitor-session-inband-src 0         1         0         0         1        
     anycast_bundleid           0         16        0         0         16       
     monitor-session-mx-excepti 0         1         0         0         1        
     monitor-session-extended   0         12        0         0         12

You may see on above resource output. Our vdc was assigned correctly by our configuration template. Let’s allocate some interfaces from the line card and verify it.

N7K-ADMIN(config-vdc)# allocate interface ethernet1/13-18
Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the ports (y/n)?  [yes]
N7K-ADMIN# show vdc N7K-PROD membership 
Flags : b - breakout port
---------------------------------

vdc_id: 3 vdc_name: N7K-PROD interfaces:
        Ethernet1/13          Ethernet1/14          Ethernet1/15          
        Ethernet1/16          Ethernet1/17          Ethernet1/18

Login to vdc N7K-PROD and verify if it already have its interfaces. Since this is a new vdc, it is behave like a new switch. It will ask you to set up password and another administration task just like when you are entering the switch after fresh bootup.

N7K-ADMIN# switchto vdc N7K-PROD 


         ---- System Admin Account Setup ----


Do you want to enforce secure password standard (yes/no) [y]: no

  Enter the password for "admin": 
  Confirm the password for "admin": 

         ---- Basic System Configuration Dialog VDC: 3 ----

This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.

Please register Cisco Nexus7000 Family devices promptly with your
supplier. Failure to register may affect response times for initial
service calls. Nexus7000 devices must be registered to receive 
entitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime
to skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): no
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
N7K-ADMIN-N7K-PROD#
N7K-ADMIN-N7K-PROD# show interface description 

-------------------------------------------------------------------------------
Port          Type   Speed   Description
-------------------------------------------------------------------------------
Eth1/13       eth    1000    --
Eth1/14       eth    1000    --
Eth1/15       eth    1000    --
Eth1/16       eth    1000    --
Eth1/17       eth    1000    --
Eth1/18       eth    1000    --

Managing VDCs

Reloading VDCs

After we create VDCs, we can modify its parameter according to your network environment needs. In this subsection we are focusing on how to do some administrative task on your VDCs. Before we execute the command, let’s verify all VDCs we have on our Nexus 7000.

N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None    
2       N7K-DEV                           active              40:55:39:0e:43:42   Ethernet    m1 m1xl m2xl f2e 
3       N7K-PROD                          active              40:55:39:0e:43:43   Ethernet    f2

Now we will pick vdc N7K-PROD as a target for this test.

N7K-ADMIN# reload vdc N7K-PROD 
Are you sure you want to reload this vdc (y/n)?  [no] yes
N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None    
2       N7K-DEV                           active              40:55:39:0e:43:42   Ethernet    m1 m1xl m2xl f2e 
3       N7K-PROD                          resume in progress  40:55:39:0e:43:43   Ethernet    f2

In order to measure how long the reload process takes, I did a continues ping to the management interface resides on N7K-PROD vdc. After 30 seconds, N7K-PROD vdc back online.

N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None    
2       N7K-DEV                           active              40:55:39:0e:43:42   Ethernet    m1 m1xl m2xl f2e 
3       N7K-PROD                          active              40:55:39:0e:43:43   Ethernet    f2

Suspending VDCs

After assign a reload command to a vdc, now we are going to put a suspend action on it.

N7K-ADMIN(config)# vdc N7K-PROD suspend 
This command will suspend the VDC. (y/n)? [no] yes
Note: Suspending vdc N7K-PROD
N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None    
2       N7K-DEV                           active              40:55:39:0e:43:42   Ethernet    m1 m1xl m2xl f2e 
3       N7K-PROD                          suspended           40:55:39:0e:43:43   Ethernet    f2

Using the same procedure above to measure how long it back online, now let’s resume the vdc.

N7K-ADMIN(config)# no vdc N7K-PROD suspend 
Note: Resuming vdc N7K-PROD

After 30 seconds we can see vdc N7K-PROD is active

N7K-ADMIN# show vdc

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None    
2       N7K-DEV                           active              40:55:39:0e:43:42   Ethernet    m1 m1xl m2xl f2e 
3       N7K-PROD                          active              40:55:39:0e:43:43   Ethernet    f2

Managing VDC Interfaces

When you create a VDC, you can allocate I/O interfaces to the VDC. One important thing regarding port allocation is, it is recommended to allocate all ports on the same port group to the same VDC. Beginning with Cisco NX-OS Release 5.2(1) for Nexus 7000 series devices, all members of a port group are automatically allocated to the VDC when you allocate an interface.

In this lab I am using two Line Cards, N7K-M148GT-11 on slot 1 and N7K-F248XP-25 on slot 2. Let’s see how we can verify port group on each line card.

Module N7K-M148GT-11

M1_Port_Group

Module N7K-F248XP-25
F2_Port_Group

We were omitted the rest of the output because those output is enough for us to understand how port group allocated on the line card. The interface number is listed in the FP port column, and the port ASIC number is listed in the MAC_0 column, which means that in slot 1 on the the above example, interfaces 1 through 12 share the same port ASIC (0) and on the slot 2, interfaces 1 through 4 share the same port ASIC (0).

When interfaces in different VDCs share the same port ASIC, reloading the VDC (with the reload vdc command) or provisioning interfaces to the VDC (with the allocate interface command) might cause short traffic disruptions (of 1 to 2 seconds) for these interfaces. If such behavior is undesirable, make sure to allocate all interfaces on the same port ASIC to the same VDC.

VDC Boot Order

Imagine you have a VDC connect to web servers, another VDC connect to app servers and Another VDC connec to database. In case your switch reload due to power outage or any force major incident, you expect specific VDC to go up first so the apps tier can comunicate properly. Another feature we can adjust on the VDC is boot order. Using boot order value you can manage which VDC should goes up first.

Use below command to adjust boot order value. By default it will have value of 1 on the boot order.

N7K-ADMIN(config)# vdc N7K-DEV
N7K-ADMIN(config-vdc)# boot-order 2
N7K-ADMIN# show vdc detail 
....
vdc id: 2
vdc name: N7K-DEV
vdc state: active
vdc mac address: 40:55:39:0e:43:42
vdc ha policy: BRINGDOWN
vdc dual-sup ha policy: SWITCHOVER
vdc boot Order: 2
CPU Share: 5
CPU Share Percentage: 33%
vdc create time: Tue Apr 11 11:53:32 2017
vdc reload count: 0
vdc uptime: 0 day(s), 0 hour(s), 52 minute(s), 25 second(s)
vdc restart count: 0
vdc type: Ethernet
vdc supported linecards: m1 m1xl m2xl f2e 
...

you cannot modify boot order on admin/default VDC. An Error will occurs when you try to modify it.

N7K-ADMIN(config)# vdc N7K-ADMIN 
N7K-ADMIN(config-vdc)# boot-order 1
ERROR: Default vdc boot order cannot be changed

Now let’s do some test by reloading the box and see the progress from each VDC.

N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                type        lc      
------  --------                          -----               ----------         ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41  Admin       None    
2       N7K-DEV                           create pending      40:55:39:0e:43:42  Ethernet    m1 m1xl m2xl f2e 
3       N7K-PROD                          create pending      40:55:39:0e:43:43  Ethernet    f2
N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None    
2       N7K-DEV                           create in progress  40:55:39:0e:43:42   Ethernet    m1 m1xl m2xl f2e 
3       N7K-PROD                          create pending      40:55:39:0e:43:43   Ethernet    f2
N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None    
2       N7K-DEV                           active              40:55:39:0e:43:42   Ethernet    m1 m1xl m2xl f2e 
3       N7K-PROD                          create in progress  40:55:39:0e:43:43   Ethernet    f2
N7K-ADMIN# show vdc 

Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 

vdc_id  vdc_name                          state               mac                 type        lc      
------  --------                          -----               ----------          ---------   ------  
1       N7K-ADMIN                         active              40:55:39:0e:43:41   Admin       None    
2       N7K-DEV                           active              40:55:39:0e:43:42   Ethernet    m1 m1xl m2xl f2e 
3       N7K-PROD                          active              40:55:39:0e:43:43   Ethernet    f2

As you can see from above output, each VDC will start to active after another. Without boot order configured, each VDC will start to active at the same time.

VDC Hostname

You can change the format of the CLI prompt for nondefault VDCs. By default, the prompt format is a combination of the default VDC name and the nondefault VDC name. You can change the prompt to only contain the nondefault VDC name using no vdc combined-hostname. You can use this command only on spesific non default VDC or for the entire VDCs. Let’s verify non default VDC hostname before we change it.

N7K-ADMIN# switchto vdc N7K-PROD 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
N7K-ADMIN-N7K-PROD#

Apply the config and see the change.

N7K-ADMIN(config)# no vdc combined-hostname
N7K-ADMIN# switchto vdc N7K-PROD 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
N7K-PROD#

Now we have non default VDC hostname without additional name from the admin/default VDC.

VDC Management Interface

Nexus SUP2E module has one physical port for management. As Infromed earlier, this management interface is belong to admin/default VDC.

N7K-ADMIN# show interface description 

-------------------------------------------------------------------------------
Interface                Description                                            
-------------------------------------------------------------------------------
mgmt0                    --

-------------------------------------------------------------------------------
Port          Type   Speed   Description
-------------------------------------------------------------------------------
Eth1/1        eth    1000    --
Eth1/2        eth    1000    --
Eth1/3        eth    1000    --
Eth1/4        eth    1000    --
Eth1/5        eth    1000    --
Eth1/6        eth    1000    --
Eth1/7        eth    1000    --
Eth1/8        eth    1000    --
Eth1/9        eth    1000    --
Eth1/10       eth    1000    --
Eth1/11       eth    1000    --
Eth1/12       eth    1000    --

When we create more VDCs other than default VDC, Management interface is distributed through the VDCs. You can configure each VDC with an IP address with same segment with other IP addresses on the other VDCs. For example I was configured N7K-ADMIN VDC with 10.10.10.1/24, N7K-DEV 10.10.10.2/24, N7K-PROD 10.10.10.3/24 and management PC using 10.10.10.100/24. Do notice that on non default VDC, management interface is not shown when you execute show interface description.

N7K-PROD# show interface description 


-------------------------------------------------------------------------------
Port          Type   Speed   Description
-------------------------------------------------------------------------------
Eth2/1        eth    10G     --
Eth2/2        eth    10G     --
Eth2/3        eth    10G     --
Eth2/4        eth    10G     --

You can configure management interface just like you configure it on the admin/default VDC.

N7K-PROD(config)# interface mgmt 0
N7K-PROD(config-if)# vrf member management 
N7K-PROD(config-if)# ip address 10.10.10.3/24
N7K-PROD(config-if)# Description ***Management_Link***
N7K-PROD(config-if)# no shut
N7K-PROD# show interface description 

-------------------------------------------------------------------------------
Interface                Description                                            
-------------------------------------------------------------------------------
mgmt0             ***Management_Link***

-------------------------------------------------------------------------------
Port          Type   Speed   Description
-------------------------------------------------------------------------------
Eth2/1        eth    10G     --
Eth2/2        eth    10G     --
Eth2/3        eth    10G     --
Eth2/4        eth    10G     --

sources:

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCE, VCP6-DCV
nantoyudi@gmail.com
Advertisements

Change Red Hat 7.x/CentOS 7.x SSH Default Port (SELinux Involved)

This article describes how to change your SSH port on Linux system (in this excercise we use CentOS 7.3) to listen to non default SSH port (TCP Port 22). We will involve SELinux modification to accomodate this change.

1. At the first time let’s verify default configuration and output of sshd service.

    1.1 Verify sshd_config for port configuration.

[root@system1 ~]# cat /etc/ssh/sshd_config | grep "Port 22"  
#Port 22

          Even port 22 is being commented, By default it will use port 22.

    1.2 Confirm that system is now listening on port 22.

[[root@server ~]# ss -tulpn | grep sshd
tcp    LISTEN     0      128       *:22            *:*                   users:(("sshd",pid=5485,fd=3))
tcp    LISTEN     0      128      :::22           :::*                   users:(("sshd",pid=5485,fd=4))

    1.3 SELinux should on Enforcing mode. If not go to /etc/sysconfig/selinux and change the SELinux mode.

[root@system1 ~]# getenforce 
Enforcing

2. Change the default SSH port and restart the service.

    2.1 Modify sshd_config file to use port 20002 as the default port.

[root@system1 ~]# vim /etc/ssh/sshd_config  
...........
Port 20002
...........

    2.2 Restart sshd service.

[root@server ~]# systemctl restart -l sshd
Job for sshd.service failed because a configured resource limit was exceeded. See "systemctl status sshd.service" and "journalctl -xe" for details.
[root@server ~]# systemctl status -l sshd
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: resources) since Sun 2017-02-26 23:43:51 WIB; 6s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 5534 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 5485 (code=exited, status=0/SUCCESS)

Feb 26 23:43:51 server.mylab.com systemd[1]: sshd.service never wrote its PID file. Failing.
Feb 26 23:43:51 server.mylab.com systemd[1]: Failed to start OpenSSH server daemon.
Feb 26 23:43:51 server.mylab.com systemd[1]: Unit sshd.service entered failed state.
Feb 26 23:43:51 server.mylab.com systemd[1]: sshd.service failed.

          You may see the service was failed to start. The easiest thing to troubleshoot this issue is to disable the SELinux (setenforce 0) to get an idea whether it is the reason why it block the sshd service. In this excercise we are going to identify using sealert to get more information regarding it.

    2.3 Check if SELinux is blocking sshd from binding to port 20002/TCP.

[root@server ~]# sealert -a /var/log/audit/audit.log 
100% done
found 3 alerts in /var/log/audit/audit.log
...
SELinux is preventing /usr/sbin/sshd from name_bind access on the tcp_socket port 20002.

*****  Plugin bind_ports (92.2 confidence) suggests   ************************

If you want to allow /usr/sbin/sshd to bind to network port 20002
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 20002
    where PORT_TYPE is one of the following: ssh_port_t, vnc_port_t, xserver_port_t.

*****  Plugin catchall_boolean (7.83 confidence) suggests   ******************

If you want to allow nis to enabled
Then you must tell SELinux about this by enabling the 'nis_enabled' boolean.

Do
setsebool -P nis_enabled 1

*****  Plugin catchall (1.41 confidence) suggests   **************************

If you believe that sshd should be allowed name_bind access on the port 20002 tcp_socket by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'sshd' --raw | audit2allow -M my-sshd
# semodule -i my-sshd.pp


Additional Information:
Source Context                system_u:system_r:sshd_t:s0-s0:c0.c1023
Target Context                system_u:object_r:unreserved_port_t:s0
Target Objects                port 20002 [ tcp_socket ]
Source                        sshd
Source Path                   /usr/sbin/sshd
Port                          20002
Host                          
Source RPM Packages           openssh-server-6.6.1p1-33.el7_3.x86_64
Target RPM Packages           
Policy RPM                    selinux-policy-3.13.1-102.el7_3.13.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     server.mylab.com
Platform                      Linux server.mylab.com 3.10.0-514.6.1.el7.x86_64
                              #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64
Alert Count                   5
First Seen                    2017-02-26 23:43:47 WIB
Last Seen                     2017-02-26 23:44:33 WIB
Local ID                      b4e40db1-4036-4c63-b35e-6ea5f7bb01c8
...

          sealert output gives us a complete information regarding the issue we are facing. Several important information we have highlighted above can be the clue to fix the issue.

    2.4 Verify SELinux port for ssh and do a necessary changes.

[root@server ~]# semanage port -l | grep ssh
ssh_port_t                     tcp      22

          by default SELinux port for ssh is bind to port 22. Add non default port on it.

[root@server ~]# semanage port -a -t ssh_port_t -p tcp 20002
[root@server ~]# semanage port -l | grep ssh
ssh_port_t                     tcp      20002, 22

    2.5 Restart sshd service.

[root@server ~]# systemctl restart sshd
[root@server ~]# systemctl status sshd
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2017-02-26 23:46:43 WIB; 5s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 5689 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 5690 (sshd)
   CGroup: /system.slice/sshd.service
           └─5690 /usr/sbin/sshd

Feb 26 23:46:43 server.mylab.com systemd[1]: Starting OpenSSH server daemon...
Feb 26 23:46:43 server.mylab.com systemd[1]: PID file /var/run/sshd.pid not readable (yet?) after start.
Feb 26 23:46:43 server.mylab.com sshd[5690]: Server listening on 0.0.0.0 port 20002.
Feb 26 23:46:43 server.mylab.com sshd[5690]: Server listening on :: port 20002.
Feb 26 23:46:43 server.mylab.com systemd[1]: Started OpenSSH server daemon.

4. Confirm that system is now listening on port 20002.

[root@server ~]# ss -tulpn | grep sshd
tcp    LISTEN     0      128       *:20002                 *:*              users:(("sshd",pid=5690,fd=3))
tcp    LISTEN     0      128      :::20002                :::*              users:(("sshd",pid=5690,fd=4))

5. Add firewall rule to allow other system accessing this system on port 20002/TCP.

    5.1 Verify current firewall rule.

[root@server ~]# firewall-cmd  --permanent --list-all
public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client https samba ssh
  ports: 3260/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  sourceports: 
  icmp-blocks: 
  rich rules:

          You may see ssh service already allowed, but do notice that this is for ssh with default TCP port ( port 22). Hence you need to add port 20002.

    5.2 Add port 20002 on the firewall and verify it.

[root@server ~]# firewall-cmd --permanent --add-port=20002/tcp
success
[root@server ~]# firewall-cmd --reload
success
[root@server ~]# firewall-cmd  --permanent --list-all
public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client https samba ssh
  ports: 20002/tcp 3260/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  sourceports: 
  icmp-blocks: 
  rich rules:

Happy labbing!!!

Source: Red Hat System Administration III

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCE, VCP6-DCV
nantoyudi@gmail.com

Nexus 7000 SUP2E Compact Flash Failure Recovery

This article describes one of the procedure to recover flash failure on Cisco Nexus 7000 using SUP2E. Cisco has published a bug id CSCus22805 (CCO account required) on their bug documentation. Before we show the procedure and CLI output during the recovery process, We are going to show how Cisco documentation explain regarding this issue.

Background

According to the documentation, Each N7K supervisor 2/2E is equipped with 2 eUSB flash devices in RAID1 configuration, one primary and one mirror. Together they provide non-volatile repositories for boot images, startup configuration and persistent application data. What can happen is over a period of months or years in service, one of these devices may be disconnected from the USB bus, causing the RAID software to drop the device from the configuration. The device can still function normally with 1/2 devices. However, when the second device drops out of the array, the bootflash is remounted as read-only, meaning you cannot save configuration or files to the bootflash, or allow the standby to sync to the active in the event it is reloaded.

Symptoms

  • Compact flash diagnostic failure
  • N7K-SUP2E# show diagnostic result module 1
    
     Current bootup diagnostic level: complete
     Module 5: Supervisor module-2  (Standby)
    
             Test results: (. = Pass, F = Fail, I = Incomplete,
             U = Untested, A = Abort, E = Error disabled)
    
              1) ASICRegisterCheck-------------> .
              2) USB---------------------------> .
              3) NVRAM-------------------------> .
              4) RealTimeClock-----------------> .
              5) PrimaryBootROM----------------> .
              6) SecondaryBootROM--------------> .
              7) CompactFlash------------------> F  <=====
              8) ExternalCompactFlash----------> U
              9) PwrMgmtBus--------------------> U
             10) SpineControlBus---------------> .
             11) SystemMgmtBus-----------------> U
             12) StatusBus---------------------> U
             13) StandbyFabricLoopback---------> .
             14) ManagementPortLoopback--------> .
             15) EOBCPortLoopback--------------> .
             16) OBFL--------------------------> .
  • Unable to perform ‘copy run start’
  • N7K-SUP2E# copy running-config startup-config
     [########################################] 100%
     Configuration update aborted: request was aborted
  • eUSB becomes read-only or is non-responsive
  • ISSU failures, usually when trying to failover to the standby supervisor

Problem Analysis

To diagnose the current state of the compact flash cards you need to use some internal commands Cisco provides on the documentation, those are show system internal raid | grep -A 1 “Current RAID status info” and show system internal file /proc/mdstat. If you have more than one supervisor, you may check it by adding slot x before the internal command, where x is the SUP2/2E slot position. Do notice, since these commands are internal, you might need to enter it completely. Don’t use tab keyboard to syntax completion it won’t working. Below are the output from those internal command related to my case.

N7K-SUP2E# show system internal raid | grep -A 1 "Current RAID status info"
 Current RAID status info:
 RAID data from CMOS = 0xa5 0xc3

From this output you want to look at the number beside of 0xa5 which is 0xc3. You can then use these keys to determine if the primary or secondary compact flash has failed, or both. The above output shows 0xc3 which tells us that both the primary and the secondary compact flashes have failed. Below is the reference table to pull up the information.

Raid Status Info Description
0xf0 No failures reported
0xe1 Primary flash failed
0xd2 Alternate (or mirror) flash failed
0xc3 Both primary and alternate failed
N7K-SUP2E# show system internal file /proc/mdstat
Personalities : [raid1]
md6 : active raid1 sdb6[2](F) sdc6[1]
      77888 blocks [2/1] [_U]
      
md5 : active raid1 sdb5[2](F) sdc5[1]
      78400 blocks [2/1] [_U]
      
md4 : active raid1 sdb4[2](F) sdc4[1]
      39424 blocks [2/1] [_U]
      
md3 : active raid1 sdb3[2](F) sdc3[1]
      1802240 blocks [2/1] [_U]

In this scenario you see that the primary compact flash is not up [_U]. A healthy output will show all blocks as [UU]. Below is the sample of the healty compact flash on my secondary SUP2E.

N7K-SUP2E# slot 2 show system internal file /proc/mdstat
Personalities : [raid1] 
md6 : active raid1 sdc6[0] sdb6[1]
      77888 blocks [2/2] [UU]
      
md5 : active raid1 sdc5[0] sdb5[1]
      78400 blocks [2/2] [UU]
      
md4 : active raid1 sdc4[0] sdb4[1]
      39424 blocks [2/2] [UU]
      
md3 : active raid1 sdc3[0] sdb3[1]
      1802240 blocks [2/2] [UU]

Scenarios

To determine which scenario you are facing, Cisco comes up with several scenarios letter. You will need to use the above commands in the “Problem Analysis” section to correlate with a scenario letter below.

Single supervisor:

Scenario Letter Active Supervisor Active Supervisor Code
A 1 Fail 0xe1 or 0xd2
B 2 Fail 0xc3

Dual supervisor:

Scenario Letter Active Supervisor Standby Supervisor Active Supervisor Code Standby Supervisor Code
C 0 Fail 1 Fail 0xf0 0xe1 or 0xd2
D 1 Fail 0 Fail 0xe1 or 0xd2 0xf0
E 1 Fail 1 Fail 0xe1 or 0xd2 0xe1 or 0xd2
F 2 Fail 0 Fail 0xc3 0xf0
G 0 Fail 2 Fail 0xf0 0xc3
H 2 Fail 1 Fail 0xc3 0xe1 or 0xd2
I 1 Fail 2 Fail 0xe1 or 0xd2 0xc3
J 2 Fail 2 Fail 0xc3 0xc3

On the table above, scenario F is highlighted. That is because we are going to show you how we were accomplished this recovery activity on our client using this scenario.

Recovery Procedure

Cisco has published a procedure for every scenarios listed on the document. When we dealing with scenario F a non-impacting recovery is possible. Below are the summary of the procedure in scenario F:

  • Backup running configuration for all vdc externally. You can use logging facility on your ssh terminal for “show running-config vdc-all” command.
  • Compare runnning configuration (show running-config vdc-all) and startup configuration (show startup-config vdc-all). Evaluate missing configuration on running configuration.
  • Perform supervisor switchover using “system switchover“.
  • New standby supervisor will begin rebooting. During this time you will want to add any missing configuration back to the new active.
  • New standby should reach “ha-standby” state. Use “show module” command to verify it alternatively you might use “show redundancy status” to ensure the all states on “Other supervisor” are “HA standby
  • If the new standby comes up in a “powered-up” state, you will need to manually bring it back online. This can be done by issuing the following commands, where “x” is the standby module stuck in a “powered-up” state:
  • (config)# out-of-service module x
    (config)# no poweroff module x
  • If you see that the standby keeps getting stuck in the powered-up state and ultimately keeps power cycling after the steps above, this is likely due to the active reloading the standby for not coming up in time. To resolve this, configure the following using ‘x’ for the standby slot that stuck in powered-up:
    (config)# system standby manual-boot
    (config)# reload module x force-dnld
  • Once the standby is back online in an “ha-standby” state, you will then need to run the recovery tool to ensure that the recovery is complete. The tool can be downloaded at the following link:
    recovery tool
  • unzipped recovery tool, and uploaded it to the bootflash of the box, you will need to execute the following command: “load bootflash:n7000-s2-flash-recovery-tool.10.0.2.gbin
  • check the recovery status with “show system internal file /proc/mdstat” command/

Procedure Output

Ok. let’s move on to the execution section. To avoid any confusion regarding the supervisor status. I will give a name to the supervisor like the following. Sup1Active means Supervisor one in active state and Sup2Standby means supervisor two on standby state. State on each supervisor will change during the procedure, please be aware with it.

Switchover Supervisor

On “Sup1Active do supevisor switchover. Sup1 will start to reboot and will be Sup1Standby.

N7K-SUP2E# system switchover 
N7K-SUP2E# 
User Access Verification
N7K-SUP2E login: 
>>>
>>>
>>>
NX7k SUP BIOS version ( 2.11 ) : Build - 01/09/2013 18:16:20
PM FPGA Version : 0x00000024 
Power sequence microcode revision - 0x00000009 : card type - 10156EEA0
Booting Spi Flash : Primary 
  CPU Signature - 0x000106e4: Version - 0x000106e0 
  CPU - 2 : Cores - 4 : HTEn - 1 : HT - 2 : Features - 0xbfebfbff 
  FSB Clk - 532 Mhz :  Freq - 2143 Mhz - 2128 Mhz 
  MicroCode Version : 0x00000002 
  Memory - 32768 MB : Frequency - 1067 MHZ 
  Loading Bootloader: Done 
  IO FPGA Version   : 0x1000d 
  PLX Version       : 861910b5
Bios digital signature verification - Passed
USB bootflash status : [1-1:0-0]
...

Below are the output from the Sup2Active, previously Sup2Standby

N7K-SUP2E(standby)# 2017 Apr 22 01:58:02  %$ VDC-1 %$ Apr 22 01:58:02 %KERN-2-SYSTEM_MSG: [18173381.026292] Switchover started by redundancy driver - kernel
2017 Apr 22 01:58:02  %$ VDC-1 %$ %SYSMGR-2-HASWITCHOVER_PRE_START: This supervisor is becoming active (pre-start phase).
2017 Apr 22 01:58:02  %$ VDC-1 %$ %SYSMGR-2-HASWITCHOVER_START: Supervisor 2 is becoming active.
2017 Apr 22 01:58:02  %$ VDC-1 %$ %SYSMGR-2-SWITCHOVER_OVER: Switchover completed.
N7K-SUP2E# show module
Mod  Ports  Module-Type                         Model              Status
---  -----  ----------------------------------- ------------------ ----------
1    0      Supervisor module-2                                    powered-up
2    0      Supervisor module-2                 N7K-SUP2E          active *
3    48     1000 Mbps Optical Ethernet XL Modul N7K-M148GS-11L     ok
4    24     10 Gbps Ethernet Module             N7K-M224XP-23L     ok
...

On my case, Sup1Standby was not able to back online. When you see highlighted lines below during the bootup process, it is a sign that your Sup is fail to boot and it will end on switch boot mode.

...
RAID assembly failed. Stopping all RAID partitions...
Trying to mount bootflash /dev/sdd3...
mount: block device /dev/sdd3 is write-protected, mounting read-only
mount: wrong fs type, bad option, bad superblock on /dev/sdd3,
       or too many mounted file systems
/dev/sdd3 mount failed, trying /dev/sdc3...
/dev/sdc3: Input/output error
mount: block device /dev/sdc3 is write-protected, mounting read-only
/dev/sdc3: Input/output error
mount: /dev/sdc3 is not a valid block device
Cannot find any valid bootflash partitions.
....
switch(boot)#

Even on switch boot mode your are not able to load the kickstart image since Sup doesn’t aware of any flash storage consist of kickstart image and operating system image.

switch(boot)# dir 

Usage for bootflash: filesystem 
   98643968 bytes used
  320786432 bytes free
  419430400 bytes total

Hence, we need to move on to the next procedure to bring Sup1Standby online. On Sup2Active do below command.

N7K-SUP2E(config)# out-of-service module 1
N7K-SUP2E(config)# 2017 Apr 22 02:00:46  %$ VDC-1 %$ %PLATFORM-2-MOD_PWRDN: Module 1 powered down (Serial number )
2017 Apr 22 02:00:46 N7K-SUP2E-VDC-4 %$ VDC-4 %$ %PLATFORM-2-MOD_PWRDN: Module 1 powered down (Serial number )
2017 Apr 22 02:00:46 N7K-SUP2E-VDC-2 %$ VDC-2 %$ %PLATFORM-2-MOD_PWRDN: Module 1 powered down (Serial number )
2017 Apr 22 02:00:46 N7K-SUP2E-VDC-3 %$ VDC-3 %$ %PLATFORM-2-MOD_PWRDN: Module 1 powered down (Serial number )
N7K-SUP2E(config)# no poweroff module 1

From Sup1Standby console, you will see it begin to bootup. When you see highlighted lines below during the bootup process, it is a sign that your Sup is in a good state.

...
Trying to mount bootflash /dev/sdd3...
Mounted primary /dev/sdd3 as /bootflash
Existing bootflash found, saving files...
Saving n7000-s2-dk9-npe.6.1.1.bin
Saving n7000-s2-dk9.6.1.2.bin
Saving n7000-s2-kickstart-npe.6.1.1.bin
Saving n7000-s2-kickstart.6.1.2.bin
Initializing the system...
Unmounting file systems...
Making partitions on physical devices...
Initializing RAID services...
Initializing startup-config and licenses...
mke2fs 1.35 (28-Feb-2004)
Checking for bad blocks (read-only test): done                        
mke2fs 1.35 (28-Feb-2004)
Checking for bad blocks (read-only test): done                        
Formatting PSS:
mke2fs 1.35 (28-Feb-2004)
Checking for bad blocks (read-only test): done                        
Formatting bootflash...
mke2fs 1.35 (28-Feb-2004)
Checking for bad blocks (read-only test): done                        
Fri Jan 3 19:04:29 2017: RAIDMON: Data(0x0) provided saved successfully to CMOS
Initialization completed - No reinit of CMOS/NVRAM
Copying saved files back to bootflash...
Checking obfl filesystem.
Checking all filesystems..... done.
Warning: switch is starting up with default configuration
rLoading system software
/bootflash//n7000-s2-dk9.6.1.2.bin read done
System image digital signature verification successful.
Uncompressing system image: bootflash:/n7000-s2-dk9.6.1.2.bin Fri Jan 3 19:06:12 UTC 2017
blogger: nothing to do.

..done Fri Jan 3 19:06:15 UTC 2017
Load plugins that defined in image conf: /isan/plugin_img/img.conf
Loading plugin 0: core_plugin...
num srgs 1
0: swid-core-sup2dc3, swid-core-sup2dc3
num srgs 1
0: swid-sup2dc3-ks, swid-sup2dc3-ks
INIT: Entering runlevel: 3



User Access Verification
N7K-SUP2E(standby) login:

Hence we need to wait until Sup1Standby reach “ha-standby” state. In this situation we would prefer use “show redundancy status” command to “show module” command from Sup2Active. Because we can see the Sup1Standby progress until “ha-standby” state.

N7K-SUP2E# show redundancy status 
Redundancy mode
---------------
      administrative:   HA
         operational:   None

This supervisor (sup-2)
-----------------------
    Redundancy state:   Active
    Supervisor state:   Active
      Internal state:   Active with HA standby

Other supervisor (sup-1)
------------------------
    Redundancy state:   Standby

    Supervisor state:   Unknown
      Internal state:   Other
...
N7K-SUP2E# show redundancy status 
Redundancy mode
---------------
      administrative:   HA
         operational:   None

This supervisor (sup-2)
-----------------------
    Redundancy state:   Active
    Supervisor state:   Active
      Internal state:   Active with HA standby

Other supervisor (sup-1)
------------------------
    Redundancy state:   Standby

    Supervisor state:   HA standby
      Internal state:   HA synchronization in progress
...
N7K-SUP2E# show redundancy status 
Redundancy mode
---------------
      administrative:   HA
         operational:   HA

This supervisor (sup-2)
-----------------------
    Redundancy state:   Active
    Supervisor state:   Active
      Internal state:   Active with HA standby

Other supervisor (sup-1)
------------------------
    Redundancy state:   Standby

    Supervisor state:   HA standby
      Internal state:   HA standby
...

Sup1Standby is the problematic Sup with the flash failure, after login prompt occurs. Login to Sup1Standby and execute command “show system internal file /proc/mdstat” to see recovery progress on this Sup (We don’t need to load recovery tool on Sup1Standby. Reload procedure will automatically recover it flash).

N7K-SUP2E(standby)#  show system internal file /proc/mdstat
Personalities : [raid1] 
md6 : active raid1 sdd6[2] sdc6[1]
      77888 blocks [2/1] [_U]
        resync=DELAYED
      
md5 : active raid1 sdd5[2] sdc5[1]
      78400 blocks [2/1] [_U]
        resync=DELAYED
      
md4 : active raid1 sdd4[2] sdc4[1]
      39424 blocks [2/1] [_U]
        resync=DELAYED
      
md3 : active raid1 sdd3[2] sdc3[1]
      1802240 blocks [2/1] [_U]
      [=========>...........]  recovery = 45.4% (819648/1802240) finish=1.2min s
peed=13142K/sec

Repeat the command above until you see the result like below, when it does your Sup1Standby is ready.

N7K-SUP2E(standby)#  show system internal file /proc/mdstat
Personalities : [raid1] 
md6 : active raid1 sdd6[0] sdc6[1]
      77888 blocks [2/2] [UU]
      
md5 : active raid1 sdd5[0] sdc5[1]
      78400 blocks [2/2] [UU]
      
md4 : active raid1 sdd4[0] sdc4[1]
      39424 blocks [2/2] [UU]
      
md3 : active raid1 sdd3[0] sdc3[1]
      1802240 blocks [2/2] [UU]

Execute Recovery Tool

As we run the procedure on scenario F, it is not necessary to execute the recovery tool on the Sup2Active, since Sup1Standby is the only problemactic Sup with flash failure. But in our case, after the supervisor switchover even though raid status info shows 0xf0, we were identified that Sup2Active raid status is not in [UU] state. You can do save configuration to startup at this state.

N7K-SUP2E# show system internal raid 
Current RAID status info:
RAID data from CMOS = 0xa5 0xf0
RAID data from driver disks 0 bad 0 name 
Bootflash: /dev/sdc
Mirrorflash: /dev/sdd

Current RAID status:
Personalities : [raid1] 
md6 : active raid1 sdc6[0]
      77888 blocks [2/1] [U_]
      
md5 : active raid1 sdc5[0]
      78400 blocks [2/1] [U_]
      
md4 : active raid1 sdc4[0]
      39424 blocks [2/1] [U_]
      
md3 : active raid1 sdc3[0]
      1802240 blocks [2/1] [U_]

Hence we need to execute the recovery tool. When you execute the tool, it will automatically copying it to the standby Sup if you have redundant Sup. Do notice on the output, since Sup1Standby already recovered it will not attempt any recovery action on it. Execute below command to run the tool.

N7K-SUP2E# load bootflash:n7000-s2-flash-recovery-tool.10.0.2.gbin
Loading plugin version 10.0(2)
###############################################################
  Warning: debug-plugin is for engineering internal use only!
  For security reason, plugin image has been deleted.
###############################################################
INFO: Running on active slot 2, checking if a ha-standby is available...
INFO: Standby present in slot 1. Copying the recovery tool...
###############################################################
  Warning: debug-plugin is for engineering internal use only!
  For security reason, plugin image has been deleted.
###############################################################
INFO: Running on the standby in slot 1, Checking RAID status...
INFO: Both disks are found to be healthy.
INFO: Verifying RAID configuration. Got primary=sdb Secondary=sdd
INFO: RAID device md3 is healthy.
INFO: RAID device md4 is healthy.
INFO: RAID device md5 is healthy.
INFO: RAID device md6 is healthy.
INFO: No recovery was attempted on module 1. All flashes left intact.
INFO: A detailed copy of the this log was saved as volatile:flash_repair_log_mod1.tgz.
INFO: Recovery procedures complete on module 1.
INFO: Please check for any errors in previous messages.
INFO: Run 'show system internal file /proc/mdstat' and check 'up status' [UU] for all disks.
INFO: Run 'show diagnostic result module ' on all available supervisor slots.
INFO: And restart CompactFlash test (7) instances if not in running state.
Loading plugin version 10.0(2)
INFO: Now starting the flash recovery procedures on active.
INFO: Primary=sdc(sdc) Secondary=sdd(sdd) Working=sdc
WARNING: Attempting recovery of secondary device sdd
INFO: Removing /dev/sdd from RAID configuration...
INFO: Resetting secondary flash...
INFO: Found secondary device sdd in 9 seconds.
INFO: Running health checks on the recovered device /dev/sdd...
INFO: Basic I/O tests passed. /dev/sdd looks healthy and responsive.
INFO: Verifying RAID configuration. Got primary=sdc Secondary=sdd
INFO: sdc3 is already a part of md3.
INFO: Adding sdd3 back into md3 RAID configuration...
INFO: sdc4 is already a part of md4.
INFO: Adding sdd4 back into md4 RAID configuration...
INFO: sdc5 is already a part of md5.
INFO: Adding sdd5 back into md5 RAID configuration...
INFO: sdc6 is already a part of md6.
INFO: Adding sdd6 back into md6 RAID configuration...
INFO: Resetting RAID status in CMOS...
WARNING: Flash recovery attempted on module 2.
INFO: A detailed copy of the this log was saved as volatile:flash_repair_log_mod2.tgz.
INFO: Recovery procedures complete on module 2.
INFO: Please check for any errors in previous messages.
INFO: Run 'show system internal file /proc/mdstat' and check 'up status' [UU] for all disks.
INFO: Run 'show diagnostic result module ' on all available supervisor slots.
INFO: And restart CompactFlash test (7) instances if not in running state.
N7K-SUP2E# show system internal file /proc/mdstat
Personalities : [raid1] 
md6 : active raid1 sdd6[2] sdc6[0]
      77888 blocks [2/1] [U_]
        resync=DELAYED
      
md5 : active raid1 sdd5[2] sdc5[0]
      78400 blocks [2/1] [U_]
        resync=DELAYED
      
md4 : active raid1 sdd4[2] sdc4[0]
      39424 blocks [2/1] [U_]
        resync=DELAYED
      
md3 : active raid1 sdd3[2] sdc3[0]
      1802240 blocks [2/1] [U_]
      [==>..................]  recovery = 14.7% (265984/1802240) finish=2.0min s
peed=12665K/sec

Wait until all blocks recover. Now you have all your flashes works.

N7K-SUP2E# show diagnostic result module 2

Current bootup diagnostic level: complete
Module 2: Supervisor module-2  (Active)

        Test results: (. = Pass, F = Fail, I = Incomplete,
        U = Untested, A = Abort, E = Error disabled)

         1) ASICRegisterCheck-------------> .
         2) USB---------------------------> .
         3) NVRAM-------------------------> .
         4) RealTimeClock-----------------> .
         5) PrimaryBootROM----------------> .
         6) SecondaryBootROM--------------> .
         7) CompactFlash------------------> .
         8) ExternalCompactFlash----------> U
         9) PwrMgmtBus--------------------> .
        10) SpineControlBus---------------> .
        11) SystemMgmtBus-----------------> .
        12) StatusBus---------------------> .
        13) StandbyFabricLoopback---------> .
        14) ManagementPortLoopback--------> .
        15) EOBCPortLoopback--------------> .
        16) OBFL--------------------------> .

Don’t forget to save all configuration to startup config.

N7K-SUP2E# copy running-config startup-config vdc-all
[########################################] 100%
Copy complete.

Source:

Nexus 7000 Supervisor 2/2E Compact Flash Failure Recovery

Contributor:
Muhammad Benny
Network Engineer 

Dirga Bramantyo
Network Engineer - CCNP

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCE, VCP6-DCV
nantoyudi@gmail.com

Reset Root Password on Red Hat7.x/CentOS7.x

Recovering the root password is a trivial task while still logged in as an administrator or a user with full sudo access, but is slightly more involved when an administrator is not logged in. To recover the root password, use the following procedure:

  1. Reboot the system, press e to edit the selected entry. Move the cursor to the kernel command line (the line that starts with linux16). Append rd.break (this will break just before control is handed from the init ramfs to the actual system). Press Crtl+x to boot with the changes. At this point, a root shell will be presented, with the root file system for the actual system mounted read-only on /sysroot.
  2. Remount /sysroot as read-write.
    switch_root:/# mount -oremount,rw /sysroot
  3. Switch into a chroot jail, where /sysroot is treated as the root of the file system tree.
    switch_root:/# chroot /sysroot
  4. Set a new root password
    sh-4.2# passwd root
  5. Make sure that all unlabeled files (including /etc/shadow at this point) get relabeled during boot.
    sh-4.2# touch /.autorelabel
  6. Type exit twice. The first will exit the chroot jail, and the second will exit the initramfs debug shell.

source:
Red Hat System Administration III

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCE, VCP6-DCV
nantoyudi@gmail.com

Register and subscribe Red Hat 7.3 Packages

This article discribes how to register your Red Hat system to Red Hat subscription manager, enable some repositories and verify it. In this article I am using Red Hat Enterprise Linux 7.3 on virtual environment. You can acquire account for this subscription process on Red Hat portal as a Red Hat developer.

Now login to your system and check your subcriptions status. At this point you will see your system is not registered to any Red Hat subscription packages.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
repolist: 0

Register your system to Red Hat subscription management. Use the following command followed by the credential you acquired from the developer portal.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager register --username=mymail@email.com --password=mypassword
Registering to: subscription.rhsm.redhat.com:443/subscription
The system has been registered with ID: abcdefg-hijkl-mnop-bf16-cfa2dfcebbb4

Once you were registered to the subscription management, you may see the available subscription you may use on your system. Use the following command.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager list --available
+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+
Subscription Name:   Red Hat Enterprise Linux Developer Suite
Provides:            Red Hat Software Collections (for RHEL Server)
                     Red Hat Container Development Kit
                     MRG Realtime
                     Red Hat EUCJP Support (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Server - Extended Update Support
                     Red Hat Beta
                     Oracle Java (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node)
                     dotNET on RHEL Beta (for RHEL Server)
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Server) - Extended Update
                     Support
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
                     Oracle Java (for RHEL Server)
                     Red Hat Container Images
                     Red Hat Enterprise Linux for Real Time
                     dotNET on RHEL (for RHEL Server)
                     Red Hat Enterprise Linux Atomic Host
                     Red Hat S-JIS Support (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server)
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Atomic Host Beta
                     Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support
                     Red Hat Container Images Beta
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Server
                     Red Hat Enterprise Linux High Availability (for RHEL Server)
                     Red Hat Software Collections Beta (for RHEL Server)
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server)
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Server)
                     Red Hat Developer Toolset (for RHEL Server)
SKU:                 RH2262474
Contract:            11293058
Pool ID:             8a85f9815af00aed015af02fffbe5bb4
Provides Management: Yes
Available:           100
Suggested:           1
Service Level:       Self-Support
Service Type:        L1-L3
Subscription Type:   Standard
Ends:                03/21/2018
System Type:         Virtual

Type yum repolist to check if we run a registered system. At this point, you may see your system is registered but is not receiving any updates. It is because you are not subcribe to any subscription package list.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is registered to Red Hat Subscription Management, but is not receiving updates. You can use subscription-manager to assign subscriptions.
repolist: 0

Enable subscription on your system use the following command followed by the pool ID from the subscription list.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager subscribe --pool=8a85f9815af00aed02846f7sffbe5bb4
Successfully attached a subscription for: Red Hat Enterprise Linux Developer Suite

To check your enabled subscription briefly, you may use the following command.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager list

+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux Server
Product ID:     69
Version:        7.3
Arch:           x86_64
Status:         Subscribed
Status Details: 
Starts:         03/21/2017
Ends:           03/21/2018

To check detail information on your enabled subscription, use the following command.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Red Hat Enterprise Linux Developer Suite
Provides:            Red Hat Enterprise Linux High Performance Networking (for RHEL Server) - Extended Update
                     Support
                     Oracle Java (for RHEL Server)
                     Red Hat EUCJP Support (for RHEL Server) - Extended Update Support
                     dotNET on RHEL Beta (for RHEL Server)
                     Red Hat Beta
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
                     MRG Realtime
                     Red Hat Developer Toolset (for RHEL Server)
                     Red Hat Enterprise Linux Atomic Host Beta
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux for Real Time
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server)
                     Red Hat Container Images Beta
                     Red Hat Enterprise Linux High Availability (for RHEL Server)
                     Red Hat Container Development Kit
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node)
                     Red Hat Enterprise Linux Server - Extended Update Support
                     Red Hat Enterprise Linux Server
                     Red Hat Enterprise Linux Atomic Host
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support
                     Oracle Java (for RHEL Server) - Extended Update Support
                     Red Hat Software Collections (for RHEL Server)
                     dotNET on RHEL (for RHEL Server)
                     Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support
                     Red Hat Software Collections Beta (for RHEL Server)
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server)
                     Red Hat Container Images
                     Red Hat S-JIS Support (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Server)
SKU:                 RH2262474
Contract:            11293058
Account:             5920534
Serial:              7701382153394857656
Pool ID:             8a85f9815af00aed02846f7sffbe5bb4
Provides Management: Yes
Active:              True
Quantity Used:       1
Service Level:       Self-Support
Service Type:        L1-L3
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              03/21/2017
Ends:                03/21/2018
System Type:         Virtual

Type yum repolist to confirm that now we have some repositories source for the system.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
repo id                                        repo name                                                    status
!rhel-7-server-rpms/7Server/x86_64             Red Hat Enterprise Linux 7 Server (RPMs)                     14,050
!rhel-7-server-rt-beta-rpms/x86_64             Red Hat Enterprise Linux for Real Time Beta (RHEL 7 Server)      15
!rhel-7-server-rt-rpms/7Server/x86_64          Red Hat Enterprise Linux for Real Time (RHEL 7 Server) (RPMs    185
!rhel-ha-for-rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux High Availability (for RHEL 7 Serve    291
!rhel-rs-for-rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux Resilient Storage (for RHEL 7 Serve    359
repolist: 14,900

Type yum repolist all to see all repository avalilable on this subscription.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum repolist all
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
repo id                                                             repo name                      status
rh-gluster-3-client-for-rhel-7-server-debug-rpms/7Server/x86_64     Red Hat Storage Native Client  disabled
...............
!rhel-7-server-rpms/7Server/x86_64                                  Red Hat Enterprise Linux 7 Ser enabled: 14,050
...............
!rhel-7-server-rt-beta-rpms/x86_64                                  Red Hat Enterprise Linux for R enabled:     15
...............
!rhel-7-server-rt-rpms/7Server/x86_64                               Red Hat Enterprise Linux for R enabled:    185
...............
!rhel-ha-for-rhel-7-server-rpms/7Server/x86_64                      Red Hat Enterprise Linux High  enabled:    291
...............
!rhel-rs-for-rhel-7-server-rpms/7Server/x86_64                      Red Hat Enterprise Linux Resil enabled:    359
...............
repolist: 14,900

You may enable or disable spesific repository with the following command. Enable or disable the repo from the available repository list.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager repos --disable=rhel-ha-for-rhel-7-server-rpms
Repository 'rhel-ha-for-rhel-7-server-rpms' is disabled for this system.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager repos --enable=rhel-7-server-extras-rpms
Repository 'rhel-7-server-extras-rpms' is enabled for this system.

You may use command yum repolist to refresh the repository lists that we are using.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
repo id                                repo name                                                            status
!rhel-7-server-extras-rpms/x86_64      Red Hat Enterprise Linux 7 Server - Extras (RPMs)                       432
!rhel-7-server-rpms/7Server/x86_64     Red Hat Enterprise Linux 7 Server (RPMs)                             14,050
!rhel-7-server-rt-beta-rpms/x86_64     Red Hat Enterprise Linux for Real Time Beta (RHEL 7 Server) (RPMs)       15
!rhel-7-server-rt-rpms/7Server/x86_64  Red Hat Enterprise Linux for Real Time (RHEL 7 Server) (RPMs)           185
repolist: 14,682

Once all set, you may need to update your system to receive latest update for each package from Red Hat Subscription Management.

    root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum update
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
rhel-7-server-extras-rpms                                                               | 3.4 kB  00:00:00     
rhel-7-server-rpms                                                                      | 3.5 kB  00:00:00     
rhel-7-server-rt-beta-rpms                                                              | 4.0 kB  00:00:00     
rhel-7-server-rt-rpms                                                                   | 4.0 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package openjpeg-libs.x86_64 0:1.5.1-10.el7 will be updated
---> Package openjpeg-libs.x86_64 0:1.5.1-16.el7_3 will be an update
---> Package tzdata.noarch 0:2017a-1.el7 will be updated
---> Package tzdata.noarch 0:2017b-1.el7 will be an update
---> Package tzdata-java.noarch 0:2017a-1.el7 will be updated
---> Package tzdata-java.noarch 0:2017b-1.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved



============================================================================================================

 Package               Arch                Version                     Repository                       Size


============================================================================================================

Updating:
 openjpeg-libs         x86_64              1.5.1-16.el7_3              rhel-7-server-rpms               86 k
 tzdata                noarch              2017b-1.el7                 rhel-7-server-rpms              443 k
 tzdata-java           noarch              2017b-1.el7                 rhel-7-server-rpms              182 k

Transaction Summary


============================================================================================================

Upgrade  3 Packages

Total download size: 711 k
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for rhel-7-server-rpms
(1/3): openjpeg-libs-1.5.1-16.el7_3.x86_64.rpm                                          |  86 kB  00:00:01     
(2/3): tzdata-java-2017b-1.el7.noarch.rpm                                               | 182 kB  00:00:01     
(3/3): tzdata-2017b-1.el7.noarch.rpm                                                    | 443 kB  00:00:03     


-----------------------------------------------------------------------------------------------------------

Total                                                                          183 kB/s | 711 kB  00:00:03     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Updating   : tzdata-2017b-1.el7.noarch                                                               1/6 
  Updating   : tzdata-java-2017b-1.el7.noarch                                                          2/6 
  Updating   : openjpeg-libs-1.5.1-16.el7_3.x86_64                                                     3/6 
  Cleanup    : tzdata-2017a-1.el7.noarch                                                               4/6 
  Cleanup    : tzdata-java-2017a-1.el7.noarch                                                          5/6 
  Cleanup    : openjpeg-libs-1.5.1-10.el7.x86_64                                                       6/6 
rhel-7-server-extras-rpms/x86_64/productid                                              | 2.1 kB  00:00:00     
rhel-7-server-rpms/7Server/x86_64/productid                                             | 2.1 kB  00:00:00     
rhel-7-server-rt-beta-rpms/x86_64/productid                                             | 2.1 kB  00:00:00     
rhel-7-server-rt-rpms/7Server/x86_64/productid                                          | 2.1 kB  00:00:00     
  Verifying  : openjpeg-libs-1.5.1-16.el7_3.x86_64                                                     1/6 
  Verifying  : tzdata-java-2017b-1.el7.noarch                                                          2/6 
  Verifying  : tzdata-2017b-1.el7.noarch                                                               3/6 
  Verifying  : tzdata-java-2017a-1.el7.noarch                                                          4/6 
  Verifying  : openjpeg-libs-1.5.1-10.el7.x86_64                                                       5/6 
  Verifying  : tzdata-2017a-1.el7.noarch                                                               6/6 

Updated:
  openjpeg-libs.x86_64 0:1.5.1-16.el7_3     tzdata.noarch 0:2017b-1.el7     tzdata-java.noarch 0:2017b-1.el7    

Complete!



Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCE, VCP6-DCV
nantoyudi@gmail.com

Cisco DMVPN Dual Hub Single Topology

Following up our previous articleΒ on DMVPN, we are going to implement another model of DMVPN deployment, DMVPN dual hub single topology. We have prepared the topology below as a guidance.

DMVPN_Dual_Hub_Single_Topology

The dual hub with single layout topology is fairly to set up. The idea in this case it to have a single DMPVN “cloud” with all hubs, and all spokes connected to this single subnet (“cloud”). On above topology, you will have two static tunnels from each spoke to the hubs (R1-HUB and R5-HUB). Since the spoke router are routing neighbors with the hub routers over the same mGRE tunnel interface, you cannot use link or interface differences (like metric, cost, delay or bandwidth) to modify the dynamic routing protocol metric toprefer one hub over the other hub when they are both up. If this preference is needed, then you can utilize the routing protocol feature to engineering traffic flow.

Router Configuration

Since we don’t have any major differences on the configuration. We will only show the additional configuration for this purpose. Any full configuration you may refer to my previous post on DMVPN.

R1-HUB

router eigrp 100
 network 10.155.145.1 0.0.0.0
 network 192.168.123.1 0.0.0.0

R5-HUB

crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 14
crypto isakmp key cisco123 address 10.155.0.0     
!
!
crypto ipsec transform-set MYTRANSFORMSET esp-aes esp-sha-hmac 
 mode tunnel
!
crypto ipsec profile MYPROFILE
 set security-association lifetime seconds 900
 set transform-set MYTRANSFORMSET 
 
interface Tunnel0
 ip address 192.168.123.5 255.255.255.0
 no ip redirects
 ip mtu 1440
 no ip next-hop-self eigrp 100
 no ip split-horizon eigrp 100
 ip nhrp authentication cisco123
 ip nhrp map multicast dynamic
 ip nhrp map 192.168.123.1 10.155.16.1
 ip nhrp map multicast 10.155.16.1
 ip nhrp network-id 1
 ip nhrp nhs 192.168.123.1
 tunnel source GigabitEthernet0/1
 tunnel mode gre multipoint
 tunnel key 12345
 tunnel protection ipsec profile MYPROFILE
 
 router eigrp 100
 network 10.155.145.5 0.0.0.0
 network 192.168.123.5 0.0.0.0

I did two changes on R1-HUB, remove the loopback IP from EIGRP proceess and put datacenter segment instead. The configuration for R5-HUB is basically the same as the R1-HUB configuration with the appropriate IP address changes. The one main difference is that R5-HUB is also a spoke (or client) of R1-HUB, making R1-HUB the primary hub and R5-HUB the secondary hub.

R2-Spoke

interface Tunnel0
 ip address 192.168.123.2 255.255.255.0
 no ip redirects
 ip mtu 1440
 ip nhrp authentication cisco123
 ip nhrp map multicast dynamic
 ip nhrp map 192.168.123.1 10.155.16.1
 ip nhrp map multicast 10.155.16.1
 ip nhrp map 192.168.123.5 10.155.56.5
 ip nhrp map multicast 10.155.56.5
 ip nhrp network-id 1
 ip nhrp nhs 192.168.123.1
 ip nhrp nhs 192.168.123.5
 tunnel source GigabitEthernet0/2
 tunnel mode gre multipoint
 tunnel key 12345
 tunnel protection ipsec profile MYPROFILE

R3-Spoke

interface Tunnel0
 ip address 192.168.123.3 255.255.255.0
 no ip redirects
 ip mtu 1440
 ip nhrp authentication cisco123
 ip nhrp map multicast dynamic
 ip nhrp map 192.168.123.1 10.155.16.1
 ip nhrp map multicast 10.155.16.1
 ip nhrp map 192.168.123.5 10.155.56.5
 ip nhrp map multicast 10.155.56.5
 ip nhrp network-id 1
 ip nhrp nhs 192.168.123.1
 ip nhrp nhs 192.168.123.5
 tunnel source GigabitEthernet0/1
 tunnel mode gre multipoint
 tunnel key 12345
 tunnel protection ipsec profile MYPROFILE

Remember that by defining the static NHRP mapping and NHS on a spoke router for a hub, you are going to run the dynamic routing protocol over this tunnel. This defines the hub and spoke routing or neighbor network.

Verifications

From the R1-HUB perpective, its peer R5-HUB was discovered through dynamic tunnel. R1-HUB treat R5-HUB as another spoke router because from the R5-HUB perspective, it need to define its “next hop server” statically. From spokes side, it will have two “next hop server using the same tunnel, “tunnel0“.

R1-HUB#show dmvpn 
----------------output omitted for brevity-----------------

Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:3, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.26.2       192.168.123.2    UP 18:41:39     D
     1 10.155.36.3       192.168.123.3    UP 18:41:39     D
     1 10.155.56.5       192.168.123.5    UP 17:59:49     D
R5-HUB#sh dmvpn 
----------------output omitted for brevity-----------------

Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub/Spoke, NHRP Peers:3, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.16.1       192.168.123.1    UP 18:01:04     S
     1 10.155.26.2       192.168.123.2    UP 18:01:00     D
     1 10.155.36.3       192.168.123.3    UP 18:01:00     D
R2-SPOKE#sh ip nhrp 
192.168.123.1/32 via 192.168.123.1
   Tunnel0 created 5d22h, never expire 
   Type: static, Flags: used 
   NBMA address: 10.155.16.1 
192.168.123.5/32 via 192.168.123.5
   Tunnel0 created 2d08h, never expire 
   Type: static, Flags: used 
   NBMA address: 10.155.56.5
R2-SPOKE#sh ip nhrp nhs 
Legend: E=Expecting replies, R=Responding, W=Waiting
Tunnel0:
192.168.123.1  RE priority = 0 cluster = 0
192.168.123.5  RE priority = 0 cluster = 0

On this dual hubs single topology layout, dynamic spoke-to-spoke tunnel is still works. As we demonstrated on earlier article, it will create a spoke-to-spoke dynamic tunnel after we triggered a traffic from one spoke to another.

R2-SPOKE#sh dmvpn 
----------------output omitted for brevity-----------------

Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.16.1       192.168.123.1    UP    2d06h     S
     1 10.155.56.5       192.168.123.5    UP    1d15h     S
R2-SPOKE#traceroute 10.150.3.3
Type escape sequence to abort.
Tracing the route to 10.150.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.123.1 2 msec
    192.168.123.3 3 msec *
R2-SPOKE#traceroute 10.150.3.3
Type escape sequence to abort.
Tracing the route to 10.150.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.123.3 4 msec *  1 msec
R2-SPOKE#show dmvpn 
----------------output omitted for brevity-----------------

Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:3, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.16.1       192.168.123.1    UP    2d06h     S
     1 10.155.36.3       192.168.123.3    UP 00:00:10     D
     1 10.155.56.5       192.168.123.5    UP    1d15h     S

After all communications establish, traffic from the spokes to the datacenter will load balance through R1-HUB and R5-HUB. This is a default behaviour since all HUBs using same dmvpn cloud.

R2-SPOKE#sh ip route 10.150.4.4
Routing entry for 10.150.4.4/32
  Known via "eigrp 100", distance 90, metric 27008256, type internal
  Redistributing via eigrp 100
  Last update from 192.168.123.1 on Tunnel0, 00:00:24 ago
  Routing Descriptor Blocks:
  * 192.168.123.5, from 192.168.123.5, 00:00:24 ago, via Tunnel0
      Route metric is 27008256, traffic share count is 1
      Total delay is 55010 microseconds, minimum bandwidth is 100 Kbit
      Reliability 255/255, minimum MTU 1440 bytes
      Loading 43/255, Hops 2
    192.168.123.1, from 192.168.123.1, 00:00:24 ago, via Tunnel0
      Route metric is 27008256, traffic share count is 1
      Total delay is 55010 microseconds, minimum bandwidth is 100 Kbit
      Reliability 255/255, minimum MTU 1440 bytes
      Loading 1/255, Hops 2
R2-SPOKE#sh ip eigrp topology 10.150.4.4 255.255.255.255
EIGRP-IPv4 Topology Entry for AS(100)/ID(10.150.2.2) for 10.150.4.4/32
  State is Passive, Query origin flag is 1, 2 Successor(s), FD is 27008256
  Descriptor Blocks:
  192.168.123.1 (Tunnel0), from 192.168.123.1, Send flag is 0x0
      Composite metric is (27008256/130816), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 1/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.4.4
  192.168.123.5 (Tunnel0), from 192.168.123.5, Send flag is 0x0
      Composite metric is (27008256/130816), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 43/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.4.4
R2-SPOKE#traceroute          
Protocol [ip]: 
Target IP address: 10.150.4.4
Source address: 
Numeric display [n]: 
Timeout in seconds [3]: 
Probe count [3]: 4
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Port Number [33434]: 
Loose, Strict, Record, Timestamp, Verbose[none]: V
Loose, Strict, Record, Timestamp, Verbose[V]: 
Type escape sequence to abort.
Tracing the route to 10.150.4.4
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.123.1 1 msec
    192.168.123.5 11 msec
    192.168.123.1 3 msec
    192.168.123.5 6 msec
  2 10.155.145.4 1 msec *  1 msec *

Same goes for traffic from datacenter to each spoke, it will load balance through R1-HUB and R5-HUB. When this happens, asymmetric routing or per-packet load balancing across the links to the two hubs and this will lead to another problem, out of order packet delivery.

R4-DATACENTER#sh ip route eigrp 
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
D        10.150.2.2/32 
           [90/27008256] via 10.155.145.5, 1d15h, GigabitEthernet0/2
           [90/27008256] via 10.155.145.1, 1d15h, GigabitEthernet0/2
D        10.150.3.3/32 
           [90/27008256] via 10.155.145.5, 1d15h, GigabitEthernet0/2
           [90/27008256] via 10.155.145.1, 1d15h, GigabitEthernet0/2
D     192.168.123.0/24 
           [90/26880256] via 10.155.145.5, 1d15h, GigabitEthernet0/2
           [90/26880256] via 10.155.145.1, 1d15h, GigabitEthernet0/2
R4-DATACENTER#sh ip eigrp topology 10.150.2.2 255.255.255.255
EIGRP-IPv4 Topology Entry for AS(100)/ID(10.150.4.4) for 10.150.2.2/32
  State is Passive, Query origin flag is 1, 2 Successor(s), FD is 27008256
  Descriptor Blocks:
  10.155.145.1 (GigabitEthernet0/2), from 10.155.145.1, Send flag is 0x0
      Composite metric is (27008256/27008000), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 1/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.2.2
  10.155.145.5 (GigabitEthernet0/2), from 10.155.145.5, Send flag is 0x0
      Composite metric is (27008256/27008000), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 58/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.2.2
R4-DATACENTER#traceroute 
Protocol [ip]: 
Target IP address: 10.150.2.2
Source address: 
Numeric display [n]: 
Timeout in seconds [3]: 
Probe count [3]: 4
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Port Number [33434]: 
Loose, Strict, Record, Timestamp, Verbose[none]: V
Loose, Strict, Record, Timestamp, Verbose[V]: 
Type escape sequence to abort.
Tracing the route to 10.150.2.2
VRF info: (vrf in name/id, vrf out name/id)
  1 10.155.145.1 1 msec
    10.155.145.5 1 msec
    10.155.145.1 7 msec
    10.155.145.5 1 msec
  2 192.168.123.2 4 msec *  8 msec *

In order to mitigate this behaviour, we will manipulate traffic flow from spokes to datacenter and vice versa. In this experiment, we engineered flow traffic to use R5-HUB as the primary traffic path. We accomplish this using an attribute on the routing protocol level.

ip access-list standard OFFSET-BRANCH
 permit 10.150.2.2
 permit 10.150.3.3
ip access-list standard OFFSET-DC
 permit 10.150.4.4
!
router eigrp 100
 network 10.155.145.1 0.0.0.0
 network 192.168.123.1 0.0.0.0
 offset-list OFFSET-DC out 500 Tunnel0 
 offset-list OFFSET-OFFSET-BRANCH out 500 GigabitEthernet0/3

Now let’s verify route information from the spokes and the datacenter. Make sure it uses R5-HUB as the gateway.

R2-SPOKE#sh ip eigrp topology 10.150.4.4 255.255.255.255
EIGRP-IPv4 Topology Entry for AS(100)/ID(10.150.2.2) for 10.150.4.4/32
  State is Passive, Query origin flag is 1, 1 Successor(s), FD is 27008256
  Descriptor Blocks:
  192.168.123.5 (Tunnel0), from 192.168.123.5, Send flag is 0x0
      Composite metric is (27008256/130816), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 43/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.4.4
  192.168.123.1 (Tunnel0), from 192.168.123.1, Send flag is 0x0
      Composite metric is (27008756/131316), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55029 microseconds
        Reliability is 255/255
        Load is 1/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.4.4
R2-SPOKE#sh ip route 10.150.4.4
Routing entry for 10.150.4.4/32
  Known via "eigrp 100", distance 90, metric 27008256, type internal
  Redistributing via eigrp 100
  Last update from 192.168.123.5 on Tunnel0, 00:02:10 ago
  Routing Descriptor Blocks:
  * 192.168.123.5, from 192.168.123.5, 00:02:10 ago, via Tunnel0
      Route metric is 27008256, traffic share count is 1
      Total delay is 55010 microseconds, minimum bandwidth is 100 Kbit
      Reliability 255/255, minimum MTU 1440 bytes
      Loading 43/255, Hops 2
R2-SPOKE#traceroute 
Protocol [ip]: 
Target IP address: 10.150.4.4
Source address: 
Numeric display [n]: 
Timeout in seconds [3]: 
Probe count [3]: 4
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Port Number [33434]: 
Loose, Strict, Record, Timestamp, Verbose[none]: V
Loose, Strict, Record, Timestamp, Verbose[V]: 
Type escape sequence to abort.
Tracing the route to 10.150.4.4
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.123.5 4 msec 5 msec 0 msec 3 msec
  2 10.155.145.4 2 msec *  1 msec * 

Also verify route information from the datacenter to the spokes. Make sure it uses R5-HUB as the gateway.

R4-DATACENTER#sh ip eigrp topology 10.150.2.2 255.255.255.255
EIGRP-IPv4 Topology Entry for AS(100)/ID(10.150.4.4) for 10.150.2.2/32
  State is Passive, Query origin flag is 1, 1 Successor(s), FD is 27008256
  Descriptor Blocks:
  10.155.145.5 (GigabitEthernet0/2), from 10.155.145.5, Send flag is 0x0
      Composite metric is (27008256/27008000), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 58/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.2.2
  10.155.145.1 (GigabitEthernet0/2), from 10.155.145.1, Send flag is 0x0
      Composite metric is (27008756/27008500), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55029 microseconds
        Reliability is 255/255
        Load is 1/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.2.2
R4-DATACENTER#sh ip route 10.150.2.2
Routing entry for 10.150.2.2/32
  Known via "eigrp 100", distance 90, metric 27008256, type internal
  Redistributing via eigrp 100
  Last update from 10.155.145.5 on GigabitEthernet0/2, 00:00:42 ago
  Routing Descriptor Blocks:
  * 10.155.145.5, from 10.155.145.5, 00:00:42 ago, via GigabitEthernet0/2
      Route metric is 27008256, traffic share count is 1
      Total delay is 55010 microseconds, minimum bandwidth is 100 Kbit
      Reliability 255/255, minimum MTU 1440 bytes
      Loading 58/255, Hops 2
R4-DATACENTER#traceroute 
Protocol [ip]: 
Target IP address: 10.150.2.2            
Source address: 
Numeric display [n]: 
Timeout in seconds [3]: 
Probe count [3]: 4
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Port Number [33434]: 
Loose, Strict, Record, Timestamp, Verbose[none]: V
Loose, Strict, Record, Timestamp, Verbose[V]: 
Type escape sequence to abort.
Tracing the route to 10.150.2.2
VRF info: (vrf in name/id, vrf out name/id)
  1 10.155.145.5 0 msec 1 msec 1 msec 1 msec
  2 192.168.123.2 1 msec *  2 msec *

High Availibility Test

In order to achive a resilience network, high availability is a must on a production network. On the first test, we tried to shut down tunnel0 interface on the R5-HUB so it will force traffic from the spokes through R1-HUB. After we shut down the interface, we could see several time out occurs.

R2-SPOKE#ping 10.150.4.4 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 10.150.4.4, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!......!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!*Dec 31 06:09:32.676: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.123.5 (Tunnel0) is down: holding time expired!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (994/1000), round-trip min/avg/max = 1/5/16 ms

On the second test, we tried to bring back interface tunnel0 operational. At the same time we were executed ping command to measure how long tunnel0 takes to be operational.

R2-SPOKE#ping 192.168.123.5 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 192.168.123.5, timeout is 2 seconds:
....!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (996/1000), round-trip min/avg/max = 1/4/66 ms
R2-SPOKE#ping 10.150.4.4 repeat 6000
Type escape sequence to abort.
Sending 6000, 100-byte ICMP Echos to 10.150.4.4, timeout is 2 seconds:
---------------------output omitted for brevity-----------------------
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
*Dec 31 06:49:08.164: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.123.5 (Tunnel0) is up: new adjacency!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
---------------------output omitted for brevity-----------------------
Success rate is 100 percent (6000/6000), round-trip min/avg/max = 1/4/66 ms

As you can see from the output above, we are not seeing any packets lost from spokes to datacenter even though it took several time outs on the tunnel0 before it become online.

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCSA, VCP6-DCV
nantoyudi@gmail.com

Python3 Threading for SSH

This article describes on how to utilize Python Threading module as a complement to make our SSH application more powerful. On threading module, it has Method of operation of the threading.Thread class: The class threading.Thread has a method start(), which can start a Thread. It triggers off the method run(), which has to be overloaded. The join() method makes sure that the main program waits until all threads have terminated. Before we integrate Python Threading module to our previous SSH application, We are going to give a basic tutorial regarding threading.

Threading Basic

At the beginning, let’s create a function like below. This function will show how Python function will execute a code without a threading.

import time
import datetime

def myfunction():
    date_time = datetime.datetime.now().strftime("%I:%M:%S %p")
    print("Start a Thread at %s" % date_time)
    time.sleep(2)
    print("End a Thread at %s" % date_time)
    print("")

for i in range(5):
    myfunction()

output:

Start a Thread at 10:28:58 PM
End a Thread at 10:28:58 PM

Start a Thread at 10:29:00 PM
End a Thread at 10:29:00 PM

Start a Thread at 10:29:02 PM
End a Thread at 10:29:02 PM

Start a Thread at 10:29:04 PM
End a Thread at 10:29:04 PM

Start a Thread at 10:29:06 PM
End a Thread at 10:29:06 PM

On above output, according to the timestamp, we can see print statement executed one by one once the the previous thread is finished. Now let’s add threading module utilized on your codes.

import threading
import datetime
import time

def myfunction2():
    date_time = datetime.datetime.now().strftime("%I:%M:%S %p")
    print("Start a Thread at %s\n" % date_time, end="")
    time.sleep(2)
    print("End a Thread at %s\n" % date_time, end="")

thread_instance = []
for i in range(5):
    trd = threading.Thread(target=myfunction2)
    trd.start()
    thread_instance.append(trd)

for thread in thread_instance:
    thread.join()

Output:

Start a Thread at 05:19:02 PM
Start a Thread at 05:19:02 PM
Start a Thread at 05:19:02 PM
Start a Thread at 05:19:02 PM
Start a Thread at 05:19:02 PM
End a Thread at 05:19:02 PM
End a Thread at 05:19:02 PM
End a Thread at 05:19:02 PM
End a Thread at 05:19:02 PM
End a Thread at 05:19:02 PM

By utilizing threading module, now we can execute the all threads at the same time. You may see the timestamp is identical for each threads.

Python Partial Codes

After we learn basic knowledge of treading module, let’s implement it on our previous SSH application.

# import threading module
import threading

# Create function for ssh threads
def SSH_Thread():
    # create list for each thread
    thread_instance = []
    # create ip address list of the devices
    list_ip = ["172.16.0.21", "172.16.0.22"]
    for ip in list_ip:
        trd = threading.Thread(target=ssh_conn, args=(ip.strip("\n"),))
        trd.start()
        thread_instance.append(trd)
         
    for trd in thread_instance:
        trd.join()

Python Full Codes

import paramiko
import time
import datetime
import re
import threading

def ssh_conn(ip):
    try:
        date_time = datetime.datetime.now().strftime("%Y-%m-%d")
        date_time_s = datetime.datetime.now().strftime("%I:%M:%S %p")
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.connect(ip, port=22, username='cisco', password='router', look_for_keys=False, timeout=None)
        connection = ssh.invoke_shell()
        connection.send("\n")
        connection.send("terminal length 0\n")
        time.sleep(1)
        connection.send("\n")
        connection.send("show ip eigrp neighbor\n")
        time.sleep(3)
        file_output = connection.recv(9999).decode(encoding='utf-8')
        hostname = (re.search('(.+)#', file_output)).group().strip('#')
        outFile = open(hostname + "-" + str(date_time) + ".txt", "w")
        outFile.writelines(file_output[1328:-3])
        outFile.close()
        ssh.close()
        if re.search('% Invalid input detected', file_output):
            print("* There was at least one IOS syntax error on device %s" % hostname)
        else:
            print("{} is done it was started at {}" .format(hostname, date_time_s))

    except paramiko.AuthenticationException:
        print("User or password incorrect, Please try again!!!")

def SSH_Thread():
    thread_instance = []
    list_ip = ["172.16.0.21", "172.16.0.22"]
    for ip in list_ip:
        trd = threading.Thread(target=ssh_conn, args=(ip.strip("\n"),))
        trd.start()
        thread_instance.append(trd)

    for trd in thread_instance:
        trd.join()

if __name__ == '__main__':
    SSH_Thread()

After you execute above codes, you will be notified that the task is completed at the same time like below.

R1 is done, it was started at 05:22:18 PM
R2 is done, it was started at 05:22:18 PM

Happy labbing!!!.

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCSA, VCP6-DCV
nantoyudi@gmail.com