Reset Root Password on Red Hat7.x/CentOS7.x

Recovering the root password is a trivial task while still logged in as an administrator or a user with full sudo access, but is slightly more involved when an administrator is not logged in. To recover the root password, use the following procedure:

  1. Reboot the system, press e to edit the selected entry. Move the cursor to the kernel command line (the line that starts with linux16). Append rd.break (this will break just before control is handed from the init ramfs to the actual system). Press Crtl+x to boot with the changes. At this point, a root shell will be presented, with the root file system for the actual system mounted read-only on /sysroot.
  2. Remount /sysroot as read-write.
    switch_root:/# mount -oremount,rw /sysroot
  3. Switch into a chroot jail, where /sysroot is treated as the root of the file system tree.
    switch_root:/# chroot /sysroot
  4. Set a new root password
    sh-4.2# passwd root
  5. Make sure that all unlabeled files (including /etc/shadow at this point) get relabeled during boot.
    sh-4.2# touch /.autorelabel
  6. Type exit twice. The first will exit the chroot jail, and the second will exit the initramfs debug shell.

source:
Red Hat System Administration III

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCE, VCP6-DCV
nantoyudi@gmail.com
Advertisements

Register and subscribe Red Hat 7.3 Packages

This article discribes how to register your Red Hat system to Red Hat subscription manager, enable some repositories and verify it. In this article I am using Red Hat Enterprise Linux 7.3 on virtual environment. You can acquire account for this subscription process on Red Hat portal as a Red Hat developer.

Now login to your system and check your subcriptions status. At this point you will see your system is not registered to any Red Hat subscription packages.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
repolist: 0

Register your system to Red Hat subscription management. Use the following command followed by the credential you acquired from the developer portal.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager register --username=mymail@email.com --password=mypassword
Registering to: subscription.rhsm.redhat.com:443/subscription
The system has been registered with ID: abcdefg-hijkl-mnop-bf16-cfa2dfcebbb4

Once you were registered to the subscription management, you may see the available subscription you may use on your system. Use the following command.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager list --available
+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+
Subscription Name:   Red Hat Enterprise Linux Developer Suite
Provides:            Red Hat Software Collections (for RHEL Server)
                     Red Hat Container Development Kit
                     MRG Realtime
                     Red Hat EUCJP Support (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Server - Extended Update Support
                     Red Hat Beta
                     Oracle Java (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node)
                     dotNET on RHEL Beta (for RHEL Server)
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Server) - Extended Update
                     Support
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
                     Oracle Java (for RHEL Server)
                     Red Hat Container Images
                     Red Hat Enterprise Linux for Real Time
                     dotNET on RHEL (for RHEL Server)
                     Red Hat Enterprise Linux Atomic Host
                     Red Hat S-JIS Support (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server)
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Atomic Host Beta
                     Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support
                     Red Hat Container Images Beta
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Server
                     Red Hat Enterprise Linux High Availability (for RHEL Server)
                     Red Hat Software Collections Beta (for RHEL Server)
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server)
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Server)
                     Red Hat Developer Toolset (for RHEL Server)
SKU:                 RH2262474
Contract:            11293058
Pool ID:             8a85f9815af00aed015af02fffbe5bb4
Provides Management: Yes
Available:           100
Suggested:           1
Service Level:       Self-Support
Service Type:        L1-L3
Subscription Type:   Standard
Ends:                03/21/2018
System Type:         Virtual

Type yum repolist to check if we run a registered system. At this point, you may see your system is registered but is not receiving any updates. It is because you are not subcribe to any subscription package list.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is registered to Red Hat Subscription Management, but is not receiving updates. You can use subscription-manager to assign subscriptions.
repolist: 0

Enable subscription on your system use the following command followed by the pool ID from the subscription list.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager subscribe --pool=8a85f9815af00aed02846f7sffbe5bb4
Successfully attached a subscription for: Red Hat Enterprise Linux Developer Suite

To check your enabled subscription briefly, you may use the following command.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager list

+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux Server
Product ID:     69
Version:        7.3
Arch:           x86_64
Status:         Subscribed
Status Details: 
Starts:         03/21/2017
Ends:           03/21/2018

To check detail information on your enabled subscription, use the following command.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Red Hat Enterprise Linux Developer Suite
Provides:            Red Hat Enterprise Linux High Performance Networking (for RHEL Server) - Extended Update
                     Support
                     Oracle Java (for RHEL Server)
                     Red Hat EUCJP Support (for RHEL Server) - Extended Update Support
                     dotNET on RHEL Beta (for RHEL Server)
                     Red Hat Beta
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
                     MRG Realtime
                     Red Hat Developer Toolset (for RHEL Server)
                     Red Hat Enterprise Linux Atomic Host Beta
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux for Real Time
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server)
                     Red Hat Container Images Beta
                     Red Hat Enterprise Linux High Availability (for RHEL Server)
                     Red Hat Container Development Kit
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Compute Node)
                     Red Hat Enterprise Linux Server - Extended Update Support
                     Red Hat Enterprise Linux Server
                     Red Hat Enterprise Linux Atomic Host
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server) - Extended Update Support
                     Oracle Java (for RHEL Server) - Extended Update Support
                     Red Hat Software Collections (for RHEL Server)
                     dotNET on RHEL (for RHEL Server)
                     Red Hat Enterprise Linux High Availability (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux Load Balancer (for RHEL Server) - Extended Update Support
                     Red Hat Software Collections Beta (for RHEL Server)
                     Red Hat Enterprise Linux Scalable File System (for RHEL Server)
                     Red Hat Container Images
                     Red Hat S-JIS Support (for RHEL Server) - Extended Update Support
                     Red Hat Enterprise Linux High Performance Networking (for RHEL Server)
SKU:                 RH2262474
Contract:            11293058
Account:             5920534
Serial:              7701382153394857656
Pool ID:             8a85f9815af00aed02846f7sffbe5bb4
Provides Management: Yes
Active:              True
Quantity Used:       1
Service Level:       Self-Support
Service Type:        L1-L3
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              03/21/2017
Ends:                03/21/2018
System Type:         Virtual

Type yum repolist to confirm that now we have some repositories source for the system.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
repo id                                        repo name                                                    status
!rhel-7-server-rpms/7Server/x86_64             Red Hat Enterprise Linux 7 Server (RPMs)                     14,050
!rhel-7-server-rt-beta-rpms/x86_64             Red Hat Enterprise Linux for Real Time Beta (RHEL 7 Server)      15
!rhel-7-server-rt-rpms/7Server/x86_64          Red Hat Enterprise Linux for Real Time (RHEL 7 Server) (RPMs    185
!rhel-ha-for-rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux High Availability (for RHEL 7 Serve    291
!rhel-rs-for-rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux Resilient Storage (for RHEL 7 Serve    359
repolist: 14,900

Type yum repolist all to see all repository avalilable on this subscription.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum repolist all
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
repo id                                                             repo name                      status
rh-gluster-3-client-for-rhel-7-server-debug-rpms/7Server/x86_64     Red Hat Storage Native Client  disabled
...............
!rhel-7-server-rpms/7Server/x86_64                                  Red Hat Enterprise Linux 7 Ser enabled: 14,050
...............
!rhel-7-server-rt-beta-rpms/x86_64                                  Red Hat Enterprise Linux for R enabled:     15
...............
!rhel-7-server-rt-rpms/7Server/x86_64                               Red Hat Enterprise Linux for R enabled:    185
...............
!rhel-ha-for-rhel-7-server-rpms/7Server/x86_64                      Red Hat Enterprise Linux High  enabled:    291
...............
!rhel-rs-for-rhel-7-server-rpms/7Server/x86_64                      Red Hat Enterprise Linux Resil enabled:    359
...............
repolist: 14,900

You may enable or disable spesific repository with the following command. Enable or disable the repo from the available repository list.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager repos --disable=rhel-ha-for-rhel-7-server-rpms
Repository 'rhel-ha-for-rhel-7-server-rpms' is disabled for this system.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ subscription-manager repos --enable=rhel-7-server-extras-rpms
Repository 'rhel-7-server-extras-rpms' is enabled for this system.

You may use command yum repolist to refresh the repository lists that we are using.

   root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
repo id                                repo name                                                            status
!rhel-7-server-extras-rpms/x86_64      Red Hat Enterprise Linux 7 Server - Extras (RPMs)                       432
!rhel-7-server-rpms/7Server/x86_64     Red Hat Enterprise Linux 7 Server (RPMs)                             14,050
!rhel-7-server-rt-beta-rpms/x86_64     Red Hat Enterprise Linux for Real Time Beta (RHEL 7 Server) (RPMs)       15
!rhel-7-server-rt-rpms/7Server/x86_64  Red Hat Enterprise Linux for Real Time (RHEL 7 Server) (RPMs)           185
repolist: 14,682

Once all set, you may need to update your system to receive latest update for each package from Red Hat Subscription Management.

    root:redhat.mylab.com in /root
πŸ˜ƒ  ➀ yum update
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
rhel-7-server-extras-rpms                                                               | 3.4 kB  00:00:00     
rhel-7-server-rpms                                                                      | 3.5 kB  00:00:00     
rhel-7-server-rt-beta-rpms                                                              | 4.0 kB  00:00:00     
rhel-7-server-rt-rpms                                                                   | 4.0 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package openjpeg-libs.x86_64 0:1.5.1-10.el7 will be updated
---> Package openjpeg-libs.x86_64 0:1.5.1-16.el7_3 will be an update
---> Package tzdata.noarch 0:2017a-1.el7 will be updated
---> Package tzdata.noarch 0:2017b-1.el7 will be an update
---> Package tzdata-java.noarch 0:2017a-1.el7 will be updated
---> Package tzdata-java.noarch 0:2017b-1.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved



============================================================================================================

 Package               Arch                Version                     Repository                       Size


============================================================================================================

Updating:
 openjpeg-libs         x86_64              1.5.1-16.el7_3              rhel-7-server-rpms               86 k
 tzdata                noarch              2017b-1.el7                 rhel-7-server-rpms              443 k
 tzdata-java           noarch              2017b-1.el7                 rhel-7-server-rpms              182 k

Transaction Summary


============================================================================================================

Upgrade  3 Packages

Total download size: 711 k
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for rhel-7-server-rpms
(1/3): openjpeg-libs-1.5.1-16.el7_3.x86_64.rpm                                          |  86 kB  00:00:01     
(2/3): tzdata-java-2017b-1.el7.noarch.rpm                                               | 182 kB  00:00:01     
(3/3): tzdata-2017b-1.el7.noarch.rpm                                                    | 443 kB  00:00:03     


-----------------------------------------------------------------------------------------------------------

Total                                                                          183 kB/s | 711 kB  00:00:03     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Updating   : tzdata-2017b-1.el7.noarch                                                               1/6 
  Updating   : tzdata-java-2017b-1.el7.noarch                                                          2/6 
  Updating   : openjpeg-libs-1.5.1-16.el7_3.x86_64                                                     3/6 
  Cleanup    : tzdata-2017a-1.el7.noarch                                                               4/6 
  Cleanup    : tzdata-java-2017a-1.el7.noarch                                                          5/6 
  Cleanup    : openjpeg-libs-1.5.1-10.el7.x86_64                                                       6/6 
rhel-7-server-extras-rpms/x86_64/productid                                              | 2.1 kB  00:00:00     
rhel-7-server-rpms/7Server/x86_64/productid                                             | 2.1 kB  00:00:00     
rhel-7-server-rt-beta-rpms/x86_64/productid                                             | 2.1 kB  00:00:00     
rhel-7-server-rt-rpms/7Server/x86_64/productid                                          | 2.1 kB  00:00:00     
  Verifying  : openjpeg-libs-1.5.1-16.el7_3.x86_64                                                     1/6 
  Verifying  : tzdata-java-2017b-1.el7.noarch                                                          2/6 
  Verifying  : tzdata-2017b-1.el7.noarch                                                               3/6 
  Verifying  : tzdata-java-2017a-1.el7.noarch                                                          4/6 
  Verifying  : openjpeg-libs-1.5.1-10.el7.x86_64                                                       5/6 
  Verifying  : tzdata-2017a-1.el7.noarch                                                               6/6 

Updated:
  openjpeg-libs.x86_64 0:1.5.1-16.el7_3     tzdata.noarch 0:2017b-1.el7     tzdata-java.noarch 0:2017b-1.el7    

Complete!



Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCE, VCP6-DCV
nantoyudi@gmail.com

Cisco DMVPN Dual Hub Single Topology

Following up our previous articleΒ on DMVPN, we are going to implement another model of DMVPN deployment, DMVPN dual hub single topology. We have prepared the topology below as a guidance.

DMVPN_Dual_Hub_Single_Topology

The dual hub with single layout topology is fairly to set up. The idea in this case it to have a single DMPVN “cloud” with all hubs, and all spokes connected to this single subnet (“cloud”). On above topology, you will have two static tunnels from each spoke to the hubs (R1-HUB and R5-HUB). Since the spoke router are routing neighbors with the hub routers over the same mGRE tunnel interface, you cannot use link or interface differences (like metric, cost, delay or bandwidth) to modify the dynamic routing protocol metric toprefer one hub over the other hub when they are both up. If this preference is needed, then you can utilize the routing protocol feature to engineering traffic flow.

Router Configuration

Since we don’t have any major differences on the configuration. We will only show the additional configuration for this purpose. Any full configuration you may refer to my previous post on DMVPN.

R1-HUB

router eigrp 100
 network 10.155.145.1 0.0.0.0
 network 192.168.123.1 0.0.0.0

R5-HUB

crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 14
crypto isakmp key cisco123 address 10.155.0.0     
!
!
crypto ipsec transform-set MYTRANSFORMSET esp-aes esp-sha-hmac 
 mode tunnel
!
crypto ipsec profile MYPROFILE
 set security-association lifetime seconds 900
 set transform-set MYTRANSFORMSET 
 
interface Tunnel0
 ip address 192.168.123.5 255.255.255.0
 no ip redirects
 ip mtu 1440
 no ip next-hop-self eigrp 100
 no ip split-horizon eigrp 100
 ip nhrp authentication cisco123
 ip nhrp map multicast dynamic
 ip nhrp map 192.168.123.1 10.155.16.1
 ip nhrp map multicast 10.155.16.1
 ip nhrp network-id 1
 ip nhrp nhs 192.168.123.1
 tunnel source GigabitEthernet0/1
 tunnel mode gre multipoint
 tunnel key 12345
 tunnel protection ipsec profile MYPROFILE
 
 router eigrp 100
 network 10.155.145.5 0.0.0.0
 network 192.168.123.5 0.0.0.0

I did two changes on R1-HUB, remove the loopback IP from EIGRP proceess and put datacenter segment instead. The configuration for R5-HUB is basically the same as the R1-HUB configuration with the appropriate IP address changes. The one main difference is that R5-HUB is also a spoke (or client) of R1-HUB, making R1-HUB the primary hub and R5-HUB the secondary hub.

R2-Spoke

interface Tunnel0
 ip address 192.168.123.2 255.255.255.0
 no ip redirects
 ip mtu 1440
 ip nhrp authentication cisco123
 ip nhrp map multicast dynamic
 ip nhrp map 192.168.123.1 10.155.16.1
 ip nhrp map multicast 10.155.16.1
 ip nhrp map 192.168.123.5 10.155.56.5
 ip nhrp map multicast 10.155.56.5
 ip nhrp network-id 1
 ip nhrp nhs 192.168.123.1
 ip nhrp nhs 192.168.123.5
 tunnel source GigabitEthernet0/2
 tunnel mode gre multipoint
 tunnel key 12345
 tunnel protection ipsec profile MYPROFILE

R3-Spoke

interface Tunnel0
 ip address 192.168.123.3 255.255.255.0
 no ip redirects
 ip mtu 1440
 ip nhrp authentication cisco123
 ip nhrp map multicast dynamic
 ip nhrp map 192.168.123.1 10.155.16.1
 ip nhrp map multicast 10.155.16.1
 ip nhrp map 192.168.123.5 10.155.56.5
 ip nhrp map multicast 10.155.56.5
 ip nhrp network-id 1
 ip nhrp nhs 192.168.123.1
 ip nhrp nhs 192.168.123.5
 tunnel source GigabitEthernet0/1
 tunnel mode gre multipoint
 tunnel key 12345
 tunnel protection ipsec profile MYPROFILE

Remember that by defining the static NHRP mapping and NHS on a spoke router for a hub, you are going to run the dynamic routing protocol over this tunnel. This defines the hub and spoke routing or neighbor network.

Verifications

From the R1-HUB perpective, its peer R5-HUB was discovered through dynamic tunnel. R1-HUB treat R5-HUB as another spoke router because from the R5-HUB perspective, it need to define its “next hop server” statically. From spokes side, it will have two “next hop server using the same tunnel, “tunnel0“.

R1-HUB#show dmvpn 
----------------output omitted for brevity-----------------

Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:3, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.26.2       192.168.123.2    UP 18:41:39     D
     1 10.155.36.3       192.168.123.3    UP 18:41:39     D
     1 10.155.56.5       192.168.123.5    UP 17:59:49     D
R5-HUB#sh dmvpn 
----------------output omitted for brevity-----------------

Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub/Spoke, NHRP Peers:3, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.16.1       192.168.123.1    UP 18:01:04     S
     1 10.155.26.2       192.168.123.2    UP 18:01:00     D
     1 10.155.36.3       192.168.123.3    UP 18:01:00     D
R2-SPOKE#sh ip nhrp 
192.168.123.1/32 via 192.168.123.1
   Tunnel0 created 5d22h, never expire 
   Type: static, Flags: used 
   NBMA address: 10.155.16.1 
192.168.123.5/32 via 192.168.123.5
   Tunnel0 created 2d08h, never expire 
   Type: static, Flags: used 
   NBMA address: 10.155.56.5
R2-SPOKE#sh ip nhrp nhs 
Legend: E=Expecting replies, R=Responding, W=Waiting
Tunnel0:
192.168.123.1  RE priority = 0 cluster = 0
192.168.123.5  RE priority = 0 cluster = 0

On this dual hubs single topology layout, dynamic spoke-to-spoke tunnel is still works. As we demonstrated on earlier article, it will create a spoke-to-spoke dynamic tunnel after we triggered a traffic from one spoke to another.

R2-SPOKE#sh dmvpn 
----------------output omitted for brevity-----------------

Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.16.1       192.168.123.1    UP    2d06h     S
     1 10.155.56.5       192.168.123.5    UP    1d15h     S
R2-SPOKE#traceroute 10.150.3.3
Type escape sequence to abort.
Tracing the route to 10.150.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.123.1 2 msec
    192.168.123.3 3 msec *
R2-SPOKE#traceroute 10.150.3.3
Type escape sequence to abort.
Tracing the route to 10.150.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.123.3 4 msec *  1 msec
R2-SPOKE#show dmvpn 
----------------output omitted for brevity-----------------

Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:3, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.16.1       192.168.123.1    UP    2d06h     S
     1 10.155.36.3       192.168.123.3    UP 00:00:10     D
     1 10.155.56.5       192.168.123.5    UP    1d15h     S

After all communications establish, traffic from the spokes to the datacenter will load balance through R1-HUB and R5-HUB. This is a default behaviour since all HUBs using same dmvpn cloud.

R2-SPOKE#sh ip route 10.150.4.4
Routing entry for 10.150.4.4/32
  Known via "eigrp 100", distance 90, metric 27008256, type internal
  Redistributing via eigrp 100
  Last update from 192.168.123.1 on Tunnel0, 00:00:24 ago
  Routing Descriptor Blocks:
  * 192.168.123.5, from 192.168.123.5, 00:00:24 ago, via Tunnel0
      Route metric is 27008256, traffic share count is 1
      Total delay is 55010 microseconds, minimum bandwidth is 100 Kbit
      Reliability 255/255, minimum MTU 1440 bytes
      Loading 43/255, Hops 2
    192.168.123.1, from 192.168.123.1, 00:00:24 ago, via Tunnel0
      Route metric is 27008256, traffic share count is 1
      Total delay is 55010 microseconds, minimum bandwidth is 100 Kbit
      Reliability 255/255, minimum MTU 1440 bytes
      Loading 1/255, Hops 2
R2-SPOKE#sh ip eigrp topology 10.150.4.4 255.255.255.255
EIGRP-IPv4 Topology Entry for AS(100)/ID(10.150.2.2) for 10.150.4.4/32
  State is Passive, Query origin flag is 1, 2 Successor(s), FD is 27008256
  Descriptor Blocks:
  192.168.123.1 (Tunnel0), from 192.168.123.1, Send flag is 0x0
      Composite metric is (27008256/130816), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 1/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.4.4
  192.168.123.5 (Tunnel0), from 192.168.123.5, Send flag is 0x0
      Composite metric is (27008256/130816), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 43/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.4.4
R2-SPOKE#traceroute          
Protocol [ip]: 
Target IP address: 10.150.4.4
Source address: 
Numeric display [n]: 
Timeout in seconds [3]: 
Probe count [3]: 4
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Port Number [33434]: 
Loose, Strict, Record, Timestamp, Verbose[none]: V
Loose, Strict, Record, Timestamp, Verbose[V]: 
Type escape sequence to abort.
Tracing the route to 10.150.4.4
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.123.1 1 msec
    192.168.123.5 11 msec
    192.168.123.1 3 msec
    192.168.123.5 6 msec
  2 10.155.145.4 1 msec *  1 msec *

Same goes for traffic from datacenter to each spoke, it will load balance through R1-HUB and R5-HUB. When this happens, asymmetric routing or per-packet load balancing across the links to the two hubs and this will lead to another problem, out of order packet delivery.

R4-DATACENTER#sh ip route eigrp 
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
D        10.150.2.2/32 
           [90/27008256] via 10.155.145.5, 1d15h, GigabitEthernet0/2
           [90/27008256] via 10.155.145.1, 1d15h, GigabitEthernet0/2
D        10.150.3.3/32 
           [90/27008256] via 10.155.145.5, 1d15h, GigabitEthernet0/2
           [90/27008256] via 10.155.145.1, 1d15h, GigabitEthernet0/2
D     192.168.123.0/24 
           [90/26880256] via 10.155.145.5, 1d15h, GigabitEthernet0/2
           [90/26880256] via 10.155.145.1, 1d15h, GigabitEthernet0/2
R4-DATACENTER#sh ip eigrp topology 10.150.2.2 255.255.255.255
EIGRP-IPv4 Topology Entry for AS(100)/ID(10.150.4.4) for 10.150.2.2/32
  State is Passive, Query origin flag is 1, 2 Successor(s), FD is 27008256
  Descriptor Blocks:
  10.155.145.1 (GigabitEthernet0/2), from 10.155.145.1, Send flag is 0x0
      Composite metric is (27008256/27008000), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 1/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.2.2
  10.155.145.5 (GigabitEthernet0/2), from 10.155.145.5, Send flag is 0x0
      Composite metric is (27008256/27008000), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 58/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.2.2
R4-DATACENTER#traceroute 
Protocol [ip]: 
Target IP address: 10.150.2.2
Source address: 
Numeric display [n]: 
Timeout in seconds [3]: 
Probe count [3]: 4
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Port Number [33434]: 
Loose, Strict, Record, Timestamp, Verbose[none]: V
Loose, Strict, Record, Timestamp, Verbose[V]: 
Type escape sequence to abort.
Tracing the route to 10.150.2.2
VRF info: (vrf in name/id, vrf out name/id)
  1 10.155.145.1 1 msec
    10.155.145.5 1 msec
    10.155.145.1 7 msec
    10.155.145.5 1 msec
  2 192.168.123.2 4 msec *  8 msec *

In order to mitigate this behaviour, we will manipulate traffic flow from spokes to datacenter and vice versa. In this experiment, we engineered flow traffic to use R5-HUB as the primary traffic path. We accomplish this using an attribute on the routing protocol level.

ip access-list standard OFFSET-BRANCH
 permit 10.150.2.2
 permit 10.150.3.3
ip access-list standard OFFSET-DC
 permit 10.150.4.4
!
router eigrp 100
 network 10.155.145.1 0.0.0.0
 network 192.168.123.1 0.0.0.0
 offset-list OFFSET-DC out 500 Tunnel0 
 offset-list OFFSET-OFFSET-BRANCH out 500 GigabitEthernet0/3

Now let’s verify route information from the spokes and the datacenter. Make sure it uses R5-HUB as the gateway.

R2-SPOKE#sh ip eigrp topology 10.150.4.4 255.255.255.255
EIGRP-IPv4 Topology Entry for AS(100)/ID(10.150.2.2) for 10.150.4.4/32
  State is Passive, Query origin flag is 1, 1 Successor(s), FD is 27008256
  Descriptor Blocks:
  192.168.123.5 (Tunnel0), from 192.168.123.5, Send flag is 0x0
      Composite metric is (27008256/130816), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 43/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.4.4
  192.168.123.1 (Tunnel0), from 192.168.123.1, Send flag is 0x0
      Composite metric is (27008756/131316), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55029 microseconds
        Reliability is 255/255
        Load is 1/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.4.4
R2-SPOKE#sh ip route 10.150.4.4
Routing entry for 10.150.4.4/32
  Known via "eigrp 100", distance 90, metric 27008256, type internal
  Redistributing via eigrp 100
  Last update from 192.168.123.5 on Tunnel0, 00:02:10 ago
  Routing Descriptor Blocks:
  * 192.168.123.5, from 192.168.123.5, 00:02:10 ago, via Tunnel0
      Route metric is 27008256, traffic share count is 1
      Total delay is 55010 microseconds, minimum bandwidth is 100 Kbit
      Reliability 255/255, minimum MTU 1440 bytes
      Loading 43/255, Hops 2
R2-SPOKE#traceroute 
Protocol [ip]: 
Target IP address: 10.150.4.4
Source address: 
Numeric display [n]: 
Timeout in seconds [3]: 
Probe count [3]: 4
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Port Number [33434]: 
Loose, Strict, Record, Timestamp, Verbose[none]: V
Loose, Strict, Record, Timestamp, Verbose[V]: 
Type escape sequence to abort.
Tracing the route to 10.150.4.4
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.123.5 4 msec 5 msec 0 msec 3 msec
  2 10.155.145.4 2 msec *  1 msec * 

Also verify route information from the datacenter to the spokes. Make sure it uses R5-HUB as the gateway.

R4-DATACENTER#sh ip eigrp topology 10.150.2.2 255.255.255.255
EIGRP-IPv4 Topology Entry for AS(100)/ID(10.150.4.4) for 10.150.2.2/32
  State is Passive, Query origin flag is 1, 1 Successor(s), FD is 27008256
  Descriptor Blocks:
  10.155.145.5 (GigabitEthernet0/2), from 10.155.145.5, Send flag is 0x0
      Composite metric is (27008256/27008000), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55010 microseconds
        Reliability is 255/255
        Load is 58/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.2.2
  10.155.145.1 (GigabitEthernet0/2), from 10.155.145.1, Send flag is 0x0
      Composite metric is (27008756/27008500), route is Internal
      Vector metric:
        Minimum bandwidth is 100 Kbit
        Total delay is 55029 microseconds
        Reliability is 255/255
        Load is 1/255
        Minimum MTU is 1440
        Hop count is 2
        Originating router is 10.150.2.2
R4-DATACENTER#sh ip route 10.150.2.2
Routing entry for 10.150.2.2/32
  Known via "eigrp 100", distance 90, metric 27008256, type internal
  Redistributing via eigrp 100
  Last update from 10.155.145.5 on GigabitEthernet0/2, 00:00:42 ago
  Routing Descriptor Blocks:
  * 10.155.145.5, from 10.155.145.5, 00:00:42 ago, via GigabitEthernet0/2
      Route metric is 27008256, traffic share count is 1
      Total delay is 55010 microseconds, minimum bandwidth is 100 Kbit
      Reliability 255/255, minimum MTU 1440 bytes
      Loading 58/255, Hops 2
R4-DATACENTER#traceroute 
Protocol [ip]: 
Target IP address: 10.150.2.2            
Source address: 
Numeric display [n]: 
Timeout in seconds [3]: 
Probe count [3]: 4
Minimum Time to Live [1]: 
Maximum Time to Live [30]: 
Port Number [33434]: 
Loose, Strict, Record, Timestamp, Verbose[none]: V
Loose, Strict, Record, Timestamp, Verbose[V]: 
Type escape sequence to abort.
Tracing the route to 10.150.2.2
VRF info: (vrf in name/id, vrf out name/id)
  1 10.155.145.5 0 msec 1 msec 1 msec 1 msec
  2 192.168.123.2 1 msec *  2 msec *

High Availibility Test

In order to achive a resilience network, high availability is a must on a production network. On the first test, we tried to shut down tunnel0 interface on the R5-HUB so it will force traffic from the spokes through R1-HUB. After we shut down the interface, we could see several time out occurs.

R2-SPOKE#ping 10.150.4.4 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 10.150.4.4, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!......!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!*Dec 31 06:09:32.676: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.123.5 (Tunnel0) is down: holding time expired!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (994/1000), round-trip min/avg/max = 1/5/16 ms

On the second test, we tried to bring back interface tunnel0 operational. At the same time we were executed ping command to measure how long tunnel0 takes to be operational.

R2-SPOKE#ping 192.168.123.5 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 192.168.123.5, timeout is 2 seconds:
....!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (996/1000), round-trip min/avg/max = 1/4/66 ms
R2-SPOKE#ping 10.150.4.4 repeat 6000
Type escape sequence to abort.
Sending 6000, 100-byte ICMP Echos to 10.150.4.4, timeout is 2 seconds:
---------------------output omitted for brevity-----------------------
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
*Dec 31 06:49:08.164: %DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.123.5 (Tunnel0) is up: new adjacency!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
---------------------output omitted for brevity-----------------------
Success rate is 100 percent (6000/6000), round-trip min/avg/max = 1/4/66 ms

As you can see from the output above, we are not seeing any packets lost from spokes to datacenter even though it took several time outs on the tunnel0 before it become online.

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCSA, VCP6-DCV
nantoyudi@gmail.com

Python3 Threading for SSH

This article describes on how to utilize Python Threading module as a complement to make our SSH application more powerful. On threading module, it has Method of operation of the threading.Thread class: The class threading.Thread has a method start(), which can start a Thread. It triggers off the method run(), which has to be overloaded. The join() method makes sure that the main program waits until all threads have terminated. Before we integrate Python Threading module to our previous SSH application, We are going to give a basic tutorial regarding threading.

Threading Basic

At the beginning, let’s create a function like below. This function will show how Python function will execute a code without a threading.

import time
import datetime

def myfunction():
    date_time = datetime.datetime.now().strftime("%I:%M:%S %p")
    print("Start a Thread at %s" % date_time)
    time.sleep(2)
    print("End a Thread at %s" % date_time)
    print("")

for i in range(5):
    myfunction()

output:

Start a Thread at 10:28:58 PM
End a Thread at 10:28:58 PM

Start a Thread at 10:29:00 PM
End a Thread at 10:29:00 PM

Start a Thread at 10:29:02 PM
End a Thread at 10:29:02 PM

Start a Thread at 10:29:04 PM
End a Thread at 10:29:04 PM

Start a Thread at 10:29:06 PM
End a Thread at 10:29:06 PM

On above output, according to the timestamp, we can see print statement executed one by one once the the previous thread is finished. Now let’s add threading module utilized on your codes.

import threading
import datetime
import time

def myfunction2():
    date_time = datetime.datetime.now().strftime("%I:%M:%S %p")
    print("Start a Thread at %s\n" % date_time, end="")
    time.sleep(2)
    print("End a Thread at %s\n" % date_time, end="")

thread_instance = []
for i in range(5):
    trd = threading.Thread(target=myfunction2)
    trd.start()
    thread_instance.append(trd)

for thread in thread_instance:
    thread.join()

Output:

Start a Thread at 05:19:02 PM
Start a Thread at 05:19:02 PM
Start a Thread at 05:19:02 PM
Start a Thread at 05:19:02 PM
Start a Thread at 05:19:02 PM
End a Thread at 05:19:02 PM
End a Thread at 05:19:02 PM
End a Thread at 05:19:02 PM
End a Thread at 05:19:02 PM
End a Thread at 05:19:02 PM

By utilizing threading module, now we can execute the all threads at the same time. You may see the timestamp is identical for each threads.

Python Partial Codes

After we learn basic knowledge of treading module, let’s implement it on our previous SSH application.

# import threading module
import threading

# Create function for ssh threads
def SSH_Thread():
    # create list for each thread
    thread_instance = []
    # create ip address list of the devices
    list_ip = ["172.16.0.21", "172.16.0.22"]
    for ip in list_ip:
        trd = threading.Thread(target=ssh_conn, args=(ip.strip("\n"),))
        trd.start()
        thread_instance.append(trd)
         
    for trd in thread_instance:
        trd.join()

Python Full Codes

import paramiko
import time
import datetime
import re
import threading

def ssh_conn(ip):
    try:
        date_time = datetime.datetime.now().strftime("%Y-%m-%d")
        date_time_s = datetime.datetime.now().strftime("%I:%M:%S %p")
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.connect(ip, port=22, username='cisco', password='router', look_for_keys=False, timeout=None)
        connection = ssh.invoke_shell()
        connection.send("\n")
        connection.send("terminal length 0\n")
        time.sleep(1)
        connection.send("\n")
        connection.send("show ip eigrp neighbor\n")
        time.sleep(3)
        file_output = connection.recv(9999).decode(encoding='utf-8')
        hostname = (re.search('(.+)#', file_output)).group().strip('#')
        outFile = open(hostname + "-" + str(date_time) + ".txt", "w")
        outFile.writelines(file_output[1328:-3])
        outFile.close()
        ssh.close()
        if re.search('% Invalid input detected', file_output):
            print("* There was at least one IOS syntax error on device %s" % hostname)
        else:
            print("{} is done it was started at {}" .format(hostname, date_time_s))

    except paramiko.AuthenticationException:
        print("User or password incorrect, Please try again!!!")

def SSH_Thread():
    thread_instance = []
    list_ip = ["172.16.0.21", "172.16.0.22"]
    for ip in list_ip:
        trd = threading.Thread(target=ssh_conn, args=(ip.strip("\n"),))
        trd.start()
        thread_instance.append(trd)

    for trd in thread_instance:
        trd.join()

if __name__ == '__main__':
    SSH_Thread()

After you execute above codes, you will be notified that the task is completed at the same time like below.

R1 is done, it was started at 05:22:18 PM
R2 is done, it was started at 05:22:18 PM

Happy labbing!!!.

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCSA, VCP6-DCV
nantoyudi@gmail.com

Python3 and Paramiko for SSH (Router and Switch)

This article describes how to utilize Python programming language to accomplish any automation task against intermediate devices (router and switch). In this demonstration, we are going to establish an SSH session to a Cisco Nexus switch, collect some output using “show ip ospf” and write it to a text file named using its router hostname plus a date information to inform us when the file is collected.

We are going to use Python3.5 as the interpreter and Paramiko module to utilize SSH function on it. Below is the flowchart of our application:

ssh_basic_flowchart

Python Codes

The application codes for this lab is pretty stright forward. We are going to develop it simplier as we can, so you can copy, use and develop it on your own.

Python partial codes
Import related Python modules

import paramiko
import time
import datetime
import re

Define the SSH function

def ssh_conn(ip):

Change exception message to raise any error related to authentication

try:
except paramiko.AuthenticationException:
    print("User or password incorrect, Please try again!!!")

Set time value to use

date_time = datetime.datetime.now().strftime("%Y-%m-%d")

Use paramiko ssh client

#Use ssh client
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(ip, port=22, username='cisco', password='router', look_for_keys=False, timeout=None)    
     
#Invoke the shell for interactive terminal
connection = ssh.invoke_shell()
connection.send("terminal length 0\n")

#hold the script n seconds second before it execute another script
time.sleep(1)

#Send router command to the device
connection.send("\n")
connection.send("show ip ospf\n")
time.sleep(3)

#Receive buffer output
file_output = connection.recv(9999).decode(encoding='utf-8')

#Create a file output name from the device hostname
hostname = (re.search('(.+)#', file_output)).group().strip('#')

#Print the output interactively to the CLI
print(file_output)

#Write output to a file
outFile = open(hostname + "-" + str(date_time) + ".txt", "w")
outFile.writelines(file_output[678:-19])# this is custom value, you may choose another value on your lab
outFile.close()

#Closing the connection
ssh.close()

#Print information if the task is done
print("%s is done" % hostname)
#call the function        
if __name__ == '__main__':
    ssh_conn("10.10.0.5")

Python Full Codes

Below are the complete codes for this experiment

import paramiko
import time
import datetime
import re


def ssh_conn(ip):
    try:
        date_time = datetime.datetime.now().strftime("%Y-%m-%d")
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.connect(ip, port=22, username='cisco', password='router', look_for_keys=False, timeout=None)
        connection = ssh.invoke_shell()
        connection.send("terminal length 0\n")
        time.sleep(1)
        connection.send("\n")
        connection.send("show ip ospf\n")
        time.sleep(3)
        file_output = connection.recv(9999).decode(encoding='utf-8')
        hostname = (re.search('(.+)#', file_output)).group().strip('#')
        print(file_output)
        outFile = open(hostname + "-" + str(date_time) + ".txt", "w")
        outFile.writelines(file_output[678:-19])
        outFile.close()
        ssh.close()
        print("%s is done" % hostname)
        
    except paramiko.AuthenticationException:
        print("User or password incorrect, Please try again!!!")

if __name__ == '__main__':
    ssh_conn("10.10.0.5")

Error Exception Test

Before we start our application, we are going generate an error related to SSH authentication. According to The Zen of Python, Errors should never pass silently. We will write down the wrong password by purpose so python will generate an error message on an elegant way. We change the password to router1. Now try to execute the script.

(myvirtualenv02)$python3 PythonSSH_Basic.py 
User or password incorrect, Please try again!!!

As we can see from the output, error messages comes up as expected. Now with the password has been fixed let’s run the application one more time.

(myvirtualenv02)$python3 PythonSSH_Basic.py

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2012, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
(myvirtualenv02)16. Python3 $python3 PythonSSH_Basic.py 
terminal length 0
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2012, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
LAB-ROUTER# terminal length 0
LAB-ROUTER# 
LAB-ROUTER# show ip ospf

 Routing Process 100 with ID 10.10.0.5 VRF default
 Stateful High Availability enabled
 Graceful-restart is configured
   Grace period: 60 state: Inactive 
   Last graceful restart exit status: None
 Supports only single TOS(TOS0) routes
 Supports opaque LSA
 This router is an autonomous system boundary
 Redistributing External Routes from
   static
 Administrative distance 110
 Reference Bandwidth is 10000 Mbps
 SPF throttling delay time of 50.000 msecs,
   SPF throttling hold time of 50.000 msecs, 
   SPF throttling maximum wait time of 5000.000 msecs
 LSA throttling start time of 0.000 msecs,
   LSA throttling hold interval of 50.000 msecs, 
   LSA throttling maximum wait time of 5000.000 msecs
 Minimum LSA arrival 15.000 msec
 LSA group pacing timer 10 secs
 Maximum paths to destination 8
 Number of external LSAs 45996, checksum sum 0x5a495484
 Number of opaque AS LSAs 0, checksum sum 0
 Number of areas is 1, 1 normal, 0 stub, 0 nssa
 Number of active areas is 1, 1 normal, 0 stub, 0 nssa
   Area (0.0.31.65) 
        Area has existed for 4y2w
        Interfaces in this area: 23 Active interfaces: 22
        Passive interfaces: 19  Loopback interfaces: 1
        No authentication available
        SPF calculation has run 7438648 times
         Last SPF ran for 0.001616s
        Area ranges are
        Number of LSAs: 1908, checksum sum 0x3c1d064
LAB-ROUTER# 
LAB-ROUTER is done

If you prefer your terminal cleaner from device output, you may comment the “print(file_output)” code, with this code deactivate you will only receive notification that your task is done (e.g “LAB-ROUTER is done”). Now check on the directory where the applications was executed. You will have a text file created contain of the device output from our last activity.

(myvirtualenv02)$ls -l | grep LAB
LAB-ROUTER-2016-12-27.txt  1435 Dec 27 15:22
(myvirtualenv02)$cat LAB-ROUTER-2016-12-27.txt
LAB-ROUTER# show ip ospf

 Routing Process 100 with ID 10.10.0.5 VRF default
 Stateful High Availability enabled
 Graceful-restart is configured
   Grace period: 60 state: Inactive 
   Last graceful restart exit status: None
 Supports only single TOS(TOS0) routes
 Supports opaque LSA
 This router is an autonomous system boundary
 Redistributing External Routes from
   static
 Administrative distance 110
 Reference Bandwidth is 10000 Mbps
 SPF throttling delay time of 50.000 msecs,
   SPF throttling hold time of 50.000 msecs, 
   SPF throttling maximum wait time of 5000.000 msecs
 LSA throttling start time of 0.000 msecs,
   LSA throttling hold interval of 50.000 msecs, 
   LSA throttling maximum wait time of 5000.000 msecs
 Minimum LSA arrival 15.000 msec
 LSA group pacing timer 10 secs
 Maximum paths to destination 8
 Number of external LSAs 45996, checksum sum 0x5a48c6cf
 Number of opaque AS LSAs 0, checksum sum 0
 Number of areas is 1, 1 normal, 0 stub, 0 nssa
 Number of active areas is 1, 1 normal, 0 stub, 0 nssa
   Area (0.0.31.65) 
        Area has existed for 4y2w
        Interfaces in this area: 23 Active interfaces: 22
        Passive interfaces: 19  Loopback interfaces: 1
        No authentication available
        SPF calculation has run 7438649 times
         Last SPF ran for 0.001670s
        Area ranges are
        Number of LSAs: 1908, checksum sum 0x3c0ad76

Comparing to the output file on the CLI, output file from the “.txt” file is neater. It is because we did a string manipulation before we wrote the terminal output to the txt file. Now we have done basic experiment on how to establish ssh session to a intermediate device. In the future article we will enhance this application to handle more complication task supporting our activity as a network engineer. Happy labbing!!!.

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCSA, VCP6-DCV
nantoyudi@gmail.com

Cisco ISR 4331 Throughput Capacity

This article is describes one of an issue we was faced on the past regarding Cisco router throughput capacity. This issue is quite interesting since I didn’t know that some of Cisco routers delivered with a throughput license feature.

At the beginning, we was received a report from our client that they were experincing slow transfer data when the link reach 90 – 95Mbps. We can see the throughput graph from below picture.

Sentraya_beforeAs a basic troubleshooting process, we tried to identify the router CPU process, we saw that all processes were normal. Also there wasn’t any packet drop on the interface. One of the key we have discovered was, slowness only happen for the traffic goes through the congested link (I said congested because our customer has 1Gbps link but it never reaches 100Mbps). Another question that came in mind how it could be slow, what was the evidence so you can say it is slow. Our customer was sent the ping comparation when the traffic is about 40-60 Mbps, ping through the router will have average delay around 2-3ms. When the congestion was occured, ping through the device will have average delay around 40-44ms. According to the graph above it even never reach 90Mbps, but when we verified it from the CLI it did.

After several tests on the network, we started to dig more information from Cisco documentation. According to Cisco, the aggregate throughput handled by isr4331 is 100Mbps to 300Mbps. By default the router is running with 100Mbps of throughput and you can increase it to maximum of 300Mbps using throughput license. you may see the throughput information summary on each ISR4000 series summary on below picture.

ISR4331_throughput

At this instance we cannot increase the router throughput capacity unless we buy the throughput license. Fortunately Cisco comes with a trial license on it, so we can do a temporary remediation to let the the current traffic utilise more bandwith space.

Before we start to activate the temporary license, let’s do some verification on the license status.

Current Throughput Level

ISR4331#show platform hardware throughput level 
The current throughput level is 100000 kb/s

Current License Status

ISR4331#sh license feature  
Feature name             Enforcement  Evaluation  Subscription   Enabled  RightToUse 
!
!output omitted for brevity
!
throughput               yes          yes         no             no       yes        
internal_service         yes          no          no             no       no
ISR4331#show license 
!
!output omitted for brevity
!
Index 7 Feature: throughput                     
        Period left: Not Activated
        Period Used: 0  minute  0  second  
        License Type: EvalRightToUse
        License State: Active, Not in Use, EULA not accepted
        License Count: Non-Counted
        License Priority: None

Now let’s enable temporary throughput license on the router. It will be available for next 60 days. Don’t forget to save your configuration and reload the chassis to take effect.

ISR4331(config)#platform hardware throughput level 300000
         Feature Name:throughput
 
PLEASE  READ THE  FOLLOWING TERMS  CAREFULLY. INSTALLING THE LICENSE OR
LICENSE  KEY  PROVIDED FOR  ANY CISCO  PRODUCT  FEATURE  OR  USING SUCH
PRODUCT  FEATURE  CONSTITUTES  YOUR  FULL ACCEPTANCE  OF  THE FOLLOWING
TERMS. YOU MUST NOT PROCEED FURTHER IF YOU ARE NOT WILLING TO  BE BOUND
BY ALL THE TERMS SET FORTH HEREIN.
 
Use of this product feature requires  an additional license from Cisco,
together with an additional  payment.  You may use this product feature
on an evaluation basis, without payment to Cisco, for 60 days. Your use
of the  product,  including  during the 60 day  evaluation  period,  is
subject to the Cisco end user license agreement
http://www.cisco.com/en/US/docs/general/warranty/English/EU1KEN_.html
If you use the product feature beyond the 60 day evaluation period, you
must submit the appropriate payment to Cisco for the license. After the
60 day  evaluation  period,  your  use of the  product  feature will be
governed  solely by the Cisco  end user license agreement (link above),
together  with any supplements  relating to such product  feature.  The
above  applies  even if the evaluation  license  is  not  automatically
terminated  and you do  not receive any notice of the expiration of the
evaluation  period.  It is your  responsibility  to  determine when the
evaluation  period is complete and you are required to make  payment to
Cisco for your use of the product feature beyond the evaluation period.
 
Your  acceptance  of  this agreement  for the software  features on one
product  shall be deemed  your  acceptance  with  respect  to all  such
software  on all Cisco  products  you purchase  which includes the same
software.  (The foregoing  notwithstanding, you must purchase a license
for each software  feature you use past the 60 days evaluation  period,
so  that  if you enable a software  feature on  1000  devices, you must
purchase 1000 licenses for use past  the 60 day evaluation period.)   
 
Activation  of the  software command line interface will be evidence of
your acceptance of this agreement.

ACCEPT? (yes/[no]): yes

Now let’s verify router status after we enable the temporary throughput license.

ISR4331#show license feature 
Feature name             Enforcement  Evaluation  Subscription   Enabled  RightToUse 
!
!output omitted for brevity
!        
throughput               yes          yes         no             yes      yes        
internal_service         yes          no          no             no       no
ISR4331#show license         
!
!output omitted for brevity
!                         
Index 7 Feature: throughput                     
        Period left: 8  weeks 4  days 
        Period Used: 0  day  0 hours 
        License Type: EvalRightToUse
        License State: Active, In Use
        License Count: Non-Counted
        License Priority: Low

And for the final information. Let me show you the throughput graph after we enable the temporary throughput license.

Sentraya_after

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCSA, VCP6-DCV
nantoyudi@gmail.com

Cisco DMVPN Single Hub

This article describes how to configure DMVPN using a single hub. We are using below topology for our lab test.

DMVPN_Single_Hub

According to our previous discussionΒ on DMVPN, we will configure static tunnel on each router, spoke routers will only have one tunnel to the hub and hub only configured with one dynamic tunnel to communicate to its spoke routers. Also we will verify spoke-to-spoke dynamic tunnel between spokes router.

Connectivity Verification

Before you configure DMVPN on your network, make sure any routers who participate on DMVPN is well establish. I will do ping test from R1-HUB to other routers.

R1-HUB#tclsh
R1-HUB(tcl)#foreach ip {
+>(tcl)#10.155.26.2
+>(tcl)#10.155.36.3
+>(tcl)#} {ping $ip
+>(tcl)#}
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.155.26.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.155.36.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/7 ms

Leveraging tcl on Cisco IOS, we can see from above output that all routers can communicate to each other.

Configuration

In this subsection, we will have there parts of configuration, cryto, tunnel and routing protocol.

Crypto Configuration

All routers Notes
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 14

crypto isakmp key cisco123 address 10.155.0.0  
   
crypto ipsec transform-set MYTRANSFORMSET esp-aes esp-sha-hmac 
 mode tunnel

crypto ipsec profile MYPROFILE
 set security-association lifetime seconds 900
 set transform-set MYTRANSFORMSET
This is basic configuration required when you want to use additional protection using IPsec. You may use your own parameter setting for the lab experiment.

Tunnel Configuration

R1-HUB Notes
interface Tunnel0
 ip address 192.168.123.1 255.255.255.0
 no ip redirects
 ip mtu 1440
 no ip next-hop-self eigrp 100
 no ip split-horizon eigrp 100
 ip nhrp authentication cisco123
 ip nhrp map multicast dynamic
 ip nhrp network-id 1
 tunnel source GigabitEthernet0/1
 tunnel mode gre multipoint
 tunnel key 12345
 tunnel protection ipsec profile MYPROFILE
  • To instruct EIGRP that the IP next hop is itself, use the ip next-hop-self eigrp command in interface configuration mode.
  • With “no ip next-hop-self eigrp 100” implemented it will bypass spoke-to-spoke traffic not using hub as the gateway. We will see it further on verification section.
  • Regarding split horizon rule, spoke router will not receive other spokes prefix unless you disable it.
R2-SPOKE Notes
interface Tunnel0
 ip address 192.168.123.2 255.255.255.0
 no ip redirects
 ip mtu 1440
 ip nhrp authentication cisco123
 ip nhrp map multicast dynamic
 ip nhrp map 192.168.123.1 10.155.16.1
 ip nhrp map multicast 10.155.16.1
 ip nhrp network-id 1
 ip nhrp nhs 192.168.123.1
 tunnel source GigabitEthernet0/2
 tunnel mode gre multipoint
 tunnel key 12345
 tunnel protection ipsec profile MYPROFILE
Since this is a static mapping, the Key point of the tunnel configuration on the spokes are nhrp mapping and nhs mapping.
R3-SPOKE Notes
interface Tunnel0
 ip address 192.168.123.3 255.255.255.0
 no ip redirects
 ip mtu 1440
 ip nhrp authentication cisco123
 ip nhrp map multicast dynamic
 ip nhrp map 192.168.123.1 10.155.16.1
 ip nhrp map multicast 10.155.16.1
 ip nhrp network-id 1
 ip nhrp nhs 192.168.123.1
 tunnel source GigabitEthernet0/1
 tunnel mode gre multipoint
 tunnel key 12345
 tunnel protection ipsec profile MYPROFILE
Since this is a static mapping, the Key point of the tunnel configuration on the spokes are nhrp mapping and nhs mapping.

Routing Protocol Configuration

R1-HUB Notes
router eigrp 100
 network 10.150.1.1 0.0.0.0
 network 192.168.123.1 0.0.0.0
We include only network from tunnel0 and loopback0 interface to participating on EIGRP route.
R2-SPOKE Notes
router eigrp 100
 network 10.150.2.2 0.0.0.0
 network 192.168.123.2 0.0.0.0
We include only network from tunnel0 and loopback0 interface to participating on EIGRP route.
R3-SPOKE Notes
router eigrp 100
 network 10.150.3.3 0.0.0.0
 network 192.168.123.3 0.0.0.0
We include only network from tunnel0 and loopback0 interface to participating on EIGRP route.

Tunnel Verification

R1-HUB

R1-HUB#show dmvpn 
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        T1 - Route Installed, T2 - Nexthop-override
        C - CTS Capable
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.26.2       192.168.123.2    UP 00:22:26     D
     1 10.155.36.3       192.168.123.3    UP 00:22:26     D

R2-SPOKE

R2-SPOKE#show dmvpn 
!
! output omitted for brevity
!
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:1, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.16.1       192.168.123.1    UP 00:20:41     S

R3-SPOKE

R3-SPOKE#sh dmvpn 
!
! output omitted for brevity
!
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:1, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.16.1       192.168.123.1    UP 00:00:15     S

On the Hub router output. We can see information from two tunnels from R2 and R3, Hub router learn the spoke tunnel dynamically. From the spoke routers perspective, since those routers statically mapped hub interface for tunnel connection it will have only one tunnel connection to the Hub and it marked as a static tunnel.

Route Verification

R1-HUB

R1#sh ip route eigrp 
!
! output omitted for brevity
!
      10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
D        10.150.2.2/32 [90/27008000] via 192.168.123.2, 1w4d, Tunnel0
D        10.150.3.3/32 [90/27008000] via 192.168.123.3, 1w4d, Tunnel0

R2-SPOKE

R2#show ip route eigrp 
!
! output omitted for brevity
!
      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
D        10.150.1.1/32 [90/27008000] via 192.168.123.1, 1w4d, Tunnel0
D        10.150.3.3/32 [90/28288000] via 192.168.123.3, 1w4d, Tunnel0

R3-SPOKE

R3#sh ip route eigrp 
!
! output omitted for brevity
!
      10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
D        10.150.1.1/32 [90/27008000] via 192.168.123.1, 1w4d, Tunnel0
D        10.150.2.2/32 [90/28288000] via 192.168.123.2, 1w4d, Tunnel0

From the routing table on each router, each router learns prefix from other routers through the EIGRP.

Connectivity Test

From the Hub router, make sure you have full connectivity to the network behind the spoke routers.

R1-HUB#tclsh
R1(tcl)#foreach ip {
+>(tcl)#10.150.2.2
+>(tcl)#10.150.3.3
+>(tcl)#} {ping $ip   
+>(tcl)#}
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.150.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/6/16 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.150.3.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/5/9 ms

Trace route from one of spoke router to another spoke router will go through the HUB.

R2-SPOKE#traceroute 10.150.3.3 source loopback 0
Type escape sequence to abort.
Tracing the route to 10.150.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.123.1 2 msec 4 msec 9 msec
  2 192.168.123.3 18 msec *  1 msec

The first trace route will establish DMVPN session between R2-SPOKE and R3-SPOKE as it will create Spoke-to-Spoke dynamic tunnel.

R2-SPOKE#show dmvpn 
!
! output omitted for brevity
!
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 10.155.16.1       192.168.123.1    UP 00:00:48     S
     1 10.155.36.3       192.168.123.3    UP 00:00:31     D

Lets repeat the traceroute command you will see packet with destination to R3-SPOKE will directly send to it. When you enable “ip next-hop-self eigrp” any spoke-to-spoke traffic will go through the Hub. To mitigate this issue you may enable “ip nhrp shortcut” in the interface tunnel on each routers.

R2-SPOKE#traceroute 10.150.3.3 source loopback 0
Type escape sequence to abort.
Tracing the route to 10.150.3.3
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.123.3 2 msec *  1 msec

Don’t forget to do the ping test to Hub site and other spoke router.

R2-SPOKE#tclsh
R2-SPOKE(tcl)#foreach ip {
+>(tcl)#10.150.1.1
+>(tcl)#10.150.3.3
+>(tcl)#} {ping $ip
+>(tcl)#}
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.150.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/7 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.150.3.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/5 ms

More on Verification

Next Hop Resolution Protocol (NHRP)

R2-SPOKE#sh ip nhrp nhs detail
Legend: E=Expecting replies, R=Responding, W=Waiting
Tunnel0:
192.168.123.1  RE priority = 0 cluster = 0  req-sent 432  req-failed 0  repl-recv 429 (00:19:31 ago)
R2-SPOKE#show ip nhrp detail 
192.168.123.1/32 via 192.168.123.1
   Tunnel0 created 1w4d, never expire 
   Type: static, Flags: used 
   NBMA address: 10.155.16.1
R2-SPOKE#show ip nhrp detail   
192.168.123.1/32 via 192.168.123.1
   Tunnel0 created 1w4d, never expire 
   Type: static, Flags: used 
   NBMA address: 10.155.16.1 
192.168.123.2/32 via 192.168.123.2
   Tunnel0 created 00:00:04, expire 01:59:55
   Type: dynamic, Flags: router unique local 
   NBMA address: 10.155.26.2 
    (no-socket) 
  Requester: 192.168.123.3 Request ID: 13
192.168.123.3/32 via 192.168.123.3
   Tunnel0 created 00:00:05, expire 01:59:55
   Type: dynamic, Flags: router nhop 
   NBMA address: 10.155.36.3

Crypto Isakmp Session Association

R2-SPOKE#show crypto isakmp sa 
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
10.155.16.1     10.155.26.2     QM_IDLE           1029 ACTIVE
R2-SPOKE#show crypto engine connections active 
Crypto Engine Connections

   ID  Type    Algorithm           Encrypt  Decrypt LastSeqN IP-Address
  621  IPsec   AES+SHA                   0       13       13 10.155.26.2
  622  IPsec   AES+SHA                  13        0        0 10.155.26.2
 1029  IKE     SHA+AES                   0        0        0 10.155.26.2

Above output occur when spoke-to-spoke session is not yet established. It consist only IKE phase 1 and two IKE phase 2 (IPsec) for traffic incoming and outgoing from R2-SPOKE perpective.

 
R2-SPOKE#show crypto isakmp sa 
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
10.155.36.3     10.155.26.2     QM_IDLE           1048 ACTIVE
10.155.16.1     10.155.26.2     QM_IDLE           1029 ACTIVE
10.155.26.2     10.155.36.3     QM_IDLE           1047 ACTIVE
R2-SPOKE#show crypto engine connections active
Crypto Engine Connections

   ID  Type    Algorithm           Encrypt  Decrypt LastSeqN IP-Address
  621  IPsec   AES+SHA                   0       41       41 10.155.26.2
  622  IPsec   AES+SHA                  42        0        0 10.155.26.2
  625  IPsec   AES+SHA                   0        0        0 10.155.26.2
  626  IPsec   AES+SHA                   0        0        0 10.155.26.2
  627  IPsec   AES+SHA                   0        0        0 10.155.26.2
  628  IPsec   AES+SHA                   0        0        0 10.155.26.2
 1029  IKE     SHA+AES                   0        0        0 10.155.26.2
 1047  IKE     SHA+AES                   0        0        0 10.155.26.2
 1048  IKE     SHA+AES                   0        0        0 10.155.26.2

When spoke-to-spoke session established, you will have two more information on crypto isakmp sa, two more IKE phase 1 tunnel and four IKE phase 2 tunnel (IPsec)

Crypto IPsec Session Association

R2-SPOKE#show crypto ipsec sa | i encaps|decaps|endpt|local|transform|Status
    Crypto map tag: Tunnel0-head-0, local addr 10.155.26.2
   local  ident (addr/mask/prot/port): (10.155.26.2/255.255.255.255/47/0)
    #pkts encaps: 123, #pkts encrypt: 123, #pkts digest: 123
    #pkts decaps: 108, #pkts decrypt: 108, #pkts verify: 108
     local crypto endpt.: 10.155.26.2, remote crypto endpt.: 10.155.16.1
        transform: esp-aes esp-sha-hmac ,
        Status: ACTIVE(ACTIVE)
        transform: esp-aes esp-sha-hmac ,
        Status: ACTIVE(ACTIVE)
R2-SPOKE#show crypto ipsec sa | i encaps|decaps|endpt|local|transform|Status
    Crypto map tag: Tunnel0-head-0, local addr 10.155.26.2
   local  ident (addr/mask/prot/port): (10.155.26.2/255.255.255.255/47/0)
    #pkts encaps: 1, #pkts encrypt: 1, #pkts digest: 1
    #pkts decaps: 1, #pkts decrypt: 1, #pkts verify: 1
     local crypto endpt.: 10.155.26.2, remote crypto endpt.: 10.155.36.3
        transform: esp-aes esp-sha-hmac ,
        Status: ACTIVE(ACTIVE)
        transform: esp-aes esp-sha-hmac ,
        Status: ACTIVE(ACTIVE)
   local  ident (addr/mask/prot/port): (10.155.26.2/255.255.255.255/47/0)
    #pkts encaps: 123, #pkts encrypt: 123, #pkts digest: 123
    #pkts decaps: 108, #pkts decrypt: 108, #pkts verify: 108
     local crypto endpt.: 10.155.26.2, remote crypto endpt.: 10.155.16.1
        transform: esp-aes esp-sha-hmac ,
        Status: ACTIVE(ACTIVE)
        transform: esp-aes esp-sha-hmac ,
        Status: ACTIVE(ACTIVE)

The second output was taken after spoke-to-spoke session is established. It add information regarding source spoke and desination spoke router. Also it shows that packet through the WAN is encrypted as expected.

Contributor:

Ananto Yudi Hendrawan
Network Engineer - CCIE Service Provider #38962, RHCSA, VCP6-DCV
nantoyudi@gmail.com