Operating Juniper Networks Routers in the Enterprise Chapter

Operating Juniper Networks Routers in the Enterprise Chapter 9: Troubleshooting 4 -1 Copyright © 2005 Juniper Networks, Inc. Proprietary and Confidential www. juniper. net

Chapter Objectives 9 -2 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -2

Agenda: Troubleshooting 9 -3 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -3

General Troubleshooting Tips 9 -4 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -4

A Layered Troubleshooting Approach 9 -6 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -6

Layered Troubleshooting Case Study Provider Network OSPF/BGP Subscriber Site 1 P Frame Relay CPE PE Ethernet PE CPE P SONET/ATM PE Subscriber Site 2 CPE Application Flows (HTTP) 9 -8 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -8

The Control and Forwarding Planes Routing Engine Control Plane Ingress FPC/PIC Forwarding Plane 0 1 2 3 Keepalives, IGP, BGP, policy, RSVP, LDP, etc. IP II 0 1 FT Packet Forwarding Engine 2 Egress FPC/PIC Physical errors, MTU, firewall filters, policers, etc. 3 9 -9 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -9

Agenda: Troubleshooting 9 -10 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -10

Troubleshooting Resources 9 -11 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -11

Troubleshooting Tool Kit 9 -12 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -12

Visual Indicators POWER LED ALARM LED STATUS LED PIM Status LED 9 -13 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -13

The JUNOS Software CLI: Key Commands 9 -14 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -14

The JUNOS Software CLI: Restarting a Software Process (daemon) (1 of 2) 9 -16 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -16

The JUNOS Software CLI: Restarting a Software Process (daemon) (2 of 2) 9 -17 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -17

The JUNOS Software CLI: Hardware Online and Offline 9 -18 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -18

The JUNOS Software CLI: Network Utilities and Applications 9 -19 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -19

System Logs and Protocol Tracing: Review 9 -20 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -20

System Logs and Protocol Tracing: Key Commands 9 -21 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -21

System Logs and Protocol Tracing: Interpreting Syslog Messages 9 -23 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -23

Core Files 9 -25 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -25

The Interactive Shell and Hidden Commands 9 -26 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -26

Hidden Command Example Hidden switch 9 -27 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -27

Agenda: Troubleshooting 9 -28 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -28

Out-of-Band Management Network Console Ports 9 -29 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -29

Monitoring Devices Using SNMP Restricts all other clients from polling local device 9 -30 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -30

Backup Configuration Files Backup occurs when commit is issued First URL listed will be used unless transfer fails Transfer options include both FTP and SCP 9 -31 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -31

Recommended Syslog Settings 9 -32 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -32

Clock Synchronization The configured list of possible synchronization sources Boot server is used to set initial NTP time during boot A simple NTP client-mode configuration 9 -33 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -33

Lab 7—Parts 1– 3: Troubleshooting 9 -35 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -35

Agenda: Troubleshooting 9 -36 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -36

Hardware Troubleshooting Tools 9 -37 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -37

Hardware Troubleshooting Chart Alarms active? Display/view alarms LED indication of component failure? View LED status/ display Craft Interface HW-related log entries? FPC/PIC/port operational? Parse/view syslogs and act accordingly Display interface and hardware status Investigate software faults 9 -38 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -38

Hardware Case Study (1 of 4) 9 -39 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -39

Hardware Case Study (2 of 4) No alarms present Log entries indicate that FPC 1 was taken offline! The FPC is offline! 9 -40 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -40

Hardware Case Study (3 of 4) 9 -41 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -41

Hardware Case Study (4 of 4) 9 -42 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -42

Agenda: Troubleshooting 9 -43 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -43

Software Troubleshooting Tools 9 -44 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -44

Software Troubleshooting Chart Hardware is OK SW-related log entries? Parse/view syslogs and act accordingly Software process running? Display running processes Core files? Determine if core files are present Investigate interface faults 9 -46 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -46

Software Case Study (1 of 3) 9 -47 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -47

Software Case Study (2 of 3) 9 -48 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -48

Software Case Study (3 of 3) 9 -49 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -49

Agenda: Troubleshooting 9 -50 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -50

Interface Troubleshooting Considerations (1 of 2) 9 -51 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -51

Interface Troubleshooting Considerations (2 of 2) 9 -52 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -52

Interface Troubleshooting Tools 9 -53 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -53

Interface Troubleshooting Chart Chassis/software OK Yes Admin Down Enable interface Interface status? Admin Up Yes Bad local port No Local loop? Yes Bad telco No Remote loop? Yes Bad remote port Admin Up, Link Up Suspect bad IP config Investigate protocol faults No Can L 2 be looped? No Yes Link Down Errors or Alarms? Ping remote end? No Suspect L 2 config Yes Local loop? No Bad local port No Bad telco Yes Remote loop? Yes Bad L 2 config 9 -55 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -55

Interface Case Study (1 of 4) 9 -56 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -56

Interface Case Study (2 of 4) Administratively up, link level down No input or output errors detected 9 -57 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -57

Interface Case Study (3 of 4) Link is up, traffic is passing (TTL expired) Local loop is possible because of L 2 configuration 9 -58 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -58

Interface Case Study (4 of 4) 9 -59 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -59

Agenda: Troubleshooting 9 -60 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -60

OSPF Troubleshooting Considerations 9 -61 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -61

OSPF Troubleshooting Tools 9 -62 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -62

Protocol Troubleshooting Chart Chassis, software, interface, and transmission line are OK Yes Route present and active? No IGP route? Yes Investigate forwarding faults Suspect IGP config No Adjacencies up? Yes Suspect policy/ or IGP config No BGP session estab. ? No Suspect config/ or IGP Yes Route hidden? No Suspect remote peer policy Yes Suspect policy/ or IGP config 9 -63 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -63

OSPF Case Study (1 of 3) 9 -64 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -64

OSPF Case Study (2 of 3) No route to remote network No OSPF neighbor for serial interface in Area 0 9 -65 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -65

OSPF Case Study (3 of 3) 9 -66 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -66

Review Questions 9 -67 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -67

Lab 7—Part 4: Troubleshooting 9 -68 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -68

9 -69 Copyright © 2007 Juniper Networks, Inc. Education Services 9 -69