MSG 323 Exchange Transport Monitoring and Troubleshooting Max

MSG 323 Exchange Transport Monitoring and Troubleshooting Max Ciccotosto Program Manager - Exchange Server Microsoft Corporation

Why Are We Here? Learn about Transport in Exchange 2003 Common Scenarios Messages are not routed <X> queue keeps growing NDRs are returned for unidentifiable reason Client has problems with messages/content Authentication failures Monitoring is essential to preventing problems, catching them before they happen

Agenda Transport in Exchange What’s new in Exchange 2003 Exchange Mailflow Common Troubleshooting Scenarios DNS and Connectivity Working with Queues Routing and Connectors Content and Message Properties Dealing with NDRs Tracking Down Messages Setting up Counters and Logging Monitoring Best Practices Q&A

New in Exchange 2003 Easier to support: More Queues (Hidden Queues) Improved Message Tracking Improved Logging (DSN Logging) New Internet Email Connection Wizard New DNS Resolver tool Routing Performance and Stability improvements New filtering and antispam features Query-based Distributed Groups (QDGs) Journaling is BCC enabled

Transport in Exchange

What Is The Transport? Handles message delivery and routing details, examples: Look up users in AD, expand DLs Dynamic routing logic Picks up and delivers from/to Store Handles SMTP protocol Handles queuing of messages Touches every message, even local-to- local user!

Transport Essentials Server-Server transport In Exchange 2003: SMTP native transport Interoperability Exchange 2003 can talk to Exchange 5. 5 via RPC Support X. 400 connectors, EDK (foreign) gateways No IMC/IMS needed for Exchange 2000 – Exchange 5. 5 interoperability MTA still there Used for X. 400 and RPC interoperability

RPC 5. 5 Server 2003 Server RPC SMTP RPC 2003 Server 5. 5 Server RPC Exchange 2000 within an Exchange 5. 5 site

SMTP Service Uses Windows® 2000 SMTP Service Protocol events - extend the SMTP protocol Transport events - extend the function of the Transport Core Multi-threaded, high-performance

Transport Core Categorizer: Component that resolves sender and recipients against Active Directory Limited Categorizer ships with Windows 2000, disabled Can do some Directory access, such as expanding mailenabled Groups Enhanced Categorizer ships with Exchange 2000 Adds Exchange features, such as Recipient Limit checking, reading Home-MDB

Transport Core Routing Advanced link-state based routing engine replaces RID Server and GWART used by Exchange 5. 5 MTA uses same engine when it needs to compute next hop, so X. 400/RPC connectors get benefit Size, priority, sender of message; cost and state of links used to compute path

Transport Core Store Driver The interface between Transport and the Store Uses “Ex. IPC” for inter-process communication, IFS for large data transfer (message body) Windows 2000 SMTP Service: NTFS store driver Exchange 2000: Exchange Web Storage store driver

Transport Dependencies Check these as necessary: Transport looks up User info in AD Routing configuration read from AD on startup, link state info kept in memory DNS used to resolve names, even internal servers! DS 2 MB replicates info to MB for Core SMTP Transport queues messages in Store, delivers to/from store End client – e. g. auth problems. NOTE: Client -DC issues are not Exchange!

Queues – Basic Flowchart NTFS SMTP Protocol Active Directory Categorizer Routing Engine Queueing Remote Delivery Queue SMTP Protocol Local Delivery Queue MAPI / OWA Client MTA (X 400) Information Store MAPI / OWA Client MTA (X 400)

A More Advanced View…

Topic 1: Internet Connectivity

Email From Internet Possible reasons: Internet DNS is mis-configured Recipient Policy does not contain the domain You maybe filtering the specific domain/IP Tips: Use a tool such as www. dnsreport. com Check the type of NDR that sender gets Specific domains or everyone? Check SMTP Logs There is a lot of information in the KBs!!! Search for Shared Domain, SMTP and DNS SMTP Greenbook

Send Mail to The Internet Possible reasons: You cannot reach the Internet DNS Smarthost mis-configured (permissions, IP) Domain not properly registered with DNS (Reverse Lookup enabled on the recipient SMTP) Tips: Use a tool such as www. dnsreport. com Check the type of NDR that sender gets (5. 7. 1) Verify info/status with your ISP Verify if you can connect to remote server Check Real-time denied lists There is a lot of information in the KBs!!! Search for Shared Domain, SMTP and DNS SMTP Greenbook

Topic 2: Working with Queues

Queue Problems Symptoms: Queue growth ‘abnormal’ – monitoring tools report queue grows beyond threshold Messages “stuck” in Queue – these messages stay in queue, do not get delivered Messages waiting to be delivered to external domains (DNS problems) Internal queues grow (waiting AD lookup, local delivery)

Troubleshooting Queues – What are they for?

Troubleshooting Queues – Local Delivery Queue Messages awaiting delivery to the Information Store Make sure store is mounted Could indicate a performance issue “Poison” Message

Troubleshooting Queues – Pre-Submission Queue Messages waiting to be processed by Transport Exposed for event sink developers Could indicate a store performance issue or issue with 3 rd party event sink

Troubleshooting Queues – Pre-Categorization Pre-Cat Queue Messages waiting to be processed by Categorizer resolves addresses Could indicate an issue when talking to Global Catalog Could indicate a permissions issue

Troubleshooting Queues – Pre-Routing Post-Cat / Pre-Routing Queue Messages waiting to be routed Slowdowns usually due to expensive restrictions Messages w/ Unreachable Destination (not shown) Indicates Routing failed to find a path for the message

Troubleshooting Queues – Remote Delivery Queue Messages being sent to a remote location Note: Messages may be physically on disk or in the Information Store! – It depends on where the message originated Use the error message to help focus your troubleshooting (netmon, nslookup, etc. )

Troubleshooting Queues – Remote Delivery Queue Error Message

Troubleshooting Queues – New in Titanium Goal was to expose “hidden” queues Queues DSN Awaiting Submission NDR messages that are being submitted Failed Message Retry Queue Messages that failed conversion Deferred Delivery Messages that have deferred delivery specified by Outlook clients Standard actions are exposed

Fixing Queue Issues To monitor and Troubleshoot: use WMI or ESM queue viewer To manipulate queues: use ESM Verify Dependencies: DNS external/internal GC Availability Store Routing Ensure there are no “stuck” messages, if so freeze or delete

Tool: Queue Viewer In ESM Shows queue state and performs actions through ESM, under: SMTP Protocol X. 400 Service Actions: Freeze, Delete, Disable queue WMI access “Stuck” messages can be frozen or deleted

Queue Viewer

Topic 3: Routing and Connectors

Routing Problems Symptoms: Links / Connectors are marked “down” Topology changes / breaks message path “Currently unreachable” queue grows External mail is not routed NDRs, delayed delivery Routing specific errors in event logs Mail “disappears” or gets queued up Only some mail gets delivered to end-user

Fixing Routing Problems Check topology status: Confirm routing configuration has not changed (Did you uninstall IIS? ) Ensure master is up, routing service is running Monitor queues, set up counters Are bridgeheads up? Are routing groups connected? DNS internal: check Network-Address AD attribute of destination server, try to resolve name Use Win. Route to debug topology, link state information New “Routing and Link. State Whitepaper”

Tool: Win. Route Available: On CD in SUPPORTUTILS (Exchange 2000) Now on “Exchange Tools Site” Read Q 281382: “How to use Win. Route” Connects to specified host, acts as read-only slave Displays Link State info packet decoded Resolves GUIDs against AD If no AD access, see GUIDs and states Configure DC hostname if running outside the domain Can save link state information to file (*. rte files)

Win. Route

Topic 4: Content and Message Properties

Content And Properties Symptoms: Messages do not appear correctly to client Content is not preserved outbound/inbound Need to check advanced message properties (X-Headers, FROM, TO) Mail message fidelity is lost Hard to monitor, rely on user info To troubleshoot, investigate original message(s), use Archive Sink tool Don’t forget “Global Settings”

SMTP vs. MAPI Submission SMTP submission happens on protocol level (port 25) MAPI submission through MAPI client (Outlook), directly to store Both submission paths go through Transport (Archive Sink) SMTP mail is MIME encoded, MAPI is MS -TNEF encoded Archive Sink captures both

Tool: Archive Sink Available: On CD in SUPPORTUTILS (Exchange 2000) Now on “Exchange Tools Site” Hooks on 2 possible Transport Events and dumps message properties (regkey) New version: three files per message: xml, . eml, p 1 stream Works on multiple Virtual SMTP servers Archives both MAPI and SMTP messages Not to be confused with Journaling – meant to be troubleshooting tool

Archive Sink

Topic 5: Dealing with NDRs

Non Delivery Reports It is a symptom Always check returned NDRs when troubleshooting Sent to end-user – but can have NDRs also sent to another account(s) We added many codes In Exchange 2003 New log category for DSN (Sev 0 -5). X-Header “Error number” Added regkey to enable pre-E 2 K behavior

Troubleshooting Delivery Status Notifications Original message (may get “Send Again” form in Outlook) Recipient Server reporting the problem DSN error code Check NDR online

NDR Troubleshooting General steps to follow: Is it permanent or transient? 4. x. x or 5. x. x? Check specific diagnostic code (e. g. 5. 4. 0) Reference cause/solution KB Q 284204 Is it a client or server problem? What’s the reporting server? Can you reach the reporting server? Can other users send messages? Worst case: use tools such as Queue viewer, Message tracking or Winroute

Topic 6: Tracking down messages

Message Tracking “I sent an email to John two days ago, he as not received it yet” Useful for: Diagnosing “missing” or “lost” messages Discovering the message path – so you can increase logging / tracing Recording successful / failed deliveries Gathering statistical data from tracking logs Tools available: Message Tracking Center in ESM Do-it-yourself scripts Third party products

$Tracking Details Per-server Writes plain text logs to share \servername. log Enabled on server$

Tracking Details Per-server Writes plain text logs to share \servername. log Enabled on server object, option to log subject Turned off by default In Exchange 2003 we added extra logs Reference Use KB Q 246959 Make sure NOT to manually modify logs – can lead to corrupt data

Message Tracking UI and Logs

Monitoring and Troubleshooting Best Practices

SMTP Protocol Logging Per-SMTP Virtual Server Common logging interface for all IIS Same formats, ODBC Automatic rollover Hourly, Daily, Weekly, Monthly, File size Extended Logging tab Cannot log all DATA Default location Winntsystem 32Logfiles

Perfmon Counters - General Processes - Working Set Memory, CPU utilization (% Processor Time, Working Set Bytes, Pool Nonpaged Bytes) Overall memory (Available Mbytes) CPU – Overall CPU utilization Disk – Physical. Disk component

Perfmon Counters - SMTP Server component Queues Categorizer Queue Length Local [Retry] Queue Length Messages Currently Undeliverable Messages Pending Routing Remote [Retry] Queue Length Performance: Msgs/sec

Best Practices Set up perfmon counters to monitor key areas, use WMI to centralize monitoring Check queue state when something goes wrong! Use NDRs to narrow down and diagnose condition (Q 284204) Check global, server settings for misconfiguration and/or changes Use advanced troubleshooting tools as necessary Introductory KB: Q 281800

Resources SMTP Greenbook: http: //www. microsoft. com/technet/treeview/defau lt. asp? url=/technet/prodtechnol/exchange/excha nge 2000/proddocs/onlinebooks/confsmtp/confs mtp. asp Tools: http: //www. microsoft. com/exchange/2003/updat es Documentation: http: //www. microsoft. com/exchange Newgroups: dev team answers questions (see Exchange transport and connectivity)

Ask The Experts Get Your Questions Answered Talk one-on-one with a community of your peers Community Experts: Microsoft product teams, consultants and Tech*Ed speakers Resources: whiteboards, internet, etc. Location: in the middle of the Exhibit Hall Hours: at least 12 -3: 30 p every day I will be at the ATE after this session

Community Resources http: //www. microsoft. com/communities/default. mspx Most Valuable Professional (MVP) http: //www. mvp. support. microsoft. com/ Newsgroups Converse online with Microsoft Newsgroups, including Worldwide http: //www. microsoft. com/communities/newsgroups/default. mspx User Groups Meet and learn with your peers http: //www. microsoft. com/communities/usergroups/default. mspx

Suggested Reading And Resources The tools you need to put technology to work! TITLE Microsoft® Exchange Server 2003 Administrator's Companion: 07356 -1979 -4 Active Directory® for Microsoft® Windows® Server 2003 Technical Reference: 0 -73561577 -2 Available 9/24/03 Today Microsoft Press books are 20% off at the Tech. Ed Bookstore Also buy any TWO Microsoft Press books and get a FREE T-Shirt

Thank You Please Fill Out The Evaluation Form massici@microsoft. com

Community Resources http: //www. microsoft. com/communities/default. mspx Most Valuable Professional (MVP) http: //www. mvp. support. microsoft. com/ Newsgroups Converse online with Microsoft Newsgroups, including Worldwide http: //www. microsoft. com/communities/newsgroups/default. mspx User Groups Meet and learn with your peers http: //www. microsoft. com/communities/usergroups/default. mspx

evaluations