troubleshooting troubleshooting DNS ../common/cw/assets/lsnMapTroubleshooting.swf courseMapBtn Course Map Course Map MainMenuButtonClicked lessonMapBtn Lesson Map Lesson Map CourseMapButtonClicked glossaryBtn Glossary Glossary GlossaryButtonClicked resourcesBtn Resources Resources ResourcesButtonClicked exitBtn Exit Exit ExitButtonClicked replayBtn Replay Replay audio ReplayButtonClicked pauseBtn Pause audio Pause audio playback PauseButtonClicked resumeBtn Resume Resume audio playback ResumeButtonClicked previousPgBtn Previous screen PreviousButtonClicked nextPgBtn Next screen NextButtonClicked dnstrb_01 1 As a systems administrator working with the DNS, you have to understand DNS performance and security concepts and methods. In this lesson, you will learn about DNS query tools, DNS packet capture and other tools, types of DNS troubleshooting scenarios, and guidelines for approaching DNS failure. This lesson has five topics, and should take approximately 60 minutes to complete. The Introduction presents lesson objectives and topics. The next two topics discuss types of DNS query tools and DNS packet capture and other tools that are useful for troubleshooting DNS failure. The next topic introduces guidelines for how to approach different kinds of troubleshooting scenarios. At the end of this lesson, there is a brief summary. Screen 1 of 26. Topic Title: Introduction. Screen Title: Objectives and Overview of Topics. Text box appears with bulleted text in support of audio. Text box appears with list of topics in support of audio. Digital clock counts down from 60 minutes to zero in support of audio. dnstrb_02 2 When DNS service fails, users are unable to resolve queries, and cannot access websites and resources that they should have access to. As a systems administrator, you may be required to troubleshoot DNS failures. Troubleshooting is not a clear-cut task, since DNS problems may have a wide variety of symptoms, and corresponding procedures that you will need to use to correct them. DNS servers may not be responding to clients, or not resolving names as expected. The DNS service may not be starting or restarting, or may be affected by some other problem. Despite this, there is a basic information-gathering approach that should be applied to most troubleshooting situations. First, you must understand your network configuration, and its expected functionality. DNS servers can be affected by other network failures and events, so it is important to understand how your network should be running. Second, you must verify whether or not your network is operating as expected. There are a variety of tools and techniques available to you to verify network and DNS operations. Some of them will be identified in this lesson; others you will become familiar with on the job. Finally, you must identify the relevant questions, based on the current situation, that you need to have answered in order to resolve the issue or issues at hand. Select the flashing forward arrow to begin examining the tools you may use to troubleshoot DNS failures. Screen 2 of 26: Topic Title: D N S Query Tools. Screen Title: Recognizing D N S Failure. There are twenty multiple choice questions. Simple D N S hierarchy appears with images of servers at each node. Bottom most node displays a laptop image. Text server cannot be found appears next to laptop image in support of audio. Image of person at a computer desk appears with callout What could be the likely point of failure here? Bulleted text appears in synch with audio. Server images are highlighted in support of audio. Callout text changes in support of audio. dnstrb_03 3 nslookup is a tool for testing and troubleshooting DNS servers. nslookup can be used non-interactively from the command line for simple queries, or interactively, for more complex queries. Select "Non-interactively" and "Interactively" to learn about each nslookup method. Select "command" to see a list of commands that appear in the nslookup Help output. dnstrb_03_01 When working in the command line, keep in mind that anything you type at the command prompt that is not a valid command will be considered either a host name or an IP address. By default, nslookup will attempt to resolve the name using a default server. In this case, that is the server ns1.example.mil. If the name is valid and can be resolved, nslookup will provide output. These are examples of simple commands that can be entered from the command line. The nslookup command should return the desired answer by querying the default server. To query a different server, or to query for different types of DNS records, enter one of the commands to query another server. Roll your cursor over "Commands to query another server" to see an example of querying other servers. Non InteractivelyPopup 1 of 3: Popup Title: Non Interactively. Computer monitor appears with command lines in support of audio. Bulleted text appears in support of audio. Text on monitor screen is highlighted in support of audio. Instructions appear to roll your cursor over Commands to query another server to see a query example. Rollover displays examples of commands to query another server. dnstrb_03_02 The nslookup interactive shell will allow you to change servers, set query options, and debug DNS. To use nslookup interactively, enter nslookup at the command line to bring up the nslookup prompt. Then you can view your current options by entering "set all.". You can change the options by entering "set" plus an option name and value. Then you can issue any nslookup commands that are necessary. By entering the "server" command, you can change the default recursive server for the remainder of this interactive nslookup session. Enter "exit" to leave nslookup. Roll your cursor over "Examples of current options" to see examples. InteractivelyPopup 2 of 3: Popup Title: Command. List of n s lookup help output appears in scrollable text box. dnstrb_03_03 This is the help output for nslookup, which displays a list of commands. Within the list, identifiers are shown in uppercase, and any brackets shown indicate an option. Use the scroll bar to view the complete output of nslookup help. CommandPopup 3 of 3: Popup Title: Interactively. Computer monitor appears with command lines in support of audio. Bulleted text appears in support of audio. Instructions appear to roll your cursor over Examples of current options see examples. Rollover displays examples of current options. Screen 3 of 26: Topic Title: D N S Query Tools. Screen Title: N S lookup. Image of computer monitor appears with c prompt n s lookup on one line, set option on the next line, exit on the third line. Bulleted text appears in support of audio. Instructions appear to select non interactively or interactively to learn more about these methods. Select command to see a list of commands. Non interactively, Command, Interactively words appear selectable as popups. dnstrb_04 4 To look up different data types within the domain name space, use the "set type" or "set q[uerytype]" command at the command prompt. Here is sample code for querying for the mail exchanger data. The first time a query is made for a remote name, the answer is authoritative, but subsequent queries for the same name are non-authoritative. The first time a query is sent for a remote name, the local DNS server contacts the DNS server that is authoritative for that domain. The local DNS server will then cache that information, so that subsequent queries for the same remote name are answered non-authoritatively out of the local server`s cache. To query another name server directly, use the server or lserver commands to switch to that name server. The lserver command uses the initial default, or local, server when looking up the address of the server to switch to. The lserver command is useful in the event that the current server is unresponsive, because the command functions correctly as long as the initial default server is functioning correctly. The server command uses the current server to get the address. Screen 4 of 26: Topic Title: D N S Query Tools. Screen Title: Looking up different data types. Sample code and text appear in support of audio. dnstrb_05 5 nslookup can be used to transfer an entire zone. Note that this capability no longer exists in recent versions of nslookup on Linux/UNIX systems, but still may be useful on Windows. By using the ls command, you can see all the hosts within a remote domain. Zone transfers will normally be restricted at the DNS server so that only authorized addresses or networks can perform this function. The following error will be returned if zone security has been set: "Can't list domain example.mil: Query refused." Screen 5 of 26: Topic Title: D N S Query Tools. Screen Title: Using N S lookup to transfer an entire zone. D N S hierarchy appears in support of audio. Text appears in support of audio. dnstrb_06 6 Knowledge Check 1 460 650 Identify whether each statement about the use of nslookup is true or false. Make your selections; then select Done. Nslookup commands can be used to query only the default server. True False Correct. You can query different servers, including the default server, using nslookup. Incorrect. You can query different servers, including the default server, using nslookup. You can change servers, set query options, and debug DNS by using nslookup interactively. True False Correct. You can use nslookup interactively from a shell to change servers, set query options, and debug DNS. Incorrect. You can use nslookup interactively from a shell to change servers, set query options, and debug DNS. Using nslookup, the first time you query a particular recursive server for a remote name, the answer is authoritative, but subsequent answers for the same remote name are non-authoritative. True False Correct. Using nslookup, the first time you query a particular recursive server for a remote name, the answer is authoritative, but subsequent answers are non-authoritative. Queries made directly to an authoritative server will normally always receive an authoritative response. Incorrect. Using nslookup, the first time you query a particular recursive server for a remote name, the answer is authoritative, but subsequent answers are non-authoritative. Queries made directly to an authoritative server will normally always receive an authoritative response. Nslookup allows you to view addresses and name server data, but does not enable you to transfer zone data. True False Correct. Nslookup.exe allows you to view addresses and name server data and perform zone transfers. Incorrect. Nslookup.exe allows you to view addresses and name server data and perform zone transfers. Check your knowledge of nslookup. Screen 6 of 26: Topic Title: D N S Query Tools. Screen Title: Knowledge Check. This is a true false question. Use your keyboard to cycle through the list of options. dnstrb_07 7 Domain information groper, or dig, is a flexible tool for querying DNS name servers. Dig is used to perform DNS lookups and display the answers that are returned from the name server(s) that were queried. Wherever both nslookup and dig are available, dig should be used. Dig provides information that is precisely tailored based on command-line options, and the level of detail in the output is very useful for troubleshooting. Although dig is normally used with command-line arguments, it also has a batch mode of operation for reading lookup requests from a file. Dig allows multiple lookups to be issued from the command line. Unless it is told to query a specific name server, dig will try each of the servers listed in /etc/resolv.conf. When no command line arguments or options are given, dig will perform an NSquery for the root. Dig does not have an interactive model; instead, all aspects of the query you would like to send are specified on the command line. Dig allows for many command-line options. Roll your cursor over this list of frequently used options to see their definitions. Select the highlighted text to learn more about dig output. dnstrb_07_01 Dig output shows you the complete DNS response message, separated into sections: the header, which provides technical details about the answer provided by the DNS server; the question section, which states the query you made; the answer section, which provides the answer to your query; the authority section, which lists the servers that can provide authoritative answers to your query; and the additional section, which typically provides the IP addresses of the servers named in the authoritative section. All the resource records in these sections are printed in master file format. If written to a text file, the resource records could easily be loaded into an authoritative DNS server. OutputPopup 1 of 1: Popup Title: Output. Image of laptop appears with text in support of audio. Screen 7 of 26: Topic Title: D N S Query Tools. Screen Title: About dig. Image of laptop appears with text dig on the screen. Bulleted text appears in support of audio. Instructions appear to Roll your cursor over each option to see its definition. Select the highlighted text to learn about dig output. Text output appears selectable as a popup. Rollovers appear with descriptions for frequently used options. dnstrb_08 8 As with nslookup, you can use dig to initiate zone transfers. Unlike nslookup, though, dig has no special command to request a zone transfer. Instead, you simply specify axfr (as the query type) and the domain name of the zone as arguments. Remember that you can only transfer a zone from a name server that is authoritative for the zone, and that there will usually be access controls in place that limit transfers only to certain IP addresses or those presenting certain credentials such as TSIG keys. Screen 8 of 26: Topic Title: D N S Query Tools. Screen Title: Zone Transfer with Dig. Image of D N S hierarchy appears with images of labeled servers in support of audio. Image of laptop appears with c prompt with command a x f r. dnstrb_09 9 Knowledge Check 1 430 650 Identify whether each statement about the use of dig is true or false. Make your selections; then select Done. Domain information groper, or dig, allows you to read lookup requests from a file for batch processing. True False Correct. Dig does have a batch mode of operation for reading lookup requests from a file. Incorrect. Dig does have a batch mode of operation for reading lookup requests from a file. By default, dig will try to query each of the servers listed in /etc/resolv.conf. True False Correct. Unless you specify a name server, dig will try to query each of the servers listed in /etc/resolv.conf. Incorrect. Unless you specify a name server, dig will try to query each of the servers listed in /etc/resolv.conf. You cannot transfer zone data using dig. True False Correct. Both dig and nslookup.exe allow you to transfer zone data. However, unlike nslookup.exe, dig does not have a special command to request a zone transfer. Instead, you type axfr as the query type, and the domain name of the zone as arguments. Incorrect. Both dig and nslookup.exe allow you to transfer zone data. However, unlike nslookup.exe, dig does not have a special command to request a zone transfer. Instead, you type axfr as the query type, and the domain name of the zone as arguments. Some DNS administrators find dig easier to use than nslookup. True False Correct. Some DNS administrators find dig easier to use than nslookup. Dig shows you the entire DNS response message, separated into three sections. Incorrect. Some DNS administrators find dig easier to use than nslookup. Dig shows you the entire DNS response message, separated into three sections. Check your knowledge of dig. Screen 9 of 26: Topic Title: D N S Query Tools. Screen Title: Knowledge Check. This is a true false question. Use your keyboard to cycle through the list of options. dnstrb_10 10 A packet is a unit of data that is sent over the Internet, or any other packet switched network, between an origin and a destination. There are three main sections of a DNS packet: a set of headers, the actual data being transmitted, and a footer. A packet sniffer or packet capture tool is a tool that allows the complete capture of traffic that is traveling between networked computers. Packet sniffers are generally used to capture the data being passed between clients and servers, and to analyze traffic on your network. Examples of packet sniffing tools include tcpdump and Wireshark. On DoD networks, access to packet sniffers is generally restricted to authorized system administration and security personnel. Also, in typical switched Ethernet networks today, a packet sniffer may be able to see only a subset of network traffic, unless a specialized capture port or span port is available. Select "Headers" to see what is included in the header section of a packet. Select the forward flashing arrow to examine tcpdump and Wireshark. dnstrb_10_01 Network packet headers typically consist of the Internet protocol, or IP, version; the length of the header; and the type of service, also called the Differentiated Services Code Point, which is rarely used. The header also includes the size of the header plus the payload in bytes, which is referred to as the datagram size, and a 16-bit number, which is the Identification. The destination computer uniquely identifies a packet by combining the Identification with the source address. It then uses these unique identifiers to reassemble the data from packets. The header includes flags, which are bits that inform a router whether it can fragment a packet or not. This is important because many networks are restricted by the maximum packet size they can forward. If packets are too large for a particular network and the packet cannot be fragmented, then an error must be returned to the packet source. Headers identify the TTL or Time to Live, which is the maximum number of hops that a packet can take; the protocol, which is the type of packet, such as TCP, UDP, ICMP, or IGMP; and the header checksum. This is a value used to detect errors or corruption. Headers also contain the source address, or the IP address where the packet originated, and the destination address, which is the IP address where the packet is going. Finally, headers may contain options, but they are rarely used. HeadersPopup 1 of 1: Popup Title: Headers. Bulleted text appears in support of audio. Screen 10 of 26: Topic Title: Packet Capture and Other Tools. Screen title: D N S Packet Sniffers. Image of document labeled D N S Packet Sections appears in support of audio. Two images of servers appears with animated data dot moving between them. Path of data dots is labeled D N S packets. Bulleted text appears in support of audio. Text headers appears selectable as a popup. dnstrb_11 11 The tcpdump utility is a command-line tool that collects and dumps data on IP networks. Most Linux distributions come with tcpdump installed by default. Because the syntax and filtering used in tcpdump are often used by other tools, it is an excellent tool to know. Tcpdump examines all packets that cross the Ethernet interface. Normally tcpdump configures the interface to examine all packets that reach it, even those not addressed to it. Tcpdump allows you to restrict the data it collects by using filters to select addresses, protocols, and port numbers for capture. These tcpdump options may be used to filter traffic. Packet capture can be narrowed further using qualifiers. The first is the type, which can be host, net, or port. The host is the IP address, such as 192.0.2.224, or domain name of the host to specify. The net is the network address, such as 192.0.2, or name from /etc/networks. The port is the port number, such as 80, or name from /etc/services. The second qualifier is the direction of the packet. You can capture the source of the packet src or the destination of the packet dst. By default, packets are captured both ways, both source and destination. The last qualifier is the protocol to capture. You can use a variety of protocols at different levels, including: arp, ether, fddi, ip, rarp , tcp, and udp. You can also combine qualifiers. Finally, you can also use the Boolean operators AND, NOT, and OR to further narrow filters. With this, you can create filters to capture traffic between more than one specific host, such as 192.0.2.224 and 192.0.2.50. You can use it to capture traffic for an entire network except one specific IP address, or you can use it to capture all traffic to and from one host or another host. Roll your cursor over each example of a tcpdump command to see its definition. Screen 11 of 26: Topic Title: Packet Capture and Other Tools. Screen Title: T C P Dump. Image of laptop appears with c prompt and command t c p dump on screen. Bulleted text appears in support of audio. Instructions appear to roll your cursor over each example of a t c p dump command to see its definition. Rollovers display a description of each command. dnstrb_12 12 Wireshark, formerly known as Ethereal, is a GUI, or Graphical User Interface -based packet sniffing tool. Like tcpdump, Wireshark performs many of the same packet-sniffing tasks, but it is a GUI-based tool, and it provides a more visual, user-friendly output. It uses the same filter system as tcpdump, so it is helpful to have a basic understanding of the filters used in tcpdump before using Wireshark. The graphical version of Wireshark provides an extensive list of possible filters via a menu. The main Wireshark display is split into three panes. The top pane is the packet summary. It displays a summary of each packet, showing the packet number, the source, the destination, the protocol, and a summary of the information provided by the packet. The middle pane is the protocol tree, which displays the various layers of the packet selected in the packet summary pane. Finally, the pane at the bottom is the hex dump of the packet. This is what the packet looks like as it travels the network. Screen 12 of 26: Topic Title: Packet Capture and Other Tools. Screen Title: Wireshark. Image of Wireshark sample screen appears with bulleted text in support of audio. Sections of the screen are highlighted in support of audio. dnstrb_13 13 There are additional sources for information on DNS troubleshooting. Logs can be vital sources of information for troubleshooting DNS failures, including firewall logs and router logs, and IDS and rndc dumpdb logs. IDS logs can be particularly useful if the affected DNS traffic is being captured or tracked, depending on local IDS configuration. Use of the rndc dumpdb command provides a dump of all the information being stored on a BIND server, which can be particularly helpful for problems related to caching behavior (such as possible cache poisoning attacks.) Tools such as ping, traceroute, and netstat can be helpful during DNS troubleshooting to confirm network status and behavior. To use these tools for troubleshooting, however, the tools must function properly during normal DNS operations, and the DNS administrator needs to understand normal behavior and indications. Screen 13 of 26: Topic Title: Packet Capture and Other Tools. Screen Title: Other tools. D N S hierarchy appears with bulleted text in support of audio. dnstrb_14 14 Knowledge Check 1 400 650 Though not necessarily applications, these resources provide useful troubleshooting information. Packet capture tcpdump Wireshark Logs, rndc, dumpdb Ping, traceroute, netstat Correct. Firewall logs, router logs, IDS logs, and rndc dumpdb are sources of information on DNS troubleshooting. Incorrect. Firewall logs, router logs, IDS logs, and rndc dumpdb are sources of information on DNS troubleshooting. Used to collect packets being passed between clients and servers, and to analyze network traffic. Packet capture tcpdump Wireshark Logs, rndc, dumpdb Ping, traceroute, netstat Correct. Packet capture tools, such as tcpdump and Wireshark, enable you to capture data on traffic that is travelling between networked computers. Incorrect. Packet capture tools, such as tcpdump and Wireshark, enable you to capture data on traffic that is travelling between networked computers. Useful for troubleshooting DNS only in situations where these tools are normally used. Packet capture tcpdump Wireshark Logs, rndc, dumpdb Ping, traceroute, netstat Correct. Use ping, traceroute, and netstat for DNS troubleshooting only in situations where these tools normally work. Incorrect. Use ping, traceroute, and netstat for DNS troubleshooting only in situations where these tools normally work. Installed on most Linux distributions by default, this command-line tool collects data on TCP/IP networks. Packet capture tcpdump Wireshark Logs, rndc, dumpdb Ping, traceroute, netstat Correct. Tcpdump is installed on most Linux distributions by default. It is an excellent tool to know, because its syntax and filtering are often used in other tools. Incorrect. Tcpdump is installed on most Linux distributions by default. It is an excellent tool to know, because its syntax and filtering are often used in other tools. To use this tool, it is useful to understand the filters used in tcpdump. Packet capture tcpdump Wireshark Logs, rndc, dumpdb Ping, traceroute, netstat Correct. This packet-sniffing tool, formerly known as Ethereal, uses the same filters used in tcpdump. Incorrect. This packet-sniffing tool, formerly known as Ethereal, uses the same filters used in tcpdump. Now check your knowledge about how to approach troubleshooting DNS failures related to the authoritative server. Select the tool that best matches the description. Screen 14 of 26: Topic Title: Packet Capture and Other Tools. Screen Title: Knowledge Check. This is a multiple choice question. Use your keyboard to cycle through the list of options. dnstrb_15 15 Troubleshooting DNS failure is a complex process, which varies according to the particular scenario. You can manage the troubleshooting process by recognizing the type of scenario you are facing, and by identifying the likely points of failure. Scenarios of DNS failure typically arise from either the recursive server, the authoritative server, or both the recursive and authoritative servers. Recursive name server scenarios include failure to resolve a name; specific resource type errors, which involve troubleshooting specific resource types, such as MX records for email or SRV records for Microsoft domain login; or PTR records; and caching and SRVFAIL errors, such as when the server cache needs to be flushed. Authoritative server scenarios include issues with the zone files on the master DNS server; zone transfer issues, including TSIG failures; and DNS dynamic update failures, including frozen or paused zones, and TSIG. Types of scenarios that can affect both recursive and authoritative servers are failure of the DNS server process to start or restart, and DNS attacks, specifically denial-of-service attacks. There are basic guidelines for approaching different scenarios. Select the flashing forward arrow to learn about troubleshooting some of these scenarios. Screen 15 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Types of D N S Troubleshooting scenarios. Image of D N S hierarchy appears in support of audio. Image of person at a computer work station appears with callouts in support of audio. Parts of the hierarchy are highlighted in support of audio. dnstrb_16 16 A basic approach for troubleshooting DNS failures involves five activities: Consider likely points of failure, Examine logs for any troublesome data, Check server caches and basic network connectivity, Check whether the intended and actual server and zone file configurations match. Either take corrective action, or if you are unable to resolve the issue, then generate questions and escalate the issue to the next level of support. In a given scenario, you may only have to identify likely points of failure, examine a log, and be ready to take corrective action. In other cases, you may go through each activity before you can take corrective action. Or, even after going through all the activities, you may have to escalate the issue to a higher level of support. The order and details of each activity also vary, depending on the specific scenario. For example, consider a simplified network configuration. This configuration includes a client, DHCP server, local recursive server, router connected to the Internet, the root server, a top-level domain such as .net, and a sub-domain such as nameserver1.net. Suppose you receive a trouble ticket related to the DNS. First ask yourself: What are the likely points of failure in this scenario? Is it the local recursive name server, the remote authoritative server, or a combination of both? Once you identify likely points of failure, you know the type of scenario you are probably dealing with. Is the problem related to the recursive server? Is the problem related to the authoritative server? Is the problem related to both the authoritative and recursive servers? Let the type of scenario you are facing guide your troubleshooting approach. In some cases, you may need to check for problems around recursive server caching and basic network connectivity. Use resolver tools such as nslookup or dig to uncover information. You may need to check the zone files and servers to see if they are configured as intended. Ultimately, the goal of troubleshooting is to take corrective action and resolve the problem. If the problem cannot be resolved, generate a list of higher-level questions to escalate the issue to the next level of support. These questions could be about configuration, or about other possible causes of the problem. Select each activity of the troubleshooting approach to learn about it. Roll your cursor over the network elements to learn more. dnstrb_16_01 In almost all cases, the first step in troubleshooting DNS failures is to identify the likely points of failure. If you need help determining the likely points of failure, check the logs for troublesome information about the master, slave, or recursive servers. Use Syslog for Unix, and Event Viewer for Windows. Once you identify the likely point of failure, see what type of scenario you are likely to be facing. Then pursue the approach recommended for that scenario. Identify likely points of failurePopup 1 of 5: Popup of Title: Identify likely points of failure. Image of five activity boxes appears with Identify likely points of failure highlighted. Bulleted text appears in support of audio. Sub bullets appear to describe bullets in support of audio. dnstrb_16_02 Examining logs can reveal critical information, if you know what to look for. In systems using Unix or Linux, the locations for BIND logging are defined in the named.conf file, often with some logging being passed to the syslog daemon. By examining syslog messages, you might discover configuration errors that resulted in failure of the DNS service to start. The BIND error messages are also very helpful. The first example indicates which line has the syntax error. Consider this example. For BIND9 command-line users, the named-checkconf utility may be used to verify configurations prior to loading. The second example reveals the problem of an incorrect file name. It should be example.com.db but is currently 1example.com.db. Consider this second example. In Windows systems, the logging destination is Event Viewer. DNS errors can be logged with numerous event IDs and sources. Log entries can be critically revealing. For example, a log can show what caused a DNS service to come to a stop. See this example of a log entry for a Windows 2008 server. Examine logsPopup 2 of 5: Popup of Title: Examine logs. Image of five activity boxes appears with Examine logs highlighted. Bulleted text appears in support of audio. Sub bullets appear to describe bullets in support of audio. dnstrb_16_03 DNS failures often result from issues around network connectivity or problems with caching on recursive servers. You can use tools such as dig and nslookup to check for connectivity issues. Also use these tools to verify if the cache on the client or on the recursive server needs to be flushed. If permitted by local situations and procedures, consider the use of packet capture tools, such as Wireshark, to check exactly what DNS query/response traffic is actually going across the network. Certain unusual failure modes in network equipment can affect DNS resolution in unexpected ways, and the only complete way to see all the traffic on the network is with a packet capture tool. Check cash and network connectivityPopup 3 of 5: Popup of Title: Check cash and network connectivity. Image of five activity boxes appears with Check cash and network connectivity highlighted. Bulleted text appears in support of audio. Sub bullets appear to describe bullets in support of audio. dnstrb_16_04 It is important to remember that when you get an unexpected answer to a query, it does not necessarily mean there is a problem with the DNS service. Often, you might get an unexpected answer because the intended configuration does not match the actual configuration. It is helpful to check whether the actual configuration of DNS servers and zone files matches the intended configuration. Check server and zone file configurationPopup 4 of 5: Popup of Title: Check server and zone file configuration. Image of five activity boxes appears with Check server and zone file configuration highlighted. Bulleted text appears in support of audio. Sub bullets appear to describe bullets in support of audio. dnstrb_16_05 If you are able to resolve the issue, take corrective action. If you are unable to resolve the issue, generate a list of higher-level questions regarding configuration and other possible causes of the problem. Escalate the issue to a higher level of support, along with your list of questions. Take corrective action or generate questions and escalate issuePopup 5 of 5: Popup of Title: Take corrective action or generate questions and escalate issue. Image of five activity boxes appears with Take corrective action or generate questions and escalate issue highlighted. Bulleted text appears in support of audio. Sub bullets appear to describe bullets in support of audio. Screen 16 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Approaching Troubleshooting Scenarios. Simple network appears with client, local recursive server, D H C P, router symbol, internet cloud, and two images of servers representing an authoritative server and a server labeled n s 1 dot example dot net. Text Approach for Troubleshooting D N S failures appears. Series of five boxes representing troubleshooting activities appears in support of audio. Each box is highlighted in support of audio. Instructions appear to select each activity in the troubleshooting approach to learn about it. Roll your cursor over the underlined network elements to learn more. Each activity appears selectable as a popup. Network elements appear as rollovers that display description of each element. dnstrb_17 17 Knowledge Check 1 650 650 When dynamic updates or zone transfers fail, this type of failure primarily affects the: Authoritative server Recursive server Both authoritative and recursive server Correct. Failure of zone transfers and dynamic updates, including paused or frozen zones and TSIG failures, is usually related to the authoritative server. Incorrect. Failure of zone transfers and dynamic updates, including paused or frozen zones and TSIG failures, is usually related to the authoritative server. When a DNS server fails to start or restart, the likely point of failure could be the: Authoritative server Recursive server Both authoritative and recursive server Correct. Failure to start or restart could involve either authoritative or recursive servers. Incorrect. Failure to start or restart could involve either authoritative or recursive servers. When a client reports "name does not resolve" errors or you suspect a caching problem, this type of failure would first be investigated at the: Authoritative server Recursive server Both authoritative and recursive server Correct. Failure to resolve names and caching problems would be investigated first at the recursive server. The root cause could be at an authoritative server, however. Incorrect. Failure to resolve names and caching problems would be investigated first at the recursive server. The root cause could be at an authoritative server, however. Now check your knowledge about likely points of DNS failures and DNS troubleshooting scenarios. Screen 17 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Knowledge Check. This is a multiple choice question. Use your keyboard to cycle through the list of options. dnstrb_18 18 A common type of DNS troubleshooting scenario involves the recursive name server. Two common scenarios for recursive name server failures are: Name does not resolve, and Specific resource record type errors. Screen 18 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Recursive Name Server Scenarios. Image of D N S hierarchy appears with image of recursive server highlighted in red with a line through it in support of audio. Bulleted text appears in support of audio. dnstrb_19 19 Suppose that a user attempts to access a resource on your network, but is unable to resolve the name. The user gets an error message. Begin by considering the most likely points of failure. The most likely culprits in this scenario are client machine caching, malware-infected client, and failure to communicate with the local recursive server and/or the remote authoritative server. Consider also any other local sources of naming information outside of DNS, such as WINS and local hosts files. On Windows, using the command "ipconfig displaydns" will list all DNS cache contents and allow you to check for a client caching issue. Check for possible malware infection on the client, which could alter client DNS settings. Given the likely causes just listed, your next step is to verify that these potential points of failure have not caused the issue. Use nslookup or dig to directly obtain DNS results, and compare with the contents of any local client DNS cache. Locate the client machine`s default recursive name server and query it directly. If the name server returns a valid answer, the problem could be with the client cache. Flush the cache to try to resolve the problem. On Windows, this can be done using the command "ipconfig flushdns." If the client cache is not the problem, locate an alternate local recursive server and use nslookup or dig to query for the name. Compare the results from the two local recursive servers with each other, and with normal expected results. If the problem has not yet been identified, next locate the remote authoritative server and use nslookup or dig to query the remote server. If the remote authoritative server returns NXDOMAIN, then the original name in fact does not exist. Normally, this result is sufficient as a diagnosis. However, if this result is not expected, it may be necessary to escalate the problem to a higher level of support, or to contact the authoritative server operator directly to verify that the zone is correctly configured on the remote side. If the remote authoritative server returns a SRVFAIL message, another scenario is involved. Flush the DNS cache on the client, and restart any web browsers to ensure that their internal caches are cleared. Verify whether high-profile sites can be directly queried from the client using nslookup or dig, for example service level .mil sites, and then coordinate with enclave-level administration to continue troubleshooting. If you are still unable to resolve the issue, generate a list of possible higher-level configuration questions, such as, how are my NS records configured? Does the network`s DNS configuration include "split DNS" with different answers for different clients? Are there other sources of name information, outside of DNS, that could give unexpected results? Also generate questions that try to uncover what else might be the cause of the problem. Elevate these concerns to the next level of support. In some cases, enclave-level support may be necessary. Screen 19 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Recursive Name Server Scenario Name Does not Resolve. Images of five text activities appear. Image of laptop appears displaying on screen text in support of audio. Activities are highlighted in support of audio. Bulleted text appears in support of audio. Sub bullets appear in support of audio. dnstrb_20 20 Sometimes when you are troubleshooting name resolution errors, it is necessary to troubleshoot specific resource record types. The symptoms of a problem with an MX record can show up in mail administration logs. For example, a mail administrator may seek your help in troubleshooting a problem related to email. In this case, remember that there may be multiple MX records in a zone. The records may have equal or different priorities. This situation is unique to MX records. The MX record with the highest value actually has the lowest priority, and the MX record with the lowest value has the highest priority. You must work with the mail administrator to troubleshoot the problem, starting by ensuring that the MX records in DNS match the desired configuration and the realities of your network. Problems with the SRV resource records can cause Active Directory, or AD, to not work properly. AD requires SRV records as pointers to the location of directory services. If SRV pointers are not correctly set, AD will not work properly. Dynamic updates must be enabled in a way that requires security. Otherwise, domain controllers will not be able to authenticate users and users will not be able to log in or access resources. To resolve SRV problems, correctly reconfigure and restart DNS services. If that does not correct the problem, start the net logon service on the domain controller to register the records. Issues with the PTR resource record can cause reverse lookup errors. In this case, the web administrator may tell you that the pointer records are configured incorrectly. Some sites, such as AOL, will reject email if PTR records are misconfigured. To troubleshoot PTR records, make sure updates to the forward and reverse zones match. If they do not match, update the reverse zone to match the forward zone. Large packets can also lead to problems with name resolution. If clients or recursive servers are having trouble receiving certain DNS information that is larger than 512 bytes, a configuration issue may be the culprit. The key symptom of this problem is that only large packets are being denied. To diagnose this problem, use event logs and command line tools. To resolve this type of problem, the firewalls throughout the organization need to be properly configured to allow UDP packets larger than 512 bytes, because of EDNS0. EDNS0 only works if clients, servers, and firewalls are properly configured. If you are unable to resolve the problem, generate a list of higher-level questions about the other possible causes, and escalate the issue to the next higher level of support. Screen 20 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Recursive Name Server Scenario Specific Resource Record Type Errors. Images of five text activities appear. Activities are highlighted in support of audio. Bulleted text appears in support of audio. Sub bullets appear in support of audio. dnstrb_21 21 Another common type of DNS troubleshooting scenario involves the authoritative name server. Troubleshooting scenarios involving the authoritative name server include problems with authoritative zone files on the master DNS server, zone transfer failures, including TSIG, and frozen or paused zones. Screen 21 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Authoritative Name Server Scenarios. Image of D N S hierarchy appears with image of authoritative server highlighted in red with a line through it in support of audio. Bulleted text appears in support of audio. dnstrb_22 22 A common DNS troubleshooting scenario involves the authoritative zone files on the master server. In this scenario, likely points of failure are your network configuration, and/or the DNS server configuration. The most common symptom of this type of failure is that the authoritative server is unable to load the zone file. If indeed the authoritative server is unable to load the zone file, the next step is to check if the problem is specific to a particular server and its zone files. Look at other zone files on the same authoritative server, and check if the problem affects other zones. Then examine event logs for any specific error messages that help identify the problem. Use the command line tools to verify the exact symptoms, and narrow down the source in the zone file. In other words, determine whether the problem is with the entire zone file or just a portion of the zone file that needs to be corrected. Increment the zone file serial number in the SOA record, and then force the zone to be reloaded. Recheck the symptoms using command line tools such as nslookup and dig, and event logs such as Event Viewer and Syslog. If you are still unable to resolve the issue, generate a list of higher-level questions about the network configuration or about other possible causes, and escalate the issue to the next level of support. Screen 22 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Authoritative Name Server Scenarios: Problems with Authoritative Zone Files on Master D N S Server. Images of five text activities appear. Image of appears appears labeled authoritative server and its zone files in support of audio. Activities are highlighted in support of audio. Bulleted text appears in support of audio. Sub bullets appear in support of audio. dnstrb_23 23 Zone transfer failures, including TSIG failures, pose another type of scenario for DNS troubleshooting. The likely points of failure in this scenario involve the ability of the master and slave servers to load updated zone files. Once you identify that the problem involves zone transfer failures, the next step is to narrow down the zone file glitch that is behind it: Normally the SOA serial number is updated on the master server whenever the zone file is changed. If the serial number is not changed, the slave servers do not recognize that the zone has changed, and so the zone is not transferred. The other likely possibility is that the zone file may have been changed on the master server, but the zone was never reloaded or the server process was not restarted. Perhaps the zone file was changed on the master server, and an attempt was made to reload the zone file, but due to some error, the zone file did not load or the server process failed. Also review event log files on both the master and slave servers to look for indications of NOTIFY messages, attempted zone transfers, and other evidence of attempted synchronization. You should also consider other network configuration possibilities, such as: Intervening firewalls should allow TCP traffic on port 53; however, it is possible that a security administrator may not have set up the firewalls to allow 53 TCP. The slave may not be configured to look to the correct master server or servers. The master server may not be allowing transfer by the correct set of slaves. There may be a difference in the TSIG shared secret key between the master and the slave. There may be a difference in the time synchronization between master and slave. The authentication provided by TSIG requires clocks to be synchronized, to protect against attacks that replay old credentials. After identifying the likely cause of the problem, you can begin taking corrective actions, such as: Query the zone SOA on the master server and compare the serial number with the zone file on the disk. If the serial number is different than what is in the zone file, then reload the zone. If the master does not respond, then investigate other causes of failure of the DNS server process or failure to reload the zone. This is usually caused by a typographical error that prevents the zone file parser from working properly. If the SOA serial matches what is on the disk, continue this path of troubleshooting. Verify that the master server allows zone transfer by the correct set of slave servers, typically by IP address and/or by TSIG keys. Also verify that the slave servers are configured with the correct master servers. If TSIG is in use, verify that the shared secret is identical between the master and slave servers. Then verify that the system time is consistent. Normally, system time should be consistent within 20 seconds across systems, and preferably all systems would use a common set of NTP servers with a much smaller deviations. Inconsistent system time will cause TSIG validation failures. Verify that the zone SOA on the slave server matches the SOA on the master server. If it does match, then try to verify whether any recently changed information is the same on both the master and the slave. If it is different, then the most likely cause is the failure to increment the SOA on the master. Consider incrementing the SOA serial number on the master and reloading the zone file. This should generate entries in syslog on the master about pending NOTIFY messages, and on the slave when the NOTIFY message is received.If the SOA serial number is not the same on the master and the slave, compare the serial number on other slave servers to see if the mismatch is systematic or isolated. Also, try to perform a zone transfer from the slave using a command line tool, such as dig or nslookup, for the entire zone with query type axfr. If you are still unable to resolve the issue, generate a list of higher-level questions about the network configuration or about other possible causes, and escalate the issue to the next level of support. Screen 23 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Authoritative Name Server Scenarios: Zone transfer failures. Images of five text activities appear. Image of appears appears labeled authoritative server and its zone files in support of audio. Activities are highlighted in support of audio. Bulleted text appears in support of audio. Sub bullets appear in support of audio. dnstrb_24 24 A common DNS troubleshooting scenario involves the failure of DNS dynamic updates, specifically including frozen or paused zones, and TSIG. The likely points of failure include the failure of the master server to receive updated zone files. This is often because the updates are sent to the wrong server, a server that is not the master server. Another possible point of failure is the zone is left frozen or paused. It is important for time to be synchronized between the client and the master server, and sometimes that is not the case. The client's TSIG secret key does not match the master server. Once you identify the most likely point of failure in this scenario, the next step is to check your event logs for additional detail about why the update failed. Check whether the status of the zone is frozen or paused. Check the time on the client and server to make sure they match. Typically, the time should be synchronized within 20 seconds. Use NTP to verify the synchronization. Verify the TSIG secret key on the client and server match. If you are unable to resolve the issue, generate a list of higher-level questions about the network configuration or about other possible causes, and escalate the issue to the next level of support. Screen 24 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Authoritative Name Server Scenarios: Failures of D N S Dynamic Update. Images of five text activities appear. Image of appears appears labeled authoritative server and its zone files in support of audio. Activities are highlighted in support of audio. Bulleted text appears in support of audio. Sub bullets appear in support of audio. dnstrb_25 25 Knowledge Check 1 650 650 First, verify the problem. Check if it is limited to one server and its zone files. Examine event logs for errors. Use dig to verify exact symptoms. Determine if the problem is with the whole zone file or a portion of it. Increment the SOA serial number and force the server to reload the zone. Recheck symptoms using dig and Event Viewer or system logs. If problem persists, generate questions and escalate the issue. Zone transfer fails to occur Master DNS server is unable to load zone files Dynamic update fails to occur Correct. You would take this approach for troubleshooting problems with the authoritative zone files on the master DNS server. Incorrect. This is the approach you take to troubleshoot problems with the authoritative zone files on the master DNS server. First, verify the problem. Then check if zone files are configured as intended. Make sure the SOA serial number on the master and slave match. Check if the master's zone file was updated but the zone was not reloaded, or whether an attempt to reload failed. Check for other configuration possibilities, such as verifying that firewalls are configured to allow TCP traffic on port 53. If TSIG is in use, verify TSIG key configuration and that server times are synchronized within acceptable limits. Take corrective actions, as needed. Zone transfer fails to occur Master DNS server is unable to load zone files Dynamic update fails to occur Correct. You would take this approach for troubleshooting zone transfer failures. Incorrect. You would take this approach for troubleshooting zone transfer failures. Verify the problem. Examine event logs for details about why the problem occurred. Verify if the zone files are configured as intended, and check the status of the zone. Use NTP to check the time on the client and server are synchronized within acceptable limits. If TSIG is in use, verify that the secret key on the client and server match. Zone transfer fails to occur Master DNS server is unable to load zone files Dynamic update fails to occur Correct. You would take this approach for troubleshooting dynamic updates, including frozen or paused zones. incorrect. You would take this approach for troubleshooting dynamic updates, including frozen or paused zones. Now check your knowledge about how to approach troubleshooting DNS failures related to the authoritative server. For each approach, select the scenario that matches best. Screen 25 of 26: Topic Title: D N S Troubleshooting Scenarios. Screen Title: Knowledge Check. This is a multiple choice question. Use your keyboard to cycle through the list of options. dnstrb_26 26 This lesson introduces techniques and approaches for troubleshooting DNS failures. Select a topic to review what you have learned. For a more thorough review, select Lesson Map and select a topic. dnstrb_26_01 D N S Query ToolsPopup 1 of 3: Popup title: D N S Query Tools. Bulleted text appears with instructions to roll your cursor over each underlined term to review its description. N s look up Command-line tool for testing and troubleshooting DNS servers. Non interactive Simple command examples: n s look up. N s look up name server. Ns look up I P underscore address server. Commands to query another server: nslookup xerxes nslookup xerxes.example.mil nslookup -querytype=ns xerxes.example.mil nslookup -querytype=a xerxes.example.mil nslookup xerxes.example.mil ns.example.mil. Interactive allows more complex queries: Change servers, Set query options, Debug DNS. Different data types lookup: At the command prompt, the set type or set q[uerytype] commands let you look up different data types within the domain name space. Zone transfer: Nslookup.exe lets you transfer an entire zone. This capability no longer exists in recent versions of nslookup on Linux/UNIX systems, but may be useful on Windows. Dig: Domain information groper, or dig, is a flexible tool for querying DNS name servers. Where both dig and nslookup are available, dig is preferable. It provides more precise information based on command-line options, and more detailed output for troubleshooting. Transferring zones with dig. Dig does not have a special command to request a zone transfer. To transfer zones, you specify axfr as the query type, and the domain name of the zone as arguments. dnstrb_26_02 Packet Capture and Other ToolsPopup 2 of 3: Popup title: Packet Capture and Other Tools. Bulleted text appears with instructions to roll your cursor over each underlined term to review its description. DNS packet sniffers are tools that allow the complete capture of traffic that is traveling between networked computers. Packet sniffers are generally used to capture data traveling between clients and servers, and to analyze traffic on your network. Tcpdump: This command-line tool collects and dumps data on IP networks. It examines all packets that cross the Ethernet interface, even those not addressed to it. Tcpdump allows you to restrict the data it collects by using filters to select addresses, protocols, and port numbers for capture. Wireshark: Formerly known as Ethereal, this GUI-based packet-capture tool performs many of the same tasks as tcpdump. However, Wireshark provides a more visual interface and user-friendly output. Other tools: Tools such as ping, traceroute, and netstat can be helpful during DNS troubleshooting to confirm network status and behavior. To use these tools for troubleshooting, the tools must function properly during normal operations, and the DNS administrator must understand normal behavior and indications. dnstrb_26_03 D N S Troubleshooting ScenariosPopup 3 of 3: Popup title: D N S Troubleshooting Scenarios. Bulleted text appears with instructions to roll your cursor over each underlined term to review its description. Recursive server scenarios: Name does not resolve: Most likely culprits in this scenario are client-machine caching. Failure to communicate with the local recursive. Remote authoritative servers. Other naming sources outside of DNS, such as WINS and local hosts files. Specific resource record type errors: Error types include: MX symptoms may appear in mail administration logs. SRV may cause Active Directory to malfunction. PTR may cause sever reverse lookup errors. Large packet errors may lead to name resolution problems for large packets. Athoritative server scenarios. Problems with authoritative zone files on master DNS server: Most common symptom of this type of failure is that the authoritative server is unable to load the zone file. Zone transfer failures: Zone transfer failures, including TSIG failures, usually involve the ability of master and slave servers to load updated zone files. Other possible causes could be related to firewall setup, a difference between the master and slave in their TSIG shared secret key or their time synchronization. Failures of DNS dynamic update: Failure of DNS dynamic updates includes frozen or paused zones, and TSIG. Likely points of failure in this scenario include failure of the master server to receive updated zone files, often because the updates were sent to the wrong server. Other possibilities include that the zone is frozen or paused, the client and master server time is not synchronized to within 20 seconds, or that the client`s TSIG key does not match the master server. Both recursive and authoritative: DNS failures that involve both recursive and authoritative servers include failure to start or restart the DNS server, and DNS attacks, specifically denial-of-service attacks. Screen 26 of 26: Topic Title: Summary and Conclusion. Screen Title: Summary. Image of D N S hierarchy appears labeled D N S query tools. Image labeled Packet capture and other tools appears with two servers and a data dot between them. Image labeled D N S troubleshooting scenarios appears with image of person at a computer workstation with callout What type of scenario appears along with images of five troubleshooting activities in text boxes. Instructions appear to select a topic to review. D N S Query tools, Packet Capture and Other Tools, and D N S Troubleshooting Scenarios appear selectable as popups.