Identifying Nmap Scanning Techniques with Wireshark

Introduction

Wireshark was originally created in 1998 by Gerald Combs, who needed a tool for network troubleshooting at his job with an ISP. Initially called Ethereal, it quickly grew in popularity among network engineers and security professionals. In 2006, the project was renamed Wireshark due to trademark issues, and it has since become the industry standard for protocol analysis and packet inspection. Wireshark is open source, cross-platform, and actively maintained by a global community of contributors.

Nmap is one of the most powerful and widely used network scanning tools in cybersecurity. From Red Teamers mapping an environment to defenders watching for suspicious activity, Nmap’s versatility makes it both an asset and a threat depending on who’s behind the keyboard. One of its most commonly used features is the ability to perform stealthy port scans, techniques designed to probe systems without triggering obvious alarms. Feel free to take a look at my post on Unlocking Network Scanning with Nmap for some more information on this great tool.

Unlocking Network Scanning with Nmap

As defenders, being able to identify these scan types is critical. While traditional tools like firewalls and intrusion detection systems may log suspicious behavior, they often miss the finer details that a packet capture can reveal. That’s where Wireshark comes in!

In this post, we’ll walk through several Nmap scanning techniques and demonstrate how each one looks in Wireshark. For every scan type, you’ll see how it works, what to look for in packet captures, and which filters to use to identify suspicious behavior, helping you become faster and more effective at recognizing network scans in the wild. With side-by-side .pcapng captures and practical filtering tips, you’ll learn how each scan behaves on the wire and how to spot its unique signature.

I will be using a set .pcap files from Hack The Box’s Intermediate Network Traffic Analysis module. The .pcap files are public facing, but since the module is not, I will not share them here. Feel free to enroll HTB Academy to follow along or to take a look at some of the great content they offer. Anyways, let’s get to it!

SYN Scan

Wireshark Filter

tcp.flags.syn == 1 && tcp.flags.ack == 0

Diving into TCP flag values:

This screenshot shows Nmap sending multiple SYN packets to various ports on the target. These are the initial probes of a SYN scan, where Nmap attempts to identify open ports without completing full TCP handshakes.

Nmap Command Used

nmap -sS 192.168.10.5

The SYN scan is the default and most widely used scan type in Nmap. It’s known as a half-open scan because it sends a TCP SYN packet to initiate a connection but intentionally never completes the three-way handshake. This behavior is designed to be stealthier than a full TCP connect scan, allowing Nmap to identify open, closed, or filtered ports without making a complete connection.

When a SYN packet is sent to a port:

If the port is open, the target replies with a SYN-ACK.
If the port is closed, the target replies with a RST (or RST-ACK) from the target.
If the port is filtered, there may be no response at all.

What we see

Source IP: 192.168.10.1 (Nmap).
Destination IP: 192.168.10.5 (the target).
TCP Flags: [SYN] only, no ACK, no other flags.
Ports: Varying destination ports (5996, 5990, 5989, etc.), this is typical of a port scan probing many services.
No full handshakes, this filter excludes SYN-ACKs and follow-up ACKs, so it clearly shows only the scan attempts.

How to Detect It

Filter by tcp.flags.syn == 1 && tcp.flags.ack == 0 to see Nmap’s initial SYN probes.
Check for SYN-ACK responses to identify open ports.
Check for RST responses from the target to identify closed ports.
Look at timing, scan packets often appear in rapid bursts.
Look for repeated SYNs with no full 3-way handshake, a strong sign of port probing.

Here you can see a response indicating port 5000 is open:

tcp.flags.syn == 1 && tcp.flags.ack == 1

This SYN-ACK packet is sent by the target in response to Nmap’s SYN probe. It indicates that port 5000 is open.

RST packet following open port response (normal teardown):

ip.src == 192.168.10.5 && tcp.flags.reset == 1

This RST packet appears after the SYN-ACK as part of the normal behavior of a SYN scan. Nmap sends a RST to avoid completing the handshake, and in some cases, the target may respond with its own RST. Although RSTs are typically associated with closed ports, in this case the RST is sent by Nmap, not by the target, it’s part of the teardown after an open port was identified.

Using this query, you can view RST/ACK responses from the target, these indicate closed ports:

tcp.flags.reset == 1

In a SYN scan, if the port is closed, the target immediately responds with a RST, which you can see in the screenshot above

FIN Scan

Wireshark Filter

tcp.flags == 0x01

Diving into TCP flag values:

nmap -sF 192.168.10.5

In a FIN scan, Nmap sends TCP packets with just the FIN flag set. Closed ports should respond with RST, while open ports will ignore the packet. This makes it a stealthier approach than a full SYN scan, especially against older or poorly configured firewalls.

The packet capture shows TCP FIN probes sent to various ports. Using a filter for RST responses, we can confirm which ports are closed.

When a FIN packet is sent to a port:

If the port is closed, the target replies with a RST.

If the port is open (or filtered), the target does not respond at all.
If the port is filtered, there may be no response or an ICMP unreachable, depending on firewall behavior.

Using this query, you can view RST responses, which indicate closed ports during a FIN scan:

tcp.flags.reset == 1

In this example, the target responds with RST, ACK packets, confirming that these ports are closed. If a port were open or filtered, there would typically be no response at all.

What we see

Source IP: 192.168.10.1 (Nmap).
Destination IP: 192.168.10.5 (target).
TCP Flags: Only FIN (0x01).
Ports: Varying destination ports (5996, 5990, 5989, etc.), this is typical of a port scan probing many services.
Closed ports should respond with a RST.
Open ports typically do not respond at all, this is how Nmap infers they’re open.

How to Detect It

Look for FIN-only packets (tcp.flags == 0x01) sent to multiple ports across a short time span.
Closed ports reply with RST packets, expect to see many RST, ACK responses.
Open or filtered ports typically show no response at all.
This behavior is uncommon in normal traffic, so even small bursts of it can indicate scanning activity.

A FIN flag without an existing connection is unusual, flag it.

NULL Scan

Wireshark Filter

tcp.flags == 0x00

Diving into the TCP flag values:

Nmap Command Used

nmap -sN 192.168.10.5

The NULL scan is a stealthy TCP scan method where packets are sent without any TCP flags set. A closed port should respond with a RST, while an open port should simply ignore the packet.

This scan technique is designed to bypass firewalls or intrusion detection systems that expect to see typical TCP behavior, like SYN packets or full connections. Because a packet with no flags is highly irregular, it may slip through poorly configured or outdated security tools.

Using this query, you can view RST responses caused by NULL scan packets:

tcp.flags.reset == 1

The target replies with RST, ACK (0x014), which means the port is closed. This is standard behavior, closed ports must respond to invalid or unexpected TCP flag combinations with a reset.

What we see

Source IP: 192.168.10.1 (Nmap).
Destination IP: 192.168.10.5 (target).
TCP Flags: None set (0x00).
Ports: Varying destination ports (e.g., 6451, 6455, 6460, etc.), which is typical of a scan targeting many services.
Closed ports respond with a RST.
Open ports typically do not respond at all, this is how Nmap infers they’re open or filtered.

How to Detect It

NULL-only packets (no flags set) sent to a wide range of ports.
Use the filter: tcp.flags == 0x00.
RST responses from closed ports indicate the scan is working as expected.
NULL scans are rare in normal traffic, so seeing multiple across varied ports is highly suspicious.

No TCP flags in use? That’s almost never legitimate.

XMAS Scan

Wireshark Filter

tcp.flags == 0x29

Diving into the TCP flag values:

Nmap Command Used

nmap -sX 192.168.10.5

A TCP XMAS scan is named because it “lights up” certain TCP flags in the packet — specifically FIN, PSH, and URG — like a Christmas tree.

This unusual flag combination is non-standard and not used in normal communication. If a port is closed, it should reply with a RST.

If a port is open, it should ignore the packet. Like FIN and NULL scans, the XMAS scan is designed to probe targets without initiating a real TCP connection, making it stealthy and hard to detect for simple firewalls or intrusion detection systems.

Using this query, you can view RST responses caused by <>:

tcp.flags.reset == 1

These RST, ACK packets are returned by the target when it receives XMAS packets on closed ports. No response would indicate an open or filtered port. This scan type is abnormal in legitimate traffic, so the presence of multiple RSTs following oddly-flagged packets is a strong indicator of reconnaissance.

What we see

Source IP: 192.168.10.1 (Nmap)
Destination IP: 192.168.10.5 (target)
TCP Flags: FIN, PSH, URG (0x29)
Ports: Varying destination ports (e.g., 199, 445, 1025, etc.), typical of a scan sweeping multiple services.
Closed ports respond with a RST.
Open or filtered ports typically do not respond, Nmap interprets this as open|filtered.

How to Detect It

Multiple packets with this unique flag combination.
RSTs from closed ports, no response from open ones.

Very uncommon flag combo, this is textbook XMAS scan activity.

ACK Scan

Wireshark Filter

tcp.flags.ack == 1 && tcp.flags.syn == 0

Diving into the TCP flag values:

Nmap Command Used

nmap -sA 192.168.10.5

Unlike other scans that try to identify whether ports are open or closed, the ACK scan is used to map firewall rules. Specifically, it determines whether a port is filtered (stateful firewall) or unfiltered (stateless) by sending TCP ACK packets to each port and observing how the target responds.

Why this works:

In a normal TCP session, ACKs are used as part of an established connection.
When an ACK arrives without a prior SYN, the target should:
Respond with RST if the port is unfiltered (i.e., no stateful inspection).
Drop the packet silently if it’s filtered by a firewall that tracks session state.

What we see

Source IP: 192.168.10.5 (Nmap)
Destination IP: 192.168.10.1 (target)
TCP Flags: ACK only (0x10)
Ports: Varying destination ports (e.g., 1700, 6000, 6880, etc.), consistent with firewall probing.
If a RST is returned, it means the port is unfiltered (the packet reached the host and the port isn’t blocked).
If there’s no response, the port is considered filtered, a firewall or device likely dropped the packet silently.

How to Detect It

Standalone ACKs with no prior SYN or handshake.
RST responses from the destination host.
Indicates probing for firewall behavior, not port discovery.

An ACK without context usually means recon, not legitimate traffic.

Example displaying RST sent by Nmap’s response:

ip.src == 192.168.10.1 && tcp.flags.reset == 1

Example displaying RST sent by the target response:

tcp.flags.reset == 1

This suggests a stateless firewall (or no firewall at all), because a stateful firewall would have dropped the ACKs silently, meaning you wouldn’t see any RSTs.

Interpretation:

If you see a RST, the port is unfiltered, it reached the host and was rejected.
If you see no response, the packet was likely dropped by a firewall, meaning the port is filtered.

Fragmented Packet Scan

What It Is

Breaks up the TCP header into small IP fragments to try to evade intrusion detection systems or firewall rules that don’t reassemble packets before inspection.

Wireshark Filter

ip.frag_offset > 0 || ip.flags.mf == 1

Diving into the TCP flag values:

Nmap Command Used

nmap -f 192.168.10.5

The -f flag in Nmap tells it to fragment the IP packets. This means each probe, normally a full TCP header in one packet, is split into multiple smaller packets (fragments). The goal is to evade firewalls and IDS that only inspect the first fragment of a packet or don’t reassemble packets at all.

This scan doesn’t change the scan type itself (you can combine it with -sS, -sF, etc.), but it changes the delivery of the scan.

What we see

Source IP: 192.168.10.5 (Nmap)
Destination IP: 192.168.10.1 (target)
Protocol: IPv4 (fragmented IP protocol)
Fragment Offset: Varies (example: offset = 0, 16, 32…)
Reassembly status: Wireshark shows packets like: IP fragmented IP protocol (proto=TCP, …) [Reassembled in #…]
Behavior: Packets appear as fragmented IP (proto=TCP), often split into multiple parts
No TCP handshake or clean SYNs are visible unless reassembled
Purpose: To bypass firewalls that only inspect the first packet in a sequence

How to Detect It

TCP traffic arriving in multiple tiny IP fragments.
Fragmentation where none is necessary (especially for SYN/ACK packets).
Abnormal patterns, like dozens of tiny fragments in short bursts.

Fragmented TCP headers? That’s a bypass attempt, not a coincidence.

Conclusion

Understanding and identifying scan patterns in packet captures is a crucial capability for defenders. While stealth scans attempt to bypass detection by avoiding standard connection flows, tools like Wireshark expose the underlying packet-level behavior.

Each .pcapng file highlights a different Nmap scanning technique, each with its own distinct network signature, whether through uncommon flag combinations, missing TCP handshakes, or fragmented headers. Targeted display filters and close inspection of TCP flag behavior can reveal these techniques even when they’re designed to appear subtle.

Detecting Nmap scans is not about isolating a single packet but recognizing coordinated patterns over time. Identifying these footprints within traffic allows for early awareness of reconnaissance activity, before further action occurs.