IP List Deduplicator
Remove repeated addresses.
IP List Deduplicator: inputs and results
Enter an IP address, CIDR prefix, or range. Then check its first and last address below.
Calculated Boundaries & ParametersReady
Key Capabilities
- •100% Client-Side: Zero telemetry or server latency.
- •RFC Compliant: Exact 32-bit and 128-bit integer precision.
- •Quick Copy: Instant access to CIDR, mask, and range bounds.
- •Cross-Platform: Works on desktop, tablet, and mobile browsers.
Related Diagnostics, Generators & UtilitiesAll Tools
IPv4 Reverse DNS Name Generator
Generate address-level reverse lookup names.
Reverse DNS Zone Planner
Identify applicable reverse delegation boundaries.
Bounded IP Address List Generator
Enumerate a selected, limited address range.
Random Address Sampler
Sample addresses within supplied ranges.
Batch Subnet Calculator
Calculate details for multiple prefixes.
IP Address Sorter
Sort addresses numerically.
How IP List Deduplicator Works?
The IP List Deduplicator works by normalizing every address to 1 canonical form, then keeping only the first occurrence of each normalized value.
Paste the address list
Enter addresses 1 per line, or separated by commas or spaces.
Normalize each entry
Whitespace is trimmed and IPv6 addresses are rewritten in RFC 5952 form.
Track seen values
A set records each normalized address. Repeats are dropped.
Output the unique list
The tool prints the unique addresses with the count of removed duplicates.
IP List Deduplicator Formula
The IP List Deduplicator formula is Unique = { normalize(x) : x in Input }, and Removed = |Input| − |Unique|.
Normalization
trim + canonical formIPv4 dotted decimal; IPv6 RFC 5952.
Unique count
|set(normalize(Input))|Distinct addresses.
Removed
|Input| − |Unique|Duplicates dropped.
Order
first occurrence keptOriginal order is preserved.
Worked example: 6 entries
Input : 10.0.0.1, " 10.0.0.1", 10.0.0.2, 2001:db8::1, 2001:DB8:0:0::1, 10.0.0.2 Unique : 10.0.0.1, 10.0.0.2, 2001:db8::1 Removed: 3
IP List Deduplicator Conversion Table
The IP List Deduplicator conversion table shows the normalized form and the keep or remove decision for 10 input entries.
| Input Entry | Normalized | Decision |
|---|---|---|
| "10.0.0.1" | 10.0.0.1 | Kept |
| " 10.0.0.1 " | 10.0.0.1 | Removed (duplicate) |
| "10.0.0.2" | 10.0.0.2 | Kept |
| "192.168.1.10" | 192.168.1.10 | Kept |
| "2001:db8::1" | 2001:db8::1 | Kept |
| "2001:DB8:0:0::1" | 2001:db8::1 | Removed (duplicate) |
| "2001:0db8:0000:0000:0000:0000:0000:0001" | 2001:db8::1 | Removed (duplicate) |
| "10.0.0.2" | 10.0.0.2 | Removed (duplicate) |
| "192.168.1.100" | 192.168.1.100 | Kept |
| "fe80::1" | fe80::1 | Kept |
What is IP List Deduplicator?
The IP List Deduplicator is a tool that removes repeated addresses from a list, treating different spellings of the same address as duplicates.
Plain text deduplication misses many repeats. 2001:db8::1 and 2001:DB8:0:0::1 are the same IPv6 address, and an entry with a trailing space looks different to a simple sort. The deduplicator compares canonical forms, so every real duplicate is caught.
Deduplication keeps the first occurrence of each address in its original position, which preserves any meaningful order in the source list.
What are the uses of IP List Deduplicator?
There are 4 main uses of the IP List Deduplicator, including blocklist cleanup, log analysis, and allow-list maintenance.
Cleaning blocklists
Merged threat feeds contain the same address many times.
Counting unique clients
Log exports reduce to the set of unique source addresses.
Maintaining allow-lists
Repeated entries waste rule capacity on firewalls.
Preparing scan targets
Unique target lists prevent scanning the same host twice.