Every part of a URL and what it is allowed to contain

Every part of a URL labelled on a worked example, which characters must be percent encoded where, how query strings actually behave, and how to spot a link that lies about its host.

A URL is one string that says which protocol to speak, which machine to speak it to, and what to ask for. Reading one correctly is mostly a matter of knowing where each part stops, because the delimiters are single characters and the first one wins.

https://alex:s3cret@www.example.co.uk:8443/docs/intro?lang=en&page=2#notes
│       │           │                 │   │           │              └─ fragment
│       │           │                 │   │           └──────────────── query
│       │           │                 │   └──────────────────────────── path
│       │           │                 └──────────────────────────────── port
│       │           └────────────────────────────────────────────────── host
│       └────────────────────────────────────────────────────────────── userinfo
└────────────────────────────────────────────────────────────────────── scheme

The parts

PartDelimited byWhat it is forReaches the server
Schemeends at the first :Which protocol and which rules apply to the restDecides the connection
Userinfoafter //, ends at @Credentials, long deprecated for http and httpsOnly if the client chooses to send them
Hostends at :, /, ? or #A domain name or IP address to resolve and connect toYes, in the Host header
Portafter :, ends at /, ? or #The TCP port, defaulting to 80 for http and 443 for httpsUsed for the connection
Pathstarts at /, ends at ? or #Which resource on that hostYes, in the request line
Querystarts at ?, ends at #Parameters for that resourceYes, in the request line
Fragmentstarts at #, runs to the endA location inside the resourceNo

Host and port together are the authority. The fragment is the outlier: it is stripped before the request is built, so a server never sees it. Browsers use it for anchors and client side routing, and some authentication flows deliberately return tokens in a fragment so those tokens never appear in a server log. It still lands in browser history, so it is not private, only unsent.

Reserved and unreserved characters

Percent encoding writes a byte as % plus two hex digits, using the UTF-8 bytes of the character. é becomes %C3%A9, two bytes and two escapes.

Four characters are unreserved alongside letters and digits, and never need encoding anywhere: - . _ ~. Everything else is either reserved, meaning it has a structural job somewhere in a URL, or has to be encoded. The reserved set is : / ? # [ ] @ and ! $ & ' ( ) * + , ; =.

The word "reserved" does not mean "always forbidden". It means the character is a delimiter in some part, and it must be encoded only in the parts where it would be read as one.

CharacterIn a path segmentIn a query value
Space%20, always%20 or +
/%2F, or it splits the segmentLegal as it stands
?%3F, or the path ends thereLegal as it stands
#%23, always%23, always
& and =Legal as they stand%26 and %3D, or the pair splits
+Legal, and means a plus%2B, or it may decode as a space

That is the practical rule: encode the character that would end the part you are in. A space in a path must be %20 because a raw space ends the URL in most parsers, while a space in a query value can be %20 or the older +. A URL Encoder is the reliable way to do this, since encoding a whole URL and encoding one component are different operations and the second is almost always what is wanted.

Query strings are a convention

The specification says only that the query is everything between ? and #, drawn from the allowed characters. The key=value&key=value shape comes from HTML form encoding, not from the URI standard. Nothing stops a server from parsing ?a:1;b:2 however it likes.

Because the shape is convention, repeated keys have no defined meaning, and every stack picked its own answer:

Behaviour for ?id=1&id=2Where
Both values, as a listPython parse_qs, Node querystring, Express
First value onlyGo Query().Get, Java getParameter
Last value onlyPHP, Rails
Joined into 1,2ASP.NET request collections

Arrays inherit the same problem. ids=1&ids=2, ids[]=1&ids[]=2, ids[0]=1&ids[1]=2 and ids=1,2 are all in wide use, and only the server decides which one it understands. Bracket notation has to be encoded to be strictly legal (ids%5B%5D=1), though most servers accept the raw brackets. Pick the form the receiving side documents, then keep it consistent.

The plus sign deserves its own warning. In application/x-www-form-urlencoded data, a form encoded request body, + means a space. In a URL path it means a literal plus. In a query string it depends on the parser, and most web frameworks apply form decoding there, so + silently becomes a space. Any value that can legitimately contain a plus, such as standard Base64 output or a phone number, must be sent as %2B.

Relative references

A relative reference is resolved against a base URL by replacing everything after the base's last /. That last slash is the whole rule, and it is why a trailing slash matters.

BaseReferenceResult
https://ex.com/docs/introguidehttps://ex.com/docs/guide
https://ex.com/docs/intro/guidehttps://ex.com/docs/intro/guide
https://ex.com/docs/intro/guidehttps://ex.com/guide
https://ex.com/docs/intro/../guidehttps://ex.com/docs/guide
https://ex.com/docs/intro?a=1?b=2https://ex.com/docs/intro?b=2
https://ex.com/docs/intro?a=1#tophttps://ex.com/docs/intro?a=1#top
https://ex.com/docs/intro//cdn.ex.com/x.jshttps://cdn.ex.com/x.js

The last row is a scheme relative reference: two leading slashes keep the base's scheme and replace the authority. A reference that begins with ? keeps the path and drops the old query, and one that begins with # keeps both.

When two URLs are the same resource

Scheme and host are case insensitive, so HTTPS://Example.COM and https://example.com are one address. Everything after the host is case sensitive as far as the standard is concerned, even if a particular server chooses to ignore case.

These pairs are equivalent:

  • https://example.com:443/a and https://example.com/a, because the port is the default
  • https://example.com and https://example.com/, because an empty path means the root
  • /a/%7Euser and /a/~user, because ~ is unreserved and percent encoding it changes nothing

These pairs are not:

  • /docs and /docs/, which are different resources, although most servers redirect one to the other
  • /p?a=1&b=2 and /p?b=2&a=1, since parameter order is part of the string
  • /index.html and /, unless the server says otherwise

Caches and CDNs key on the exact bytes, so an added tracking parameter or a reordered query splits one cached object into two and halves the hit rate. Search engines treat the variants as duplicate pages unless a canonical link points at one chosen form. The fix is to pick a single shape, redirect the rest to it with a 301, and strip parameters that do not change the response.

Length, logs and privacy

No length limit exists in the specification, but limits exist everywhere else. Common server defaults cap the whole request line at about 8 KB, some proxies and appliances at 4 KB, and older clients at around 2 KB. Anything that must survive email clients, QR codes and link shorteners is safest under about 2000 characters.

The stronger argument for keeping URLs short is that a query string is not private. It is written into access logs, passed to third parties in the Referer header, kept in browser history, saved with bookmarks, and copied whenever someone shares the link. Session tokens, password reset codes, API keys and personal data therefore do not belong there. Data that is large or sensitive goes in a request body, which has no practical size ceiling and is not logged by default.

The authority begins after :// and ends at the very first /, ? or #. Read that span, then read its last two or three labels. That is the real host, and nothing to its left changes it.

https://www.paypal.com@198.51.100.7/secure/login
        ^^^^^^^^^^^^^^ ^^^^^^^^^^^^
        userinfo       real host

Common dressing to recognise:

  • Text before an @ is userinfo, never the host, and it can be any brand name at all.
  • paypal.com.secure-login.example is a subdomain of secure-login.example. Labels read right to left.
  • https://short.example/https://www.bank.com/login puts a whole convincing URL in the path.
  • A host beginning xn-- is Punycode, the ASCII form of an internationalised domain name produced by IDNA. münchen.de is xn--mnchen-3ya.de on the wire, which is legitimate, but the same mechanism enables homographs: Cyrillic а (U+0430) is visually identical to Latin a, and a domain using it encodes to something like xn--pple-43d.com. Browsers show the Punycode form when a label mixes scripts, though the rules vary and are not a guarantee.
  • Encoded delimiters such as %2F or %40 inside a host are a deliberate attempt to confuse a parser.

Pasting the link into a URL Parser settles it in one step, since a real parser applies the same first-delimiter-wins rule the browser will and shows the host on its own. For checking many links at once, a Regex Tester is a quick way to confirm which of them actually have the authority you expect.