Skip to main content

Functions for Working with URLs

All these functions do not follow the RFC. They are maximally simplified for improved performance.

Functions that Extract Parts of a URL​

If the relevant part isn’t present in a URL, an empty string is returned.

protocol​

Extracts the protocol from a URL.

Examples of typical returned values: http, https, ftp, mailto, tel, magnet…

domain​

Extracts the hostname from a URL.

domain(url)

Arguments

The URL can be specified with or without a scheme. Examples:

svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
https://clickhouse.com/time/

For these examples, the domain function returns the following results:

some.svn-hosting.com
some.svn-hosting.com
clickhouse.com

Returned values

  • Host name. If ClickHouse can parse the input string as a URL.
  • Empty string. If ClickHouse can’t parse the input string as a URL.

Type: String.

Example

SELECT domain('svn+ssh://some.svn-hosting.com:80/repo/trunk');
β”Œβ”€domain('svn+ssh://some.svn-hosting.com:80/repo/trunk')─┐
β”‚ some.svn-hosting.com β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

domainWithoutWWW​

Returns the domain and removes no more than one β€˜www.’ from the beginning of it, if present.

topLevelDomain​

Extracts the the top-level domain from a URL.

topLevelDomain(url)

Arguments

The URL can be specified with or without a scheme. Examples:

svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
https://clickhouse.com/time/

Returned values

  • Domain name. If ClickHouse can parse the input string as a URL.
  • Empty string. If ClickHouse cannot parse the input string as a URL.

Type: String.

Example

SELECT topLevelDomain('svn+ssh://www.some.svn-hosting.com:80/repo/trunk');
β”Œβ”€topLevelDomain('svn+ssh://www.some.svn-hosting.com:80/repo/trunk')─┐
β”‚ com β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

firstSignificantSubdomain​

Returns the β€œfirst significant subdomain”. The first significant subdomain is a second-level domain if it is β€˜com’, β€˜net’, β€˜org’, or β€˜co’. Otherwise, it is a third-level domain. For example, firstSignificantSubdomain (β€˜https://news.clickhouse.com/’) = β€˜clickhouse’, firstSignificantSubdomain (β€˜https://news.clickhouse.com.tr/’) = β€˜clickhouse’. The list of β€œinsignificant” second-level domains and other implementation details may change in the future.

cutToFirstSignificantSubdomain​

Returns the part of the domain that includes top-level subdomains up to the β€œfirst significant subdomain” (see the explanation above).

For example:

  • cutToFirstSignificantSubdomain('https://news.clickhouse.com.tr/') = 'clickhouse.com.tr'.
  • cutToFirstSignificantSubdomain('www.tr') = 'tr'.
  • cutToFirstSignificantSubdomain('tr') = ''.

cutToFirstSignificantSubdomainWithWWW​

Returns the part of the domain that includes top-level subdomains up to the β€œfirst significant subdomain”, without stripping "www".

For example:

  • cutToFirstSignificantSubdomain('https://news.clickhouse.com.tr/') = 'clickhouse.com.tr'.
  • cutToFirstSignificantSubdomain('www.tr') = 'www.tr'.
  • cutToFirstSignificantSubdomain('tr') = ''.

cutToFirstSignificantSubdomainCustom​

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain. Accepts custom TLD list name.

Can be useful if you need fresh TLD list or you have custom.

Configuration example:

<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
<top_level_domains_lists>
<!-- https://publicsuffix.org/list/public_suffix_list.dat -->
<public_suffix_list>public_suffix_list.dat</public_suffix_list>
<!-- NOTE: path is under top_level_domains_path -->
</top_level_domains_lists>

Syntax

cutToFirstSignificantSubdomain(URL, TLD)

Parameters

  • URL β€” URL. String.
  • TLD β€” Custom TLD list name. String.

Returned value

  • Part of the domain that includes top-level subdomains up to the first significant subdomain.

Type: String.

Example

Query:

SELECT cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list');

Result:

β”Œβ”€cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐
β”‚ foo.there-is-no-such-domain β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

See Also

cutToFirstSignificantSubdomainCustomWithWWW​

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. Accepts custom TLD list name.

Can be useful if you need fresh TLD list or you have custom.

Configuration example:

<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
<top_level_domains_lists>
<!-- https://publicsuffix.org/list/public_suffix_list.dat -->
<public_suffix_list>public_suffix_list.dat</public_suffix_list>
<!-- NOTE: path is under top_level_domains_path -->
</top_level_domains_lists>

Syntax

cutToFirstSignificantSubdomainCustomWithWWW(URL, TLD)

Parameters

  • URL β€” URL. String.
  • TLD β€” Custom TLD list name. String.

Returned value

  • Part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www.

Type: String.

Example

Query:

SELECT cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list');

Result:

β”Œβ”€cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list')─┐
β”‚ www.foo β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

See Also

firstSignificantSubdomainCustom​

Returns the first significant subdomain. Accepts customs TLD list name.

Can be useful if you need fresh TLD list or you have custom.

Configuration example:

<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
<top_level_domains_lists>
<!-- https://publicsuffix.org/list/public_suffix_list.dat -->
<public_suffix_list>public_suffix_list.dat</public_suffix_list>
<!-- NOTE: path is under top_level_domains_path -->
</top_level_domains_lists>

Syntax

firstSignificantSubdomainCustom(URL, TLD)

Parameters

  • URL β€” URL. String.
  • TLD β€” Custom TLD list name. String.

Returned value

  • First significant subdomain.

Type: String.

Example

Query:

SELECT firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list');

Result:

β”Œβ”€firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐
β”‚ foo β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

See Also

port(URL[, default_port = 0])​

Returns the port or default_port if there is no port in the URL (or in case of validation error).

path​

Returns the path. Example: /top/news.html The path does not include the query string.

pathFull​

The same as above, but including query string and fragment. Example: /top/news.html?page=2#comments

queryString​

Returns the query string. Example: page=1&lr=213. query-string does not include the initial question mark, as well as # and everything after #.

fragment​

Returns the fragment identifier. fragment does not include the initial hash symbol.

queryStringAndFragment​

Returns the query string and fragment identifier. Example: page=1#29390.

extractURLParameter(URL, name)​

Returns the value of the β€˜name’ parameter in the URL, if present. Otherwise, an empty string. If there are many parameters with this name, it returns the first occurrence. This function works under the assumption that the parameter name is encoded in the URL exactly the same way as in the passed argument.

extractURLParameters(URL)​

Returns an array of name=value strings corresponding to the URL parameters. The values are not decoded in any way.

extractURLParameterNames(URL)​

Returns an array of name strings corresponding to the names of URL parameters. The values are not decoded in any way.

URLHierarchy(URL)​

Returns an array containing the URL, truncated at the end by the symbols /,? in the path and query-string. Consecutive separator characters are counted as one. The cut is made in the position after all the consecutive separator characters.

URLPathHierarchy(URL)​

The same as above, but without the protocol and host in the result. The / element (root) is not included.

URLPathHierarchy('https://example.com/browse/CONV-6788') =
[
'/browse/',
'/browse/CONV-6788'
]

encodeURLComponent(URL)​

Returns the encoded URL. Example:

SELECT encodeURLComponent('http://127.0.0.1:8123/?query=SELECT 1;') AS EncodedURL;
β”Œβ”€EncodedURL───────────────────────────────────────────────┐
β”‚ http%3A%2F%2F127.0.0.1%3A8123%2F%3Fquery%3DSELECT%201%3B β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

decodeURLComponent(URL)​

Returns the decoded URL. Example:

SELECT decodeURLComponent('http://127.0.0.1:8123/?query=SELECT%201%3B') AS DecodedURL;
β”Œβ”€DecodedURL─────────────────────────────┐
β”‚ http://127.0.0.1:8123/?query=SELECT 1; β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

encodeURLFormComponent(URL)​

Returns the encoded URL. Follows rfc-1866, space( ) is encoded as plus(+). Example:

SELECT encodeURLFormComponent('http://127.0.0.1:8123/?query=SELECT 1 2+3') AS EncodedURL;
β”Œβ”€EncodedURL────────────────────────────────────────────────┐
β”‚ http%3A%2F%2F127.0.0.1%3A8123%2F%3Fquery%3DSELECT+1+2%2B3 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

decodeURLFormComponent(URL)​

Returns the decoded URL. Follows rfc-1866, plain plus(+) is decoded as space( ). Example:

SELECT decodeURLFormComponent('http://127.0.0.1:8123/?query=SELECT%201+2%2B3') AS DecodedURL;
β”Œβ”€DecodedURL────────────────────────────────┐
β”‚ http://127.0.0.1:8123/?query=SELECT 1 2+3 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

netloc​

Extracts network locality (username:[email protected]:port) from a URL.

Syntax

netloc(URL)

Arguments

Returned value

Type: String.

Example

Query:

SELECT netloc('http://[email protected]:80/');

Result:

β”Œβ”€netloc('http://[email protected]:80/')─┐
β”‚ [email protected]:80 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Functions that Remove Part of a URL​

If the URL does not have anything similar, the URL remains unchanged.

cutWWW​

Removes no more than one β€˜www.’ from the beginning of the URL’s domain, if present.

cutQueryString​

Removes query string. The question mark is also removed.

cutFragment​

Removes the fragment identifier. The number sign is also removed.

cutQueryStringAndFragment​

Removes the query string and fragment identifier. The question mark and number sign are also removed.

cutURLParameter(URL, name)​

Removes the β€˜name’ URL parameter, if present. This function works under the assumption that the parameter name is encoded in the URL exactly the same way as in the passed argument.