system.tokenizers

Querying in ClickHouse Cloud

The data in this system table is held locally on each node in ClickHouse Cloud. Obtaining a complete view of all data, therefore, requires the clusterAllReplicas function. See here for further details.

Shows all available tokenizers. These can be used in functions tokens, hasAllTokens, hasAnyTokens, and the text index.

Columns:

• name (String) — Name of the tokenizer
• type (String) — Type of the tokenizer

Example

SELECT * FROM system.tokenizers;

┌─name────────────┬─type────────────┐
│ ngrams          │ Ngrams          │
│ splitByNonAlpha │ SplitByNonAlpha │
│ sparseGrams     │ SparseGrams     │
│ tokenbf_v1      │ SplitByNonAlpha │
│ ngrambf_v1      │ Ngrams          │
│ array           │ Array           │
│ splitByString   │ SplitByString   │
│ sparse_grams    │ SparseGrams     │
└─────────────────┴─────────────────┘