This Book describes Kalem's main branch, which may be ahead of the latest release.

Completers, modes and highlighters

1. Completers
2. Document modes
3. Layers over a mode
4. Highlighters
5. Language packs

A file type gets three things from Kalem beyond plain editing: a completer, which offers completions at the cursor; a highlighter, which colors its code; and a document mode, which renders it (Org as a styled document, CSV as a grid, LaTeX with formulas typeset). The design gives each one contract shared by the built-in ones and by plugins (D25). This chapter says where each contract stands today.

Contract	State today	Code	Plugin side
Completers	Implemented; the built-in completers use it, and plugins' through the `completer` export (API 0.2.11)	`kalem-core/src/completers.rs`	API 0.2.11 `completer`
Highlighters	syntect with the built-in syntax definitions, and language plugins' (no code)	`kalem-highlight/src/lib.rs`	T3.1.9g
Document modes	The contract with its conformance check; CSV, Markdown, LaTeX and Org on it, the editors drawing each mode through `mode_view`	`kalem-core/src/modes.rs`	T2.7c.10, T3.1.9g
Language packs	Implemented and wired; no pack built in	`kalem-core/src/packs.rs`	T3.1.9c, T3.1.9e
Layers	Implemented: a plugin's overlays over Markdown and Org, applied by `mode_view`; the graph plugin's	`kalem-core/src/layers.rs`	API 0.2.10 `layer`

1. Completers

A completer is a source of completions: Org's keywords and link targets, LaTeX's commands and citation keys, the words of the document. One contract serves all of them (design §11.12): the core runs the completers that apply at the cursor, merges and ranks their items, and shows one menu, the same in both editors. The contract is a Rust trait in kalem_core::completers, implemented inside Kalem; a plugin offers a completer through the plugin API's completer export, which Kalem wraps in the same trait (see A plugin's completer, below).

The menu opens by itself when a completer's trigger fires as you type, and Ctrl+Space or Alt+/ (edit.complete) asks for it anywhere. Down and Up choose an item, Escape closes the menu, Tab takes the chosen item and so does Enter, except for a word from the document, where Enter closes the menu and starts a new line. Taking an item is one undo step. kalem complete FILE:LINE:COL prints the items at a place, best first, one a line, as label, kind and completer separated by tabs; it completes as if asked with Ctrl+Space and waits up to two seconds for slow completers.

$ kalem complete notes.org:4:5      # line 4 is "#+ti"
#+title:	keyword	org

ID	Applies in	Trigger	Priority	Items
`org`	Org documents	`#+`, `[[`, `[fn:` or `:` before the cursor	10	`#+` keywords and blocks, link targets, footnote labels, tags
`latex`	LaTeX files	`\`, `{`, `,` or `[` before the cursor	10	Commands, environments, citation keys, labels, files, packages, options
`words`	Every mode but the file manager	A word of three letters	0	Words of the document, nearest to the cursor first

The words completer takes words of at least three letters that start with the word being typed, ignoring case, from the megabyte of text around the cursor, nearest first, at most fifty; in Org documents it stays quiet where Org's own completion applies. A dictionary with spell checking and a model are not implemented; a language server completes the files of a language plugin that names one.

1.1. The contract

pub trait Completer: Send + Sync {
    fn id(&self) -> &'static str;                 // "org", "words"
    fn priority(&self) -> i32 { 0 }               // higher first among equals
    fn scope(&self) -> Scope { Scope::all() }     // the text types it serves
    fn applies(&self, ctx: &Context) -> bool;     // where else it applies
    fn trigger(&self) -> Trigger;                 // what opens its menu
    fn slow(&self) -> bool { false }              // run on another thread
    fn budget(&self) -> Duration { DEFAULT_BUDGET } // a slow one's time: 100 ms
    fn complete(&self, ctx: &Context, doc: Option<&DocumentState>,
                cancel: &Cancel) -> Vec<Item>;
    fn resolve(&self, _item: &Item) -> Option<String> { None }
    fn hover(&self, _ctx: &Context, _doc: Option<&DocumentState>) -> Option<String> { None }
}

scope gives the text types the completer serves, as a command's scope does, by the type at the cursor, so a completer for python also runs in a Python source block; applies decides the rest from the context. A fast completer gets the document (doc is Some) and runs on the editor's thread, so it must return quickly; a completer whose slow is true runs on another thread with the context only, and long work checks cancel.cancelled(). resolve gives documentation for an item when it is chosen; hover gives text about the thing under the cursor (the editors' hover card shows a language server's, not yet a completer's).

Trigger	Fires when
`Strings(&[...])`	One of the strings occurs in the line before the cursor, or on request
`WordPrefix(n)`	The word before the cursor has at least `n` characters; on request, one
`Request`	Only on request (Ctrl+Space, Alt+/)

The Context holds text (the document's text up to a megabyte, else the megabyte around the cursor, starting at base), point, mode, text_type, path and requested; its methods slice(range), line_before(), paragraph_before() and word_prefix() give the text around the cursor. An Item has label, insert, range (what it replaces, ending at the cursor), cursor (where the cursor goes in insert), kind (word, keyword, link, tag, footnote, symbol, snippet), detail and source; Item::new(label, insert, range, kind) makes one.

At each keystroke that may complete, the Registry keeps the completers that apply and whose trigger fires, runs the fast ones at once and the slow ones on their own threads, adding their items as they arrive; the next keystroke drops the old completion and sets its cancel flag. Items with the same insertion over the same range count once, and the list is ordered by items whose label starts with the word being typed, then items accepted recently, then the completer's priority. A slow completer that misses its budget (budget, 100 ms) shows nothing for that keystroke and the menu keeps the others' items.

1.2. Writing a completer today

Inside Kalem, a new completer is a type that implements Completer, added to Registry::with_builtins in completers.rs (the LaTeX completer in latex_complete.rs is a complete example). A sketch that completes a fixed list of words in Markdown after two letters:

use kalem_core::command::Scope;
use kalem_core::completers::{Cancel, Completer, Context, Item, Kind, Trigger};
use kalem_core::DocumentState;

struct Glossary;

impl Completer for Glossary {
    fn id(&self) -> &'static str { "glossary" }
    fn scope(&self) -> Scope { Scope::only(&["markdown"]) }
    fn applies(&self, _ctx: &Context) -> bool { true }
    fn trigger(&self) -> Trigger { Trigger::WordPrefix(2) }
    fn complete(&self, ctx: &Context, _doc: Option<&DocumentState>, _c: &Cancel) -> Vec<Item> {
        let (start, prefix) = ctx.word_prefix();
        ["adjunction", "category", "functor", "morphism"]
            .iter()
            .filter(|w| w.starts_with(prefix))
            .map(|w| {
                let mut item = Item::new(*w, *w, start..ctx.point, Kind::Word);
                item.detail = "glossary".into();
                item
            })
            .collect()
    }
}

1.3. A plugin's completer

A plugin offers completions through the completer interface it exports (plugin API 0.2.11, kalem-plugin's feature completer). Its manifest declares its completers, which Kalem runs with its own:

"completers": [{
  "id": "refs", "types": ["markdown", "org"], "layers": ["logseq"],
  "triggers": ["[[", "#", "(("], "replaces": ["wiki", "words"], "priority": 12
}]

Field	What it says
`types`	The text types it serves, as a completer's scope; all when missing
`layers`	The plugin's own layers: when given, it applies only in a document one of them serves
`triggers`	Strings before the cursor on its line that open its menu, as `Trigger::Strings`
`word`	A word of this many letters opens it too; with neither, only Ctrl+Space asks it
`replaces`	Completers it stands in for: it is asked first, and where it gives items they are not asked
`priority`	Its rank among the others when items rank the same: Kalem's own are 10, the words 0; 5 when missing

Kalem asks the plugin's complete(completer, request) on the editor's thread as the user types: the request holds the document's file, its text (all of it up to a megabyte, else the megabyte around the cursor, from base), the cursor and the text type at it; each item names the byte where the text it replaces starts (it ends at the cursor), what it inserts and where the cursor goes in it, its kind, a detail and documentation. An item that does not fit the text is left out. While another call of the plugin runs, the completer offers nothing for that keystroke. The graph plugin of getkalem/plugins completes pages, tags and blocks with it in a Logseq graph or an Obsidian vault.

2. Document modes

The modes today are a fixed set in the core (DocumentMode in mode.rs):

Mode	Name in settings and `editorMode`	Files
Org	`org`	`.org`, `.org_archive`
Markdown	`markdown`	`.md`, `.markdown`, `.mdown`, `.mkd`, `.gfm`
CSV	`csv`	`.csv`, `.tsv`, `.tab`
LaTeX	`latex`	`.tex`, `.latex`, `.ltx`
BibTeX	`text`, language `bib`	`.bib`, shown as a grid of entries
Plain text	`text`	Everything else that is text, with a language for highlighting
Binary	`binary`	Not opened for editing
File manager	`directory`	Folders

A mode is chosen in this order: the mode chosen with Set Document Mode (view.setMode), which the workspace settings remember by path in files.modes; a mode line on the first line (-*- mode: org -*- or -*- org -*-; a name that is not a mode opens plain text with that language); the extension; a #! line (#!/usr/bin/env python3 opens plain text in python); plain text without a language. A file whose start has a NUL byte, or bytes that are not UTF-8 with many control characters, is binary.

One contract serves every mode, the one built-in modes use and plugins will use (§11.11): the trait ModeSpec in kalem_core::modes, with id and detect (extensions and a sniff function over the first bytes), parse(text, edit?, previous?) (a flat tree of nodes, each with a kind from a fixed vocabulary, a byte range and its parent), grid(text) (rows and cells with ranges, for table-like formats), edit (Enter, Tab and toggles, each returning edits with ranges), the language pack hooks outline, format and diagnostics, and an optional to_html. A mode returns ranges into the text, never text, so it cannot break the byte-exact round trip; modes::check is the conformance check every mode runs: every range inside the text and inside its parent, two parses equal, an incremental parse equal to a full one. CSV, Markdown, LaTeX and Org (org_mode::OrgMode) are written against it (T2.7c.10). LaTeX maps its syntax tree to the kinds: sectioning commands as headings, levels counted from the document's highest (a book's chapters, an article's sections are level 1); itemize, enumerate and description as lists, each item from its \item to the next with what it holds; tables with rows ended by \\ and cells split by &; code environments with their language (minted's argument, lstlisting's language=); displayed and inline mathematics; \emph, \textbf, \texttt, \verb, \href and \url with their targets, \includegraphics with its file, \footnote; any other command or environment by its name. Its outline is the headings' titles, its formatter kalem fmt's, its diagnostics kalem check's, Enter and Tab those of the editor; every file of the arXiv sample passes modes::check. Both editors draw each mode through kalem_core::mode_view, one trait for a mode's lines, blocks and outline. Still to come: the plugin side, kalem::modes::register (T3.1.9g).

3. Layers over a mode

A layer changes how a mode draws a document without parsing it again: a plugin that knows a dialect of Markdown or Org (Logseq's blocks and properties, Obsidian's callouts, block ids and comments) says what its constructs become, and the mode draws the rest as it always does, its tables, links, formulas and pictures included. It is the way every "Markdown plus" is a plugin without a second Markdown parser.

A plugin declares its layers in its manifest:

"layers": [
  { "id": "logseq", "markers": ["logseq/config.edn"], "modes": ["markdown", "org"] },
  { "id": "obsidian", "markers": [".obsidian"], "modes": ["markdown"] }
]

A layer serves the documents of its modes whose folder, or a folder above it, holds one of its markers. For such a document Kalem asks the plugin's layer export (API 0.2.10) for overlays, by byte range of the text, once for each version of the text, and draws each line as its mode draws it with them applied:

Overlay	Away from the cursor	With the cursor in it
A span hidden	Not shown, as a mode's markers	The source
A span replaced	Shown as the layer's text, in its style	The source
A span styled	Drawn in the style (bold, a tag, a keyword…)	The same
Lines hidden	Not shown at all	Shown
Lines folded	Folded to their first line, as a drawer	Shown

The setting editor.show_source_markers rules them as it rules a mode's markers (always shows hidden and replaced spans as source, never never does). The source view (Show Source) shows the text as it is. A span that would cut a run the mode already replaced (a wiki link showing its title) is left out; overlapping spans and lines are left out after the first. A layer asked while a plugin's call runs answers at the next drawing; one that fails leaves the document as its mode draws it. A plugin calls layers.refresh when what its layers show depends on other files that changed (a block reference showing its block's text): every document a layer serves asks again.

The folder holding a layer's marker, once it is a project, is watched whole: a file another program writes there (Logseq or Obsidian open on the same graph, a sync) reaches the plugins as workspace-file-changed, as an open document's does, so that a plugin's index of the folder follows it; a file in a .git folder does not. An open document of it is read again, or the user asked, as always.

A layer may draw lines under a line of the text that are not part of it (API 0.2.13): a plugin built with kalem-plugin's feature layer-2 answers overlays_2, whose overlays add below, for the line holding a byte, lines of styled pieces, each so many columns in from where that line's text starts. The graph plugin shows an embedded block's children, an embedded page's blocks and a query's results under their line, as Logseq shows them in place. They are drawn under their line wherever it shows, the cursor on it or not, and not in the source view or when the line is folded or hidden away; the cursor steps over them, a search does not find them, a copy leaves them out, and a click on one puts the cursor at the end of their line.

Such a line may show the text of a file, to be typed into where it is drawn (API 0.2.15): a plugin built with the feature layer-3 answers overlays_3, whose lines under lines may each name a source, the file, its line and the byte of it the text starts at, and the text it held, to the line's end. The graph plugin's embedded blocks name the block's line in its page. A click on such a line, or Edit Line Under Line (view.editRow) on its line, makes it a one-line input showing that text as the file holds it, the cursor in it: Enter writes what was typed in place of that text, Escape drops it, Up, Down and Tab write it and go to the line before or after under the same line. It is written in the file's open document when that has unsaved changes, a step to undo there, else in the file, which an open document of it then reads again and the layer is asked about anew. Only files in Kalem's projects are written (the plugins' workspace, as fs:write:workspace reaches), and only where they still hold the text the layer saw; else Kalem says why and writes nothing. The text is that of one line: a line feed typed is a space.

A layer may name the plugin's command that opens the link at the cursor in the documents it serves ("open": "graph.openLink", API 0.2.12): a Ctrl-click runs it in place of the mode's own Open Link, so that a link is followed by the plugin's rules (a Logseq page's aliases, an Obsidian note's shortest path); the command leaves to the mode what is not its.

The plugin's keys in the documents a layer serves are bound with a when-clause naming it, editorLayer = graph.logseq=, and such keys come before the profile's and Vim's there, as a plugin's keys in its own documents do: an outliner's Enter or Tab in a list is the plugin's, not Markdown's.

The contract is kalem_core::layers; the plugin's side is kalem_plugin::layer with the feature layer, whose Plugin trait answers overlays(layer, path, text), with the feature layer-2 overlays_2(layer, path, text), or with the feature layer-3 overlays_3(layer, path, text).

4. Highlighters

Code is highlighted by kalem-highlight, one engine for plain text files and for Org source blocks (D16). It uses syntect, which reads Sublime Text syntax definitions, with pure Rust regular expressions; the definitions are syntect's default set with those of bat added (the two-face crate), built into Kalem: about two hundred languages, listed in Plain text and code. A language name is looked up, ignoring case, first in a table of Org's source block names that differ from syntect's (sh, shell, bash and zsh are shell, elisp is Lisp, js JavaScript, py Python, tex LaTeX, dot Graphviz), then as a syntax's name token, then as a file extension; a name that finds nothing is not highlighted.

The highlighter gives each span a kind from the syntax's scope, not a color: comment, string, number, constant, keyword, operator, function, type, variable, tag, macro, invalid. The editors color the kinds with the theme, whose six code colors (keyword, string, comment, number, function, type) you can change in themes/light.toml and themes/dark.toml. In plain text files the highlighter keeps its state at each line, so after an edit only the changed lines, and those after them until the state is what it was, are highlighted again. A language plugin adds Sublime syntax definitions from its folder, with no code: they come before the built-in ones for their extensions, and extends (which syntect does not know, and which Sublime Text's own syntaxes use) is resolved by Kalem. The set with the plugins' syntaxes takes most of a second to build, so it is built once and kept in Kalem's state folder until a plugin changes.

5. Language packs

A format whose view is its source (JSON, YAML, TOML, XML, a diff, a ledger journal, a gettext catalogue) gets no document mode but a language pack: four hooks on top of its highlighter, the same four a plugin gives (kalem_core::packs, T2.7a.7).

Hook	What the pack gives	Where Kalem uses it
`outline`	Headings with their level, title and start	The outline sidebar of both editors
`format`	The formatted text, or a refusal with its place	Format Document (`hasFormatter`), `kalem fmt`, `kalem check`
`diagnostics`	Problems with a range, a code and a message	The status bar, `kalem check`
`completer`	A completer on the contract above	The completion menu

A pack serves text types, the highlighter's language names (json, diff), and a document in plain text mode with one of them gets the pack's hooks. The status bar shows the problem on the cursor's line or how many there are; Format Document refuses a text the formatter refuses and puts the cursor at the place; kalem fmt lists it as "not formatted" and fails, and kalem check prints each problem as FILE:LINE:COL: error[CODE]: MESSAGE. The packs themselves are to be plugins in getkalem/plugins (D29). The plugin API gives three of the hooks so far, completer (0.2.11), problems for the diagnostics (0.2.14) and formatter (0.2.16), and not the outline; the contract is in the core with its wiring and tests, and no pack is built in.