Language plugins

From the editor: the Kalem menu (and the command palette) has Browse Plugins, Install Plugin…, Install Plugin from GitHub… and Installed Plugins.

• Browse Plugins reads the index of getkalem/plugins and lists its plugins with what is installed; choosing one installs it. Its first line, Install from a GitHub link…, is the command below.
• Install Plugin from GitHub… asks for the link of a plugin of your own, or of anyone's not in the index: a repository whose root is the plugin (https://github.com/OWNER/REPO, OWNER/REPO, or the git@github.com:OWNER/REPO.git of GitHub's Clone button), a folder in one (…/tree/BRANCH/FOLDER, or the link of its plugin.json), or a release's page (…/releases/tag/TAG: the repository at that tag). A language plugin installs as its folder is. A plugin with a component has its source in the repository but not its build (dist/ is not committed): Kalem takes the component from the repository's GitHub releases, the asset named as the file main names (zig.wasm), from the release of the link's tag, else one tagged with the plugin's version (v1.2.0, 1.2.0 or FOLDER-v1.2.0), else the newest. To publish one: kalem plugin build, then attach the .wasm to a release on GitHub.
• Install Plugin… asks for a plugin: a name from the index (elixir), a link, or a folder or archive on disk. Links Kalem understands: a folder of a GitHub repository, https://github.com/getkalem/plugins/tree/main/plugins/elixir; a repository whose root is the plugin, https://github.com/OWNER/REPO; an archive, https://…/NAME.tar.gz.
• Kalem downloads it in the background, then shows what it is before anything is installed: name, version, the languages it serves, the programs it may run (its language servers) and its permissions, and where it came from. Install puts it in ~/.config/kalem/plugins/ID and records it in ~/.config/kalem/plugins.toml (version, source, date, permissions); it works at once, in the files already open. Cancel leaves nothing behind.
• Installed Plugins lists them, with a newer version when the index has one; choosing one offers Update (from where it came), Remove (asked once more) and Show Its Folder. After an update or a removal the plugin's language servers stop and start again for the open files with the plugin as it is now.
• Once a day, when Kalem starts, it reads the index in the background and says in the status bar which installed plugins have a newer version (the setting plugins.check_updates turns this off). Only the version in plugin.json counts: a change published without a new version is found by Update, not by the check.

On the command line the same: kalem plugin browse, kalem plugin install NAME|LINK|PATH (--yes not to be asked), kalem plugin list, kalem plugin remove ID. The indexes are read from the settings plugins.sources (yours, first) and plugins.index (the index of getkalem/plugins by default); a file:// link reads a copy on disk.

Plugins with a WebAssembly component install as language plugins do; their component runs in Kalem's sandbox (see Plugins). A plugin from an index is checked against the SHA-256 the index gives; the releases of getkalem/plugins are also signed with Sigstore, which Kalem does not check yet (their README says how to by hand). A plugin from a link is installed as it is (a component is checked to be one), which is why Kalem shows where it comes from before installing.

Anyone can change how a plugin behaves, or publish plugins of their own, without waiting for getkalem/plugins: fork that repository on GitHub (or start one), change or add plugin folders, and add the fork's index as a source. Kalem ▸ Plugin Sources… lists the sources: Add a source asks for an index's address, a source of yours is removed by choosing it, and the official index can be set there too (empty for the default).

• Your sources (the setting plugins.sources) are read before the official index (plugins.index): a plugin in a fork stands for the plugin of the same ID in the official index, so Browse Plugins, Install Plugin… and the update check take the fork's.
• An index's source and download may be written relative to the index itself: in https://raw.githubusercontent.com/ada/plugins/main/index.json, the source wordcount is the folder wordcount of ada/plugins at main, so a fork's index needs no links rewritten. An index on disk (file://) names folders beside it.
• A source is an index's address: a raw.githubusercontent.com link, a github.com/.../blob/... link, any https address or a file:// path. A single plugin installs without an index too: Install Plugin… takes a GitHub folder's link or a folder on disk.

A plugin is a folder holding a plugin.json. Kalem looks in ~/.config/kalem/plugins/ (the plugins folder of the settings folder), where it installs them, and in the folders of the environment variable KALEM_PLUGIN_PATH, a list like PATH whose entries are plugin folders or folders of plugins (for developing a plugin in its repository). Both copies of a plugin load when it is installed too, and the installed one serves its files: remove it first (kalem plugin remove org.kalem.NAME, by its full ID) to try the other; a plugin found only there has no page in the settings panel. Plugins load when the settings load, before the first file opens; kalem lsp status lists them and the problems met loading them.

Beside the fields every plugin has (id, name, version, description, api), a language plugin has these:

{
  "syntaxes": ["syntaxes/Elixir.sublime-syntax", "syntaxes/HTML (HEEx).sublime-syntax"],
  "syntaxBases": ["syntaxes/bases/HTML.sublime-syntax"],
  "languages": [
    {
      "id": "elixir",
      "name": "Elixir",
      "syntax": "Elixir",
      "extensions": ["ex", "exs"],
      "filenames": ["mix.lock"],
      "shebangs": ["elixir"],
      "comment": { "line": "#" },
      "servers": ["expert", "elixir-ls"]
    }
  ],
  "servers": {
    "elixir-ls": {
      "name": "ElixirLS",
      "command": ["elixir-ls"],
      "candidates": ["language_server.sh", "~/.elixir-ls/release/language_server.sh"],
      "localDirs": [],
      "env": {},
      "rootMarkers": ["mix.exs"],
      "rootOutermost": true,
      "requireRoot": true,
      "busyLog": { "start": ["Starting project"], "done": ["Compiled "] },
      "initializationOptions": {},
      "settings": { "elixirLS": { "mixEnv": "test" } },
      "install": "brew install elixir-ls"
    }
  },
  "commands": { "format": ["mix", "format", "--stdin-filename", "{file}", "-"] }
}

• syntaxes are Sublime syntax files (D16), added to the highlighter. A plugin's syntax comes before a built-in one for the same extension. syntaxBases are files that other syntaxes extend and that are not languages of their own: a plugin's copy of HTML must not take .html files from Kalem's.

• A language is a server's languageId, the files it claims (by whole name first, then by the longest extension, so page.html.heex is HEEx and not HTML, then by the interpreter of a #! line), its syntax's name (its names and extensions are highlighted with that syntax, whatever the syntax's own extensions: eex with HTML (EEx)), its comment tokens (Toggle Comment uses them), its servers in order of preference (the first found serves it), and the servers that serve its files beside that one (alongside): a linter beside the type server, a server of one part of the file. The Rust plugin claims Cargo.toml by name, highlighted as TOML, for Taplo (its keys from SchemaStore's Cargo schema) with crates-lsp beside it (the crates' versions); another .toml file stays Kalem's own:

  { "id": "toml", "name": "Cargo manifest", "syntax": "TOML", "filenames": ["Cargo.toml"],
    "servers": ["taplo"], "alongside": ["crates-lsp"] }
  

• A server is a program and its arguments. When the program is not found, candidates are tried in order: bare names on the PATH (after localDirs under the project's root, such as node_modules/.bin), or paths, with ~ for the home folder. The root is the folder up from the file holding one of rootMarkers: the nearest, or the outermost with rootOutermost (an umbrella project); without one, the file's folder, or no server at all with requireRoot (a script beside no mix.exs is highlighted, and the status bar says why no server runs). busyLog is for servers that report their work only in their log (Expert): the lines from one containing a start text to one containing a done text show as progress in the status bar. settings answer the server's workspace/configuration requests, are sent once after it starts, and again whenever the user's settings change. install is what the status bar says when no candidate is found, or when the one found does not start: Kalem never installs a server by itself. version are the arguments that make the program print its version and end (["--version"] for rust-analyzer); kalem lsp status runs them, in the root and with the server's environment, and prints the version or why the program does not run. A program found is not always one that runs: rustup puts a rust-analyzer on the PATH whether or not the component is installed. Leave version out when the program would not end (a script that starts the server whatever it is asked). capabilities are added to what Kalem tells the server it can do, where Kalem says nothing: the extensions a server sends only to a client that asks for them. status names the notification in which a server tells its state (rust-analyzer's, clangd's file status, Metals's status) and where in it Kalem finds the text, the level and whether the server is idle, as JSON pointers:

  "capabilities": { "experimental": { "serverStatusNotification": true } },
  "status": { "method": "experimental/serverStatus", "text": "/message",
              "level": "/health", "warning": "warning", "error": "error",
              "idle": { "/quiescent": true } }
  

  While the server says it works only in part (warning) or not at all (error), the status bar says its text; while it is not idle it counts as busy, and kalem lsp check waits for it. ignore names capabilities of the server's that Kalem leaves unused, by their names in its answer to initialize: a feature that is wrong, or that the plugin does otherwise. They are taken out of that answer, so nothing asks the server for them: with "ignore": ["documentFormattingProvider"] Format Document takes the plugin's commands.format, or says there is no formatter. initializationOptions are sent as the server starts, for what it reads only then.

• commands.format formats a document when no server formats it: the text on its standard input, the formatted text on its output, run in the background, {file} filled in. It is optional: a plugin states a formatter only when its language has one. A formatter the user sets in the formatters setting for the type comes before the plugin's; when a plugin has none, an extension plugin's formatter for the type is asked (the chapter Plugins), then the language's usual program runs when it is installed (the chapter Plain text and code).
• commands.run, commands.test and commands.testAtPoint are the project's run and test commands, ["cargo", "test"] for Rust and ["mix", "test", "{file}:{line}"] for the test at the cursor in Elixir: Run Project (project.run, SPC p R), Test Project (project.test, SPC p T) and Test at Cursor (project.testAtCursor) run the active file's plugin's in the project's root as its server finds it (the nearest Cargo.lock), {file} and {line} filled in. The output, standard error too, shows in a read-only document as it comes, its last line saying how the command ended, and the status bar says it runs; the files are run as saved. The same command run again, or its document closed, stops the one under way. Where the file's language gives no such command, Run Project and Test Project ask for one and run it in the shell at the project's root.

• requests give a server's own requests to Kalem's commands, which know the shapes of questions and answers, not the methods, so one command serves every server that has such a request:

  "requests": {
    "code.expandMacro": { "method": "rust-analyzer/expandMacro", "shape": "text",
                          "answer": ["/expansion"], "title": "/name", "language": "rust" },
    "code.openDocs": { "method": "experimental/externalDocs", "shape": "url" },
    "code.moveItemUp": { "method": "experimental/moveItem", "params": "range",
                         "extra": { "direction": "Up" }, "shape": "edits" },
    "code.structuralReplace": { "method": "experimental/ssr", "shape": "workspaceEdit",
                                "extra": { "query": "{search} ==>> {replace}",
                                           "parseOnly": false, "selections": "{selections}" } },
    "edit.newline": { "method": "experimental/onEnter", "shape": "edits" }
  }
  

  params is what is asked about: position (the document and the cursor, the default), document, range (the selection), ranges (the selection in a list) or none, with extra fields added. In them, {search} and {replace} in a text are what the command asks for, one after the other, and the text {selections} is the selection in a list, empty when nothing is selected. shape is what the answer is: text (shown as documentation is, under its title, as plain text), url (an http or https page, opened in the browser), location (gone to), edits (applied to the document as one undoable change, the cursor where a snippet in them puts it), workspaceEdit (edits of the project's files, offered as a rename's are before anything changes) or none. answer is where the text or address is in the answer (JSON pointers tried in order, the whole answer by default); server names the one of the plugin's servers that has the request. The commands: Expand Macro (code.expandMacro), Open Documentation in the Browser (code.openDocs), Go to Parent Module (code.parentModule), Go to Project File (code.openManifest), Reload Project (code.reloadProject), Join Lines (Language Server) (code.joinLines), Move Item Up and Down (code.moveItemUp, code.moveItemDown), Structural Search and Replace (code.structuralReplace: a pattern and its replacement, such as rust-analyzer's foo($a) ==>> bar($a)), each in the palette where the document's server has it.

  The request keyed edit.newline is asked as Enter makes a new line in code (and in Vim's Insert mode). Nothing waits for it: Kalem's new line is made at once, and the server's edits take its place in the same undo step as they come, the cursor where they put it, unless the text changed meanwhile; an answer of nothing leaves Kalem's. rust-analyzer's experimental/onEnter continues a // or /// comment so.

A user's settings for a plugin live under its ID in settings.toml, and the plugin's page in the settings panel (installed plugins) changes them without the file: the server, the settings the manifest describes under "settings" (see Plugins), each server's program and environment:

[plugins."org.kalem.elixir"]
server = "auto"     # a server's key, or "off"

[plugins."org.kalem.elixir".settings.elixirLS]
mixEnv = "dev"      # merged over the server's own settings

[plugins."org.kalem.elixir".servers.elixir-ls]
command = ["/opt/elixir-ls/language_server.sh"]
env = { ELS_INSTALL_PREFIX = "/opt/elixir-ls" }

[plugins."org.kalem.rust".servers.crates-lsp]
enabled = false     # a server beside the language's own, not started

[plugins."org.kalem.pta".servers.beancount-language-server]
initializationOptions = { journal_file = "~/money/main.beancount" }
ignore = ["documentFormattingProvider"]

A server's initializationOptions are merged over the manifest's, as its settings are: a server that reads part of its configuration only when it starts (beancount-language-server's main journal) is told the user's, and a plugin's manifest may describe such a setting (servers.beancount-language-server.initializationOptions.journal_file) so that its page shows it. ignore stands in place of the manifest's: the server's capabilities Kalem leaves unused, [] for all of them used (a server whose formatting the plugin turned off because one program of that name formats wrongly, used with another that does not).

A server starts when the first file it serves opens and stops when the last one closes. A language may have servers beside its own (alongside): each gets every file of the language, kept in step as the first. Their problems are shown with the first's, each under its source; completion lists the items of each server that completes, the language's own first; code actions are one list of every server's; a question one server answers (documentation, a definition, a rename, formatting) goes to the first that has it, the language's own server first. Language Server Status (code.serverStatus) and kalem lsp status name them, or say why one does not run; one not installed is said there only, not as a notice, and Restart Language Server starts it again once it is installed. A file a server named in an answer outside its root (a definition in Rust's standard library, a dependency's source, a system header) is served by that server when it opens, since it knows the file as part of its project; a root of the file's own would start another server there (a crate from crates.io has its own Cargo.lock, and rust-analyzer started in the standard library's folder cannot load it). A server already running in the file's own root serves it instead. Kalem keeps it in step: the whole text when a file opens, then each change as the range it replaced (as UTF-16 or UTF-8 positions, whichever the server takes; Kalem's own offsets are bytes), and the save. Nothing waits for a server on the editor's thread: an answer arrives while you type, and a server that never answers costs nothing but its silence. A server that exits is started again after one second, then two, four, eight and sixteen; at its sixth exit it stays stopped, and the status bar says so (exits are forgotten after ten minutes of running). A server whose process ends before it answers initialize is not started again, since it would end the same way (a toolchain's proxy for a component not installed, a version manager's shim with no version chosen, initialize refused, no answer in two minutes): the status bar says at once why, in the server's words (the last lines it wrote on its standard error) and with the plugin's install text. Restart Language Server (code.restartServer) tries again once it is mended.

gd, gD and K act only in files a server serves. A SPC c key with no server says why: no plugin for the file's type, a server not installed (with the plugin's install text), servers turned off, or a server that does not provide the feature. Several places are a list to choose from; one definition is a jump. The status bar shows the problem on the cursor's line, else the server's work in progress (ElixirLS compiling), else the counts of errors and warnings, after the server's own word on its state while it does not work fully (rust-analyzer's "cargo check failed to start"). Formatting is applied as one undoable change, and only when the document has not changed since it was asked. A server that answers null is said to have changed nothing, not to have found the file formatted (an empty list of edits is "Already formatted"): rust-analyzer answers so when the file is formatted, and also when rustfmt fails (not installed for the project's toolchain, or a syntax error), which its log tells (kalem lsp check --log).

A server's problems come two ways, and Kalem takes both: those it sends on its own, and those it gives when asked (the protocol's pull model). rust-analyzer does both: cargo's errors and clippy's come as the check on save ends, its own (type mismatches, names, unresolved imports, a file no module includes) when Kalem asks, after the file opens, after each change once the server has answered the last question, and when the server says they changed. A server busy loading the project may cancel a question ("content modified"); Kalem asks again after half a second, one and two, and says it is still busy only after that.

Problems are underlined in the text, errors in red, warnings in orange (yellow in the terminal) and the rest (information, hints) in blue, a problem over several lines on each of them; the line's number is colored in the gutter, and the message shows under the mouse (in the window). The problems list (SPC c X) has the project's files the server reports, open or not. A rename shows how many changes in how many files before anything changes; edits of open documents are made in their editors as one undoable change each (skipped, and said, for a document changed since it was asked), files not open are written. A code action's edit is applied and its command run; the edits a server then asks to apply (workspace/applyEdit) are applied the same way. Files changed outside the editor in a server's project (a checkout, a dependency fetched) are told to the server, which compiles again.

Inlay hints, the short texts a server would show in a line (a type, a parameter's name, an elided amount, a running balance), are drawn in place, dimmed and in italics, in both editors; the cursor passes over them. Kalem asks for those of the whole file once the server is ready, after each change once the last question is answered, and again when the server says they changed; until the answer, the hints shown move with the edits, and those inside an edit go. editor.inlay_hints (on) shows them, and Toggle Inlay Hints (code.toggleInlayHints) hides or shows them until the settings are read again. The server decides which it gives: hledger-lsp's running balances, for one, come only once its inlayHints.showRunningBalances is set.

kalem lsp status lib/hello.ex          # the plugins, what serves the file, its version
kalem lsp check lib/*.ex               # the servers' diagnostics; exits 1 on errors
kalem lsp check --format json lib/hello.ex
kalem lsp check --log lib/hello.ex     # and the server's log on standard error
kalem lsp ask hover lib/hello.ex 6:40  # documentation at line 6, column 40
kalem lsp ask definition lib/hello.ex 6:40
kalem lsp ask format lib/hello.ex      # the formatted text
kalem lsp ask code.expandMacro src/main.rs 12:16   # a server's own request, by its command
kalem lsp ask code.structuralReplace src/main.rs 1:1 --input 'search=foo($a)' --input 'replace=bar($a)'

status starts no server; it runs a server's version when the plugin gives one, and exits 1 when the program found does not run. check and ask start the server, wait until it is ready and quiet (--wait seconds at most), ask, and stop it; a server that does not start is said at once.

A completion item's documentation is the server's: given with the items (Expert, ElixirLS), or fetched with completionItem/resolve when an item is chosen, from servers that answer it.

Semantic tokens, folding ranges, code lenses, workspace symbols and the server's own command list are the rest of T3.8.2. Completion items are inserted as plain text until the snippet engine (T3.8.3): a function is inserted as name() with the cursor between the brackets, other snippets with their places left empty. Edits that create, rename or delete files are refused with the reason. The server's log (its log messages and standard error) is kept in memory and printed by kalem lsp check --log; the log view is T2.7g.2.