Plain text and code

For a plain text file the oracle is the file itself: opened and saved without an edit, it stays byte for byte; opened, edited and saved, it differs only in the bytes of the edit. Kalem has no model of a text file beyond its characters, so there is nothing else to agree with.

What is tested: the byte order mark and CR LF surviving a round trip, line feeds inserted by commands getting their carriage return in a CR LF file, UTF-16 LE and BE with a byte order mark round-tripping exactly, Windows-1254 text guessed from its bytes and written back to the same bytes, characters an encoding cannot hold, reopening in a chosen encoding (the decoding and encodings tests of files.rs); atomic saving, backups, change detection, symbolic links and permissions (saving, links_and_permissions); and an Org file with CR LF edited, saved and reloaded (files in document.rs). There is no corpus test of arbitrary files yet.

A document has one mode (DocumentMode): Org, Markdown, CSV, LaTeX, text with an optional language, binary (not opened) or a folder (the file manager). When a file is opened, the mode is decided in this order; the first rule that applies wins.

1. Binary. Unless the file starts with a UTF-16 byte order mark, its first 8,192 bytes (after a UTF-8 byte order mark) are checked (looks_binary): a NUL byte, or bytes that are not UTF-8 (not merely a character cut at the end of the sample) with control characters making up more than a twentieth of the sample, make it binary. Text in a legacy encoding such as Windows-1254 has no control characters and is not taken for binary. A binary file is not opened (see Files that are not text).
2. The user's choice. Set Document Mode (view.setMode) records the mode for the file in the workspace settings, in the table files.modes of the nearest .kalem/settings.toml above the file (created beside it when there is none), keyed by the file's path relative to that workspace with / between names. The value is org, markdown, csv, latex or text, or the language chosen for a text file (python). When the file is opened again, in either editor, this mode replaces the detected one.
3. A mode line. On the first line (after a byte order mark), or on the second when the first is a #! line (as Emacs reads it), the text between two -*- markers: either mode: NAME among ;-separated key: value pairs, or a bare NAME when there is no colon. The name is lower-cased. With a trailing -mode removed, a name of one of the modes below selects that mode (org-mode, markdown, csv, latex…); any other name is taken as the language of a text file, as written (so -*- python -*- highlights Python, while -*- mode: python-mode -*- names the unknown language python-mode).
4. A Local Variables: block. In the last 3,000 characters of the file, a line holding Local Variables: starts a block that ends at a line holding End:; the text before Local Variables: on its line is a prefix, and the text after it a suffix, removed from each line of the block (so # mode: org and /* mode: org */ both work). A mode: NAME line there names the mode as a mode line does. As in Emacs's set-auto-mode, it comes before the file name.

5. The extension. The file name's last extension, lower-cased:

   Extensions	Mode
   org, org_archive	Org
   md, markdown, mdown, mkd, gfm	Markdown
   csv, tsv, tab	CSV
   tex, latex, ltx	LaTeX
   any other	Text, the extension as its language

   Any extension decides, so a script named run.sh is shell by its extension whatever its first line says, and notes.txt is text with the language txt. One exception: a .txt file whose first lines (up to twenty non-empty ones, at least two) all hold the same number of tabs, at least one, with a value before the first tab in most of them, is CSV: this is Excel's "Unicode Text" export (UTF-16 with a byte order mark, tab-separated).

6. A shebang. Only for a file without an extension: a first line starting with #!. The program is its first word, or the word after it when the first word ends in /env; its last path component with trailing digits and dots removed is the language (#!/usr/bin/env python3 gives python, #!/bin/sh gives sh).
7. Plain text, without a language.

A file that does not exist yet (opened by name to be created) gets its mode from its name alone, UTF-8 without a byte order mark, and LF line endings (CR LF on Windows).

The names Set Document Mode accepts are org, markdown, csv, text or plain, the names and extensions of the table above (md, tex, latex…), and any other word, which is taken as a language: view.setMode with python gives text highlighted as Python. Open as Plain Text in a CSV file switches to text for the session only.

In when-clauses the mode is editorMode (org, markdown, csv, latex, text, binary, directory); the language of a text file is the command scope's text type, lower-cased (rs, python, bib).

A text file with a language is coloured by the highlighter (kalem_highlight), which runs Sublime Text syntax definitions through syntect with pure Rust regular expressions (fancy-regex). The language name is resolved (Language::find) in this order, ignoring letter case:

1. Org's source-block language names that differ from the syntaxes' names (the alias table below);
2. a syntax whose name or extensions match the name (syntect's token search);
3. a syntax that claims the name as an extension.

The syntax "Plain Text" is never used, so txt files are not coloured. A name that resolves to nothing leaves the file uncoloured, with every editing feature of text files still working.

The aliases:

Names	Syntax
sh shell bash zsh	Bourne Again Shell (bash)
elisp emacs-lisp lisp scheme	Lisp
clojure	Clojure
js javascript	JavaScript
ts typescript	TypeScript
tsx	TypeScriptReact
h	C
toml	TOML
ini conf	INI
python py	Python
rust	Rust
c	C
cpp c++	C++
latex tex	LaTeX
sql sqlite	SQL
R	R
haskell ocaml java go ruby perl php html css xml yaml json markdown makefile diff lua matlab scala erlang	The syntax of that name
dot	Graphviz (DOT)

The syntaxes and the extensions they claim: the default set of syntect 5.3 with the syntaxes of bat 0.26.1 added, through the two-face crate (0.5.2). An entry that is a whole file name (Makefile, Cargo.lock) is listed as the syntax claims it, but the language is looked up by extension only (see below). The table is made by cargo run -p kalem-highlight --example syntax_table.

Syntax	Extensions
ActionScript	as
Ada	adb ads gpr
Apache Conf	envvars htaccess HTACCESS htgroups HTGROUPS htpasswd HTPASSWD .htaccess .HTACCESS .htgroups .HTGROUPS .htpasswd .HTPASSWD
AppleScript	applescript script editor
AsciiDoc (Asciidoctor)	adoc ad asciidoc
ASP	asa
Authorized Keys	authorized_keys pub authorized_keys2
AWK	awk
Batch File	bat cmd
BibTeX	bib
Bourne Again Shell (bash)	sh bash zsh ash .bash_aliases .bash_completions .bash_functions .bash_login .bash_logout .bash_profile .bash_variables .bashrc .profile .textmate_init .zlogin .zlogout .zprofile .zshenv .zshrc PKGBUILD ebuild eclass
C	c h
C#	cs csx
C++	cpp cc cp cxx c++ C h hh hpp hxx h++ inl ipp
Cabal	cabal
camlp4
Cargo Build Results
CFML	cfml cfm cfc
CFML Script
CFML Script (Tags)
Clojure	clj cljc cljs edn
CMake	CMakeLists.txt cmake
CMake C Header	h.in
CMake C++ Header	hh.in hpp.in hxx.in h++.in
CMakeCache	CMakeCache.txt
CMakeCommands
CoffeeScript	coffee Cakefile coffee.erb cson
Comma Separated Values
Command Help	cmd-help help
commands-builtin-shell-bash
CpuInfo	cpuinfo
Crontab	tab crontab cron.d
Crystal	cr
CSS	css css.erb css.liquid
D	d di
Dart	dart
debsources	sources.list
Diff	diff patch
DMD Output
Dockerfile	Dockerfile dockerfile .Dockerfile
Dockerfile (with bash)
DotENV	.env .env.dist .env.local .env.sample .env.example .env.template .env.test .env.test.local .env.testing .env.dev .env.development .env.development.local .env.prod .env.production .env.production.local .env.dusk.local .env.staging .env.default .env.defaults .envrc .flaskenv env env.example env.sample env.template
Elixir	ex exs
Elm	elm
Elm Compile Messages
Elm Documentation
Email	eml msg mbx mboxz
Erlang	erl hrl Emakefile emakefile escript
F#	fs fsi fsx
Fish	fish
Fortran (Fixed Form)	f F f77 F77 for FOR fpp FPP
Fortran (Modern)	f90 F90 f95 F95 f03 F03 f08 F08
Fortran Namelist	namelist
fstab	fstab crypttab mtab
GDScript (Godot Engine)	gd
GFortran Build Results
Git Attributes	attributes gitattributes .gitattributes
Git Commit	COMMIT_EDITMSG MERGE_MSG TAG_EDITMSG
Git Common
Git Config	gitconfig .gitconfig .gitmodules
Git Ignore	exclude gitignore .gitignore
Git Link	.git
Git Log	gitlog
Git Mailmap	.mailmap mailmap
Git Rebase Todo	git-rebase-todo
GLSL	vs fs gs vsh fsh gsh vshader fshader gshader vert frag geom tesc tese comp glsl mesh task rgen rint rahit rchit rmiss rcall
gnuplot	gp gpl gnuplot gnu plot plt
Go	go
Gomod	go.mod go.work
Gosum	go.sum
GraphQL	graphql graphqls gql
Graphviz (DOT)	dot DOT gv
Groff/troff	groff troff 1 2 3 4 5 6 7 8 9
Groovy	groovy gvy gradle Jenkinsfile
group	group
Haskell	hs
Highlight non-printables	show-nonprintable
HTML	html htm shtml xhtml
HTML (ASP)	asp
HTML (EEx)	html.eex html.leex
HTML (Erlang)	yaws
HTML (Jinja2)	htm.j2 html.j2 xhtml.j2 xml.j2
HTML (Rails)	rails rhtml erb html.erb
HTML (Tcl)	adp
HTML (Twig)	twig html.twig
HTTP Request and Response	http
Idris	idr
INI	ini INI inf INF reg REG lng cfg CFG desktop url URL .editorconfig .coveragerc .pylintrc .gitlint .hgrc hgrc
Java	java bsh
Java Properties	properties
Java Server Page (JSP)	jsp
Javadoc
JavaScript	js htc
JavaScript (Rails)	js.erb
Jinja2	j2 jinja2 jinja
JQ	jq
JSON	json sublime-settings sublime-menu sublime-keymap sublime-mousemap sublime-theme sublime-build sublime-project sublime-completions sublime-commands sublime-macro sublime-color-scheme ipynb Pipfile.lock
JSON (Terraform)	tfstate
jsonnet	jsonnet libsonnet libjsonnet
Julia	jl
Known Hosts	known_hosts known_hosts.old
Kotlin	kt kts
LaTeX	tex ltx
LaTeX Log
Lean 4	lean
Less	less css.less
Lisp	lisp cl clisp l mud el scm ss lsp fasl sld
Literate Haskell	lhs
LiveScript	ls Slakefile ls.erb
LLVM	ll
log	log
Lua	lua
Make Output
Makefile	make GNUmakefile makefile Makefile makefile.am Makefile.am makefile.in Makefile.in OCamlMakefile mak mk
Manpage	man
Markdown	md mdown markdown markdn
MATLAB	matlab
MediawikerPanel
MediaWiki	mediawiki wikipedia wiki
MemInfo	meminfo
MultiMarkdown
NAnt Build File	build
Navigational Bar SV
Navigational Bar VHDL
nginx	conf.erb conf nginx.conf mime.types fastcgi_params scgi_params uwsgi_params
Nim	nim nims nimble
Ninja	ninja
Nix	nix
NSIS	nsi nsh bnsi bnsh nsdinc
Objective-C	m h
Objective-C++	mm M h
OCaml	ml mli
OCamllex	mll
OCamlyacc	mly
Odin	odin
OpenMP (Fortran)
orgmode	org
Pascal	pas p dpr
passwd	passwd
Perl	pl pc pm pmc pod t
PHP	php php3 php4 php5 php7 phps phpt phtml
PHP Source
Pipe Separated Values
Private Key
Protocol Buffer	proto protobuf protodevel
Protocol Buffer (TEXT)	pb.txt proto.text textpb pbtxt prototxt textproto
Puppet	pp epp
PureScript	purs
Python	py py3 pyw pyi pyx pyx.in pxd pxd.in pxi pxi.in rpy cpy SConstruct Sconstruct sconstruct SConscript gyp gypi Snakefile vpy wscript bazel bzl
QML	qml qmlproject
R	R r Rprofile
R Console
Racket	rkt
Rd (R Documentation)	rd
Rego	rego
Regular Expression	re
Regular Expressions (Javascript)
Regular Expressions (PHP)
Regular Expressions (Python)
Requirements.txt	requirements.txt requirements.in pip
resolv	resolv.conf
reStructuredText	rst rest
Robot Framework	robot resource
Ruby	rb Appfile Appraisals Berksfile Brewfile capfile cgi Cheffile config.ru Deliverfile Fastfile fcgi Gemfile gemspec Guardfile irbrc jbuilder Podfile podspec prawn rabl rake Rakefile Rantfile rbx rjs ruby.rail Scanfile simplecov Snapfile thor Thorfile Vagrantfile
Ruby Haml	haml sass
Ruby on Rails	rxml builder
Ruby Slim	slim skim
Rust	rs
Sass	sass
Scala	scala sbt sc
SCSS	scss
Semi-Colon Separated Values
Separated Values	csv
Shell-Unix-Generic
SML	sml cm sig
Solidity	sol
SQL	sql ddl dml
SQL (Rails)	erbsql sql.erb
SSH Common
SSH Config	ssh_config
SSH Crypto
SSHD Config	sshd_config
Strace	strace
Stylus	styl stylus
Svelte	svlt svelte
Swift	swift
syslog	syslog
SystemVerilog	sv v svh vh
Tab Separated Values	tsv
Tcl	tcl
Terraform	tf tfvars hcl
TestBox
TeX	sty cls
Textile	textile
Todo.txt	todo.txt done.txt
TOML	toml tml Cargo.lock Gopkg.lock Pipfile pdm.lock poetry.lock uv.lock
TypeScript	ts mts cts
TypeScriptReact	tsx
Typst	typ
Typst Build Results
Typst Code
varlink	varlink
Verilog	v V
VHDL	vhd vhdl vho vht
VimL	vim vimrc gvimrc .vimrc .gvimrc .exrc .nvimrc _vimrc _gvimrc _exrc
Vue Component	vue
Vyper	vy
WGSL	wgsl
x86_64 Assembly	yasm nasm asm inc mac
XML	xml xsd xslt tld dtml rng rss opml svg xaml
YAML	yaml yml sublime-syntax
Zig	zig zon

Consequences worth knowing: .h files are coloured as C (Emacs's c-or-c++-mode guesses between C and C++; Kalem does not), .m files as Objective-C, not MATLAB, as Emacs's auto-mode-alist has it; .conf and .cfg files as INI. PowerShell has no syntax, so .ps1 files are not coloured. A file whose extension is the only clue gets no colour when it has no extension (a file named Makefile or Dockerfile has no extension and no shebang, and is plain text).

The highlighter gives each span a kind (comment, string, number, constant, keyword, operator, function, type, variable, tag, macro, invalid) from its scope; the editors colour kinds with their theme. After an edit only the changed lines are highlighted again, and the lines after them until the parser's state is what it was. A file larger than 4 MiB is highlighted a window at a time: 400 lines around the one asked for, from a fresh state 200 lines before the window, so a construct opened further back (a long block comment) may be coloured as code. A line longer than 16 KiB is shown only in the part around the cursor, with … for the rest.

Toggle Comment (edit.toggleComment, Ctrl+Alt+C) uses the marker of the language (code::comment_style), from the same name the mode gives (the extension, a shebang's program, a mode line's name). It is not offered in text without a language, in CSV, in folders or in binary documents.

Names	Marker
rs rust c h cc cpp c++ hpp cxx java js javascript jsx ts typescript tsx go swift kt kotlin scala cs csharp dart groovy zig d php json5 jsonc proto glsl hlsl less scss sass v	//
py python sh bash zsh fish shell rb ruby pl perl r yaml yml toml make makefile mk cmake conf cfg dockerfile nix jl julia tcl ps1 powershell elixir ex exs nim coffee gitignore awk sed properties env crystal cr graphql	#
lua sql hs haskell elm ada adb ads vhdl purs	--
lisp el elisp emacs-lisp clj clojure cljs scm scheme rkt racket fnl fennel ini asm s	;
tex latex sty cls bib erl erlang m matlab octave pro prolog ps postscript	%
vim vimscript	"
f90 f95 fortran	!
bat cmd	REM
org	#
html htm xml xhtml svg md markdown vue plist	<!-- … --> around each line
css	/* … */ around each line
ml ocaml fs fsharp	(* … *) around each line

The lines the selection covers are commented (a selection ending at the start of a line leaves that line out). When every non-blank line already starts with the marker (and, for the enclosing kinds, ends with the closing one), they are uncommented: the marker and one space after it are removed. Otherwise every non-blank line gets the marker and a space at the smallest indentation among them. Blank lines are left alone.

The indentation step of a document (indent_unit) comes from its text (text::detect_indent), looking at the first 10,000 lines within the first megabyte:

• tabs, when more lines start with a tab than with a space;
• otherwise spaces, the step being the most frequent increase of indentation from one non-blank line to the next among 2, 4, 3 and 8 (4 when no increase is seen);
• when no line is indented: tabs for the languages makefile, make, mk and go, four spaces for everything else.

Tab and Shift+Tab (in every mode but Org, where Tab folds and moves in tables, and CSV, where they move between fields):

• With a selection spanning lines, Tab adds one step at the start of every non-blank line; Shift+Tab removes one tab, or up to one step of leading spaces, from each line.
• Otherwise Tab inserts a tab, or the spaces up to the next multiple of the step, replacing a selection within the line; Shift+Tab removes one tab, or up to one step of leading spaces, from the start of the line.

Enter (edit.newline, in every mode but Org and LaTeX, which have their own):

• In a text file with a language (code::newline), the new line gets the current line's indentation; one step more when the text before the cursor ends with (, [ or { (or with : in py, python, yaml, yml and nim); and when the cursor sits between a bracket and its partner ({|}), the closing bracket goes to a line of its own at the original indentation, the cursor on an indented line between.
• In text without a language, Markdown and CSV (input::newline), the new line gets the current line's indentation, when the cursor is past it.

Typing }, ) or ] in a text file with a language, where only indentation precedes the cursor on its line and that indentation ends with one full step, removes that step first (code::dedent_for).

Go to Matching Bracket (Ctrl+Alt+B) and the marking of the bracket matching the one at the cursor pair (, [ and { by counting, up to 256 KiB each way; brackets inside strings and comments count like any other.

Kalem does not convert tabs to spaces or spaces to tabs, and never re-indents lines it did not touch.

A file's bytes are decoded as follows (files::decode_with):

1. A UTF-32 byte order mark (FF FE 00 00 or 00 00 FE FF): the file is not opened ("The file is in UTF-32, which Kalem cannot read"), as neither decoding nor saving supports it; it is not mistaken for UTF-16.
2. A UTF-8 byte order mark: the rest goes through the binary check (see Which mode a file gets) and must be valid UTF-8, or the file is not opened ("The file is not UTF-8 (invalid byte at offset n)"); the mark is removed from the text and remembered.
3. A UTF-16 LE or BE byte order mark: the rest is decoded as UTF-16; the mark is remembered.
4. Otherwise, after the binary check: valid UTF-8 is UTF-8, unless a coding: file variable (rule 5) names another encoding.
5. Otherwise, when the file names its encoding in a coding: file variable (in the -*- line, -*- coding: latin-5 -*-, or in the Local Variables: block), that encoding: Emacs's names (latin-1, iso-latin-5, turkish-iso-8bit, utf-8-unix, cp1254) or any WHATWG label, the -unix, -dos and -mac suffixes ignored.
6. Otherwise the encoding is guessed from the bytes with chardetng, the way Emacs guesses a coding system, and the status bar says "opened as" that encoding. The guess takes the system locale's region as a hint (tr_TR gives tr), or, with no region, its language or the interface language (tr, ja as jp, el as gr…); so a short Turkish text reads as Windows-1254 on a Turkish system rather than as Windows-1252, which it also fits. When the guessed encoding cannot read every byte, the file is read as Windows-1252, which reads each byte as a character and writes it back as the same byte.

When decoding meets bytes the encoding cannot read (a UTF-16 file with a lone surrogate, or Reopen with Encoding in an encoding that does not fit), they become U+FFFD and the editor says so on opening: saving writes U+FFFD in their place, so such a file does not survive a save unchanged. Save with Encoding accepts that.

The status bar names the encoding when it is not UTF-8. Reopen with Encoding (file.reopenWithEncoding) reads the file again in the encoding chosen, whatever its bytes, dropping a byte order mark of that encoding and turning bytes it cannot read into U+FFFD; it refuses while the document has unsaved changes. Save with Encoding (file.saveWithEncoding) makes the document's encoding the one chosen and saves; UTF-16 is always written with a byte order mark, UTF-8 keeps the mark if the file had one, other encodings have none. Both offer UTF-8, UTF-16LE, UTF-16BE, windows-1252, ISO-8859-1, windows-1254, ISO-8859-9, ISO-8859-15, windows-1250, windows-1251, KOI8-R, Shift_JIS, EUC-JP, GBK, gb18030, Big5 and EUC-KR, and accept any label the WHATWG Encoding Standard knows (latin1, utf-16le…). As in that standard (and in web browsers), ISO-8859-1 is Windows-1252 and ISO-8859-9 is Windows-1254: the status bar names them so, and bytes 0x80–0x9F are the Windows characters (€, curly quotes) rather than control codes, which text files do not use. When a file is reloaded after another program changed it, the document's encoding is used again unless a byte order mark names another.

On save the text is encoded in the document's encoding, the byte order mark written if the file had one. When a character cannot be written in the encoding (a ł in a Windows-1254 file), the save stops and says which character and encoding: take the character out, or Save with Encoding UTF-8, which writes the whole file in UTF-8 from then on. A file read with replacement characters (bytes that are not in the encoding chosen, shown as �) is not saved over: Save says so, and Reopen with Encoding reads it in the right one. Save with Encoding and Save As write it as it shows, the � included, on purpose.

The text keeps its line endings as they are in the file; carriage returns are characters of the text. A document's line ending (files::line_ending_of) is CR LF when the file has at least one line feed and every line feed follows a carriage return, as Emacs decides between dos and unix; CR when the file has carriage returns and no line feed (Emacs's mac, the files of classic Mac OS); otherwise LF.

• Lines are separated at line feeds. A CR file is the exception to keeping line endings in the text: it is read with every carriage return turned into a line feed, and saved with every line feed turned back into a carriage return, so it has lines and is written as it was. In other files a lone carriage return is not a line break.
• In a CR LF document, commands and typing insert bare line feeds, and saving writes a carriage return before every line feed that lacks one (files::encode). Pasted text has its CR LF made LF first and, in a CR LF document, every line feed made CR LF again.
• In an LF document nothing is added: a file with mixed line endings keeps each one as it was, and new lines get LF.
• No final line feed is added or removed.

The setting editor.trim_trailing_whitespace (off by default) removes spaces and tabs before the end of each line on saving, carriage returns excepted. It does nothing in CSV files, where trailing tabs are empty fields and blanks belong to values, nor in Markdown files, where two trailing spaces are a hard line break.

Saving is atomic: the bytes go to a temporary file beside the target (.NAME.kalem-PID-N.tmp), which is synced and renamed over it, and the folder is synced. The file's permissions are kept; a symbolic link is followed, so the link stays a link and its target is written, and changes to the target are watched as the link's. The group is kept when the user belongs to it. A file with several hard links, owned by another user, or in a folder Kalem may not write in, is written in place instead, since a rename would detach the links or change the owner, or cannot be made. A write-protected file is not saved over. The rename does not keep extended attributes (macOS Finder tags), ACLs or the creation date; a file written in place keeps them. With files.backup, the previous file is first copied to NAME.bak.

Before saving, Kalem compares the file with what it last read or wrote (length, time, and a hash of the contents when those differ); when another program changed it, saving stops unless forced. A document without unsaved changes reloads when another program changes its file.

A file found binary (rule 1 of the modes above) is not opened; opening it fails with "The file is not a text file". The graphical editor shows the reason in its status bar (a window opened on such a file shows an empty document with the reason); the terminal editor shows "Cannot open path: The file is not a text file". The file manager's preview of a file with a NUL byte in its first 64 KiB says "Not text: no preview"; pictures are previewed as pictures. Kalem never writes to a file it did not open.

UTF-16 text without a byte order mark contains NUL bytes and is therefore taken for binary; Reopen with Encoding cannot help, since the file is never opened.

• A file named only by convention (Makefile, Dockerfile) gets no language.
• A shebang is looked at only when the file has no extension.
• Bracket matching and Enter's bracket rules ignore strings and comments.
• Very large files are highlighted in windows and may colour a long construct wrongly.