The Kalem Book GitHub

This Book describes Kalem's main branch, which may be ahead of the latest release.

Markdown

This chapter states what Kalem does with Markdown today and what remains. The two are kept apart: the sections under Today describe the code as it is; the last section, What remains of Markdown mode, lists the open work by its task.

In short: Markdown is one of Kalem's built-in formats. Kalem reads it with comrak, the Rust port of GitHub's own parser, in Kalem's fork (D19), and draws a Markdown file as it reads: markers hidden away from the cursor, headings by level, emphasis, code, links, images, formulas, task list checkboxes, tables and front matter. The file is edited as text and never rewritten: Kalem writes nothing into a Markdown file that you did not type, and never converts it in order to open it. Kalem also writes Markdown as an export of Org documents, in two flavours, Emacs's ox-md Markdown byte for byte and GitHub Flavored Markdown, and converts a Markdown file to Org without pandoc when you ask (Convert to Org, kalem export FILE.md --to org).

1. The oracle

The standard is CommonMark with the GitHub extensions (GFM). The oracle named by the design is the CommonMark and GFM specification test suites, every example of each specification a test, with cmark-gfm as the reference implementation where the suites are silent, plus a byte-exact round trip of untouched text on a corpus of real READMEs and note vaults. The suites run as Kalem's own test (markdown::spec, T2.7c.11): CI downloads CommonMark 0.31.2's and GFM 0.29's specifications, which are CC BY-SA and not in the repository (KALEM_COMMONMARK_SPEC and KALEM_GFM_SPEC name copies on disk; without them the test passes without checking). Every example's HTML, as comrak renders it, is compared with the specification's: all 652 CommonMark examples agree with CommonMark's options, and all 24 extension examples with GFM's. Every example's nodes lie inside the text and their parents, every line is drawn with the cursor on it and away, an edit in the middle reparses as a full parse does, and the view shows the text the specification's HTML shows, in all 676 examples.

With every extension Kalem reads a file with, as the editor does, 639 of the 652 CommonMark examples agree. The thirteen others are the extensions' own cases, and the count only goes up:

  • four of Autolinks: bare addresses and mail addresses are links, as GitHub's extension makes them;
  • six of HTML blocks: <script>, <style> and <textarea> are escaped, as GitHub's tag filter does;
  • one of Links and one of Images: [[…]] is a wiki link;
  • one of Setext headings: a document starting with --- is front matter in Kalem, as in GitHub, Jekyll and Obsidian.

The Markdown Kalem writes has a different oracle, Org's exporter: the md back-end is compared byte for byte with Emacs's ox-md (see Markdown export from Org below).

2. Today: Markdown files in the editor

2.1. Which files

A file is in Markdown mode when its extension is md, markdown, mdown, mkd or gfm (any letter case), when its first line has a mode line naming markdown (-*- mode: markdown -*-, -*- markdown -*-, or markdown-mode), or when the user chose Markdown with Set Document Mode (view.setMode with mode: markdown), which is remembered for the file under files.modes in the workspace settings. The mode's name in when-clauses and scopes is markdown (editorMode == markdown, text type markdown); the status bar says "Markdown".

2.2. What the mode does

Markdown mode (DocumentMode::Markdown) draws the file as it reads and edits it as plain text. Concretely:

  • The file is read, decoded and saved as every text file is (the chapter Plain text and code): encoding, byte order mark and line endings kept, untouched bytes unchanged. Kalem never rewrites a Markdown file.
  • Each line is drawn from comrak's parse (kalem_core::markdown), with Org's reveal rule: the markers of an object (the * of emphasis, the ** of strong emphasis, the ~~ of strike-through, the backticks of code, the [, ](url) of a link) show only while the cursor is inside it, from its first character to its last; the # of a heading only while the cursor is on its line. Headings are drawn by level (an underlined heading's line of = or — dimmed), emphasis in italics, strong emphasis bold, code in the code style, links as links, footnote references as such; an image is drawn in place of ![alt](src) and a formula in place of $…$ or $$…$$ on one line away from the cursor (a $$ block over several lines stays source); a task list item's [ ] or [x] is a checkbox while the cursor is off its line. Block quote markers and thematic breaks are dimmed; code blocks, HTML blocks, tables and front matter are monospace, fences and front matter dimmed. A code block is drawn on the code background, its fences included; away from the cursor its opening fence shows only its language, as a label, and its closing fence nothing. A table away from the cursor is drawn as a grid: each column as wide as its widest cell and lined up as its delimiter cell says (:-- left, --: right, :-: centered), the bars as lines and the delimiter row as a rule, without changing the text; a table wider than the window has its widest columns narrowed and their cells' text wrapped inside them, as a web page lays out a table, in the graphical and the terminal editor. With the cursor in the table it shows its source, and Tab aligns the source itself. Away from the cursor YAML front matter (---) folds to its first line, as a property drawer does in Org, and opens when the cursor goes into it. The setting editor.show_source_markers (cursor, always, never) applies as in Org, and Toggle Source View shows the text as it is. After an edit only the top-level blocks around it are parsed again, with a block more on each side out to a blank line and the next block as a guard: if the guard reads differently, or a fence is left open, or two lists meet, or the document has link reference definitions or footnotes (which reach across it), the whole text is parsed. A random-edit test checks that the result is always that of a full parse. The new region's nodes take the old ones' place and those after it are moved where they lie, as are the line starts; the nodes a line shows are found from the top-level blocks when it is drawn. A file over 2 MiB is parsed in the background when it opens and drawn as source until then (about 1.5 s for 10 MB); after that a keystroke reparses only around it.
  • A corpus of real Markdown (tests/corpus/markdown, T2.7c.8): eleven READMEs of well-known projects (ripgrep, Rust, serde, tokio, bat, fd, fzf, React, VS Code, Mermaid, Kubernetes) and a vault of 88 notes linked by wiki links (Foam's documentation), each under its licence. Every file is drawn line by line with the cursor on each line and away, every shown character comes from its line, and it saves unedited byte for byte; both editors show the same words on every line (kalem-ui/tests/parity.rs); the terminal's screens of a Markdown document are snapshots (kalem-tui/tests/snapshots.rs).
  • The outline sidebar lists the headings.
  • Links open with Open Link (markdown.openLink, Command-click on macOS and Control-click elsewhere in the graphical editor, SPC o l in the Vim profile): a web or mail address in the system's application, a file relative to the document (percent-encoded names such as My%20Note.md decoded; a #heading after it opens the file at that heading), a #heading alone the heading of this document whose GitHub anchor it is (lower case, - for spaces, punctuation left out, repeated titles numbered -1, -2), and a wiki link as Obsidian and Logseq write them, [[Page]] or [[Page|title]]: Page.md in the document's folder, else the first file of that name, ignoring case, anywhere in the project, else a new Page.md beside the document. A wiki link shows its title, its brackets and target hidden away from the cursor. Typing [[ offers the project's Markdown files by name and completes the closing ]].
  • Copy as HTML and Copy as Rich Text copy the selection, or the whole document, as comrak renders it (raw HTML left out), as they do for Org.
  • A click on a task list item's checkbox, or Toggle Checkbox (Ctrl+Shift+C, markdown.toggleCheckbox) on its line, turns [ ] into [x] and back; one character changes.
  • The lines of a fenced code block that names its language are coloured in that language, each with the state of the block's lines before it (a string or comment over several lines is one).
  • In a table (GitHub's pipe tables), Tab aligns the table and goes to the next cell, past the last cell adding an empty row, and Shift+Tab goes back, as in an Org table (markdown.table.nextField, markdown.table.previousField, when inMarkdownTable); Align Table (markdown.table.align) only aligns. Aligning pads each column to its widest cell by display width, keeps the delimiter row's alignments (:---, :---:, ---:) with as many dashes as the column is wide, pads cells as their column is aligned, gives short rows empty cells, keeps escaped pipes \| inside their cell, and keeps the first line's indentation. Next Row (markdown.table.nextRow) aligns the table and goes to the same column of the row below, from the header to the first body row, past the last row adding an empty one, as Enter in an Org table does; it has no key, since Enter in a Markdown table stays a line break for typing a row as text, and binding it to Enter in keymap.json ("when": "editorMode = markdown && inMarkdownTable"=) makes Enter Org's.
  • The table keys of Org move a table's rows and columns: Alt+Up and Alt+Down move the body row (the header stays, and no row goes above the delimiter row), Alt+Left and Alt+Right the column with its alignment; Alt+Shift+Down inserts an empty row above, Alt+Shift+Up deletes the row, Alt+Shift+Right inserts an empty column to the left, Alt+Shift+Left deletes the column (markdown.table.moveRowUp and the others). Sort Rows by Column (markdown.table.sort, C-c ^ in the Emacs keymap) sorts the body rows by the cursor's column, as numbers when they all are, else as text ignoring case; reverse sorts descending. Outside a table these keys move lines as everywhere.
  • In a list item, Alt+Up and Alt+Down move the item with what it holds above the item before or below the one after (markdown.list.moveUp, markdown.list.moveDown); a numbered list is numbered again from the number it started at. Renumber List (markdown.list.renumber) puts the numbers in order after other edits, as markdown-mode's cleanup of list numbers does. Tab nests an item, with what it holds, under the item before it, its lines indented to that item's text (markdown.list.indent); Shift+Tab takes it out again (markdown.list.outdent). A list's first item has nothing to nest under and stays.
  • Pictures: dropping picture files on the document (graphical editor), pasting a copied picture or the paths of picture files (a terminal pastes a dropped file's path), or Insert Image (markdown.insert.image, which asks for the file) writes ![name](images/name.png) at the cursor. A picture from outside the document's folder is copied into images/ beside it (the setting markdown.assets_dir), under a free name; a pasted picture is saved there as pasted-DATE.png. Deleting a picture's link leaves its file (undo can bring the link back, and another document can use it); Remove Unused Images (file.removeUnusedImages, in Org documents too, for their org.assets_dir) moves to the Trash the pictures of that folder whose names no Org, Markdown, LaTeX or HTML file of the document's folder, nor the document as it is in the editor, mentions.
  • Edit Properties (markdown.frontMatter.edit) lists the front matter's fields as Obsidian's properties panel does; choosing one offers its value to change, and the list has Add a property… and a delete for each. YAML (---, key: value, lists as [a, b] or as - item lines) and TOML (+++, folded as YAML's is) are read; only the changed field's line is rewritten, quoted when YAML needs it, a list kept a list (its items separated by commas, an item with a comma in double quotes), and a document without front matter gets one.
  • Formulas: a table followed by lines <!-- TBLFM: … --> (as Obsidian's Advanced Tables writes them, and invisible elsewhere) is computed by F9 (Recalculate Table, markdown.table.recalculate) with Org's formula language and engine: $4=$2*$3, @>$4=vsum(@I..@II), Calc's functions. The delimiter row is Org's first rule, so column formulas leave the header alone; numbers come out as Emacs writes them (6. for a float).
  • The text of a source view is coloured by the Markdown syntax definition bundled with the highlighter (the Sublime Text Markdown syntax that syntect ships, language name "Markdown"), looked up as md. That syntax is a set of regular expressions, not a CommonMark parser: its colouring is an approximation and can disagree with CommonMark on edge cases (lazy continuation lines, nested emphasis, HTML blocks). Files over 4 MiB are coloured a window of lines at a time.
  • Tab indents: with a selection over lines, those lines one step; otherwise a tab or the spaces to the next step. The step is the one the file uses (tabs, or 2, 3, 4 or 8 spaces), else four spaces. Shift+Tab outdents.
  • Enter in a list item or a block quote continues it (markdown.newline, when inMarkdownList): the new line gets the same indentation, > markers and bullet, the next number (9. is followed by 10., the . or ) kept), and an empty [ ] after a task item; the text after the cursor moves to the new item. On a nested item that holds nothing but its markers, Enter takes it out a level, as Shift+Tab does (and as Org's Enter does); on a top-level item or quoted line that holds nothing but its markers, Enter removes them, which ends the list or quote, and leaves a blank line before the cursor, so that what is typed next is a paragraph of its own and not a lazy continuation of the item before (CommonMark reads a line right after an item's text as more of it). A tab after the bullet or number is kept as written, and a CR LF file gets CR LF. In a numbered list the items after the new one are numbered on from it. At the end of an opening fence (``` or ~~~, with or without a language) that nothing closes, Enter adds an empty line and the closing fence and puts the cursor between them. Over a selection the selection goes first, in the same step. Elsewhere, and in code blocks, Enter (edit.newline) starts a new line with the current line's indentation.
  • Toggle Comment (Ctrl+Alt+C) wraps each selected line in <!-- … -->, or unwraps lines that all have them.
  • The line commands (Duplicate Lines, Move Lines Up and Down, Join Lines, Sort Lines, Trim Trailing Whitespace), Go to Matching Bracket, Expand Selection, find and replace and the other commands for all files work as in plain text.
  • The export commands of the Export menu apply to Org documents only; they are not offered for a Markdown file.

Since these commands know nothing of Markdown, some change its meaning:

  • Join Lines removes each line break with the blanks around it and puts one space in its place. Two trailing spaces (a hard line break) go with it, and a heading, list item or quote line joined to the next line takes that line into itself (# Title and Text become # Title Text).
  • Sort Lines sorts whole lines by their text (code points). An item that continues on further lines, a nested list or a paragraph under an item is split apart; a numbered list is sorted by its numbers as text (10. before 2.), not by its items' words.
  • Trim Trailing Whitespace (the command) removes the two trailing spaces of hard line breaks. The setting editor.trim_trailing_whitespace does not apply to Markdown files, so saving keeps them.
  • Toggle Comment wraps lines in <!-- … -->, which inside a code block or an indented code line is text, not a comment.

A link in an Org document to a .md file opens it in Kalem. The macOS application bundle declares Markdown as a type it can open.

2.3. Converting Markdown to Org

Convert to Org (markdown.convertToOrg, in the File menu of a Markdown document) and kalem export FILE.md --to org convert without pandoc, from comrak's tree (kalem_core::markdown_org): headings become headlines by level, *em* /em/, **strong** *strong*, `code` ~code~ (==code== when it holds a tilde), ~~gone~~ +gone+, links [[url][text]] ([[url]] when the text is the address), images [[src]], lists with their bullets or numbers and task items with [ ] or [X], block quotes #+begin_quote, fenced code #+begin_src LANG (#+begin_example without a language, lines starting with * or #+ quoted with a comma as Org does), raw HTML blocks #+begin_export html and inline HTML @@html:…@@, tables with a rule under the header row, thematic breaks -----, footnotes [fn:NAME] with their definitions, $x$ \(x\) and $$x$$ \[x\], hard line breaks \\, wiki links [[Page|title]] as [[file:Page.md][title]], a heading inside a quote or a list as a bold line (a headline would end it), and the front matter's keys as #+KEY: lines (title, author, date become Org's). Text Org would read otherwise stays text, with a zero-width space as Org's manual advises: after the opening character of what would be markup (\*x\*, /x/ and =x= as text), and before a line that would be structure (\# x, a | that makes no table, \- x). The command writes NAME.org beside the file, never over an existing one, and opens it; the command line writes it, or to -o. The Markdown file does not change.

kalem export FILE.md --to html writes NAME.html, a page of comrak's HTML (raw HTML left out, as Copy as HTML has it) titled by the front matter's title (else the file's name); its formulas, $x$ and $$x$$, are written \(x\) and \[x\] for MathJax, loaded as Org's HTML export loads it. --body-only writes the HTML alone. Other formats are refused for a Markdown file: convert it to Org first.

Import as Org (the command file.import, and kalem import FILE.md on the command line) converts a Markdown file to Org through pandoc, when pandoc is installed. The file (.md, .markdown, .mdown, .mkd or .gfm) is read with pandoc's gfm reader, CommonMark with GitHub's extensions (tables, task lists, strike-through, footnotes, bare URLs as links) as Kalem's Markdown mode takes it, plus a YAML metadata block for the title (gfm+yaml_metadata_block); Pandoc's own Markdown dialect, which differs from CommonMark in places, is not used. The result is cleaned up, written as NAME.org beside the file (never over an existing one, unless --force is given on the command line), its pictures in NAME_assets, and opened. The Markdown file does not change. pandoc's conversion also brings the pictures beside the file.

3. Today: Markdown export from Org

Kalem writes Markdown from Org documents with two back-ends of its Rust port of Org's export engine (crates/org-export):

Back-end Command Command line Reference
md Export as Markdown (export.markdown), Export Subtree as Markdown kalem export FILE --to md Emacs's ox-md, byte for byte
gfm Export as GitHub Markdown (export.gfm) kalem export FILE --to gfm The ox-gfm package, byte for byte

Both write NAME.md (or where #+EXPORT_FILE_NAME says), honour the export options (#+OPTIONS, #+SETUPFILE, EXPORT_ properties, body only), and read Org: for a Markdown file kalem export refuses them and says so on standard error (it exports Markdown to HTML and to Org only, Converting Markdown to Org above). The output is the input's name with its extension replaced by .md, except when that is the input itself: README.md exports to README.md.md (Emacs refuses with "Output file cannot be the same as the input file").

3.1. The md back-end

The md back-end (org_export::Markdown, md.rs) is a port of ox-md.el. Like ox-md, it derives from the HTML back-end: what Markdown cannot express is written as HTML. Its output is identical to Emacs's on the 37 export cases of tests/export/expected, on all 287 Worg files of the extended corpus (crates/org-export/tests/corpus.rs, worg_md), and on the sample book of examples/book.

What it writes, element by element:

Org Markdown written
Headline at most H: deep and at most level 6 ATX heading: a blank line, # times the level, the TODO keyword, the priority as [#A] (with pri:t only), the title, the tags after five spaces as :a:b:, a blank line
A headline listed by a table of contents or the target of an id or CUSTOM_ID link The heading preceded by <a id="REF"></a> and a blank line
Headline below H: or deeper than 6 A list item (- or its number), its contents indented four spaces
Bold, italic **…**, *…*
Code, verbatim, inline source Backticks; double backticks when the text contains one, padded with spaces when it starts or ends with one
Underline, strike-through, subscript, superscript, entities and other objects As the HTML back-end writes them
Paragraph Its lines, blank lines inside removed; a leading # escaped
Plain list item - and three spaces, or N. padded to four columns; [X], [-], [ ] and a space for checkboxes; a description item's tag as **tag:** and a space; contents indented four spaces
Quote block Every line prefixed with > and a space
Example block, source block, fixed width Indented four spaces (no fence, no language); line numbers and coderefs as org-export-format-code-default writes them
Table, center block, special block, inline task HTML, as the HTML back-end writes it
Horizontal rule ---
Line break (\\) Two spaces and a line feed
LaTeX fragment (with tex:t) $…$ for \(…\), $$…$$ for \[…\], others as written
LaTeX environment (with tex:t) As written, with \label{REF} after its first line when it is named
Link with a description [description](path); a file link to X.org becomes X.md
Link without a description <path>
Link to a headline or target [title or number](#REF)
Image link without a description ![img](path), with the caption as title: ![img](path "caption")
Radio link <a href="#REF">…</a>
Property drawer Its lines indented four spaces (empty ones left out)
#+MARKDOWN: and #+MD: keywords, #+begin_export markdown or md Their text as it is
#+begin_export html, other keywords As the HTML back-end writes them
Export snippet @@md:…@@ or @@html:…@@ Its text; other back-ends' snippets are dropped
#+TOC: headlines [N] [local] A table of contents at that place
Footnotes A "Footnotes" heading at the end, each as <sup><a id="fn.N" href="#fnr.N">N</a></sup> text

The document's title is not written (ox-md's template has none). With toc:t (the default) or toc:N a "Table of Contents" heading and a nested list of [title](#REF) links come first; a document without headlines gets the heading over an empty list, as in Emacs. The heading style is always ATX and the top level always #: ox-md's org-md-headline-style (setext, mixed) and org-md-toplevel-hlevel are not options in Kalem. Elements are separated by one blank line, except inside items and for a first paragraph followed by an item's last sub-list. In plain text, ` * _ and \ are escaped with a backslash, as are a # at the start of a line and a ![; with ':t quotes become typographic HTML entities, with -:t special strings become their HTML entities, and with \n:t every line ends in two spaces. The headings Kalem adds ("Table of Contents", "Footnotes") are translated by #+LANGUAGE as the HTML back-end translates them.

3.2. The gfm back-end

The gfm back-end (org_export::Gfm, gfm.rs) is a port of the ox-gfm package (larstvei/ox-gfm, the one MELPA ships), which is not part of Org. Like it, it is the md back-end with these changes:

  • A paragraph is written on one line: its words joined by single spaces (so a line break \\ disappears), unless \n:t.
  • Source and example blocks are fenced: three backticks and the source block's language, the code, three backticks.
  • Strike-through becomes ~~…~~.
  • An Org table (not a table.el table) becomes a pipe table. Each cell is padded with spaces to its column's width (the longest cell of the column, counted in characters); a row is | a | b |. A rule is kept only as the table's second row, written |---- |--- | (dashes to the column's width, at least three, no alignment colons); other rules are dropped. A table of one row gets an empty header and a rule above it; a table of several rows without a rule gets none, which GFM reads as a header row without its delimiter row, that is, as a paragraph. Cells are not escaped: a | inside a cell (written \vert{}) becomes &vert;, as the HTML back-end writes the entity.
  • The table of contents (toc:t, the default, or toc:N) has no heading: a list of - [title](#REF) links, indented two spaces per level of the headline (its level as written, not relative), then a blank line.
  • The footnotes come under ## Footnotes (always level 2 and in English, not translated), each as <sup><a id="fn.N" class="footnum" href="#fnr.N">N</a></sup> text, separated by blank lines.
  • The whole body is trimmed of leading and trailing blank lines.
  • Export snippets @@gfm:…@@ and #+begin_export gfm are written as they are.
| Name | n |                | Name     | n  |
|------+---|                |-------- |--- |
| *a*  | 1 |      →         | **a**    | 1  |
| b\vert{}c | 22 |          | b&vert;c | 22 |

Its output is identical to that of ox-gfm (pinned at commit 4f774f1, run on Emacs's Org 9.7) on the 37 export cases of tests/export/expected (the .gfm files) and on ORG-NEWS: tools/fetch-ox-gfm.sh fetches the package, which, being GPL-3.0, is not in the repository, and tools/export-expected.sh uses it when it is there.

3.3. Known differences from CommonMark and GitHub

Both back-ends, like ox-md and ox-gfm, escape only what ox-md escapes in plain text (` * _ \ , a # at the start of a line, ![). A reader following CommonMark can therefore read some Org text as Markdown constructs it was not meant as:

  • a line of a paragraph that starts with > (a quote), -, + or 1. followed by a space (a list), or = = (four spaces: code), or that is --- or *** (a rule, or a setext heading underline);
  • raw HTML in the text (<b>, <!--), which passes through;
  • [x](y) or <https://…> written in the text, which becomes a link;
  • &amp;-style entity references, which are decoded.

On GitHub, the anchors ox-md writes (<a id"orgXXXXXXX"></a>= before a heading, the fn.N and fnr.N footnote anchors) are rewritten by GitHub's sanitizer to user-content-… ids; links to them still work through GitHub's page script, but not in other renderers that apply the same sanitizer. GFM's own footnote syntax ([^1]) and GitHub's heading slugs are not used, since ox-gfm does not use them. In a table with / column groups, a row has a | before a cell only where a group starts (the other cells are separated by a space), which GFM reads as fewer columns.

3.4. Markdown from LaTeX

A LaTeX document can be exported as Markdown through pandoc (latex.export.markdown, Export as Markdown (pandoc)). Pandoc's markdown writer produces it; Kalem passes the file and, when the document names one, its bibliography.

4. What remains of Markdown mode

The mode is in use. What is left of group 6 of docs/todo.md (the former 2.7c), by task:

  • The comrak fork (T2.7c.1a): each fix an editor needs, every node inside its parent and byte offsets beside line and column, offered upstream as a pull request; Kalem stays pinned to the fork until they land.
  • The mode contract (T2.7c.10): Markdown implements ModeSpec and passes its conformance check, but the editors still draw it through kalem_core::markdown's own view code. Drawing every mode from its tree is open, as is the plugin side of the contract (T3.1.9g).
  • Large files (T2.7c.6): a keystroke in a 10 MB file reparses in 10 ms at the median and 23 ms at the 99th percentile; most of it is moving the positions of the nodes after the edit. A representation with relative positions would take that off. A document with a link reference definition ([x]: url) or a footnote's ([^1]: …) outside code is parsed whole at each edit, since a definition reaches every reference (about 1.4 s at 10 MB); a footnote mark without a definition, or a definition in a code block, does not do so. Parsing only the edited blocks with the definitions is open.
  • A known-differences chapter of its own, following the skeleton the other formats use (T2.10.8); today the thirteen differences are listed under The oracle above.

The rule of the standard modes holds: Kalem writes nothing into a Markdown file that you did not type.