Markdown
Table of Contents
This chapter states what Kalem does with Markdown today and what remains. The two are kept apart: the sections under Today describe the code as it is; the last section, What remains of Markdown mode, lists the open work by its task.
In short: Markdown is one of Kalem's built-in formats. Kalem reads it
with comrak, the Rust port of GitHub's own parser, in Kalem's fork
(D19), and draws a Markdown file as it reads: markers hidden away from
the cursor, headings by level, emphasis, code, links, images, formulas,
task list checkboxes, tables and front matter. The file is edited as
text and never rewritten: Kalem writes nothing into a Markdown file that
you did not type, and never converts it in order to open it. Kalem also
writes Markdown as an export of Org documents, in two flavours, Emacs's
ox-md Markdown byte for byte and GitHub Flavored Markdown, and
converts a Markdown file to Org without pandoc when you ask (Convert to
Org, kalem export FILE.md --to org).
1. The oracle
The standard is CommonMark with the GitHub extensions (GFM). The oracle
named by the design is the CommonMark and GFM specification test suites,
every example of each specification a test, with cmark-gfm as the
reference implementation where the suites are silent, plus a byte-exact
round trip of untouched text on a corpus of real READMEs and note
vaults. The suites run as Kalem's own test (markdown::spec, T2.7c.11):
CI downloads CommonMark 0.31.2's and GFM 0.29's specifications, which
are CC BY-SA and not in the repository (KALEM_COMMONMARK_SPEC and
KALEM_GFM_SPEC name copies on disk; without them the test passes
without checking). Every example's HTML, as comrak renders it, is
compared with the specification's: all 652 CommonMark examples agree
with CommonMark's options, and all 24 extension examples with GFM's.
Every example's nodes lie inside the text and their parents, every line
is drawn with the cursor on it and away, an edit in the middle reparses
as a full parse does, and the view shows the text the specification's
HTML shows, in all 676 examples.
With every extension Kalem reads a file with, as the editor does, 639 of the 652 CommonMark examples agree. The thirteen others are the extensions' own cases, and the count only goes up:
- four of Autolinks: bare addresses and mail addresses are links, as GitHub's extension makes them;
- six of HTML blocks:
<script>,<style>and<textarea>are escaped, as GitHub's tag filter does; - one of Links and one of Images:
[[…]]is a wiki link; - one of Setext headings: a document starting with
---is front matter in Kalem, as in GitHub, Jekyll and Obsidian.
The Markdown Kalem writes has a different oracle, Org's exporter:
the md back-end is compared byte for byte with Emacs's ox-md (see
Markdown export from Org below).
2. Today: Markdown files in the editor
2.1. Which files
A file is in Markdown mode when its extension is md, markdown,
mdown, mkd or gfm (any letter case), when its first line has a
mode line naming markdown (-*- mode: markdown -*-, -*- markdown
-*-, or markdown-mode), or when the user chose Markdown with Set
Document Mode (view.setMode with mode: markdown), which is
remembered for the file under files.modes in the workspace settings.
The mode's name in when-clauses and scopes is markdown
(editorMode == markdown, text type markdown); the status bar says
"Markdown".
2.2. What the mode does
Markdown mode (DocumentMode::Markdown) draws the file as it reads and
edits it as plain text. Concretely:
- The file is read, decoded and saved as every text file is (the chapter Plain text and code): encoding, byte order mark and line endings kept, untouched bytes unchanged. Kalem never rewrites a Markdown file.
- Each line is drawn from comrak's parse (
kalem_core::markdown), with Org's reveal rule: the markers of an object (the*of emphasis, the**of strong emphasis, the~~of strike-through, the backticks of code, the[,](url)of a link) show only while the cursor is inside it, from its first character to its last; the#of a heading only while the cursor is on its line. Headings are drawn by level (an underlined heading's line of=or — dimmed), emphasis in italics, strong emphasis bold, code in the code style, links as links, footnote references as such; an image is drawn in place ofand a formula in place of$…$or$$…$$on one line away from the cursor (a$$block over several lines stays source); a task list item's[ ]or[x]is a checkbox while the cursor is off its line. Block quote markers and thematic breaks are dimmed; code blocks, HTML blocks, tables and front matter are monospace, fences and front matter dimmed. A code block is drawn on the code background, its fences included; away from the cursor its opening fence shows only its language, as a label, and its closing fence nothing. A table away from the cursor is drawn as a grid: each column as wide as its widest cell and lined up as its delimiter cell says (:--left,--:right,:-:centered), the bars as lines and the delimiter row as a rule, without changing the text; a table wider than the window has its widest columns narrowed and their cells' text wrapped inside them, as a web page lays out a table, in the graphical and the terminal editor. With the cursor in the table it shows its source, and Tab aligns the source itself. Away from the cursor YAML front matter (---) folds to its first line, as a property drawer does in Org, and opens when the cursor goes into it. The settingeditor.show_source_markers(cursor,always,never) applies as in Org, and Toggle Source View shows the text as it is. After an edit only the top-level blocks around it are parsed again, with a block more on each side out to a blank line and the next block as a guard: if the guard reads differently, or a fence is left open, or two lists meet, or the document has link reference definitions or footnotes (which reach across it), the whole text is parsed. A random-edit test checks that the result is always that of a full parse. The new region's nodes take the old ones' place and those after it are moved where they lie, as are the line starts; the nodes a line shows are found from the top-level blocks when it is drawn. A file over 2 MiB is parsed in the background when it opens and drawn as source until then (about 1.5 s for 10 MB); after that a keystroke reparses only around it. - A corpus of real Markdown (
tests/corpus/markdown, T2.7c.8): eleven READMEs of well-known projects (ripgrep, Rust, serde, tokio, bat, fd, fzf, React, VS Code, Mermaid, Kubernetes) and a vault of 88 notes linked by wiki links (Foam's documentation), each under its licence. Every file is drawn line by line with the cursor on each line and away, every shown character comes from its line, and it saves unedited byte for byte; both editors show the same words on every line (kalem-ui/tests/parity.rs); the terminal's screens of a Markdown document are snapshots (kalem-tui/tests/snapshots.rs). - The outline sidebar lists the headings.
- Links open with Open Link (
markdown.openLink, Command-click on macOS and Control-click elsewhere in the graphical editor,SPC o lin the Vim profile): a web or mail address in the system's application, a file relative to the document (percent-encoded names such asMy%20Note.mddecoded; a#headingafter it opens the file at that heading), a#headingalone the heading of this document whose GitHub anchor it is (lower case,-for spaces, punctuation left out, repeated titles numbered-1,-2), and a wiki link as Obsidian and Logseq write them,[[Page]]or[[Page|title]]:Page.mdin the document's folder, else the first file of that name, ignoring case, anywhere in the project, else a newPage.mdbeside the document. A wiki link shows its title, its brackets and target hidden away from the cursor. Typing[[offers the project's Markdown files by name and completes the closing]]. - Copy as HTML and Copy as Rich Text copy the selection, or the whole document, as comrak renders it (raw HTML left out), as they do for Org.
- A click on a task list item's checkbox, or Toggle Checkbox
(Ctrl+Shift+C,
markdown.toggleCheckbox) on its line, turns[ ]into[x]and back; one character changes. - The lines of a fenced code block that names its language are coloured in that language, each with the state of the block's lines before it (a string or comment over several lines is one).
- In a table (GitHub's pipe tables), Tab aligns the table and goes to
the next cell, past the last cell adding an empty row, and Shift+Tab
goes back, as in an Org table (
markdown.table.nextField,markdown.table.previousField, wheninMarkdownTable); Align Table (markdown.table.align) only aligns. Aligning pads each column to its widest cell by display width, keeps the delimiter row's alignments (:---,:---:,---:) with as many dashes as the column is wide, pads cells as their column is aligned, gives short rows empty cells, keeps escaped pipes\|inside their cell, and keeps the first line's indentation. Next Row (markdown.table.nextRow) aligns the table and goes to the same column of the row below, from the header to the first body row, past the last row adding an empty one, as Enter in an Org table does; it has no key, since Enter in a Markdown table stays a line break for typing a row as text, and binding it to Enter inkeymap.json("when": "editorMode =markdown && inMarkdownTable"=) makes Enter Org's. - The table keys of Org move a table's rows and columns: Alt+Up and
Alt+Down move the body row (the header stays, and no row goes above the
delimiter row), Alt+Left and Alt+Right the column with its alignment;
Alt+Shift+Down inserts an empty row above, Alt+Shift+Up deletes the
row, Alt+Shift+Right inserts an empty column to the left,
Alt+Shift+Left deletes the column (
markdown.table.moveRowUpand the others). Sort Rows by Column (markdown.table.sort,C-c ^in the Emacs keymap) sorts the body rows by the cursor's column, as numbers when they all are, else as text ignoring case;reversesorts descending. Outside a table these keys move lines as everywhere. - In a list item, Alt+Up and Alt+Down move the item with what it holds
above the item before or below the one after (
markdown.list.moveUp,markdown.list.moveDown); a numbered list is numbered again from the number it started at. Renumber List (markdown.list.renumber) puts the numbers in order after other edits, as markdown-mode's cleanup of list numbers does. Tab nests an item, with what it holds, under the item before it, its lines indented to that item's text (markdown.list.indent); Shift+Tab takes it out again (markdown.list.outdent). A list's first item has nothing to nest under and stays. - Pictures: dropping picture files on the document (graphical editor),
pasting a copied picture or the paths of picture files (a terminal
pastes a dropped file's path), or Insert Image
(
markdown.insert.image, which asks for the file) writesat the cursor. A picture from outside the document's folder is copied intoimages/beside it (the settingmarkdown.assets_dir), under a free name; a pasted picture is saved there aspasted-DATE.png. Deleting a picture's link leaves its file (undo can bring the link back, and another document can use it); Remove Unused Images (file.removeUnusedImages, in Org documents too, for theirorg.assets_dir) moves to the Trash the pictures of that folder whose names no Org, Markdown, LaTeX or HTML file of the document's folder, nor the document as it is in the editor, mentions. - Edit Properties (
markdown.frontMatter.edit) lists the front matter's fields as Obsidian's properties panel does; choosing one offers its value to change, and the list has Add a property… and a delete for each. YAML (---,key: value, lists as[a, b]or as- itemlines) and TOML (+++, folded as YAML's is) are read; only the changed field's line is rewritten, quoted when YAML needs it, a list kept a list (its items separated by commas, an item with a comma in double quotes), and a document without front matter gets one. - Formulas: a table followed by lines
<!-- TBLFM: … -->(as Obsidian's Advanced Tables writes them, and invisible elsewhere) is computed by F9 (Recalculate Table,markdown.table.recalculate) with Org's formula language and engine:$4=$2*$3,@>$4=vsum(@I..@II), Calc's functions. The delimiter row is Org's first rule, so column formulas leave the header alone; numbers come out as Emacs writes them (6.for a float). - The text of a source view is coloured by the Markdown syntax definition bundled with
the highlighter (the Sublime Text Markdown syntax that
syntectships, language name "Markdown"), looked up asmd. That syntax is a set of regular expressions, not a CommonMark parser: its colouring is an approximation and can disagree with CommonMark on edge cases (lazy continuation lines, nested emphasis, HTML blocks). Files over 4 MiB are coloured a window of lines at a time. - Tab indents: with a selection over lines, those lines one step; otherwise a tab or the spaces to the next step. The step is the one the file uses (tabs, or 2, 3, 4 or 8 spaces), else four spaces. Shift+Tab outdents.
- Enter in a list item or a block quote continues it
(
markdown.newline, wheninMarkdownList): the new line gets the same indentation,>markers and bullet, the next number (9.is followed by10., the.or)kept), and an empty[ ]after a task item; the text after the cursor moves to the new item. On a nested item that holds nothing but its markers, Enter takes it out a level, as Shift+Tab does (and as Org's Enter does); on a top-level item or quoted line that holds nothing but its markers, Enter removes them, which ends the list or quote, and leaves a blank line before the cursor, so that what is typed next is a paragraph of its own and not a lazy continuation of the item before (CommonMark reads a line right after an item's text as more of it). A tab after the bullet or number is kept as written, and a CR LF file gets CR LF. In a numbered list the items after the new one are numbered on from it. At the end of an opening fence (```or~~~, with or without a language) that nothing closes, Enter adds an empty line and the closing fence and puts the cursor between them. Over a selection the selection goes first, in the same step. Elsewhere, and in code blocks, Enter (edit.newline) starts a new line with the current line's indentation. - Toggle Comment (Ctrl+Alt+C) wraps each selected line in
<!--…-->, or unwraps lines that all have them. - The line commands (Duplicate Lines, Move Lines Up and Down, Join Lines, Sort Lines, Trim Trailing Whitespace), Go to Matching Bracket, Expand Selection, find and replace and the other commands for all files work as in plain text.
- The export commands of the Export menu apply to Org documents only; they are not offered for a Markdown file.
Since these commands know nothing of Markdown, some change its meaning:
- Join Lines removes each line break with the blanks around it and
puts one space in its place. Two trailing spaces (a hard line break)
go with it, and a heading, list item or quote line joined to the next
line takes that line into itself (
# TitleandTextbecome# Title Text). - Sort Lines sorts whole lines by their text (code points). An item
that continues on further lines, a nested list or a paragraph under
an item is split apart; a numbered list is sorted by its numbers as
text (
10.before2.), not by its items' words. - Trim Trailing Whitespace (the command) removes the two trailing
spaces of hard line breaks. The setting
editor.trim_trailing_whitespacedoes not apply to Markdown files, so saving keeps them. - Toggle Comment wraps lines in
<!--…-->, which inside a code block or an indented code line is text, not a comment.
A link in an Org document to a .md file opens it in Kalem. The macOS
application bundle declares Markdown as a type it can open.
2.3. Converting Markdown to Org
Convert to Org (markdown.convertToOrg, in the File menu of a
Markdown document) and kalem export FILE.md --to org convert without
pandoc, from comrak's tree (kalem_core::markdown_org): headings become
headlines by level, *em* /em/, **strong** *strong*, `code`
~code~ (==code== when it holds a tilde), ~~gone~~ +gone+, links
[[url][text]] ([[url]] when the text is the address), images
[[src]], lists with their bullets or numbers and task items with
[ ] or [X], block quotes #+begin_quote, fenced code
#+begin_src LANG (#+begin_example without a language, lines
starting with * or #+ quoted with a comma as Org does), raw HTML
blocks #+begin_export html and inline HTML @@html:…@@, tables with
a rule under the header row, thematic breaks -----, footnotes
[fn:NAME] with their definitions, $x$ \(x\) and $$x$$ \[x\],
hard line breaks \\, wiki links [[Page|title]] as
[[file:Page.md][title]], a heading inside a quote or a list as a bold
line (a headline would end it), and the front matter's keys as #+KEY:
lines (title, author, date become Org's). Text Org would read
otherwise stays text, with a zero-width space as Org's manual advises:
after the opening character of what would be markup (\*x\*, /x/
and =x= as text), and before a line that would be structure
(\# x, a | that makes no table, \- x). The command writes
NAME.org beside the file, never over an existing one, and opens it;
the command line writes it, or to -o. The Markdown file does not
change.
kalem export FILE.md --to html writes NAME.html, a page of comrak's
HTML (raw HTML left out, as Copy as HTML has it) titled by the front
matter's title (else the file's name); its formulas, $x$ and
$$x$$, are written \(x\) and \[x\] for MathJax, loaded as Org's
HTML export loads it. --body-only writes the HTML alone. Other
formats are refused for a Markdown file: convert it to Org first.
Import as Org (the command file.import, and kalem import FILE.md
on the command line) converts a Markdown file to Org through pandoc,
when pandoc is installed. The file (.md, .markdown, .mdown,
.mkd or .gfm) is read with pandoc's gfm reader, CommonMark with
GitHub's extensions (tables, task lists, strike-through, footnotes,
bare URLs as links) as Kalem's Markdown mode takes it, plus a YAML
metadata block for the title (gfm+yaml_metadata_block); Pandoc's own
Markdown dialect, which differs from CommonMark in places, is not used.
The result is cleaned up, written as NAME.org beside the file (never
over an existing one, unless --force is given on the command line),
its pictures in NAME_assets, and opened. The Markdown file does not
change. pandoc's conversion also brings the pictures beside the file.
3. Today: Markdown export from Org
Kalem writes Markdown from Org documents with two back-ends of its
Rust port of Org's export engine (crates/org-export):
| Back-end | Command | Command line | Reference |
|---|---|---|---|
md |
Export as Markdown (export.markdown), Export Subtree as Markdown |
kalem export FILE --to md |
Emacs's ox-md, byte for byte |
gfm |
Export as GitHub Markdown (export.gfm) |
kalem export FILE --to gfm |
The ox-gfm package, byte for byte |
Both write NAME.md (or where #+EXPORT_FILE_NAME says), honour the
export options (#+OPTIONS, #+SETUPFILE, EXPORT_ properties, body
only), and read Org: for a Markdown file kalem export refuses
them and says so on standard error (it exports Markdown to HTML and
to Org only, Converting Markdown to Org above). The output is the input's
name with its extension replaced by .md, except when that is the
input itself: README.md exports to README.md.md (Emacs refuses with
"Output file cannot be the same as the input file").
3.1. The md back-end
The md back-end (org_export::Markdown, md.rs) is a port of
ox-md.el. Like ox-md, it derives from the HTML back-end: what
Markdown cannot express is written as HTML. Its output is identical to
Emacs's on the 37 export cases of tests/export/expected, on all 287
Worg files of the extended corpus (crates/org-export/tests/corpus.rs,
worg_md), and on the sample book of examples/book.
What it writes, element by element:
| Org | Markdown written |
|---|---|
Headline at most H: deep and at most level 6 |
ATX heading: a blank line, # times the level, the TODO keyword, the priority as [#A] (with pri:t only), the title, the tags after five spaces as :a:b:, a blank line |
A headline listed by a table of contents or the target of an id or CUSTOM_ID link |
The heading preceded by <a id="REF"></a> and a blank line |
Headline below H: or deeper than 6 |
A list item (- or its number), its contents indented four spaces |
| Bold, italic | **…**, *…* |
| Code, verbatim, inline source | Backticks; double backticks when the text contains one, padded with spaces when it starts or ends with one |
| Underline, strike-through, subscript, superscript, entities and other objects | As the HTML back-end writes them |
| Paragraph | Its lines, blank lines inside removed; a leading # escaped |
| Plain list item | - and three spaces, or N. padded to four columns; [X], [-], [ ] and a space for checkboxes; a description item's tag as **tag:** and a space; contents indented four spaces |
| Quote block | Every line prefixed with > and a space |
| Example block, source block, fixed width | Indented four spaces (no fence, no language); line numbers and coderefs as org-export-format-code-default writes them |
| Table, center block, special block, inline task | HTML, as the HTML back-end writes it |
| Horizontal rule | --- |
Line break (\\) |
Two spaces and a line feed |
LaTeX fragment (with tex:t) |
$…$ for \(…\), $$…$$ for \[…\], others as written |
LaTeX environment (with tex:t) |
As written, with \label{REF} after its first line when it is named |
| Link with a description | [description](path); a file link to X.org becomes X.md |
| Link without a description | <path> |
| Link to a headline or target | [title or number](#REF) |
| Image link without a description | , with the caption as title:  |
| Radio link | <a href="#REF">…</a> |
| Property drawer | Its lines indented four spaces (empty ones left out) |
#+MARKDOWN: and #+MD: keywords, #+begin_export markdown or md |
Their text as it is |
#+begin_export html, other keywords |
As the HTML back-end writes them |
Export snippet @@md:…@@ or @@html:…@@ |
Its text; other back-ends' snippets are dropped |
#+TOC: headlines [N] [local] |
A table of contents at that place |
| Footnotes | A "Footnotes" heading at the end, each as <sup><a id="fn.N" href="#fnr.N">N</a></sup> text |
The document's title is not written (ox-md's template has none). With
toc:t (the default) or toc:N a "Table of Contents" heading and a
nested list of [title](#REF) links come first; a document without
headlines gets the heading over an empty list, as in Emacs. The
heading style is always ATX and the top level always #: ox-md's
org-md-headline-style (setext, mixed) and
org-md-toplevel-hlevel are not options in Kalem. Elements are separated by one blank
line, except inside items and for a first paragraph followed by an
item's last sub-list. In plain text, ` * _ and \ are escaped
with a backslash, as are a # at the start of a line and a links, indented two spaces per level of the headline (its level as written, not relative), then a blank line. - The footnotes come under
## Footnotes(always level 2 and in English, not translated), each as<sup><a id="fn.N" class="footnum" href="#fnr.N">N</a></sup> text, separated by blank lines. - The whole body is trimmed of leading and trailing blank lines.
- Export snippets
@@gfm:…@@and#+begin_export gfmare written as they are.
| Name | n | | Name | n |
|------+---| |-------- |--- |
| *a* | 1 | → | **a** | 1 |
| b\vert{}c | 22 | | b|c | 22 |
Its output is identical to that of ox-gfm (pinned at commit
4f774f1, run on Emacs's Org 9.7) on the 37 export cases of
tests/export/expected (the .gfm files) and on ORG-NEWS:
tools/fetch-ox-gfm.sh fetches the package, which, being GPL-3.0, is
not in the repository, and tools/export-expected.sh uses it when it
is there.
3.3. Known differences from CommonMark and GitHub
Both back-ends, like ox-md and ox-gfm, escape only what ox-md
escapes in plain text (` * _ \ , a # at the start of a line,
or<https://…>written in the text, which becomes a link;&-style entity references, which are decoded.
On GitHub, the anchors ox-md writes (<a id"orgXXXXXXX"></a>= before
a heading, the fn.N and fnr.N footnote anchors) are rewritten by
GitHub's sanitizer to user-content-… ids; links to them still work
through GitHub's page script, but not in other renderers that apply the
same sanitizer. GFM's own footnote syntax ([^1]) and GitHub's heading
slugs are not used, since ox-gfm does not use them. In a table with
/ column groups, a row has a | before a cell only where a group
starts (the other cells are separated by a space), which GFM reads as
fewer columns.
3.4. Markdown from LaTeX
A LaTeX document can be exported as Markdown through pandoc
(latex.export.markdown, Export as Markdown (pandoc)). Pandoc's
markdown writer produces it; Kalem passes the file and, when the
document names one, its bibliography.
4. What remains of Markdown mode
The mode is in use. What is left of group 6 of docs/todo.md (the former
2.7c), by task:
- The comrak fork (T2.7c.1a): each fix an editor needs, every node inside its parent and byte offsets beside line and column, offered upstream as a pull request; Kalem stays pinned to the fork until they land.
- The mode contract (T2.7c.10): Markdown implements
ModeSpecand passes its conformance check, but the editors still draw it throughkalem_core::markdown's own view code. Drawing every mode from its tree is open, as is the plugin side of the contract (T3.1.9g). - Large files (T2.7c.6): a keystroke in a 10 MB file reparses in
10 ms at the median and 23 ms at the 99th percentile; most of it is
moving the positions of the nodes after the edit. A representation
with relative positions would take that off. A document with a link
reference definition (
[x]: url) or a footnote's ([^1]: …) outside code is parsed whole at each edit, since a definition reaches every reference (about 1.4 s at 10 MB); a footnote mark without a definition, or a definition in a code block, does not do so. Parsing only the edited blocks with the definitions is open. - A known-differences chapter of its own, following the skeleton the other formats use (T2.10.8); today the thirteen differences are listed under The oracle above.
The rule of the standard modes holds: Kalem writes nothing into a Markdown file that you did not type.