Introduction

Topiary aims to be a uniform formatter for simple languages, as part of the Tree-sitter ecosystem. It is named after the art of clipping or trimming trees into fantastic shapes.

Topiary is designed for formatter authors and formatter users. Authors can create a formatter for a language without having to write their own formatting engine or even their own parser. Users benefit from uniform code style and, potentially, the convenience of using a single formatter tool, across multiple languages over their codebases, each with comparable styles applied.

Motivation

The style in which code is written has, historically, been mostly left to personal choice. Of course, this is subjective by definition and has led to many wasted hours reviewing formatting choices, rather than the code itself. Prescribed style guides were an early solution to this, spawning tools that lint a developer's formatting and ultimately leading to automatic formatters. The latter were popularised by gofmt, whose developers had the insight that "good enough" uniform formatting, imposed on a codebase, largely resolves these problems.

Topiary follows this trend by aspiring to be a "universal formatter engine", which allows developers to not only automatically format their codebases with a uniform style, but to define that style for new languages using a simple DSL. This allows for the fast development of formatters, providing a Tree-sitter grammar is defined for that language.

Design principles

Topiary has been created with the following goals in mind:

• Use Tree-sitter for parsing, to avoid writing yet another grammar for a formatter.

• Expect idempotency. That is, formatting of already-formatted code doesn't change anything.

• For bundled formatting styles to meet the following constraints:

  • Be compatible with attested formatting styles used for that language in the wild.

  • Be faithful to the author's intent: if code has been written such that it spans multiple lines, that decision is preserved.

  • Minimise changes between commits such that diffs focus mainly on the code that's changed, rather than superficial artefacts. That is, a change on one line won't influence others, while the formatting won't force you to make later, cosmetic changes when you modify your code.

  • Be well-tested and robust, so that the formatter can be trusted in large projects.

• For end users -- i.e., not formatting style authors -- the formatter should:

  • Prescribe a formatting style that, while customisable, is uniform and "good enough" for their codebase.

  • Run efficiently.

  • Afford simple integration with other developer tools, such as editors and language servers.

Installation

Topiary can be installed in a few different ways. For more information on these, see the respective chapter:

• Package managers
• Building from source
• Using Nix

Dependencies

The Topiary CLI will build Tree-sitter grammars on demand, optionally fetching them from a Git remote first. For this to work, your environment will need a C/C++ toolchain (i.e., compiler and linker) available. If this is available, it should "just work"; otherwise, refer to the underlying mechanism's documentation for configuration advice.

Alternatively, the Tree-sitter CLI can be used to build Tree-sitter grammars outside of Topiary, which can then be loaded through configuration from your local filesystem.

Package managers

Topiary has been packaged for some package managers. However, note that packaged versions may lag behind the current release.

Cargo (Rust package manager)

The Topiary CLI, amongst other things, is available on crates.io:

cargo install topiary-cli

OPAM (OCaml Package Manager)

Topiary is available through OPAM for the purposes of formatting OCaml code:

opam install topiary

Development of this package can be found on GitHub at tweag/topiary-opam.

Nix (nixpkgs)

Topiary exists within nixpkgs and can therefore be installed in whichever way you prefer. For example:

NixOS (configuration.nix)

environment.systemPackages = with pkgs; [
  topiary
];

Home Manager (home.nix)

home.packages = with pkgs; [
  topiary
];

Nix install

# Using flakes:
nix profile install nixpkgs#topiary

# Or, without flakes:
# (Note: Use nixos.topiary on NixOS)
nix-env -iA nixpkgs.topiary

nix-shell

To temporarily add Topiary to your path, use:

# Using flakes:
nix shell nixpkgs#topiary

# Or, without flakes:
nix-shell -p topiary

Arch Linux (AUR)

Topiary is available on the Arch user repository:

yay -S topiary

Building from source

Assuming you have the Topiary repository cloned locally, you can build Topiary in two ways.

Using Nix

To build Topiary using Nix simply call nix build (assuming you have flakes and nix-command enabled).

Alternatively, the Topiary flake also has a Topiary package that doesn't fetch and build the grammars but instead takes them from nixpkgs pinned by the flake. To build this version use nix build .#topiary-cli-nix.

Using Cargo

Building Topiary using the standard Rust build tools requires not only those tools, but also some external dependencies. Our flake provides a devshell that sets all required environment variables and fetches all dependencies. To enter this devshell use nix develop or setup direnv.

If you are not a Nix user, you will need to set up all dependencies and required environment variables. You must ensure at least pkg-config and openssl are available.

From there use cargo build, as usual.

Using Nix

Topiary provides a flake with several outputs. The main being topiary-cli, which builds a version of the CLI that doesn't come with any Tree-sitter grammars. However, this version cannot be used in Nix. For that purpose the flake also provides the topiary-cli-nix package. This package utilises the Tree-sitter grammars from the nixpkgs flake input.

Note that some Tree-sitter grammars haven't been added to nixpkgs yet, so respective languages are disabled in these cases.

Git hooks

Topiary integrates seamlessly with pre-commit-hooks.nix: add Topiary as input to your flake and, in pre-commit-hooks.nix's setup, use:

pre-commit-check = nix-pre-commit-hooks.run {
  hooks = {
    nixfmt.enable = true; ## keep your normal hooks
    ...
    ## Add the following:
    topiary = topiary.lib.${system}.pre-commit-hook;
  };
};

Tree-sitter and its queries

From the Tree-sitter documentation:

Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited.

Any non-trivial formatter needs to work with the code's syntax tree. Topiary uses Tree-sitter to this end, which precludes the need to write a new parser each time formatting for a new language is needed.

A number of Tree-sitter grammars already exist and these languages are, therefore, potential targets for Topiary. Even in the case when a Tree-sitter grammar doesn't exist for some target language, writing one is significantly easier than writing a parser from scratch and benefits from other tooling in the Tree-sitter ecosystem.

Tree-sitter queries

Tree-sitter exposes a query interface, which allows you to express syntactic patterns in a simple DSL that Tree-sitter will match against your source code. You can think of it a little bit like "regular expressions for syntax trees". While regular expressions match linear strings, and thus look like a bit like a string, Tree-sitter queries match against syntax trees, so they look a bit like a tree. They are expressed using S-expressions, similar to Lisp, using the node names that are defined in the grammar.

For example, say you wanted to find for loops with an if statement as the first child. The query for this might look something like:

(for_loop
  .
  (if_statement)
)

The syntax for Tree-sitter queries is described in the Tree-sitter documentation,¹ which we won't go into here. However, there is one very important concept in Tree-sitter queries, as far as Topiary is concerned: what Tree-sitter calls "capture names".

When you write a query and you want to capture a specific node in that query, rather than the entire subtree, you can annotate the node with a "capture name". These are represented with an @ character, followed by an identifier. (To stretch our regular expression analogy, these would be similar to named groups.)

In our above example, say we actually only care about if statements that appear as the first child of for loops; we don't care about the rest of the subtree. Then we could add a capture name as follows:

(for_loop
  .
  (if_statement) @my_important_node
)

It's this mechanism that Topiary relies on to perform formatting. That is, queries are used to identify syntactic constructions of interest, then specific capture names -- for which Topiary defines particular semantics -- apply respective formatting to those nodes within that and every matching subtree.

Say, for example, we always want an if statement that appears as the first child of a for loop to be on a new line. For this, we could use the @prepend_hardline capture name:

(for_loop
  .
  (if_statement) @prepend_hardline
)

This is the essence of Topiary.

Topiary language query files

A language query file, usually given then .scm extension, is a collection of such queries with appropriate formatting capture names. Taken in aggregate, when applied to source code files by Topiary, they define a formatting style for the language in question.

command
  command_name
    word
  word
  word

; Naive query (incorrect; neglects arguments)
; Results in: command arg1arg2
(command_name) @append_space


; Improved queries (incorrect; too specific)
; Results in: command arg1 arg2
; Failure mode if the first argument is a string: command "arg1"arg2
(command
  (command_name) @append_space
)

(command
  (word) @append_space
)


; Correct query (if we ignore everything besides commands)
; Results in: command arg1 arg2
; Handles string arguments: command "arg1" arg2
(command
  (_) @append_space
  .
  (_)
)

Usage

The Topiary CLI uses a number of subcommands to delineate functionality. These can be listed with topiary --help; each subcommand then has its own, dedicated help text.

CLI app for Topiary, the universal code formatter.

Usage: topiary [OPTIONS] <COMMAND>

Commands:
  format      Format inputs
  visualise   Visualise the input's Tree-sitter parse tree
  config      Print the current configuration
  prefetch    Prefetch all languages in the configuration
  coverage    Checks how much of the tree-sitter query is used
  completion  Generate shell completion script
  help        Print this message or the help of the given subcommand(s)

Options:
  -C, --configuration <CONFIGURATION>  Configuration file [env: TOPIARY_CONFIG_FILE]
  -M, --merge-configuration            Enable merging for configuration files
  -v, --verbose...                     Logging verbosity (increased per occurrence)
  -h, --help                           Print help
  -V, --version                        Print version

See the respective chapter for usage documentation on each subcommand:

• format
• visualise
• config
• completion
• prefetch
• coverage

Example

Once built, Topiary can be run like this:

echo '{"foo":"bar"}' | topiary format --language json

topiary can also be built and run from source via either Cargo or Nix, if you have those installed:

echo '{"foo":"bar"}' | cargo run -- format --language json
echo '{"foo":"bar"}' | nix run . -- format --language json

This will output the following formatted code:

{ "foo": "bar" }

Format

Format inputs

Usage: topiary format [OPTIONS] <--language <LANGUAGE>|FILES>

Arguments:
  [FILES]...
          Input files and directories (omit to read from stdin)

          Language detection and query selection is automatic, mapped from file extensions defined
          in the Topiary configuration.

Options:
  -t, --tolerate-parsing-errors
          Consume as much as possible in the presence of parsing errors

  -s, --skip-idempotence
          Do not check that formatting twice gives the same output

  -l, --language <LANGUAGE>
          Topiary language identifier (when formatting stdin)

  -q, --query <QUERY>
          Topiary query file override (when formatting stdin)

  -L, --follow-symlinks
          Follow symlinks (when formatting files)

  -C, --configuration <CONFIGURATION>
          Configuration file

          [env: TOPIARY_CONFIG_FILE]

  -M, --merge-configuration
          Enable merging for configuration files

  -v, --verbose...
          Logging verbosity (increased per occurrence)

  -h, --help
          Print help (see a summary with '-h')

Note
fmt is a recognised alias of the format subcommand.

When formatting inputs from disk, language selection is detected from the input files' extensions. To format standard input, you must specify the --language and, optionally, --query arguments, omitting any input files.

Valid language identifiers, as specified with --language, are defined as part of your Topiary configuration. See the configuration chapter for more details.

# This won't work
topiary format <(some_command)

# Do this instead
some_command | topiary format --language LANGUAGE

Visualise

topiary visualise converts the input's Tree-sitter parse tree to a graph representation in the selected format. By default, Topiary outputs a DOT file, which can be rendered using a visualisation tool such as the Graphviz suite. For example, using Graphviz's dot: topiary visualise input.ocaml | dot -T png -o output.png.

Visualise the input's Tree-sitter parse tree

Visualise generates a graph representation of the parse tree that can be rendered by external
visualisation tools, such as Graphviz. By default, the output is in the DOT format.

Usage: topiary visualise [OPTIONS] <--language <LANGUAGE>|FILE>

Arguments:
  [FILE]
          Input file (omit to read from stdin)

          Language detection and query selection is automatic, mapped from file extensions defined
          in the Topiary configuration.

Options:
  -f, --format <FORMAT>
          Visualisation format

          [default: dot]

          Possible values:
          - dot:  GraphViz DOT serialisation
          - json: JSON serialisation

  -l, --language <LANGUAGE>
          Topiary language identifier (when formatting stdin)

  -q, --query <QUERY>
          Topiary query file override (when formatting stdin)

  -C, --configuration <CONFIGURATION>
          Configuration file

          [env: TOPIARY_CONFIG_FILE]

  -M, --merge-configuration
          Enable merging for configuration files

  -v, --verbose...
          Logging verbosity (increased per occurrence)

  -h, --help
          Print help (see a summary with '-h')

Note
vis, visualize and view are recognised aliases of the visualise subcommand.

When visualising inputs from disk, language selection is detected from the input file's extension. To visualise standard input, you must specify the --language and, optionally, --query arguments, omitting the input file. The visualisation output is written to standard out.

Valid language identifiers, as specified with --language, are defined as part of your Topiary configuration. See the configuration chapter for more details.

Configuration

Print the current configuration

Usage: topiary config [OPTIONS]

Options:
  -C, --configuration <CONFIGURATION>  Configuration file [env: TOPIARY_CONFIG_FILE]
  -M, --merge-configuration            Enable merging for configuration files
  -v, --verbose...                     Logging verbosity (increased per occurrence)
  -h, --help                           Print help

Note
cfg is a recognised alias of the config subcommand.

Shell completion generation

Shell completion scripts for Topiary can be generated with the completion subcommand. The output of which can be sourced into your shell session or profile, as required.

Generate shell completion script

Usage: topiary completion [OPTIONS] [SHELL]

Arguments:
  [SHELL]  Shell (omit to detect from the environment) [possible values: bash, elvish, fish,
           powershell, zsh]

Options:
  -C, --configuration <CONFIGURATION>  Configuration file [env: TOPIARY_CONFIG_FILE]
  -M, --merge-configuration            Enable merging for configuration files
  -v, --verbose...                     Logging verbosity (increased per occurrence)
  -h, --help                           Print help

For example, in Bash:

source <(topiary completion)

Grammar prefetching

Topiary dynamically downloads, builds, and loads the Tree-sitter grammars. In order to ensure offline availability or speed up start up time, the grammars can be prefetched and compiled.

Prefetch all languages in the configuration

Usage: topiary prefetch [OPTIONS]

Options:
  -f, --force                          Re-fetch existing grammars if they already exist
  -C, --configuration <CONFIGURATION>  Configuration file [env: TOPIARY_CONFIG_FILE]
  -M, --merge-configuration            Enable merging for configuration files
  -v, --verbose...                     Logging verbosity (increased per occurrence)
  -h, --help                           Print help

Query coverage

This subcommand checks how much of the language query file is used to process the input. Specifically, it checks the percentage of queries in the query file that match the given input, and prints the queries that don't match anything.

Checks how much of the tree-sitter query is used

Usage: topiary coverage [OPTIONS] <--language <LANGUAGE>|FILE>

Arguments:
  [FILE]
          Input file (omit to read from stdin)

          Language detection and query selection is automatic, mapped from file extensions defined
          in the Topiary configuration.

Options:
  -l, --language <LANGUAGE>
          Topiary language identifier (when formatting stdin)

  -q, --query <QUERY>
          Topiary query file override (when formatting stdin)

  -C, --configuration <CONFIGURATION>
          Configuration file

          [env: TOPIARY_CONFIG_FILE]

  -M, --merge-configuration
          Enable merging for configuration files

  -v, --verbose...
          Logging verbosity (increased per occurrence)

  -h, --help
          Print help (see a summary with '-h')

The coverage subcommand will exit with error code 1 if the coverage is less than 100%.

Configuration

Topiary is configured using languages.ncl files. The .ncl extension relates to Nickel, a configuration language created by Tweag. There are up to four sources where Topiary checks for such a file.

Configuration sources

At build time the languages.ncl in the root of the Topiary repository is embedded into Topiary. This file is parsed at runtime. The purpose of this languages.ncl file is to provide sane defaults for users of Topiary (both the library and the CLI binary).

The next two are read by the Topiary binary at runtime and allow the user to configure Topiary to their needs. The first is intended to be user specific, and can thus be found in the configuration directory of the OS:

This file is not automatically created by Topiary.

The next source is intended to be a project-specific settings file for Topiary. When running Topiary in some directory, it will ascend the file tree until it finds a .topiary directory. It will then read any languages.ncl file present in that directory.

Finally, an explicit configuration file may be specified using the -C/--configuration command line argument (or the TOPIARY_CONFIG_FILE environment variable). This is intended for driving Topiary under very specific use-cases.

To summarise, Topiary consumes configuration from these sources in the following order (highest to lowest):

1. The explicit configuration file specified as a CLI argument.
2. The project-specific Topiary configuration.
3. The user configuration file in the OS's configuration directory.
4. The built-in configuration file.

Configuration merging

By default, Topiary only considers the configuration file with the highest priority. However, if the -M/--merge-configuration option is provided to the CLI, then all available configurations are merged together, as per the Nickel specification.

In which case, if one of the sources listed above attempts to define a language configuration already present in the built-in configuration, or if two configuration files have conflicting values, then Topiary will display a Nickel error.

To understand why, one can read the Nickel documentation on merging. However, the short answer is that a priority must be defined. The built-in configuration has everything defined with priority 0. Any priority above that will replace any other priority. For example, to override the entire Bash configuration, use the following Nickel file.

{
  languages = {
    # Alternatively, use `priority 1`, rather than `force`
    bash | force = {
      extensions = [ "sh" ],
      indent = "    ",
    },
  },
}

To override only the indentation, use the following Nickel file:

{
  languages = {
    bash = {
      indent | force = "    ",
    },
  },
}

Configuration options

The configuration file contains a record of languages. That is, it defines language identifiers and configures them accordingly.

For instance, the configuration for Nickel is defined as such:

nickel = {
  extensions = ["ncl"],
},

The language identifier is used by Topiary to associate the language entry with the respective query file, as well exposing it to the user through the --language CLI argument. This value should be written in lowercase.

File extensions

The list of extensions is mandatory for every language, but does not necessarily need to exist in every configuration file. It is sufficient if, for every language, there is a single configuration file that defines the list of extensions for that language.

Indentation

The optional field, indent, exists to define the indentation method for that language. Topiary defaults to two spaces " " if it cannot find the indent field in any configuration file for a specific language.

Specifying the grammar

Topiary fetches and builds the grammar for you, or a grammar can be provided by some other method. To have Topiary fetch the grammar for you, specify the grammar.source.git attribute of a language:

nickel = {
  extensions = ["ncl"],
  grammar.source.git = {
    git = "https://github.com/nickel-lang/tree-sitter-nickel",
    rev = "43433d8477b24cd13acaac20a66deda49b7e2547",
  },
},

To specify a prebuilt grammar, specify the grammar.source.path attribute, which must point to a compiled grammar file on your file system:

nickel = {
  extensions = ["ncl"],
  grammar.source.path = "/path/to/compiled/grammar/file.so",
},

Note
If you want to link to a grammar file that has already been compiled by Topiary itself, those look like ~/.cache/topiary/<LANGUAGE>/<GIT_HASH>.so (or the equivalent for your platform).

For usage in Nix, a prefetchLanguages.nix file provides utilities allowing to transform a Topiary configuration into one where languages have been pre-fetched and pre-compiled in Nix derivations. The only caveat is that, for each Git source, the configuration must contain a nixHash for that source. For instance:

nickel = {
  extensions = ["ncl"],
  grammar.source.git = {
    git = "https://github.com/nickel-lang/tree-sitter-nickel",
    rev = "43433d8477b24cd13acaac20a66deda49b7e2547",
    nixHash = "sha256-9Ei0uy+eGK9oiH7y2KIhB1E88SRzGnZinqECT3kYTVE=",
  },
},

The simplest way to obtain the hash is to use nix-prefetch-git (and look for the hash field in its output). The second simplest way is to compile, which will show something like:

evaluation warning: Language `nickel`: no nixHash provided - using dummy value
error: hash mismatch in fixed-output derivation '/nix/store/jgny7ll7plh7rfdnvdpgcb82kd51aiyx-tree-sitter-nickel-43433d8.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-9Ei0uy+eGK9oiH7y2KIhB1E88SRzGnZinqECT3kYTVE=
error: 1 dependencies of derivation '/nix/store/0q20rk8l4g0n5fzr0w45agxx0j9qy65v-nickel-grammar-43433d8477b24cd13acaac20a66deda49b7e2547.drv' failed to build
error: 1 dependencies of derivation '/nix/store/s5phxykjyzqay7gc33hc6f8kw4ndba25-languages-prefetched.json.drv' failed to build
error: 1 dependencies of derivation '/nix/store/5w15p3b3xfw5nd6mxz58ln09v10kvf8v-languages-prefetched.ncl.drv' failed to build
error: 1 dependencies of derivation '/nix/store/7zzyha67jw09kc37valp28bp5h6i7dka-topiary-0.6.0.drv' failed to build

Runtime dialogue

Environment variables

Topiary needs to find language query files (*.scm) to function properly. By default, Topiary looks for a languages directory in the current working directory.

This won't work if you are running Topiary from a directory other than its repository. In order to use Topiary without this restriction, you must set the environment variable TOPIARY_LANGUAGE_DIR to point to the directory where Topiary's language query files are located.

By default, you should set it to <local path of the topiary repository>/topiary-queries/queries, for example:

export TOPIARY_LANGUAGE_DIR=/home/me/tools/topiary/topiary-queries/queries
topiary format ./projects/helloworld/hello.ml

TOPIARY_LANGUAGE_DIR can alternatively be set at build time. Topiary will pick the correspond path up and embed it into the topiary binary. In that case, you don't have to worry about making TOPIARY_LANGUAGE_DIR available at runtime any more. When TOPIARY_LANGUAGE_DIR has been set at build time and is set at runtime as well, the runtime value takes precedence.

See the contributor's guide for details on setting up a development environment.

Logging

By default, the Topiary CLI will only output error messages. You can increase the logging verbosity with a respective number of -v/--verbose flags:

Exit codes

The Topiary process will exit with a zero exit code upon successful formatting. Otherwise, the following exit codes are defined:

Negative results with error code 1 only happen when Topiary is called with the coverage sub-command, if the input does not cover 100% of the query.

When given multiple inputs, Topiary will do its best to process them all, even in the presence of errors. Should any errors occur, Topiary will return a non-zero exit code. For more details on the nature of these errors, run Topiary at the warn logging level (with -v).

Web playground

The Topiary web playground is a browser-based tool for experimenting with Topiary. It provides three editor panes:

1. Query: This editor pane lets you define the Topiary formatting queries to format the given language.

2. Input: This editor pane allows you to set the input code in the given language.

3. Output: This editor pane (which is read only) shows the result of Topiary formatting (i.e., the defined queries run against the input).

The given language is set by a drop-down at the top of the window. Changing this will set the appropriate Tree-sitter grammar and populate the "Query" and "Input" panes with the default formatting queries for that language, as shipped with Topiary, and the sample testing input for that language; the "Output" pane will then be generated appropriately.

By default, the web playground will format the input on-the-fly (i.e., on changes to either the "Query" or "Input" panes). This option can be disabled with the appropriate checkbox and the "Format" button can be used as an alternative. Otherwise, options are provided that mimic the --skip-idempotence and --tolerate-parsing-errors flags to the Topiary CLI (see topiary format).

Terminal-based playground

Nix users may also find the playground script to be helpful in aiding the interactive development of query files. When run in a terminal, inside the devshell defined in the Nix flake, it will format the given source input with the requested query file, updating the output on any inotify event against those files.

Usage: playground LANGUAGE [QUERY_FILE] [INPUT_SOURCE]

LANGUAGE can be one of the supported languages (e.g., "ocaml", "rust",
etc.). The packaged formatting queries for this language can be
overridden by specifying a QUERY_FILE.

The INPUT_SOURCE is optional. If not specified, it defaults to trying
to find the bundled integration test input file for the given language.

The script itself is very basic -- that cannot be overstated! -- but it provides a fast enough feedback loop to be useful. For example, the playground can be run in a terminal emulator/multiplexer pane, with your editor of choice open in another.

Adding a new language

This chapter illustrates how to add a supported language to Topiary, provided it already has a Tree-sitter grammar.

We will use C as the running example. The following steps are enough to bootstrap the formatting of a new language.

Register the grammar in topiary-config/languages.ncl:

    clang = {
      extensions = ["c", "h"],
      grammar.source.git = {
        git = "https://github.com/tree-sitter/tree-sitter-c.git",
        rev = "6c7f459ddc0bcf78b615d3a3f4e8fed87b8b3b1b",
      },
    },

Create the query file

touch topiary-queries/queries/clang.scm

Testing

You can now check that Topiary is able to "format" your new language with:

$ echo 'void main();' | cargo run -- format -s --language clang
voidmain();

At this point, there are no formatting queries, so no formatting is applied and we get mangled output. This is why the -s (--skip-idempotence) flag is set, as Topiary will complain when formatting doesn't reach a fixed point (which this output is likely to result in).

Add the new language to the test suite

Create input/expected files

Topiary's I/O tester will format input files and check them against an expected output. For the time being, let's stick with the mangled output, so we can get the tests to run and pass.

echo 'void main ();' > topiary-cli/tests/samples/input/clang.c
echo 'voidmain();' > topiary-cli/tests/samples/expected/clang.c

Add Cargo feature flags

Each language is gated behind a feature flag. We'll use the clang feature flag for C -- to match the language name in the configuration -- which needs to be added in the appropriate places.

In topiary-cli/Cargo.toml

experimental = [
  "clang",
]

clang = ["topiary-config/clang", "topiary-queries/clang"]

In topiary-config/Cargo.toml

clang = []

all = [
  "clang",
]

In topiary-queries/Cargo.toml

clang = []

Add tests in topiary-cli/tests/sample-tester.rs

To register the I/O and coverage tests for the new language, we need to add it to the test suite.

In fn get_file_extension

You will need to add a mapping from the language (feature name) to the file extension under test:

#![allow(unused)]
fn main() {
fn get_file_extension(language: &str) -> &str {
    match language {

        [...]

        "clang" => "c",

        [...]

    }
}
}

In mod test_fmt

Then you'll need to add the language to the lang_test! macro calls in the test_fmt module, respectively:

#![allow(unused)]
fn main() {
    lang_test!(

        [...]

        "clang",

        [...]

        fmt_input
    );
}

In mod test_coverage

Likewise in the test_coverage module:

#![allow(unused)]
fn main() {
    lang_test!(

        [...]

        "clang",

        [...]

        coverage_input
    );
}

Testing

You should be able to successfully run the new tests with:

cargo test --no-default-features -F clang -p topiary-cli --test sample-tester

Include the query file in Topiary at compile time

In topiary-queries/src/lib.rs

#![allow(unused)]
fn main() {
/// Returns the Topiary-compatible query file for C.
#[cfg(feature = "clang")]
pub fn clang() -> &'static str {
    include_str!("../queries/clang.scm")
}
}

In topiary-cli/src/io.rs

#![allow(unused)]
fn main() {
fn to_query<T>(name: T) -> CLIResult<QuerySource>

[...]

        #[cfg(feature = "clang")]
        "clang" => Ok(topiary_queries::clang().into()),
}

This will allow your query file to by considered as the default fallback query, when no other file can be found at runtime for your language.

Iterate

Once the above steps have been completed, Topiary will be able to use the C Tree-sitter grammar and the formatting queries in clang.scm to format C code.

You can now iterate on the formatting queries and the respective input and expected sample files to build your formatter, using the I/O and coverage tests to guide the process. Also see the following chapter on query development for more information.

Suggested query development workflow

In order to work productively on query files, the following is one suggested way to work:

1. If you're working on a new language, first follow the steps in the previous chapter.

2. Say you are working on formatting mylanguage code, then from the previous step (or otherwise), there should be two files that drive the test suite:

   • topiary-cli/tests/samples/input/mylanguage.code
   • topiary-cli/tests/samples/expected/mylanguage.code

   These respectively define the input, which will be formatted by Topiary when under test, and the expected output, which will be compared against the formatted input. (Note that the .code extension here is arbitrary, for illustrative purposes, but an appropriate extension is required.)

   Add a snippet of code to each of these files that exhibit the formatting you wish to implement as Tree-sitter queries. These snippets can be identical, but it would be a better test if the input version was intentionally misformatted.

   If code already exists in these files, please ensure that the new snippet is both syntactically valid, in the context of the other code, and inserted at the same relative position in both files.

3. Run:

   cargo test \
     --no-default-features \
     -F mylanguage \
     -p topiary-cli \
     input_output_tester \
     -- --nocapture
   

   Provided it works, it should output a lot of log messages. Copy that output to a text editor. You are particularly interested in the CST output that starts with a line like this: CST node: {Node compilation_unit (0, 0) - (5942, 0)} - Named: true.

Note
As an alternative to using the debugging output, the visualise subcommand exists to output the Tree-sitter syntax tree in a variety of formats.

4. The test run will output all the differences between the actual output and the expected output, e.g. missing spaces between tokens. Pick a difference you would like to fix, and find the line number and column in the input file.

5. In the CST debug or visualisation output, find the nodes in this region, such as the following:

   [DEBUG atom_collection] CST node:   {Node constructed_type (39, 15) - (39, 42)} - Named: true
   [DEBUG atom_collection] CST node:     {Node type_constructor_path (39, 15) - (39, 35)} - Named: true
   [DEBUG atom_collection] CST node:       {Node type_constructor (39, 15) - (39, 35)} - Named: true
   [DEBUG atom_collection] CST node:     {Node type_constructor_path (39, 36) - (39, 42)} - Named: true
   [DEBUG atom_collection] CST node:       {Node type_constructor (39, 36) - (39, 42)} - Named: true
   

6. This may indicate that you would like spaces after all type_constructor_path nodes:

   (type_constructor_path) @append_space
   

   Or, more likely, you just want spaces between pairs of them:

   (
     (type_constructor_path) @append_space
     .
     (type_constructor_path)
   )
   

   Or maybe you want spaces between all children of constructed_type:

   (constructed_type
     (_) @append_space
     .
     (_)
   )
   

Note
Using the #query_name! predicate will help identify which query does what.

7. Run cargo test again, to see if the output has improved, then return to step 4.

Writing a formatter has never been so easy: a Topiary tutorial

Yann Hamdaoui has written a step-by-step guide on how to write a new formatter for your own language, starting from zero.

Part 1

Part 1 introduces Topiary and motivates its use to create a formatter for a toy language, Yolo. Yann first guides you through the creation of a Tree-sitter grammar for Yolo, then plugs it into Topiary to write its formatting queries.

Contributing to Topiary

The Topiary team is greatly appreciative of all contributions and will endeavour to give each the attention it deserves. Issues, pull requests and discussions are all welcome. If you have any immediate questions, or just want to hang out, feel free to say "Hi!" on our Discord channel.

For language formatter authors

The most effective way of contributing to Topiary is by developing formatting queries for a language you are interested or invested in that is not currently supported by Topiary.

The other guides in this section outline the steps needed to bootstrap this process:

• Adding a new language
• Suggested query development workflow

More thorough, end-to-end tutorials are also available:

• Writing a formatter has never been so easy: a Topiary tutorial

The section below, for developers, may also be useful to this end.

Maturity policy

As described in Language support, formatting queries for languages come in two levels of maturity: supported and experimental. Cargo feature flags are used to distinguish these.

Formatting queries from external contributors are also subject to these levels. However, the Topiary team will not necessarily be familiar with the language in question and therefore will not be able to accurately assess the maturity of contributed formatting queries.

In the (current) absence of automated tools that can quantify grammar coverage of a set of queries, we leave it to contributors to make this judgement honestly. This can be done by asking yourself the following question:

Would I ask a colleague to format their production code with this?

If the answer is "no", because, say, not enough of the language's grammar is covered, then these queries fall under "experimental" support. Even if the answer is "maybe", for whatever reason, err on the side of "experimental"; bearing in mind that, once merged, the Topiary team will take a best effort approach to fixing any post-contribution issues, but won't actively maintain these queries.

For developers

Nix devshell

A Nix devshell is available, which includes all development dependencies, for contributing to Topiary. Enter this shell with:

nix develop

Performance profiling

You can check performance before or after changes by running cargo bench.

If you have flamegraph installed, you can also generate a performance flamegraph with, for example:

CARGO_PROFILE_RELEASE_DEBUG=true \
cargo flamegraph -p topiary-cli -- \
  format --language ocaml \
  < topiary-cli/tests/samples/input/ocaml.ml \
  > /dev/null

This will produce a flamegraph.svg plot.

Code coverage

Code coverage metrics can be generated via LLVM profiling data generated by the build. These can be created by setting appropriate environment variables to cargo test:

CARGO_INCREMENTAL=0 \
RUSTFLAGS='-Cinstrument-coverage' \
LLVM_PROFILE_FILE='cargo-test-%p-%m.profraw' \
cargo test

This will build and run the test suite and output cargo-test-*-*.profraw files in the working directory. (Outside of the Nix devshell, you may need binutils installed.)

These files can be used by grcov to render a variety of output reports. For example, the following renders HTML output in target/coverage/html:

grcov \
  --branch \
  --output-type html \
  --source-dir topiary-cli/src \
  --binary-path target/debug/deps \
  --output-path target/coverage/html \
  .

Website and web playground

Website

If you have Deno installed, you can start a local web server like this:

deno run -A local-web-server.ts

The website should then be running on http://localhost:8080.

Web playground WASM assets

In order to build or update the web playground, you can run the following within the Nix devshell:

update-wasm-app

Similarly, to update the Tree-sitter grammar WASM binaries, again within the Nix devshell, you can run:

update-wasm-grammars

Alternatively, if you have git, tree-sitter and emcc (Emscripten) in your PATH, you can run the bin/update-wasm-grammars.sh script directly.

To use Docker instead, the legacy approach can still be used (using JSON as an example):

1. Make sure you have Docker running and that you are member of the docker group, so you can run it without being root.

2. npm install tree-sitter-cli, or via some other method.

3. npm install tree-sitter-json or Git clone the grammar repository.

   • If you used NPM, tree-sitter-json will be fetched under node_modules/tree-sitter-json.

   • If you used Git, it will be wherever you cloned the repository (most likely tree-sitter-json).

   Whichever of these options you pick, we will use GRAMMAR_PATH as a stand-in for the directory where grammar.js can be found.

4. Run npx tree-sitter build-wasm GRAMMAR_PATH. If you get a Docker permission error, you may need to add yourself to the docker group.

5. mv tree-sitter-json.wasm web-playground/public/scripts

Note
Some grammar repositories are slightly different because they can contain multiple grammars or grammars under an unconventional path; OCaml, for example. In such cases, step 4 (above) should be changed such that GRAMMAR_PATH points to the directory containing the appropriate grammar.js file.

Web playground frontend

The playground frontend is a small React app. You can run a development server with the following:

cd web-playground
npm install
npm run dev

If you want to build the playground so it works with the full website running with Deno, as above, you can now just do npm run build.

Language support

Topiary ships with formatting support for a number of languages. The formatting styles for these languages come in two levels of maturity:

• Supported languages are actively maintained by the Topiary team.

• Experimental languages do not cover a significant proportion of the target grammar, may contain formatting bugs and could even be deprecated. You should not use these to format production code.

We also ship formatting styles for languages from a number of contributors. The Topiary team does not actively maintain these and -- while not necessarily -- where indicated, they should also be considered experimental.

Supported

These formatting styles cover their target language and fulfil Topiary's stated design goals. They are exposed, in Topiary, through the --language command line flag, or language detection (based on file extension).

• Bash
• JSON
• Nickel
• OCaml (both implementations and interfaces)
• OCamllex
• TOML
• Tree Sitter Queries

Contributed

These languages' formatting styles have been generously provided by external contributors. They are built in, by default -- unless marked as experimental -- so are exposed in the same way as supported languages.

• CSS, by Eric Lavigne
• OpenSCAD, by Mikhail Katychev
• SDML, by Simon Johnston

Experimental

These languages' formatting styles -- from either the Topiary team or external contributors -- are subject to change and/or not yet considered production-ready. They are not built by default and are gated behind a feature flag (either experimental, for all of them, or by their individual name). Once included, they can be accessed in Topiary in the usual way.

• Rust

The formatting pipeline

Query matching

As discussed in Tree-sitter and its queries, the essence of Topiary is to apply the formatting queries against an input. That is, each Tree-sitter query will be applied to the input's syntax tree, taking into account the relevant capture names against all matching subtrees. The result of this is effectively serialised into a list of atomic units -- formatting directives, defined by the matching capture names, and leaf node content (see also @leaf) -- along with the necessary metadata to drive the process.

Atom processing

The list of atoms from the first step are then processed into a cleaned-up, canonical form. Specifically:

• Processes scopes;
• Processes deletions;
• Processes case modifications;
• Merges runs of whitespace into a single atom -- or removes them entirely, in the presence of an antispace -- and sorts some remaining, adjacent atoms (e.g., hardlines always before spaces, etc.).

Note
In the code, this step is referred to as "post-processing"; as in "post-query matching-processing", rather than a final step.

Pretty printing

The pretty printer goes through the processed atom collection and renders each atom into a stream of text output. For example, an "indentation start" atom will increase the indent level; that is, all atoms immediately following a hardline will now be prefixed with the appropriate indent string, until the respective "indentation end" atom is reached.

Whitespace trimming

The output from pretty printing can leave trailing horizontal whitespace (at the ends of lines) and vertical whitespace (at the end of the file). This final step clears that up.

Idempotence checking

The Topiary CLI performs "idempotence checking" by default. That is, it checks that formatting (i.e., per the pipeline as described above) an already-formatted input makes no further changes.

Anecdotally, this incurs a negligible performance penalty to Topiary: its formatting time is dominated by start-up overhead (e.g., parsing the query files). However, the check can be disabled; this is often useful while developing or debugging formatting queries.

Note
See the usage documentation for topiary format for details on how this is exposed. The web playground also provides this option.

Capture names

As long as there is a Tree-sitter grammar defined for a language, Tree-sitter can parse it and provide a concrete syntax tree (CST). Tree-sitter will also allow us to run queries against this tree. We can make use of that to define how a language should be formatted, by annotating nodes with "capture names" that define formatting directives.

This chapter assumes you are already familiar with the Tree-sitter query language and documents all the capture names recognised by Topiary to drive formatting.

Example

Note
This example is derived from the Topiary announcement blog post; see the post for additional detail.

(
  [
    (infix_operator)
    "if"
    ":"
  ] @append_space
  .
  (_)
)

This will match any node that the grammar has identified as an infix_operator, or the anonymous nodes containing if or : tokens, immediately followed by any named node (represented by the (_) wildcard pattern). The query matches on subtrees of the same shape, where the annotated node within it will be "captured" with the name @append_space. Topiary runs through all matches and captures, and when we process any capture called @append_space, we append a space after the annotated node.

Before rendering the output, Topiary does some further processing, such as squashing consecutive spaces and newlines, trimming extraneous whitespace, and ordering indentation and newline instructions consistently. This means that you can, for example, prepend and append spaces to if and true, and Topiary will still output if true with just one space between the words.

General advice

This chapter contains general advice on writing Tree-sitter queries and also some specific, important Topiary semantics.

@leaf

Some nodes should not have their contents formatted at all; the classic example being string literals. The @leaf capture will mark such nodes as leaves -- even if they admit their own structure, by virtue of the grammar -- and leave them unformatted.

Example

; Don't format strings or comments
[
  (string)
  (comment)
] @leaf

@do_nothing

If any of the captures in a query match are @do_nothing, then the entire match will be ignored. This is useful for cancelling a formatting query based on context.

Example

; Put a semicolon delimiter after field declarations, unless they already have
; one, in which case we do nothing.
(
  (field_declaration) @append_delimiter
  .
  ";"* @do_nothing

  (#delimiter! ";")
)

#query_name!

When the logging verbosity is set to -vv or higher (see runtime dialogue), Topiary outputs information about which queries are matched, for instance:

[2024-10-08T15:48:13Z INFO  topiary_core::tree_sitter] Processing match: LocalQueryMatch { pattern_index: 17, captures: [ {Node "," (1,3) - (1,4)} ] } at location (286,1)

Here, pattern_index: 17 means that the 17th (0-based) pattern in the query file has matched. Counting patterns in the query file -- not to mention the potential for off-by-one errors -- is not a great developer experience!

As such, the optional predicate #query_name!, taking a string argument, can be added to any query. It will modify the log line to display its argument, to aid debugging.

Example

Considering the log line above, and let us assume that the query at location (286,1) is:

(
  "," @append_space
  .
  (_)
)

If we add a query_name predicate:

(
  "," @append_space
  .
  (_)

  (#query_name! "comma spacing")
)

Then the log line will become:

[2024-10-08T15:48:13Z INFO  topiary_core::tree_sitter] Processing match of query "comma spacing": LocalQueryMatch { pattern_index: 17, captures: [ {Node "," (1,3) - (1,4)} ] } at location (286,1)

Tree-sitter predicates

Tree-sitter supports a number of predicates by default, which allow for fine-tuning queries. These are discussed in the Tree-sitter documentation and outlined here:

• #eq?: Checks a direct match against a capture or string.
• #match?: Checks a match against a regular expression.
• #any-of?: Checks a match against a list of strings.
• Prefixing not- negates any of the above predicates.

Note
Topiary does not allow arbitrary capture names; just those it defines for formatting. The Tree-sitter predicates expect a capture name and, as such, this can make using them with Topiary a little unwieldy (see issue #824).

Query and capture precedence

Formatting is not necessarily invariant over the order of queries. For example, queries that add delimiters or remove nodes can have a different effect on the formatted output depending on the order in which they appear in the query file.

Consider, say, the following two queries for the Bash grammar:

; Query A: Append semicolon
(
  (word) @append_delimiter
  .
  ";"? @do_nothing

  (#delimiter! ";")
)

; Query B: Surround with quotes
(
  "\""? @do_nothing
  .
  (word) @prepend_delimiter @append_delimiter
  .
  "\""? @do_nothing

  (#delimiter! "\"")
)

In the order presented above (A, then B), then the input foo will be formatted as:

"foo;"

In the opposite order (B, then A), Topiary will however produce the following output:

"foo";

A similar consideration exists for capture names. That is, while most captures do not meaningfully affect one another, there are three notable exceptions:

1. @do_nothing (see above) will cancel all other captures in a matched query. This takes the highest priority.

2. @delete (see insertion and deletion) will delete any matched node, providing the matching query is not cancelled.

3. @leaf (see above) will suppress formatting within that node, even if it admits some internal structure. However, leaf nodes are still subject to deletion.

Note
While not in the same league as the above, also note that antispaces will cancel out all inserted spaces (see horizontal spacing).

Captures are always postfix

Note that a capture is put after the node it is associated with. If you want to put a space in front of a node, for example, you do so like this:

(infix_operator) @prepend_space

This, on the other hand, will not work:

@append_space (infix_operator)

A note on anchors

The behaviour of "anchors" can be counter-intuitive. Consider, for instance, the following query:

(
  (list_entry) @append_space
  .
)

One might assume that this query only matches the final element in the list but this is not true. Since we did not explicitly march a parent node, the engine will match on every list_entry. After all, when looking only at the nodes in the query, the list_entry is indeed the last node.

To resolve this issue, match explicitly on the parent node:

(list
  (list_entry) @append_space
  .
)

Or even implicitly:

(_
  (list_entry) @append_space
  .
)

if this; then that; fi

(if_statement
  (_)  ; will match "this"
  .
  (_)  ; will match "that"
)

(list
  (element) @append_space
  .
  (element)
)

Horizontal spacing

This chapter discusses horizontal spacing explicitly. However, horizontal spacing can also be introduced with softlines (see vertical spacing) and indentation (see indentation).

@append_space / @prepend_space

The matched nodes will have a space appended (or, respectively, prepended) to them. Note that this is the same as @append_delimiter / @prepend_delimiter, with a space as the delimiter (see insertion and deletion).

Example

[
  (infix_operator)
  "if"
  ":"
] @append_space

@append_antispace / @prepend_antispace

It is often the case that tokens need to be juxtaposed with spaces, except in a few isolated contexts. Rather than writing complicated rules that enumerate every exception, an "antispace" can be inserted with @append_antispace / @prepend_antispace; this will destroy all horizontal whitespace (besides any added through indentation) from that node, including those added by other formatting rules.

Example

[
  ","
  ";"
  ":"
  "."
] @prepend_antispace

Indentation

Indentation is a special form of horizontal spacing, where "blocks" are defined using start and end markers. Each block, which can be nested, sets an increasing indentation level which, when formatted in the output, will be prefixed with an appropriate number of indentation strings (defined in the language configuration).

@append_indent_start / @prepend_indent_start

The matched nodes will trigger indentation before (or, respectively, after) them. This will only apply to the lines following, until an indentation end is signalled. If indentation is started and ended on the same line, nothing will happen. This is useful, because we get the correct behaviour whether a node is formatted as single-line or multi-line. It is important that all indentation starts and ends are balanced.

Note
If indentation is not balanced, the formatting will (probably) not fail, but a warning will be logged.

Example

; Start an indented block after these
[
  "begin"
  "else"
  "then"
  "{"
] @append_indent_start

@append_indent_end / @prepend_indent_end

The matched nodes will trigger that indentation ends before (or, respectively, after) them.

Example

; End the indented block before these
[
  "end"
  "}"
] @prepend_indent_end

; End the indented block after these
[
  (else_clause)
  (then_clause)
] @append_indent_end

@multi_line_indent_all

To be used on comments, or other leaf nodes, to indicate that we should indent all its lines, not just the first.

Example

(comment) @multi_line_indent_all

@single_line_no_indent

The matched node will be printed alone, on a single line, regardless of any indentation level. That is, this capture temporarily suspends normal indentation for the node that is matched.

Example

; line number directives must be alone on their line, and can't be indented
(line_number_directive) @single_line_no_indent

Vertical spacing

Hardlines and softlines

Topiary makes a distinction between "hardlines" and "softlines". Hardlines are inserted regardless of the context; whereas softlines are more like points where line breaks can be inserted, based on context.

In particular, Topiary goes through the CST nodes and detects all that span more than one line. This is interpreted as an indication from the programmer who wrote the input that the node in question should be formatted as multi-line; while any other nodes will be formatted as single-line. Whenever a query match has inserted a softline, it will be expanded to a line break if the node is multi-line (otherwise it will be formatted depending on the capture name used). See:

• @append_hardline / @prepend_hardline
• @append_empty_softline / @prepend_empty_softline
• @append_spaced_softline / @prepend_spaced_softline
• @append_input_softline / @prepend_input_softline

Note
The single- and multi-lined context can also be used to drive arbitrary queries, not necessarily related to vertical spacing; see below.

Understanding the different line break captures

"Input softlines" are rendered as line breaks whenever the targeted node follows/precedes (for append/prepend, respectively) a line break in the input. Otherwise, they are rendered as spaces.

Example

Consider the following JSON, which has been hand-formatted to exhibit every context under which the different line break capture names operate:

{
  "single-line": [1, 2, 3, 4],
  "multi-line": [
    1, 2,
    3
    , 4
  ]
}

We'll apply a simplified set of JSON formatting queries that:

1. Opens (and closes) an indented block for objects;
2. Each key-value pair gets its own line, with the value split onto a second;
3. Applies the different line break capture name on array delimiters.

That is, iterating over each @LINEBREAK type, we apply the following:

(object . "{" @append_hardline @append_indent_start)
(object "}" @prepend_hardline @prepend_indent_end .)
(object (pair) @prepend_hardline)
(pair . _ ":" @append_hardline)

(array "," @LINEBREAK)

The first four formatting queries are just for clarity's sake. The last query is what's important; the results of which are demonstrated below:

@append_hardline

{
  "single-line":
  [1,
  2,
  3,
  4],
  "multi-line":
  [1,
  2,
  3,
  4]
}

A line break is added after each delimiter.

@prepend_hardline

{
  "single-line":
  [1
  ,2
  ,3
  ,4],
  "multi-line":
  [1
  ,2
  ,3
  ,4]
}

A line break is added before each delimiter.

@append_empty_softline

{
  "single-line":
  [1,2,3,4],
  "multi-line":
  [1,
  2,
  3,
  4]
}

In the single-line context, all whitespace is consumed around delimiters; in the multi-line context, a line break is added after each delimiter.

@prepend_empty_softline

{
  "single-line":
  [1,2,3,4],
  "multi-line":
  [1
  ,2
  ,3
  ,4]
}

In the single-line context, all whitespace is consumed around delimiters; in the multi-line context, a line break is added before each delimiter.

@append_spaced_softline

{
  "single-line":
  [1, 2, 3, 4],
  "multi-line":
  [1,
  2,
  3,
  4]
}

In the single-line context, a space is added after each delimiter; in the multi-line context, a line break is added after each delimiter.

@prepend_spaced_softline

{
  "single-line":
  [1 ,2 ,3 ,4],
  "multi-line":
  [1
  ,2
  ,3
  ,4]
}

In the single-line context, a space is added before each delimiter; in the multi-line context, a line break is added before each delimiter.

@append_input_softline

{
  "single-line":
  [1, 2, 3, 4],
  "multi-line":
  [1, 2,
  3, 4]
}

In the single-line context, a space is added after each delimiter; in the multi-line context, a line break is added after the delimiters that had a line break after them in the original input.

@prepend_input_softline

{
  "single-line":
  [1 ,2 ,3 ,4],
  "multi-line":
  [1 ,2 ,3
  ,4]
}

In the single-line context, a space is added before each delimiter; in the multi-line context, a line break is added before the delimiters that had a line break before them in the original input.

Testing context with predicates

Sometimes, similarly to what happens with softlines, we want a query to match only if the context is single-line, or multi-line. Topiary has several predicates that achieve this result.

#single_line_only! / #multi_line_only!

These predicates allow the query to trigger only if the matched nodes are in a single-line (or, respectively, multi-line) context.

Note
There are scoped equivalents to these predicates; see scopes for details.

Example

; Allow (and enforce) the optional "|" before the first match case
; in OCaml if and only if the context is multi-line
(
  "with"
  .
  "|" @delete
  .
  (match_case)
  (#single_line_only!)
)

(
  "with"
  .
  "|"? @do_nothing
  .
  (match_case) @prepend_delimiter
  (#delimiter! "| ")
  (#multi_line_only!)
)

@allow_blank_line_before

The matched nodes will be allowed to have a blank line before them, if specified in the input. For any other nodes, blank lines will be removed.

Example

; Allow comments and type definitions to have a blank line above them
[
  (comment)
  (type_definition)
] @allow_blank_line_before

@append_hardline / @prepend_hardline

The matched nodes will have a line break appended (or, respectively, prepended) to them.

Example

; Consecutive definitions must be separated by line breaks
(
  (value_definition) @append_hardline
  .
  (value_definition)
)

@append_empty_softline / @prepend_empty_softline

The matched nodes will have an empty softline appended (or, respectively, prepended) to them. This will be expanded to a line break for multi-line nodes and to nothing for single-line nodes.

Example

; Put an empty softline before dots, so that in multi-line constructs we start
; new lines for each dot.
(_
  "." @prepend_empty_softline
)

@append_spaced_softline / @prepend_spaced_softline

The matched nodes will have a spaced softline appended (or, respectively, prepended) to them. This will be expanded to a line break for multi-line nodes and to a space for single-line nodes.

Example

; Append spaced softlines, unless there is a comment following.
(
  [
    "begin"
    "else"
    "then"
    "->"
    "{"
    ";"
  ] @append_spaced_softline
  .
  (comment)* @do_nothing
)

@append_input_softline / @prepend_input_softline

The matched nodes will have an input softline appended (or, respectively, prepended) to them. An input softline is a line break if the node has a line break after (or, respectively, before) it in the input document, otherwise it is a space.

Example

; Input softlines before and after all comments. This means that the input
; decides if a comment should have line breaks before or after. But don't put a
; softline directly in front of commas or semicolons.

(comment) @prepend_input_softline

(
  (comment) @append_input_softline
  .
  [ "," ";" ]* @do_nothing
)

Scopes

So far, we've expanded softlines into line breaks depending on whether the CST node they are associated with is multi-line (see hardlines and softlines). Sometimes, CST nodes define scopes that are either too big or too small for our needs. For instance, consider this piece of OCaml code:

(1,2,
3)

Its CST is the following:

{Node parenthesized_expression (0, 0) - (1, 2)} - Named: true
  {Node ( (0, 0) - (0, 1)} - Named: false
  {Node product_expression (0, 1) - (1, 1)} - Named: true
    {Node product_expression (0, 1) - (0, 4)} - Named: true
      {Node number (0, 1) - (0, 2)} - Named: true
      {Node , (0, 2) - (0, 3)} - Named: false
      {Node number (0, 3) - (0, 4)} - Named: true
    {Node , (0, 4) - (0, 5)} - Named: false
    {Node number (1, 0) - (1, 1)} - Named: true
  {Node ) (1, 1) - (1, 2)} - Named: false

We would like to add a line break after the first comma, but because the CST structure is nested, the node containing this comma (product_expression (0, 1) - (0, 4)) is not multi-line Only the top-level node product_expression (0, 1) - (1, 1) is multi-line.

To solve this issue, we introduce user-defined scopes and softlines.

Note
Similar to the non-scoped single- and multi-lined context, the scoped equivalents can also be used to drive arbitrary queries, not necessarily related to vertical spacing; see below.

@append_begin_scope / @prepend_begin_scope / @append_end_scope / @prepend_end_scope

These capture names are used to define custom scopes. In conjunction with the #scope_id! predicate, they define scopes that can span multiple CST nodes, or only part of one. For instance, this scope matches anything between parenthesis in a parenthesized_expression:

(parenthesized_expression
  "(" @append_begin_scope
  ")" @prepend_end_scope

  (#scope_id! "tuple")
)

Scoped softlines

We have four predicates that insert softlines in custom scopes, in conjunction with the #scope_id! predicate:

• @append_empty_scoped_softline / @prepend_empty_scoped_softline
• @append_spaced_scoped_softline / @prepend_spaced_scoped_softline

When one of these scoped softlines is used, their behaviour depends on the innermost encompassing scope with the corresponding scope_id. If that scope is multi-line, the softline expands into a line break. In any other context, they behave as their non-scoped counterparts.

Example

This Tree-sitter query:

(parenthesized_expression
  "(" @append_begin_scope @append_empty_softline @append_indent_start
  ")" @append_end_scope @prepend_empty_softline @prepend_indent_end

  (#scope_id! "tuple")
)

(product_expression
  "," @append_spaced_scoped_softline

  (#scope_id! "tuple")
)

formats this snippet of OCaml code:

(1,2,
3)

as:

(
  1,
  2,
  3
)

Whereas the single-lined (1, 2, 3) is kept as is.

If we used @append_spaced_softline rather than @append_spaced_scoped_softline, the 1, would be followed by a space rather than a newline, because it's inside a single-line product_expression node.

Testing context with predicates

Sometimes, similarly to what happens with scoped softlines, we want a query to match only if the scoped context is single-line, or multi-line. Topiary has several predicates that achieve this result.

#single_line_scope_only! / #multi_line_scope_only!

These predicates allow the query to trigger only if the associated custom scope containing the matched nodes are is single-line (resp. multi-line).

Note
There are non-scoped equivalents to these predicates; see vertical spacing for details.

Example

; Allow (and enforce) the optional "|" before the first match case
; in function expressions in OCaml if and only if the scope is multi-line
(function_expression
  (match_case)? @do_nothing
  .
  "|" @delete
  .
  (match_case)

  (#single_line_scope_only! "function_definition")
)

(function_expression
  "|"? @do_nothing
  .
  (match_case) @prepend_delimiter

  (#multi_line_scope_only! "function_definition")
  (#delimiter! "| ") ; sic
)

Measuring scopes

@append_begin_measuring_scope / @prepend_begin_measuring_scope / @append_end_measuring_scope / @prepend_end_measuring_scope

Sometimes, custom scopes are not enough: we may want to format a node depending on the multi-line-ness of a piece of code that does not include the node in question. For instance, consider this function application in OCaml:

foo bar (fun x -> qux)

We may want to format it as either of the following two, depending on the actual length of foo, bar, and qux:

foo bar (fun x ->
  qux
)

foo
  bar
  (fun x ->
    qux
  )

If foo bar is single-line, we don't want to wrap (fun x -> qux) onto a new line. However, if foo bar is multi-line, then we do, to be consistent with bar.

Because custom scopes can only impact the behaviour of nodes inside the scope, we can't use them to solve this issue. This is why we need "measuring scopes".

Measuring scopes are opened/closed with a similar syntax as "regular" custom scopes, with any of the following capture names, in conjunction with the #scope_id! predicate:

• @append_begin_measuring_scope / @prepend_begin_measuring_scope
• @append_end_measuring_scope / @prepend_end_measuring_scope

Measuring scopes behave as follows:

• A measuring scope must always be contained in a regular custom scope with the same #scope_id!. There can't be two measuring scopes with the same #scope_id! inside the same regular custom scope.

• If a regular custom scope contains a measuring scope, then all captured nodes contained in the regular scope that depend on its multi-line-ness will instead depend on the multi-line-ness of the measuring scope (hence the name: the inner, measuring scope measures the multi-line-ness of the outer, regular scope).

Example

The example below solves the problem of indenting function application in OCaml stated above, using measuring scopes.

(application_expression
  .
  (_) @append_indent_start @prepend_begin_scope @prepend_begin_measuring_scope
  (#scope_id! "function_application")
  (_) @append_end_scope
  .
)

; The end of the measuring scope depends on the last argument: if it's a function,
; end it before the function, otherwise end it after the last argument. In that case,
; it's the same as the regular scope.
(application_expression
  (#scope_id! "function_application")
  (_
    [
      (fun_expression)
      (function_expression)
    ]? @do_nothing
  ) @append_end_measuring_scope
  .
)

(application_expression
  (#scope_id! "function_application")
  (_
    [
      (fun_expression)
      (function_expression)
    ] @prepend_end_measuring_scope
  )
  .
)

; If the measuring scope is single-line, end indentation _before_ the last node.
; Otherwise, end the indentation after the last node.
(application_expression
  (#multi_line_scope_only! "function_application")
  (_) @append_indent_end
  .
)

(application_expression
  (#single_line_scope_only! "function_application")
  (_) @prepend_indent_end
  .
)

@append_empty_scoped_softline / @prepend_empty_scoped_softline

The matched nodes will have an empty softline appended (or, respectively, prepended) to them. This will be expanded to a line break for multi-line nodes within the scope defined by the #scope_id! predicate and to nothing for single-line nodes.

Example

; Define a scope immediately following a command up to (but not
; including) the next node. If that scope is multi-line, then insert a
; line break after the command; otherwise, insert nothing.
(program
  (command) @append_begin_scope @append_empty_scoped_softline
  .
  _ @prepend_end_scope

  (#scope_id! "program_line_break")
)

@append_spaced_scoped_softline / @prepend_spaced_scoped_softline

The matched nodes will have a spaced softline appended (or, respectively, prepended) to them. This will be expanded to a line break for multi-line nodes within the scope defined by the #scope_id! predicate and to a space for single-line nodes.

Example

; Define a scope after the equals sign in a let binding that's followed
; by a function expression. If that scope is multi-line, then insert a
; line break after the function arrow; otherwise, insert a space.
(let_binding
  "=" @prepend_begin_scope
  .
  (fun_expression
    "->" @append_spaced_scoped_softline
  ) @append_end_scope

  (#scope_id! "fun_definition")
)

Insertion and deletion

@append_delimiter / @prepend_delimiter

The matched nodes will have a delimiter appended (or, respectively, prepended) to them. The delimiter must be specified using the predicate #delimiter!.

Note that @append_delimiter is a generalisation of @append_space (and, respectively, for prepending) where the delimiter is set to " " (i.e., a space); see horizontal spacing.

Note
A delimiter can be any string; it is not limited to a single character.

Example

; Put a semicolon delimiter after field declarations, unless they already have
; one, in which case we do nothing.
(
  (field_declaration) @append_delimiter
  .
  ";"* @do_nothing

  (#delimiter! ";")
)

If there is already a semicolon, the @do_nothing instruction (see general advice) will be activated and prevent the other instructions in the query (the @append_delimiter, here) from applying. Otherwise, the ";"* captures nothing and, in this case, the associated instruction (@do_nothing) does not activate.

@delete

Remove the matched node from the output.

Example

; Move semicolon after comments.
(
  ";" @delete
  .
  (comment)+ @append_delimiter

  (#delimiter! ";")
)

Note
The above example uses a combination of @delete and @append_delimiter (see above) to effectively implement a rewrite rule.

Modification

@lower_case / @upper_case

Set the capitalisation of all text in the matched node and its children (besides designated leaf nodes, which remain unchanged).

Example

; Example for SQL, since that's where this makes sense
; (Grammar: https://github.com/DerekStride/tree-sitter-sql)

; Make keywords "select" and "from" lowercase
[
  (keyword_select)
  (keyword_from)
] @lower_case

; Make keyword "WHERE" uppercase
(keyword_where) @upper_case

Topiary as a Rust library

Topiary is published on crates.io, which means that its documentation can be found on docs.rs. Of main interest is the formatter function that performs the actual formatting. The example in the documentation of that function is kept up to date.

For a more complete example, see the client-app example in the Topiary repository.

Related Tools

Tree-sitter specific

• Syntax Tree Playground: An interactive, online playground for experimenting with Tree-sitter and its query language.
• Neovim Treesitter Playground: A Tree-sitter playground plugin for Neovim.
• Difftastic: A tool that utilises Tree-sitter to perform syntactic diffing.

Meta and multi-language formatters

• format-all: A formatter orchestrator for Emacs.
• null-ls.nvim: An LSP framework for Neovim that facilitates formatter orchestration.
• Prettier: A formatter with support for multiple (web-development related) languages.
• treefmt: A general formatter orchestrator, which unifies formatters under a common interface.

Related formatters

• gofmt: The de facto standard formatter for Go, and major source of inspiration for the style of our formatters.
• ocamlformat: A formatter for OCaml.
• ocp-indent: A tool to indent OCaml code.
• Ormolu: Tweag's formatter for Haskell, which follows similar design principles as Topiary.
• rustfmt: The de facto standard formatter for Rust.
• shfmt: A parser, formatter and interpreter for Bash et al.