Most programs start by reading text that someone else wrote. Splitting a line on commas, or matching it with a regular expression, works until the input contains a quoted comma, a missing field, or a value of the wrong kind. Then the code either crashes, silently produces wrong data, or grows a pile of special cases.

A parser describes the shape of valid input once, as a Roc value, and checks every input against it. With roc-parser:

Compared with a regular expression, a combinator parser is longer to write for a one-off match, but it can describe nested structure (a regular expression cannot match balanced brackets), it produces a value rather than a list of captured substrings, and it reads as ordinary Roc code.

This program decodes a CSV file with a header row into typed records and prints them. Getting started walks through it.

Output:

Each module follows a published specification. Where a module implements only part of one, its chapter says which part, and input outside that part is rejected with an error rather than misread.

The format chapters describe each module’s exact behaviour. The API reference lists every public function.

A combinator is a function that takes parsers and returns a new parser. The Utf8 module provides parsers for the smallest pieces of text, such as one byte, a digit, or an exact word. The Parser module combines them: one after another, one of several alternatives, repeated, or with the result transformed. Each format module also exposes its parser as a Parser value, so your own parsers and the library’s can be mixed.

Your first parser teaches the combinators by building a small parser one step at a time.

roc-parser fits programs that read text formats of moderate size completely into memory and want typed results with clear errors. Consider something else when:

roc-parser tracks recent nightly builds of the Roc compiler. Because Roc is still changing, the package’s API may change between releases. Each release names the compiler build it was tested with, and the release notes describe any changes you need to make when upgrading.

You need:

Check the compiler:

It prints the name of the nightly build, which should match the one in the release notes:

A Roc app names each package it uses by URL in its header. The compiler downloads the package the first time it builds the app and checks it against the hash in the URL.

Create a file named main.roc with this header, replacing the parser URL with the one you copied:

The examples in this manual are tested against the package source in the repository, so their headers say parser: "../../package/main.roc" instead. Everything else in them is the same.

For complete programs to start from, download roc-parser-examples-<version>.zip from the same release. Its apps are already pinned to that release; unzip it and follow the README.md inside, which runs them with roc <name>.roc from its examples/ directory.

Add the rest of the program below the header:

Run it:

Output:

The first run takes longer while the compiler downloads the platform and the package.

import parser.CSV makes the CSV module from the package named parser available.

Planet is the record type each row becomes. The first line of the input is a header row, and each field of Planet reads the column with the same name.

planets calls CSV.parse, which splits the text into rows and decodes each row into a record. CSV.parse does not take the row type as an argument: the compiler infers it from the type annotation on planets, and finds the decoding code through the static dispatch method parser_for, which every record type has. Change the annotation and the same call decodes a different record.

The result is Ok with a list of Planet records, or Err with one of two tags:

describe handles all three cases with a match. Nothing is printed until every row has been read successfully.

To see the error path, change Mars,2 to Mars,two and run the program again. It prints a message that names line 4 and quotes two. Rename the moons column and it prints no moons column. Report parse errors shows how to report every kind of failure, and Read CSV data shows optional columns, empty cells and hand-built row parsers that match columns by position.

Roc has type-directed decoding built in. A format (such as the builtin Json) knows how to read strings, numbers, lists and records. Your type knows which of those it is made of. The compiler combines the two, so the annotation alone decides what is accepted:

The parsers chapter of the language reference describes the mechanism. Every type has a parser_for method, which the compiler derives for records, lists and tuples, and a format is a type module whose methods read one value at a time. Static dispatch joins the two. A field of type Try(_, [Missing]) is optional, a missing required field is reported by name, and when the parser is a top-level constant it is assembled at compile time.

Choose this when:

roc-parser’s CSV and Yaml modules are formats for this mechanism. CSV.parse reads a file with a header row into a list of records, matching columns to fields by name, and Yaml.decode reads a document into a record. Neither takes the type as an argument: the annotation chooses it. A required field with no column or key fails with MissingRequiredField(name), a tag the compiler adds to the error type, so the annotation names it alongside the format’s own error:

Its limits come from the same design. The format decides the representation, so you cannot ask it to accept yes as a boolean or to read a date in your own layout. Where a wrong value is depends on the format: CSV.parse reports the record, field, line and column, and Yaml.decode the line and column. And the type must be known when you compile.

roc-parser’s Yaml, Xml and Markdown modules read a whole document into a tree that mirrors the format, and HTTP reads a message into a record of its parts. You then walk the tree with pattern matching, or with the tree methods such as get_path and as_i64 on a Yaml value and attribute and children_named on an Xml.Node:

Choose this when:

The cost is code: you write the walk, and a missing or mistyped key is something you check, not something the type system checks for you. The format chapters (Read YAML configuration and frontmatter, Read XML documents, Markdown and HTTP messages) show helper functions that keep the walk short.

When no format module reads your input, write a parser from the Parser and Utf8 combinators. A parser is a value made of smaller parsers, so the code has the shape of the grammar:

Choose this when:

Combinators are also how you extend a format module. CSV.record and CSV.field are combinators, every format module has a parser value to embed, and you can mix Utf8 parsers into any parser of your own. A combinator parser’s failure is the furthest failure, with a byte offset. Your first parser teaches the method and Parse a format of your own applies it to a real format.

Parser combinators are not always the right tool, and roc-parser’s own format modules show where the line falls.

The same choice as a sequence of questions:

The questions are ordered from least code to most. You can also combine answers: read a Markdown file as a tree, read its frontmatter as a YAML tree, and parse one custom field inside it with a combinator.

combinator-primer-step1.roc

A key is one or more lowercase letters or underscores. Start with a parser for one such byte, repeat it, and turn the bytes into a Str:

Three ideas appear here:

Utf8.parse_str runs a parser on a whole string. These tests pass:

The second fails because N is not a lowercase letter, so not even one key byte can be read.

combinator-primer-step2.roc

A setting is a key, an =, and a value. For now, the value is everything up to the end of the line:

This is the pattern for reading things in sequence:

Parser.chomp_while(pred) reads bytes while pred accepts them. It always succeeds, even when it reads nothing, which is why "empty=" gives an empty value:

Running the step prints:

combinator-primer-step3.roc

A value is a number, true or false, or text in double quotes. Write one parser for each kind, then try them in order with Parser.one_of:

Parser.one_of tries each parser on the same input and returns the first success. If one fails part-way through, the next one starts again from the beginning, so text does not see input already examined by number or flag.

Utf8.digits reads a whole number as a U64, and Utf8.string("true") reads exactly that text. .between(open, close) reads open, the parser, and close, and keeps only the middle result.

The entry parser from step 2 now uses this value. Running the step prints:

combinator-primer-step4.roc

A settings file is entries separated by line breaks. sep_by reads that shape and drops the separators:

Utf8.parse_str returns one of three results, and a real program should handle each:

Output:

Each line of output shows one outcome:

The second line is worth remembering. A repetition such as sep_by or many stops at the first element it cannot read, but that element’s failure is kept, so a mistake in the middle of a file is reported where it is. Report parse errors explains how to get better messages.

Combinators by task covers the remaining combinators and the rules for alternatives and repetition in more depth.

When the first row names the columns, CSV.parse decodes every other row into the record type you annotate. Each field reads the column whose header is exactly its name, so the column order does not matter:

Running it on a good file, on a file without a quantity column, and on a file whose second data row is short:

MissingRequiredField(name) is not declared by CSV.parse: the compiler adds it to the error type of any decoder whose record has required fields, so include it in your annotation. Columns that no field names are ignored, and when two columns share a name the last one wins.

A cell decodes according to its field’s type:

These match Rust’s from_str for each number type. A field that is a record, a tuple or a tag with a payload fails with an InvalidCsv error when it is decoded; a field that is a list or a dictionary is a compile-time error, because a CSV cell cannot hold one.

CSV.parse_headerless decodes every row into a tuple, one element per column, with the same cell rules. Every row must have as many fields as the tuple has elements:

Use CSV.parse_records when the columns vary or you want to inspect rows yourself. It returns one CSV.Record, a List(List(U8)), per row:

Given a header row, a quoted field containing "" and ,, and a row whose last field is empty:

CSV.split_header separates the first record as column names:

CSV.parser is the same reader as a Parser(Utf8.Bytes, List(CSV.Record)), for use inside larger parsers. It accepts any bytes.

When a column needs custom handling, such as a list stored in one cell, describe one row with CSV.record, giving it a function that takes one argument per column, then add one .keep(CSV.field(...)) per column in order. A field parser is any Parser(Utf8.Bytes, a); CSV.string, CSV.u64 and CSV.f64 read cells with the rules above:

Run it on text with CSV.parse_with, or on records you already have with CSV.decode. Columns are matched by position, and the record parser must read every field in the row: a row with more fields than .keep calls fails, and so does a row with fewer.

Every function stops at the first problem and returns InvalidCsv(error). The CSV.Error record says where the problem is:

The dialect is RFC 4180 with the relaxations that Python’s csv module (in strict mode) and Rust’s csv crate also accept:

For every function and type in the module, see the CSV API reference.

Reading every field as a string runs at 120–150 MB/s on an Apple M2, from a 1.6 KB export to a 1 MiB file: on par with Rust’s csv crate, about 2.3 times slower than Go’s encoding/csv and nearly twice as fast as Python’s C csv module. Delimiters and quotes are found 16 bytes at a time with SIMD, and fields without doubled quotes are slices of the input. The parser is a byte scanner and stays linear on very wide rows and long runs of escaped quotes. Decoding typed columns adds the cost of each conversion. The whole file and every field are held in memory: about 12 MiB of peak memory for a 1 MiB file. See Performance for the method and caveats.

Yaml.decode reads a document straight into your own Roc type. You do not pass the type: it is inferred from how the result is used, usually from a type annotation. This configuration, held in a Roc multi-line string (each \\ starts one line of the file):

is read by:

summarize(config_text) returns:

The rules that map YAML onto Roc types:

A value of the wrong shape or out of range fails with InvalidYaml at that value’s line and column, the same error invalid YAML gives. A required field whose key is absent fails with MissingRequiredField(name). Your code does not declare that tag: the compiler adds it to the error type of every decoder of a record with required fields, so name it in the annotation:

When the shape is not known in advance, Yaml.parse_str returns a Yaml tree:

A mapping keeps its entries in source order. get looks up a key, at an index, and get_path follows several of them, using decimal strings such as "0" for sequence indexes. Each returns Err(Missing) when there is nothing there. as_str, as_i64, as_bool and as_list unwrap one variant or return Err(WrongType):

For config_text, and then for "owners: []":

Empty input, or a document holding only comments, parses as Null. To combine YAML with other parsers, Yaml.parser is a Parser(Utf8.Bytes, Yaml) that reads the rest of its input; its ParseError offset is the byte offset of the error’s line and column.

Frontmatter is a YAML document between two --- lines at the start of a Markdown file. Split the text at the closing --- and parse the first part. The parser accepts the opening --- (and a closing ...) as document markers, so you do not need to strip them:

For "---\ntitle: Hello\ntags: [roc, yaml]\n---\n# Hello\n", printing Yaml.to_inspect(meta) and then the body gives:

Yaml.decode works on the same text when you know the frontmatter’s fields.

In a Yaml tree, an unquoted value is resolved with the YAML 1.2 core schema. Anything that matches none of these forms is Text:

The last four values show what stays text: a version number such as 1.2.3, a number with _ separators, anything in quotes, and dates (YAML 1.2 has no timestamp type). tRUE and 0X1 are text too, because the core schema allows only the spellings in the table. In a tree, quote a value whenever it must stay text; Yaml.decode reads any non-null plain scalar into a Str field as written.

Yaml.to_inspect prints a tree in Roc notation, escaping line breaks, tabs and other control characters so each value stays on one line.

Use a block scalar for text that spans lines. | (literal) keeps line breaks; > (folded) joins lines with spaces and keeps a blank line as a paragraph break. A chomping indicator controls the final line break: none keeps one, - removes it, and + keeps every trailing blank line. A digit sets the content indentation, so the text can start with spaces:

A sequence item may start with a mapping on the same line as its - (compact), may hold another sequence the same way, and may sit at the same indentation as its parent key (indentless):

Short lists and mappings can also use flow style on one line, as in ports: [80, 443] or labels: { tier: web }.

A single-quoted string has no escapes except '' for one quote. A double-quoted string supports every YAML 1.2 escape, including \t, \n, \xXX, \uXXXX and \UXXXXXXXX. A # inside quotes is text; outside quotes and after a space it starts a comment:

The following are outside the supported subset. Each one is an InvalidYaml error, never a silently different value:

Mapping keys are kept as their source text, not resolved like values. 1 and 01 are different keys, and true is the string key "true".

InvalidYaml carries a Yaml.Error record: a one-based line and column and a message. Columns count bytes from the start of the line:

For the full module interface, see the Yaml API reference.

Parsing takes time linear in the size of the document, at about 51–55 MB/s on an Apple M2 from 4 KiB to 1 MiB: a 4 KiB configuration in under 0.1 ms and a 1 MiB one in about 20 ms. That is about as fast as yaml-rust2 (Rust), 1.6 times faster than serde_yaml, 2.6 times faster than Go’s yaml.v3 and seven times faster than PyYAML with libyaml. Nesting is limited to 100 levels. Decoding into a record builds the same document first, then walks it once. See Performance for the measurements and the comparison with other libraries.

Xml.parse_str takes the whole document and returns an Xml record with the document’s root element and its declaration. Each node in the tree is one of:

An Element holds its name, its attributes in source order, and its children. There is no query language, but Xml.Node has four helpers for the common lookups, and you can pattern match on the tree for anything else:

Given this document:

parse it and describe each entry element under the root:

The whitespace between elements is kept as Text nodes; children_named skips them because it only returns elements. &, the CDATA section and the character reference 😀 have already been turned into ordinary text.

The tree is what an XML processor reports, not the literal source:

This document has every one of these in a single element:

Printing the root with Str.inspect gives:

When a document is not well-formed, Xml.parse_str returns InvalidXml({ line, column, message }). Lines and columns are one-based, and columns count UTF-8 bytes, so a column after non-ASCII text is larger than the number of characters a user sees.

If you are combining XML with other parsers, Xml.parser is the same parser as a Parser(Utf8.Bytes, Xml). It leaves any input after the document for the next parser. Its failures are ParseError({ message, offset }), with the same message as Xml.parse_str and the byte offset of the problem.

These limits are by design:

For the full module interface, see the Xml API reference.

Xml.parse_str builds the tree at about 120–135 MB/s on an Apple M2, and the rate stays flat from 4 KiB to 1 MiB documents: the parser is linear in the input size. On the generated documents that is about 2.2 times as fast as Go’s encoding/xml, and 1.4–2 times slower than Rust’s quick-xml and roxmltree. Character data and attribute values are found 16 bytes at a time with SIMD, the input is validated as UTF-8 once, and names, text and attribute values with nothing to unescape are slices of the input string, so they keep the input alive. Elements are parsed with an explicit stack, so 5,000 levels of nesting are safe, and duplicate-attribute checks are linear: an element with 5,000 attributes parses 19 times faster than with the Rust libraries. See Performance for the measurements and caveats.

Parse a whole document with Markdown.parse_str. It returns a List(Markdown): one value per top-level block, in document order.

Markdown has no syntax errors. Every input is a valid document, so Markdown.parse_str returns the blocks directly rather than a Try; text that looks like broken syntax becomes ordinary paragraph text. To use the document parser inside your own combinators, Markdown.parser is the same parser as a Parser(Utf8.Bytes, List(Markdown)).

The following app parses a document that uses each kind of block and prints each block with Str.inspect, which shows what the parser produced.

A list in which no items are separated by blank lines is tight; otherwise it is loose. The loose field records this, which matters when you render HTML: a tight list normally shows its item paragraphs without <p> tags. Ordered lists keep their starting number:

Output:

The block variants are:

Block quotes and lists nest at most 1,000 levels deep. Deeper > and list markers are read as paragraph text. Emphasis, strikethrough, links and images also nest at most 1,000 levels deep (counted separately); the delimiters and brackets of deeper ones are read as text. Code that walks the tree recursively therefore cannot run out of stack on hostile input.

The syntax tree types support == and hashing, so blocks and inlines can be Dict keys and Set elements.

Link reference definitions ([label]: /url) do not appear in the tree. The parser resolves them into the links that use them.

Inline content is the text inside a paragraph, heading or table cell. When you already have a single line or fragment, such as a title from a database, parse it directly with Markdown.parse_inlines, which returns a List(Inline). Markdown.inline_parser is the same parser for use in combinators.

Output:

The output shows these rules:

Many static-site tools put metadata at the top of a Markdown file between two --- lines. When the document’s first line is exactly --- and a later line is exactly ---, Markdown.parse_str returns the lines between them as the first block, Frontmatter(raw), and parses the rest as Markdown. Markdown.frontmatter(blocks) returns that text, or Err(Missing) when the document has none. The parser does not interpret the text. Pass it to the YAML parser (see Read YAML configuration and frontmatter) to read the fields:

Output:

Without a closing --- line, the opening --- is ordinary Markdown: a thematic break, or a Setext heading underline.

To produce output, write two functions: one that matches each block variant and one that matches each inline variant. Each function calls the other for nested content. This section builds a small HTML renderer and a table-of-contents extractor for the following release notes:

Escape the characters that are significant in HTML: