RocSyntax

:= { source : List(U8), tokens : List(Token), nodes : List(Node), children : List(U64), diagnostics : List(Diagnostic), status : Status, root : U64, consumed : Bool }

Lossless parsing of Roc source into tokens, syntax trees and diagnostics. Byte spans are half-open. Comments and whitespace are tokens.

Expressions include interpolated and multiline strings, typed literals, calls, field access, operators, pipes, nominal construction and control flow. Binding patterns, type declarations, annotations, constraints, imports and module, app, package and platform headers retain their syntax structure. Parsing does not resolve names or establish that a program typechecks.

Parsing retains the source once and appends nodes and children to flat lists that each parser receives and returns directly, so they stay uniquely owned. Nesting is bounded at 128; documents over 1,048,576 tokens retain all tokens but skip syntax construction with Limited status. Malformed input returns a partial tree and diagnostics, including arbitrary non-UTF-8 source bytes.

RocSyntax.validate and quoted RocSyntax values require complete file syntax without recovery diagnostics. Validation also rejects ... placeholder expressions by default; strings and comments containing ... are ordinary source text. This checks syntax, not name resolution or type correctness.

from_quote : Str -> Try(RocSyntax, [BadQuotedBytes(Str)])

Check quoted file source using the default validation options. The source, tokens, nodes and diagnostics are available as record fields.

validate : ValidateProps -> Try(Parsed, [BadQuotedBytes(Str)])

Parse one complete Roc source file and return its existing syntax tree. Recovery, unsupported syntax, resource limits, and unread input fail. Disable only the placeholder lint with lints: { placeholders: False }.

tokenize_str : Str -> Tokenized

Lossless lexer entry point for UTF-8 source text.

highlight_category : TokenKind -> HighlightCategory

Map one lexical kind to a theme category without allocating. Semantic highlighting can refine names using the syntax tree.

parse_str : Str -> Parsed

Parse source text; offsets refer to UTF-8 bytes, not Unicode characters.

parse_bytes : List(U8) -> Parsed

Parse arbitrary source bytes, retaining invalid bytes and incomplete syntax.

parse_expression : Str -> Parsed

Parse one expression. Extra significant tokens produce a diagnostic.

parse_expr_bytes : List(U8) -> Parsed

Parse one expression from arbitrary bytes, preserving malformed input.

parse_statement_str : Str -> Parsed

Parse one statement in a body, including expression statements and locals.

parse_header_str : Str -> Parsed

Parse one file header and require complete input consumption.

Span : { start : U64, end : U64 }

A UTF-8 byte interval; end is excluded.

TokenKind : [Whitespace, Comment, LowerIdent, UpperIdent, Keyword, Number, StringStart, MultilineStringStart, StringPart, StringEnd, InterpolationStart, InterpolationEnd, SingleQuote, OpenRound, NoSpaceOpenRound, CloseRound, OpenSquare, CloseSquare, OpenCurly, CloseCurly, Operator, Punctuation, Invalid, EndOfFile]

Lexical classes. Operator and punctuation spelling is available by span.

Token : { kind : TokenKind, start : U64, end : U64 }

Lossless token. Only EOF has an empty span.

NodeKind : [File, Declaration, VariableDeclaration, Expect, For, ForStream, While, ExpressionStatement, Lambda, If, Match, MatchBranch, Pattern, IdentifierPattern, WildcardPattern, PatternStringLiteral, CapturePattern, MutablePattern, TagPattern, NominalPattern, TuplePattern, ListPattern, RecordPattern, PatternField, RestPattern, AliasPattern, AlternativePattern, Block, Record, RecordUpdate, UnsetField, RecordBuilder, NominalRecord, NominalApply, RecordField, Header, Dependencies, DependencyField, PlatformReference, Import, FileImport, ImportPath, ImportAlias, ExposedItem, ExposedWildcard, TypeAnnotation, TypeExpression, Identifier, NumberLiteral, StringLiteral, MultilineStringLiteral, StringSegment, Interpolation, TypedLiteral, ArrowCall, MethodArrowCall, CharacterLiteral, Unary, Binary, Call, FieldAccess, OptionalFieldAccess, Suffix, Debug, Crash, Return, Break, Ellipsis, ListLiteral, Tuple, Parenthesized, ErrorNode, UnsupportedNode, TypeName, TypeVariable, TypeWildcard, TypeApply, TypeFunction, TypeTuple, TypeRecord, TypeField, TypeTagUnion, TypeExtension, TypeParenthesized, TypeDeclaration, TypeHeader, WhereClause, WhereMethod, WhereAlias, AssociatedBlock, PlatformRequires, RequiredValue, RequiredType, PlatformProvides, PlatformHosted, SymbolMapping, PlatformTargets, TargetInputsDirectory, TargetEntry, TargetConfig, TargetField, TargetList, TargetFile]

Syntax categories. Children are indices into Parsed.nodes.

Node : { kind : NodeKind, start : U64, end : U64, children_start : U64, children_len : U64 }

Flat node. Its children occupy a contiguous interval in Parsed.children.

DiagnosticCode : [InvalidByte, InvalidUtf8, BidiControlInSource, Utf8ByteOrderMark, UnclosedString, UnclosedDelimiter, UnexpectedToken, InvalidNumber, InvalidEscape, InvalidCharacter, UnsupportedSyntax, ResourceLimit]

Structured issues; human-facing text can be rendered outside parsing.

Diagnostic : { code : DiagnosticCode, start : U64, end : U64 }

An issue code and its precise source byte interval.

Status : [Complete, Recovered, Unsupported, Limited]

Complete means the implemented grammar consumed the input without issues. It does not establish type correctness or complete language conformance.

Parsed : { source : List(U8), tokens : List(Token), nodes : List(Node), children : List(U64), diagnostics : List(Diagnostic), status : Status, root : U64, consumed : Bool }

Source-backed tokens and a flat, indexed syntax tree with recovery status.

LintProps

:= { placeholders : Bool ?? True }

Optional checks applied after complete file syntax has been established.

ValidateProps

:= { source : Str, name : Str ?? "<literal>", lints : LintProps ?? {} }

Complete file source, its diagnostic name, and optional syntax lints.

Tokenized : { source : List(U8), tokens : List(Token), diagnostics : List(Diagnostic), status : Status }

Lexer-only result; suitable for highlighting without constructing a tree.

HighlightCategory : [Trivia, CommentText, Name, TypeName, ReservedWord, NumericLiteral, StringText, Symbol, InvalidText]

Stable presentation categories, independent of theme colors.