package participle

import "github.com/alecthomas/participle"

Package participle constructs parsers from definitions in struct tags and parses directly into those structs. The approach is philosophically similar to how other marshallers work in Go, "unmarshalling" an instance of a grammar into a struct.

The supported annotation syntax is:

• `@<expr>` Capture expression into the field.
• `@@` Recursively capture using the fields own type.
• `<identifier>` Match named lexer token.
• `( ... )` Group.
• `"..."` Match the literal (note that the lexer must emit tokens matching this literal exactly).
• `"...":<identifier>` Match the literal, specifying the exact lexer token type to match.
• `<expr> <expr> ...` Match expressions.
• `<expr> | <expr>` Match one of the alternatives.

The following modifiers can be used after any expression:

• `*` Expression can match zero or more times.
• `+` Expression must match one or more times.
• `?` Expression can match zero or once.
• `!` Require a non-empty match (this is useful with a sequence of optional matches eg. `("a"? "b"? "c"?)!`).

Supported but deprecated:

• `{ ... }` Match 0 or more times (**DEPRECATED** - prefer `( ... )*`).
• `[ ... ]` Optional (**DEPRECATED** - prefer `( ... )?`).

Here's an example of an EBNF grammar.

type Group struct {
    Expression *Expression `"(" @@ ")"`
}

type Option struct {
    Expression *Expression `"[" @@ "]"`
}

type Repetition struct {
    Expression *Expression `"{" @@ "}"`
}

type Literal struct {
    Start string `@String` // lexer.Lexer token "String"
    End   string `("…" @String)?`
}

type Term struct {
    Name       string      `  @Ident`
    Literal    *Literal    `| @@`
    Group      *Group      `| @@`
    Option     *Option     `| @@`
    Repetition *Repetition `| @@`
}

type Sequence struct {
    Terms []*Term `@@+`
}

type Expression struct {
    Alternatives []*Sequence `@@ ("|" @@)*`
}

type Expressions []*Expression

type Production struct {
    Name        string      `@Ident "="`
    Expressions Expressions `@@+ "."`
}

type EBNF struct {
    Productions []*Production `@@*`
}

Index ¶

• Variables
• func AnnotateError(pos lexer.Position, err error) error
• func ErrorWithTokenf(tok lexer.Token, format string, args ...interface{}) error
• func Errorf(pos lexer.Position, format string, args ...interface{}) error
• func Wrapf(pos lexer.Position, err error, format string, args ...interface{}) error
• type Capture
• type Error
• type Mapper
• type Option
• func CaseInsensitive(tokens ...string) Option
• func Elide(types ...string) Option
• func Lexer(def lexer.Definition) Option
• func Map(mapper Mapper, symbols ...string) Option
• func Unquote(types ...string) Option
• func Upper(types ...string) Option
• func UseLookahead(n int) Option
• type ParseOption
• func AllowTrailing(ok bool) ParseOption
• type Parseable
• type Parser
• func Build(grammar interface{}, options ...Option) (parser *Parser, err error)
• func MustBuild(grammar interface{}, options ...Option) *Parser
• func (p *Parser) Lex(r io.Reader) ([]lexer.Token, error)
• func (p *Parser) Lexer() lexer.Definition
• func (p *Parser) Parse(r io.Reader, v interface{}, options ...ParseOption) (err error)
• func (p *Parser) ParseBytes(b []byte, v interface{}, options ...ParseOption) error
• func (p *Parser) ParseFromLexer(lex *lexer.PeekingLexer, v interface{}, options ...ParseOption) error
• func (p *Parser) ParseString(s string, v interface{}, options ...ParseOption) error
• func (p *Parser) String() string
• type UnexpectedTokenError
• func (u UnexpectedTokenError) Error() string
• func (u UnexpectedTokenError) Message() string
• func (u UnexpectedTokenError) Token() lexer.Token

Variables ¶

var (
	// MaxIterations limits the number of elements capturable by {}.
	MaxIterations = 1000000

	// NextMatch should be returned by Parseable.Parse() method implementations to indicate
	// that the node did not match and that other matches should be attempted, if appropriate.
	NextMatch = errors.New("no match") // nolint: golint
)

var DropToken = errors.New("drop token") // nolint: golint

DropToken can be returned by a Mapper to remove a token from the stream.

Functions ¶

func AnnotateError ¶

func AnnotateError(pos lexer.Position, err error) error

AnnotateError wraps an existing error with a position.

If the existing error is a lexer.Error or participle.Error it will be returned unmodified.

func ErrorWithTokenf ¶

func ErrorWithTokenf(tok lexer.Token, format string, args ...interface{}) error

ErrorWithTokenf creats a new Error with the given token as context.

func Errorf ¶

func Errorf(pos lexer.Position, format string, args ...interface{}) error

Errorf creats a new Error at the given position.

func Wrapf ¶

func Wrapf(pos lexer.Position, err error, format string, args ...interface{}) error

Wrapf attempts to wrap an existing participle.Error in a new message.

Types ¶

type Capture ¶

type Capture interface {
	Capture(values []string) error
}

Capture can be implemented by fields in order to transform captured tokens into field values.

type Error ¶

type Error interface {
	error
	// Unadorned message.
	Message() string
	// Closest token to error location.
	Token() lexer.Token
}

Error represents an error while parsing.

The error will contain positional information if available.

type Mapper ¶

type Mapper func(token lexer.Token) (lexer.Token, error)

Mapper function for mutating tokens before being applied to the AST.

If the Mapper func returns an error of DropToken, the token will be removed from the stream.

type Option ¶

type Option func(p *Parser) error

An Option to modify the behaviour of the Parser.

func CaseInsensitive ¶

func CaseInsensitive(tokens ...string) Option

CaseInsensitive allows the specified token types to be matched case-insensitively.

func Elide ¶

func Elide(types ...string) Option

Elide drops tokens of the specified types.

func Lexer ¶

func Lexer(def lexer.Definition) Option

Lexer is an Option that sets the lexer to use with the given grammar.

func Map ¶

func Map(mapper Mapper, symbols ...string) Option

Map is an Option that configures the Parser to apply a mapping function to each Token from the lexer.

This can be useful to eg. upper-case all tokens of a certain type, or dequote strings.

"symbols" specifies the token symbols that the Mapper will be applied to. If empty, all tokens will be mapped.

func Unquote ¶

func Unquote(types ...string) Option

Unquote applies strconv.Unquote() to tokens of the given types.

Tokens of type "String" will be unquoted if no other types are provided.

func Upper ¶

func Upper(types ...string) Option

Upper is an Option that upper-cases all tokens of the given type. Useful for case normalisation.

func UseLookahead ¶

func UseLookahead(n int) Option

UseLookahead allows branch lookahead up to "n" tokens.

If parsing cannot be disambiguated before "n" tokens of lookahead, parsing will fail.

Note that increasing lookahead has a minor performance impact, but also reduces the accuracy of error reporting.

type ParseOption ¶

type ParseOption func(p *parseContext)

ParseOption modifies how an individual parse is applied.

func AllowTrailing ¶

func AllowTrailing(ok bool) ParseOption

AllowTrailing tokens without erroring.

That is, do not error if a full parse completes but additional tokens remain.

type Parseable ¶

type Parseable interface {
	// Parse into the receiver.
	//
	// Should return NextMatch if no tokens matched and parsing should continue.
	// Nil should be returned if parsing was successful.
	Parse(lex *lexer.PeekingLexer) error
}

The Parseable interface can be implemented by any element in the grammar to provide custom parsing.

type Parser ¶

type Parser struct {
	// contains filtered or unexported fields
}

A Parser for a particular grammar and lexer.

func Build ¶

func Build(grammar interface{}, options ...Option) (parser *Parser, err error)

Build constructs a parser for the given grammar.

If "Lexer()" is not provided as an option, a default lexer based on text/scanner will be used. This scans typical Go- like tokens.

See documentation for details

func MustBuild ¶

func MustBuild(grammar interface{}, options ...Option) *Parser

MustBuild calls Build(grammar, options...) and panics if an error occurs.

func (*Parser) Lex ¶

func (p *Parser) Lex(r io.Reader) ([]lexer.Token, error)

Lex uses the parser's lexer to tokenise input.

func (*Parser) Lexer ¶

func (p *Parser) Lexer() lexer.Definition

Lexer returns the parser's builtin lexer.

func (*Parser) Parse ¶

func (p *Parser) Parse(r io.Reader, v interface{}, options ...ParseOption) (err error)

Parse from r into grammar v which must be of the same type as the grammar passed to participle.Build().

This may return a participle.Error.

func (*Parser) ParseBytes ¶

func (p *Parser) ParseBytes(b []byte, v interface{}, options ...ParseOption) error

ParseBytes is a convenience around Parse().

This may return a participle.Error.

func (*Parser) ParseFromLexer ¶

func (p *Parser) ParseFromLexer(lex *lexer.PeekingLexer, v interface{}, options ...ParseOption) error

ParseFromLexer into grammar v which must be of the same type as the grammar passed to participle.Build().

This may return a participle.Error.

func (*Parser) ParseString ¶

func (p *Parser) ParseString(s string, v interface{}, options ...ParseOption) error

ParseString is a convenience around Parse().

This may return a participle.Error.

func (*Parser) String ¶

func (p *Parser) String() string

String returns the EBNF for the grammar.

Productions are always upper case. Lexer tokens are always lower case.

type UnexpectedTokenError ¶

type UnexpectedTokenError struct {
	Unexpected lexer.Token
	Expected   string
}

UnexpectedTokenError is returned by Parse when an unexpected token is encountered.

This is useful for composing parsers in order to detect when a sub-parser has terminated.

func (UnexpectedTokenError) Error ¶

func (u UnexpectedTokenError) Error() string

func (UnexpectedTokenError) Message ¶

func (u UnexpectedTokenError) Message() string

func (UnexpectedTokenError) Token ¶

func (u UnexpectedTokenError) Token() lexer.Token

Source Files ¶

api.go context.go doc.go ebnf.go error.go grammar.go map.go nodes.go options.go parser.go stringer.go struct.go

Directories ¶