package stateful

import "github.com/alecthomas/participle/lexer/stateful"

Package stateful defines a nested stateful lexer.

This lexer is based heavily on the approach used by Chroma (and Pygments).

The lexer is a state machine defined by a map of rules keyed by state. Each rule is a named regex and optional operation to apply when the rule matches.

As a convenience, any Rule starting with a lowercase letter will be elided from output.

Lexing starts in the "Root" group. Each rule is matched in order, with the first successful match producing a lexeme. If the matching rule has an associated Action it will be executed. The name of each non-root rule is prefixed with the name of its group to yield the token identifier used during matching.

A state change can be introduced with the Action `Push(state)`. `Pop()` will return to the previous state.

To reuse rules from another state, use `Include(state)`.

As a special case, regexes containing backrefs in the form \N (where N is a digit) will match the corresponding capture group from the immediate parent group. This can be used to parse, among other things, heredocs.

See the example and tests in this package for details.

Index ¶

• type Action
• func Pop() Action
• func Push(state string) Action
• type ActionFunc
• type Definition
• func New(rules Rules) (*Definition, error)
• func NewSimple(rules []Rule) (*Definition, error)
• func (d *Definition) Lex(r io.Reader) (lexer.Lexer, error)
• func (d *Definition) Symbols() map[string]rune
• type Lexer
• func (l *Lexer) Next() (lexer.Token, error)
• type Rule
• func Include(state string) Rule
• func Return() Rule
• type Rules
• type RulesAction

Examples ¶

• New

Types ¶

type Action ¶

type Action interface {
	// contains filtered or unexported methods
}

A Action is applied when a rule matches.

func Pop ¶

func Pop() Action

Pop to the previous state.

func Push ¶

func Push(state string) Action

Push to the given state.

The target state will then be the set of rules used for matching until another Push or Pop is encountered.

type ActionFunc ¶

type ActionFunc func(*Lexer, []string) error

ActionFunc is a function that is also a Action.

type Definition ¶

type Definition struct {
	// contains filtered or unexported fields
}

Definition is the lexer.Definition.

func New ¶

func New(rules Rules) (*Definition, error)

New constructs a new stateful lexer from rules.

{
	type Terminal struct {
		String *String `  @@`
		Ident  string  `| @Ident`
	}

	type Expr struct {
		Left  *Terminal `@@`
		Op    string    `( @Oper`
		Right *Terminal `  @@)?`
	}

	type Fragment struct {
		Escaped string `(  @Escaped`
		Expr    *Expr  ` | "${" @@ "}"`
		Text    string ` | @Char)`
	}

	type String struct {
		Fragments []*Fragment `"\"" @@* "\""`
	}

	def, err := New(interpolatedRules)
	if err != nil {
		log.Fatal(err)
	}
	parser, err := participle.Build(&String{}, participle.Lexer(def))
	if err != nil {
		log.Fatal(err)
	}

	actual := &String{}
	err = parser.ParseString(`"hello ${user + "??"}"`, actual)
	if err != nil {
		log.Fatal(err)
	}
	repr.Println(actual)
}

func NewSimple ¶

func NewSimple(rules []Rule) (*Definition, error)

NewSimple creates a new stateful lexer with a single "Root" state.

func (*Definition) Lex ¶

func (d *Definition) Lex(r io.Reader) (lexer.Lexer, error)

func (*Definition) Symbols ¶

func (d *Definition) Symbols() map[string]rune

type Lexer ¶

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer implementation.

func (*Lexer) Next ¶

func (l *Lexer) Next() (lexer.Token, error)

type Rule ¶

type Rule struct {
	Name    string
	Pattern string
	Action  Action
}

A Rule matching input and possibly changing state.

func Include ¶

func Include(state string) Rule

Include rules from another state in this one.

func Return ¶

func Return() Rule

Return to the parent state.

Useful as the last rule in a sub-state.

type Rules ¶

type Rules map[string][]Rule

Rules grouped by name.

type RulesAction ¶

type RulesAction interface {
	// contains filtered or unexported methods
}

RulesAction is an optional interface that Actions can implement.

It is applied during rule construction to mutate the rule map.

Source Files ¶

stateful.go