Some notes on pattern matching

When I first designed Language 84, I included a traditional pattern matching construct that looks like this:

Match numbers {
| 'cons.{n _} n
| 'nil 0
}

That expression evaluates to 0 if the numbers list is empty and evaluates to the first number in the list if numbers is not empty.

More recently, I added an alternative way to use pattern matching. It looks like this:

If [Pattern 'cons.{n _} Matches numbers]
    n
    0

That expression does almost the same thing as the previous example.

(It's different in that it only checks for the cons case. Normally, the only other possibility is nil but the code here only distinguishes between cons and not-cons.)

It is equivalent to the following expression:

Match numbers {
| 'cons.{n _} n
| _ 0
}

(The _ is used as a match-anything, bind-no-variables pattern.)

In Language 84, [Pattern 'cons.{n _} Matches numbers] is an expression and can be used anywhere a boolean value is expected.

But as the example shows, if the expression is in a control flow predicate context, it also binds variables appearing in the pattern for use in the branch taken when the condition evaluated to True.

Something interesting happens when you start using this style with And and Or operators.

If (And [Pattern 'sym.t Matches token] (STRING.equal s t))
    [TRUE_BRANCH]
    [FALSE_BRANCH]

In that expression, the t variable is bound both in the (STRING.equal s t) expression and in the [TRUE_BRANCH] of the If conditional.

Translating (naively) to the other pattern matching style, you get the following:

Match token {
| 'sym.t
    If (STRING.equal s t)
        [TRUE_BRANCH]
        [FALSE_BRANCH]
| _
    [FALSE_BRANCH]
}

Note that [FALSE_BRANCH] appears twice. To correctly expand the new style into the old style, I'll need to introduce join points into my intermediate languages to avoid code duplication. I also have to overcome the problem that the naive expansion shown above incorrectly creates a binding for t in the first [FALSE_BRANCH], where 'sym.t matches token but (STRING.equal s t) evaluates to False.

I know that Rust has something similar in its if let construct and I did have that in mind as I worked on this feature even though I haven't used Rust except to write a couple of demo programs to explore its features.

Are you familiar with similar features in a language you use? Is it something that works well? Are there issues to watch out for? I'm curious because I haven't really used a language that works like this so I'm somewhat wary about including it in my own language design.

One thing that worries me is that it feels somehow non-compositional. If you look at the above example that starts If (And ...), you can see that the condition within the If expression affects the translation of the enclosing If expression itself.

For fun, here's a curious function definition I wrote using this new style.

Define (any items pred)
    Iterate items
        (And [Pattern 'cons.{item items} Matches items]
            (Or (pred item) (Continue items)))

20 comments · 100% upvoted · other discussions

comments

scottmcmrust

Rust's if let has turned out to be a bit unfortunate, because it doesn't compose elegantly. That leads to things like the matches! macro, at which point one can just if matches!(x, Some(y)) instead of if let Some(y) = x.

Infix Matches like you have seems like a good idea; it's been brought up for Rust as well. (Though in the other order for parsing reasons, like x is Some(y).) Though beware variable scoping from it -- where they should be in-scope and usable aren't obvious questions.

WalkerCodeRanger

Can you expand on what you mean when you say if let "it doesn't compose elegantly." Or, link to something that explains it. I don't know that I have heard about those issues.

scottmcmrust

Well, if x > 0 && let Some(y) = z or if let Some(y) = x && y.is_power_of_two() don't work naturally, which gets frustrating. (And they, in some sense, can't work, since if let true = y || b does work, and has a different parenthesization than the preceeding examples want, as it's if let true = (y || b).)

And making let be an expression is also awkward, since let x = 4; on its own is the syntax for bindings, which are supposed to be irrefutable patterns.

It would have been much nicer to have a "real" expression for x is Some(y) from the beginning.

More discussion in https://github.com/rust-lang/rfcs/pull/2497

DerpDick90

cough bow gaping governor whistle impossible plate payment existence crawl

This post was mass deleted and anonymized with Redact

matthieum

Interesting, since the if let syntax is Rust was copied from Swift :p

matthieum

I find the is expression quite elegant here: it reads "naturally" as plain English, and it is expected to have a boolean result making it composable.

The compiler requires some boolean logic to be able to handle:

if x is Some(x') and (x' is Some(y') or y is Some(y')) {

where y' is always defined after evaluating the clause, but would not be if the and was an or... however handling conjunctives/disjunctives seems relatively painless.

moosekk

F# has User-defined match patterns, which are a compositional way [at least, within the structure of the match expression] to structure the check. However, AFAIK they are used for syntactic clarity and function more like a chain of if/else rather than lifting the `Pattern 'sym.t Matches token` part into a common check.

A 3-part demonstration of F#'s pattern-matching:

Part 1: the builtin feature: string pattern matching

match token with
   | Sym "foo" -> true_expr               // if you don't need t
   | Sym ("foo" as t) -> true_expr_of t   // if you need t
   | _ -> false_expr

Part 2: when-conditions on matches

match token with
   | Sym t when t = "foo" -> true_expr_of t
   | _ -> false_expr

Part 3: custom match pattern

Affectionately called "banana brackets", F#'s custom match functions combine parameters and return values on the match line. Below, in the SymbolMatch "foo" t ->`line, "foo" is the parameter s to SymbolMatch and t is the returned value Some(s)

let (|SymbolMatch|_|) s token = 
   if token = Sym s
      Some s
   else
      None

match token with
   | SymbolMatch "foo" t -> true_expr_of t
   | _ -> false_expr
ericbb [OP]

Those look quite useful. I haven't implemented any of them yet but I've considered them before. The third example seems especially interesting to me. It strikes me as an embedded miniKanren-like language to use for pattern matching within a host language.

ghkbrew

But as the example shows, if the expression is in a control flow predicate context, it also binds variables appearing in the pattern for use in the branch taken when the condition evaluated to True

This is the bit I would expect you and your users to stumble over the most. It's a pretty unique feature as far as I know. The closest feature I can think of is flow typing in some languages, but even that doesn't introduce new bindings.

The scoping of those bound variables seems like it will be complex.

Are they bound in other parts of the conditional expression? Which parts?

What about matches in negative contexts? e.g.

If (Not ['some.x Matches item])
   Print x

Can they be nested under user-defined functions?

Def If_Not cond if_result else_result = 
  If ( Not cond )
    if_result
    else_result

If (If_Not ['some.x Matches item] True x )
   ...
ericbb [OP]

Exactly. These are the kinds of concerns I have too.

The scoping of those bound variables seems like it will be complex.

There is definitely some complexity (that is unique compared to other binding features of the language) but I think it can be made precise:

Each control flow operator that dispatches based on truth values has its own interactions with bindings created by embedded pattern match forms:

If TEST THEN ELSE

Variables bound in TEST are in scope in THEN and not in scope in ELSE.

(And TEST1 TEST2 TEST3 ...)

Variables bound in TEST[m] are in scope in TEST[n] for all n > m.

When one control flow form is nested within another, bindings flow outward (with respect to nesting) based on logical implication, in some sense. (This part is harder to state clearly.)

The Or case has a wrinkle of its own: if there is a pattern matching subexpression, then all subexpressions must be pattern matching expressions that bind the same variables. It's pretty complicated but I haven't run into a use for that case yet. (Mostly, the composed form I find uses for is If (And ...).)

What about matches in negative contexts?

If (Not ['some.x Matches item])
    Print x

I think the answer there has to be that x would not be bound in the Print x expression. If a pattern doesn't match, then there are no bindings to introduce. For what it's worth, the syntax for your example would typically be:

When ![Pattern 'some.x Matches item] {
    (print x)
}

The ! operator is basically just a function and not really a control flow operator in the way that If, And, and Or are. It's like ! versus && and || in C, in case you are familiar with C.

Can they be nested under user-defined functions?

That is a really good question and I think it relates to "custom match pattern" of F# mentioned in another comment here. For now, I have no support for such constructs but it may be possible to find a workable design based on that idea, though I think the "user-defined function" would have to be more of a miniKanren-style "user-defined relation".

Edit: Slight word choice changes (s/bound/in scope/).

ghkbrew

I agree that these are pretty much the only sensible answers to those questions. But at that point you're basically adding a 3rd class of syntax (besides expression and patterns) with it's own set of rules/restrictions and means of abstraction.

I wonder if it's worth the complexity, given that for the most part it's macro-translatable in to match expressions.

What about a slightly different alternative where you just extend the pattern syntax with And, Or, and Not operators. And maybe add a rust-style if let statement.

ericbb [OP]

But at that point you're basically adding a 3rd class of syntax (besides expression and patterns) with it's own set of rules/restrictions and means of abstraction.

Yes. I think that that's the right way to think about it. And, in fact, once you look at it that way, maybe you get compositionality back because now If is defined in terms of composition of two expression branches with one thing of this new class, whose composition rules are not the composition rules of expressions.

I wonder if it's worth the complexity, given that for the most part it's macro-translatable in to match expressions.

Yeah, it could be that it's not worth the complexity. My experience so far is that I do appreciate having these new forms available and I enjoy using them. They seem like fairly natural extensions of the existing concepts in the language. Still, it's important that they carry their weight and don't have nasty corner cases or bad interactions with other features.

What about a slightly different alternative where you just extend the pattern syntax with And, Or, and Not operators. And maybe add a rust-style if let statement.

It's worth a thought but I suspect that going that route might lead back to what I already have. (?)

Edit: To expand on this last point: One reason not to use if let is that Language 84 has a number of different boolean-based conditional operators and I'd like this feature to work with all of them (orthogonally) rather than just with If. Here are the operators (everywhere you see TEST, that's where I'd like to be able to use patterns):

If TEST THEN ELSE

(And TEST1 TEST2 ...)

(Or TEST1 TEST2 ...)

Cond {
| TEST1 BLOCK1
| TEST2 BLOCK2
...
}

When TEST { STATEMENTS }

Begin Cond {
| TEST1 STATEMENTS1
| TEST2 STATEMENTS2
...
}
ericbb [OP]

Negation is more interesting than I had realized. Example:

Define (all items pred)
    Iterate items
        (Or ![Pattern 'cons.{item items} Matches items]
            (And (pred item) (Continue items)))

One might reasonably expect that definition to be well-formed.

The general pattern is (Or ... !TEST1 ... TEST2 ...). In evaluation of TEST2 we know that TEST1 evaluated to True and, therefore, any bindings created in TEST1 should be in scope within TEST2.

Edit: Similarly, in If !TEST THEN ELSE, bindings created in TEST should be in scope within ELSE.

choeger

I think you cannot let the binding depend on the value of some computation. Negation could be replaced by an arbitrary function and you would have no clue as to where you should put your bindings. I think only a dedicated "match form" of if would if you restrict it to an understandable form, something like:

IF xs IS [n] AND n > 5 THEN

Here, the match is syntactically special and it binds to everything beyond the AND and into the body.

You can also have

IF x IS NOT [n] OR n < 5 THEN ... ELSE

Where n is bound to everything beyond the OR and below the ELSE.

In the case of:

IF x IS NOT [n] AND ...

The n cannot be bound behind the AND. The same holds for the IS/OR case, I think.

ericbb [OP]

I think you cannot let the binding depend on the value of some computation. Negation could be replaced by an arbitrary function and you would have no clue as to where you should put your bindings.

Yes, I think that's exactly right. I didn't write it in my comment but I was just starting to realize that I'll need to have a syntactic negation operator in order to make things work the way they really ought to. Interestingly, it only needs to be syntactic for matching predicates, which means that it can be integrated into that part (as it is in your syntax sketch).

To revise my example, it could become something like:

Define (all items pred)
    Iterate items
        (Or [items Is Not 'cons.{item items}]
            (And (pred item) (Continue items)))
shponglespore

The pcase macro in Emacs Lisp (doc) does something similar, but it works by introducing a more powerful pattern-matching construct rather than adding binding semantics to boolean expressions.

Your token example would look something like this with pcase:

(pcase token
  (((and `(sym ,t)
         (guard (equal s t))))
   [TRUE_BRANCH])
  (_
   [FALSE_BRANCH]))

The and in the code above is part of the pcase expression language, not the ordinary Elisp and operator. The guard form allows an ordinary boolean expression to be embedded into the expression language, so equal is just an ordinary function call. The scope of bindings is complicated but intuitive; for example, a variable bound in one branch of an or subexpression doesn't exist outside that subexpression, but a variable bound in both branches does.

abecedarius

Are you familiar with similar features in a language you use?

http://erights.org/elang/kernel/MatchBindExpr.html

This depends on E's reference semantics -- broken references in particular.

ericbb [OP]

That does seem closely related but I wonder why E uses broken references instead of just leaving the variables out-of-scope in branches that are reached when the condition evaluates to false.

abecedarius

It's simpler, given that broken references were already in the design.

gergoerdi