Add LexKind

[ratmice]

This is a follow up to PR #483 we now have a number of ways in which RegexOptions can be used to alter the lrlex format primarily by modifying the interpretation of regular expressions.

In that PR it had been mentioned doing follow up work adding a LexKind similar to YaccKind

One thing I have been thinking about which is kind of tangential but related to this is issue #349 which delays the time that YaccKind is known until parser reading time. What seems like it should be a trivial refactor, but also a breaking change. I think it ends up a little bit less trivial than I think it first appears since it introduces some new kinds of propositions for directives that current directives doesn't require and the format parsing code has no current way to enforce.

1. YaccKind affects other directives in the file format so presumably it needs to be first before other directives
2. It should also probably be unique in that we don't want multiple *Kinds.

Thus my thoughts have been that it would be good to try and come to some kind of conclusion on #349 as we introduce a LexKind, just to either avoid introducing any new code that could require this same delicate refactoring, or close #349 abandoning the idea of having self describing formats.

Anyhow I'll post some more new thoughts related to that other issue where they belong.

[ltratt]

I think a couple of things are in our favour here:

1. %yacckind would only make sense for Yacc inputs. So we already know that we're dealing with a Yacc-ish format. If we ever support non-Yacc formats, they'll go down a different codepath.
2. We could easily say "%yacckind, if it exists, must be the first declaration in the file. If it exists and is not the first declaration, an error is raised".

Would those be enough to enable progress?

[ratmice]

I put a bunch more comments in #349 some of it overlaps the big issue is backwards compatibility, we currently need to specify the YaccKind not through the builder, but for YaccGrammar::new so there is kind of a chicken-egg problem in my comment over there I tried to get around that by introducing a new variant YaccKind::SelfDescribing. I think that is the biggest impediment to progress.