Skip to content

Commit 385fac3

Browse files
sharwellsharwell
committed
Merge remote-tracking branch 'origin/optimized' into tunnelvisionlabs_master
2 parents a9f7939 + 8965d37 commit 385fac3

File tree

2 files changed

+132
-0
lines changed

2 files changed

+132
-0
lines changed

‎README.md

+4
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@
44

55
**ANTLR** (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files. It's widely used to build languages, tools, and frameworks. From a grammar, ANTLR generates a parser that can build parse trees and also generates a listener interface (or visitor) that makes it easy to respond to the recognition of phrases of interest.
66

7+
## This is a fork
8+
9+
This is the "optimized" fork of ANTLR 4, which contains many features and optimizations not present in the reference release. For information about these features, see [doc/optimized-fork.md](doc/optimized-fork.md).
10+
711
## Build Status
812

913
[![Build Travis-CI Status](https://travis-ci.org/sharwell/antlr4.svg?branch=optimized)](https://travis-ci.org/sharwell/antlr4)

‎doc/optimized-fork.md

+128
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Optimized Fork
2+
3+
The optimized fork of ANTLR 4 is maintained by Sam Harwell at Tunnel Vision Laboratories, LLC. This "bleeding edge" implementation of ANTLR 4 contains numerous features and performance optimizations which are not included in the reference release. In general, these features are not included in the reference release for one or more of the following reasons.
4+
5+
* **Implementation complexity:** Many features of the optimized fork were developed to meet specific concerns found during the use of ANTLR 4 for large-scale commercial applications. In some cases, the implementation of the feature is *extremely* complex, while the target audience likely to benefit from the change is small. The reference release of ANTLR is widely used in both educational scenarios and small- to medium-sized parsing applications where these optimizations are not needed.
6+
* **Incomplete specification and/or implementation:** Some features of the optimized fork were developed to address a specific use case without specific concern to other use cases. These features may or may not work in other use cases.
7+
* **Configuration required:** Some optimizations present in the optimized fork, including some of the most powerful, have the ability to dramatically improve performance for some grammars while reducing performance for others. Optimal use of these features requires manual configuration with respect to a specific grammar, and in some cases knowledge about characteristics of the machine where the application is being executed. While these features are essential for certain applications, including them all in a reference release of ANTLR would be quite overwhelming for new users.
8+
9+
## Features
10+
11+
### Base Context
12+
13+
**Reason for exclusion:** Incomplete implementation (not known to work in all grammars)
14+
15+
This release has the ability to instruct multiple rules in a grammar to use the same context object. This feature was originally intended for use in cases like the following, which is extracted from a [Java 8 grammar](https://github.com/sharwell/antlr4/blob/java8-grammar/tool/test/org/antlr/v4/test/Java.g4) derived from the language specification (many uses of the feature are seen in this grammar).
16+
17+
```antlr
18+
primitiveType
19+
: annotation* numericType
20+
| annotation* 'boolean'
21+
;
22+
23+
unannPrimitiveType
24+
options { baseContext=primitiveType; }
25+
: numericType
26+
| 'boolean'
27+
;
28+
```
29+
30+
In the above example, the rule `unannPrimitiveType` does not allow annotations, but the parse tree will still contain `PrimitiveTypeContext`
31+
32+
### Automated left factoring
33+
34+
**Reason for exclusion:** Incomplete implementation (not known to work in all grammars)
35+
36+
This release has the ability to automatically left-factor a named rule reference from alternatives of a rule. This feature was created a few releases into GoWorks in an effort to reduce memory consumption by reducing the lookahead requirements for a specific rule in the grammar. Many features of GoWorks were already written with a specific parse tree shape in mind, and changing the shape of the parse tree in order to manually left factor the offending rule would be both time-consuming and risky for introducing bugs. To address this, I created a parse-tree-preserving grammar transformation which left-factors a rule without changing the shape of the final parse tree. GoWorks [uses the feature](https://github.com/tunnelvisionlabs/goworks/blob/5543a633dfc0d9b0e2ab407445a82bc70b24f100/goworks.editor/src/org/tvl/goworks/editor/go/parser/GoParser.g4#L435-L444) to left factor its *expression* rule out of the *simpleStmt* rule.
37+
38+
```antlr
39+
simpleStmt
40+
@version{1}
41+
@leftfactor{expression}
42+
: shortVarDecl
43+
| sendStmt
44+
| incDecStmt
45+
| assignment
46+
| expressionStmt
47+
| emptyStmt
48+
;
49+
```
50+
51+
> :warning: While the parse tree shape will not be changed for *correct* input, it is possible that use of the automatic left factoring functionality could change the behavior in error handling scenarios. Specifically, it may be possible for erroneous input to result in parse trees where nodes are placed in locations that appear impossible according to the grammar.
52+
53+
### Indirect left recursion elimination
54+
55+
**Reason for exclusion:** Incomplete implementation (not known to work in all grammars)
56+
57+
This release has the ability to eliminate indirect left recursion (reported by the reference release as "mutual" left recursion). This feature is an early expansion of the ability to automatically left factor a grammar.
58+
59+
### Backwards compatibility
60+
61+
**Reason for exclusion:** Implementation complexity (quite constraining to the benefit of only a few users)
62+
63+
The optimized fork releases have a stronger emphasis on preserving compatibility. Unlike the reference release, grammars typically do not need to be regenerated when upgrading the runtime. However, we still recommend that grammars be generated using the new version as it may improve performance or the available features at runtime.
64+
65+
### `@Nullable` and `@NotNull` annotations
66+
67+
**Reason for exclusion:** Implementation complexity
68+
69+
### Rule versioning
70+
71+
**Reason for exclusion:** Implementation complexity
72+
73+
The optimized fork includes the [rule versioning](https://github.com/sharwell/antlr4/wiki/Rule-Versioning) feature.
74+
75+
## Optimizations
76+
77+
### Improved SLL termination condition
78+
79+
**Reason for exclusion:** Implementation complexity
80+
81+
This release of ANTLR 4 uses a stronger termination condition for SLL prediction. In some cases it is possible for this implementation to detect an SLL conflict with fewer symbols of lookahead than the reference release. In general, this change would not be observable. However, aside from the performance benefit it is possible for the shorter lookahead to allow for better error handling in some edge cases.
82+
83+
### Full context DFA
84+
85+
**Reason for exclusion:** Implementation complexity, unproven theoretical impact on algorithmic complexity of ALL(*)
86+
87+
The reference release of ANTLR 4 only uses a DFA for local-context prediction (SLL). The optimized fork expands on that by allowing the use of a DFA for full-context prediction (LL) as well.
88+
89+
> :bulb: This optimization is disabled by default. It can be enabled by setting `ParserATNSimulator.enable_global_context_dfa` to true:
90+
>
91+
> ```java
92+
> Parser myParser = new MyParser(input);
93+
> myParser.getInterpreter().enable_global_context_dfa = true;
94+
> ```
95+
>
96+
> :warning: This feature can substantially increase the memory consumed by the DFA. For grammars and applications that rarely need to use full context prediction, especially in combination with two-stage parsing, the overhead of this feature may not provide gains that justify its use. I recommend leaving this feature disabled initially, and only experiment with it if you find that other options do not provide an acceptable level of performance. With that said, there are multiple known grammars which are practically unusable without this feature.
97+
98+
### Tail call elimination
99+
100+
**Reason for exclusion:** Incomplete specification (work in progress)
101+
102+
The optimized fork of ANTLR compacts `PredictionContext` instances associated with states in the DFA by eliminating certain unnecessary return states. As an added bonus, this feature reduced overall DFA storage requirements by allowing `DFAState` instances to be shared in scenarios where the reference release of ANTLR believes the states to be distinct.
103+
104+
> :bulb: This feature includes a configuration option `ParserATNSimulator.tail_call_preserves_sll`, which has a default value of `true`. Before enabling this feature, be aware of the following advantages and disadvantages.
105+
>
106+
> * Advantages of `tail_call_preserves_sll=true` (the default)
107+
> * Preserves maximum accuracy of `PredictionMode.SLL`. When the setting is false, it is possible (varies by grammar and input) for `PredictionMode.SLL` to report a parse error even though the input successfully parses when the setting is true.
108+
> * Minimizes lookahead of SLL decisions. When the setting is false, it is possible (varies by grammar and input) for SLL prediction - which is enabled even for `PredictionMode.LL` - to increase the lookahead. In one case it was observed that overall performance was degraded due to the impact of this.
109+
> * Advantages of `tail_call_preserves_sll=false`
110+
> * Reduces DFA size. For cases where two-stage parsing is used (so reduced accuracy of SLL mode is acceptable) and lookahead doesn't hit a pathological case, explicitly setting `tail_call_preserves_sll` to false can substantially reduce the size of the DFA.
111+
112+
### DFA edge optimization
113+
114+
**Reason for exclusion:** Implementation complexity (the reference release uses a subset of this optimization)
115+
116+
The optimized fork uses several different map implementations based on the number of outgoing edges held in a DFA state. This feature minimizes the size of DFA states, especially in infrequently used areas of the DFA.
117+
118+
### ATN configuration optimization
119+
120+
**Reason for exclusion:** Implementation complexity (most small- and medium-sized applications don't run into DFA-related memory problems)
121+
122+
The optimized fork uses several `ATNConfig` classes to reduce the size of the DFA. For configurations returning default values from most properties, a small `ATNConfig` instance is used. The larger types are only used for configurations that need to represent non-default values from one or more methods. As a simple example, configurations appearing in the lexer DFA need to store a few more fields than configurations appearing in the parser DFA. By removing these fields from `ATNConfig` instances used in the parser DFA, some applications observe marked reductions in the memory overhead for the parser DFA (we've seen 20MiB or more for large applications).
123+
124+
### Prediction context optimization
125+
126+
**Reason for exclusion:** Implementation complexity
127+
128+
The optimized release uses an exact implementation for merging `PredictionContext` instances. In some cases, the reference release produces prediction context graphs which are not fully reduced (maximum sharing of nodes in the graph). The algorithm used by the optimized fork implements an exact merge for these contexts, so the `PredictionContext` instances appearing in the DFA cache are fully reduced.

0 commit comments

Comments
 (0)