Add line numbers to AST nodes

[sdeal]

I couldn’t find a way to access code comments or line numbers through BOA (I assume comments are simply ignored by the scanner and not even passed to the parser), because there are not part of the AST. Adding line numbers to each node (start line, end line) would help a lot, because then one could simply checkout the files from GitHub and access the code comments or any other information that is not included in the BOA dataset. Furthermore, line numbers would also facilitate to align information mined with BOA with information mined with other tools.

[psybers]

Just an FYI - Boa had support for positional information on the AST nodes from day one. However, we found that keeping that information made the dataset substantially larger (and at the time, too big for our servers to handle) and so we disabled this.

See:

compiler/src/proto/ast.proto

Line 500 in ef92bab

message PositionInfo {

See:

compiler/src/java/boa/datagen/util/JavaVisitor.java

Line 102 in ef92bab

public void preVisit(ASTNode node) {

See:

compiler/src/java/boa/datagen/util/JavaVisitor.java

Line 614 in ef92bab

protected void buildPosition(final ASTNode node) {

Comments are actually in there too, but were stripped again for similar reasons.

See:

compiler/src/java/boa/datagen/util/JavaVisitor.java

Line 143 in ef92bab

public boolean visit(BlockComment node) {

We might revisit these two points in the future and see if we are able to incorporate them now. We have picked up a few extra tricks over the years that might allow this.