fix: properly parse octal escape sequences

[ayazhafiz]

Closes #1480.

Currently, octal escape sequences are parsed incorrectly because the
sequence is evaluated only partially. The "\" delimiter in an octal
sequence like "\101" is already siphoned off during escape sequence
matching, so it is enough to simply parse the match of the numeric
sequence "101".

[ayazhafiz]

I think this one should be an error, since \88 is not a valid octal number.

You're right. Should be fixed now.

Right now the test method is rather crude, we check the string representation of the entire AST. Please let me know if you'd prefer a tree visitor or tests specific to the tokenizer rather than the parser; the latter may be more appropriate in this case.

[blerner]

Mm, that is slightly awkward, yes. Seems to me that an easier way to do this would be a pair of tests: one that only tested parsing and only tested string literals (rather than let-bound string literals), and a second one that evaluated the string literal and checked that it had the desired value (since Pyret string expressions evaluate to JS strings, in the end).

[ayazhafiz]

@blerner is there already infrastructure for testing the tokenizer? If not, is there documentation of setting up new test files? I would like to get some idea of how you all prefer to do that.

[blerner]

Yup: look higher in this file, at

pyret-lang/tests/parse/parse.js

Lines 115 to 131 in 7656aa3

	it("should have tight lexical extents for all tokens", function() {
	lex("");
	expect(numToks).toBe(1);
	expect(toks[0].name).toBe("EOF");
	for (var ws1 = 0; ws1 < ws.length; ws1++) {
	for (var i = 0; i < all_toks.length; i++) {
	for (var ws2 = 0; ws2 < ws.length; ws2++) {
	var str = "" + ws[ws1] + all_toks[i] + ws[ws2]
	lex(str);
	test(numToks, 2, str, toks) &&
	testPos(toks[0], all_toks[i], str, toks) &&
	test(toks[1].name, "EOF", str, toks);
	}
	}
	}
	console.log("Finished single token tests");
	});

. Basically, call lex and then examine the toks array.

[ayazhafiz]

@blerner please take a look

[blerner]

Have you run make parse-test on this? (The parse tests don't run automatically in make test.)

[ayazhafiz]

Have you run make parse-test on this? (The parse tests don't run automatically in make test.)

Yep.

[blerner]

Oh my bad; I misread the test code -- I got concerned by the single-quotes in the string literals, since Pyret always prints its string literals with double-quotes...but you're checking the output of the lexer, which preserves the quotes of the actual token.

Yep, this looks good, thanks!

[jpolitz]

Thanks @blerner, @ayazhafiz !