Page MenuHomePhabricator

Investigate templatizing value storage in Token
Open, p3: NextPublic


Currently the Token class stores its literal content as a onestring named lexeme. The Java implementation of Lox included a bit of extra code that would parse numbers and store them as an appropriate number type (ie, int or float) using an Object class variable. I didn't see an easy way to set this up in C++ when I was creating the tokenizing code but it might be nice to have, maybe a template that either uses an alternate data type for lexeme in numbers, or instantiates a value variable in number type Tokens as needed (in which case a decision would have to be made about the default value of a non-number token).

Alternatively, we could leave creating a value out of a Token's lexeme to the Parser.

There is a relevant TODO note at line 89 of tokens.hpp, but part of the actual relevant code - if we go with the templatized Token object - would go in the Lexer::number() function in lexer.cpp, other than changes to the Token class itself to account for templates and constructor changes.


Task Type
Not Set
Proposed Urgency
g2: Minor
f2: Street
r3: Moderate
Volatility (Caught At)
Not a Bug
Not a Bug/Unknown

Event Timeline

ardunster triaged this task as p3: Next priority.Mar 15 2021, 2:34 PM
ardunster created this task.
jcmcdonald moved this task from Restricted Project Column to Backlog on the Ratscript [Project] board.Jun 19 2021, 9:51 AM
jcmcdonald edited projects, added Ratscript [Project]; removed Unknown Object (Project).