Page MenuHomePhabricator
Differential D383

Lexer/Tokenizer
ClosedPublic
Actions

Authored by ardunster on Feb 5 2021, 5:43 PM.

Details

Reviewers
jcmcdonald
wdede
memateo
mahussain
Maniphest Tasks
T1430: Setup Ratscript to use Goldilocks tests
T1378: Scanner/Lexer
Commits
rR64bdd6c9bcec: Lexer/Tokenizer
Summary

Related to tasks T1378 and T1403, setting up tokenizing for math inputs.

Test Plan

Goldilocks 1 tests integrated into console.

Revert Plan

na

Diff Detail

Repository
rR Ratscript
Branch
T1378
Lint
Lint Passed

Event Timeline

ardunster created this revision.Feb 5 2021, 5:43 PM
Herald added projects: Programming [Dept], Ratscript [Project]. · View Herald TranscriptFeb 5 2021, 5:43 PM
Restricted Application completed remote builds in Restricted Buildable.Feb 5 2021, 5:43 PM
ardunster requested review of this revision.Feb 5 2021, 5:43 PM
ardunster updated this revision to Diff 1797.Feb 10 2021, 5:10 PM
  • Rename token_types, work on lexer
Restricted Application completed remote builds in Restricted Buildable.Feb 10 2021, 5:10 PM
ardunster updated this revision to Diff 1799.Feb 11 2021, 5:20 PM
  • Work on lexer logic, tokens, models, comments; change int to uint
Restricted Application completed remote builds in Restricted Buildable.Feb 11 2021, 5:20 PM
ardunster updated this revision to Diff 1812.Feb 17 2021, 3:37 PM
  • Additional work on Lexer, definitions, models
Restricted Application completed remote builds in Restricted Buildable.Feb 17 2021, 3:37 PM
ardunster updated this revision to Diff 1813.Feb 19 2021, 2:53 PM
  • rename Models to Model, continue work on lexer, add hashmap to token types
Restricted Application completed remote builds in Restricted Buildable.Feb 19 2021, 2:53 PM
ardunster updated this revision to Diff 1847.Feb 26 2021, 1:59 PM
  • Merge devel
  • Update clang-format
  • Continue work on lexer
  • Lexer first pass mostly done, start work on console and file interface
Restricted Application completed remote builds in Restricted Buildable.Feb 26 2021, 1:59 PM
ardunster mentioned this in D397: Update build system.Feb 26 2021, 2:03 PM
ardunster updated this revision to Diff 1849.Feb 27 2021, 12:05 PM
  • Remove unused files
  • Update gitignore and change directory names to prepare for merge
  • Merge build system update to 3.1.0
Restricted Application completed remote builds in Restricted Buildable.Feb 27 2021, 12:05 PM
ardunster updated this revision to Diff 1853.Mar 2 2021, 3:48 PM
  • Merge build system 3.2.0 changes
  • Rearrange code out of console and into ratscript files
Restricted Application completed remote builds in Restricted Buildable.Mar 2 2021, 3:48 PM
ardunster added reviewers: No Test [Control], No Build [Control].Mar 2 2021, 3:48 PM
ardunster removed reviewers: No Test [Control], No Build [Control].
ardunster added subscribers: No Test [Control], No Build [Control].
ardunster updated this revision to Diff 1855.Mar 5 2021, 3:14 PM
  • Fix multiple definition errors by moving Model namespace to lexer
  • Work on getting console input working
Restricted Application completed remote builds in Restricted Buildable.Mar 5 2021, 3:14 PM
ardunster added a comment.Mar 5 2021, 3:16 PM

TODO next: figure out how to stringify Tokens, figure out how to convert onestrings to std::string and get hashmap working.

ardunster updated this revision to Diff 1857.Mar 6 2021, 1:14 PM
  • Set up keyword hashmap and overload stream insertion operator for TokenType
Restricted Application completed remote builds in Restricted Buildable.Mar 6 2021, 1:14 PM
ardunster updated this revision to Diff 1861.Mar 8 2021, 3:51 PM
  • Stringify tokens, work on debug in lexer
Restricted Application completed remote builds in Restricted Buildable.Mar 8 2021, 3:51 PM
ardunster mentioned this in T1462: IOCat shut_up() not working.EditedMar 8 2021, 4:06 PM

Todo: Fix IOCat commands if working properly, debug line numbers, modify for testing and write tests. Also figure out whether comments are parsing wrong or if I was confused, and fix.

jenkins added a comment.Mar 9 2021, 12:44 PM

Build has FAILED

Building Ratscript on mpm-bionic-000294d0fnwi8 as build 4.

Link to build: https://jenkins.mousepawmedia.net:8449/job/ratscript_build/4/
See console output for more information: https://jenkins.mousepawmedia.net:8449/job/ratscript_build/4/console

jenkins added a comment.Mar 9 2021, 12:48 PM

Build has FAILED

Building Ratscript on mpm-bionic-000297qa31j3x as build 5.

Link to build: https://jenkins.mousepawmedia.net:8449/job/ratscript_build/5/
See console output for more information: https://jenkins.mousepawmedia.net:8449/job/ratscript_build/5/console

jenkins added a comment.Mar 9 2021, 12:51 PM

Build is green

Building Ratscript on mpm-bionic-000299fb4s19n as build 6.

See https://jenkins.mousepawmedia.net:8449/job/ratscript_build/6/ for more details.

ardunster removed subscribers: No Build [Control], No Test [Control].Mar 9 2021, 12:52 PM
jenkins added a comment.Mar 9 2021, 1:05 PM

Build is green

Building Ratscript on mpm-bionic-00029k5eboub5 as build 7.
Running tests.

See https://jenkins.mousepawmedia.net:8449/job/ratscript_build/7/ for more details.

jenkins added a comment.Mar 9 2021, 1:37 PM

Build is green

Building Ratscript on mpm-bionic-0002a9a1w0l69 as build 8.

See https://jenkins.mousepawmedia.net:8449/job/ratscript_build/8/ for more details.

ardunster updated this revision to Diff 1862.Mar 9 2021, 3:22 PM
  • Adjusted outer makefile to make console when using make ready
  • Setup test framework, get includes working
  • Overload == and != for Token, work on tests
jenkins added a comment.Mar 9 2021, 3:23 PM

Build is green

Building Ratscript on mpm-bionic-0002chvbpa7wd as build 9.
Running tests.

See https://jenkins.mousepawmedia.net:8449/job/ratscript_build/9/ for more details.

Restricted Application completed remote builds in Restricted Buildable.Mar 9 2021, 3:23 PM
ardunster edited the test plan for this revision. (Show Details)Mar 9 2021, 3:27 PM
ardunster updated this revision to Diff 1863.Mar 10 2021, 2:00 PM
  • Adjust outer.mk
  • Troubleshoot tests, write more tests
jenkins added a comment.Mar 10 2021, 2:01 PM

Build is green

Building Ratscript on mpm-bionic-00035dpz1375f as build 10.
Running tests.

See https://jenkins.mousepawmedia.net:8449/job/ratscript_build/10/ for more details.

Restricted Application completed remote builds in Restricted Buildable.Mar 10 2021, 2:01 PM
ardunster added a comment.Mar 10 2021, 2:02 PM

TODO: Test numbers, non keyword identifiers, newline, comments.

ardunster updated this revision to Diff 1867.Mar 12 2021, 2:30 PM
  • Run format on all cpp and hpp files
  • Clean up formatting and comments
  • Add and troubleshoot numerical test
  • Test non-keyword identifiers and troubleshoot
  • Write test for newlines, add COLON to tokentypes
  • Add tests for comments, moved comment logic and newline logic to functions
  • Add testing for strings and troubleshoot
jenkins added a comment.Mar 12 2021, 2:32 PM

Build is green

Building Ratscript on mpm-bionic-0004v9skzor6a as build 11.
Running tests.

See https://jenkins.mousepawmedia.net:8449/job/ratscript_build/11/ for more details.

Restricted Application completed remote builds in Restricted Buildable.Mar 12 2021, 2:32 PM
ardunster added reviewers: jcmcdonald, wdede, memateo, mahussain.Mar 12 2021, 2:35 PM

I think this is finally ready for review. I know it's super long :( If any of you have time to look over even a file or two at a time, that'd be great! All my tests are passing but I'm insecure about how well I wrote the tests. Totally open to suggestions for more things to test!

ardunster updated this revision to Diff 1869.Mar 13 2021, 12:04 PM
  • Adjust run_prompt to run in loop instead of single run
jenkins added a comment.Mar 13 2021, 12:05 PM

Build is green

Building Ratscript on mpm-bionic-0005ms5dvvvs2 as build 12.
Running tests.

See https://jenkins.mousepawmedia.net:8449/job/ratscript_build/12/ for more details.

Restricted Application completed remote builds in Restricted Buildable.Mar 13 2021, 12:05 PM
mahussain requested changes to this revision.EditedMar 13 2021, 12:28 PM

Looks fine to me

wait i didn't request changes i thought i had accepted it.

This revision now requires changes to proceed.Mar 13 2021, 12:28 PM
mahussain accepted this revision.Mar 13 2021, 1:03 PM
This revision is now accepted and ready to land.Mar 13 2021, 1:03 PM
memateo accepted this revision.Mar 14 2021, 8:05 PM

Looks good!

Closed by commit rR64bdd6c9bcec: Lexer/Tokenizer (authored by ardunster). · Explain WhyMar 15 2021, 1:50 PM
This revision was automatically updated to reflect the committed changes.
ardunster added a commit: rR64bdd6c9bcec: Lexer/Tokenizer.
ardunster mentioned this in T1379: Command Line Console - Link to scanner.Mar 15 2021, 1:57 PM
ardunster mentioned this in T1403: Ratscript: Math parser.Mar 15 2021, 2:03 PM
ardunster mentioned this in T1430: Setup Ratscript to use Goldilocks tests.Mar 15 2021, 2:06 PM
ardunster added a task: T1430: Setup Ratscript to use Goldilocks tests.
ardunster mentioned this in T1373: Parsing and tokenizing basic math operators.Mar 15 2021, 2:27 PM
ardunster mentioned this in T1372: Parsing and tokenizing decimals (number).
ardunster mentioned this in T1371: Parsing and tokenizing integers (int).

Revision Contents
Changeset List

PathSize
console/
23 lines
language/
15 lines
include/
ratscript/
75 lines
lexer_models.hpp
tokentypes.hpp
88 lines
token_types.hpp
tokentypes.hpp
4 lines
63 lines
src/
62 lines
token_types.cpp
tokentypes.cpp
2 lines
15 lines

Diff 1799

View Options

console/main.cpp

Loading...
View Options

language/CMakeLists.txt

Loading...
View Options

language/include/ratscript/lexer.hpp

Loading...
View Options

language/include/ratscript/lexer_models.hpp

Loading...
View Options

language/include/ratscript/token_types.hpp

Loading...
View Options

language/include/ratscript/tokens.hpp

Loading...
View Options

language/include/ratscript/tokentypes.hpp

Loading...
View Options

language/src/lexer.cpp

Loading...
View Options

language/src/token_types.cpp

Loading...
View Options

language/src/tokens.cpp

Loading...
View Options

language/src/tokentypes.cpp

Loading...
Copyright (C) 2019 MousePaw Media. All Rights Reserved. · Community Rules · Terms of Use · Privacy Policy · Terms of Development