Partial lexer implementation #8

wpkelso · 2025-01-14T19:15:38Z

Adds partial support for lexing token chains that include the AND, ADD, LD, LDI, LDR, LEA, ST, STI, STR, and NOT instructions, as well as passing tests for all but NOT. Merging this unblocks working on the assembler frontend, but does not guarantee compatibility with all (indeed most) programs.

Adds strum as a dependency to ease enum matching

src/assembler/lexer.rs

@@ -0,0 +1,327 @@
+use crate::{
+    assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},


src/assembler/lexer.rs

@@ -0,0 +1,327 @@
+use crate::{
+    assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},
+    defs::{LC3Word, RegAddr},


src/assembler/lexer.rs

+use crate::{
+    assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},
+    defs::{LC3Word, RegAddr},
+    instruction::{ADD_OPCODE, AND_OPCODE, ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, ALL_LOAD_OPCODES, ALL_STORE_OPCODES, TRAP_OPCODE, NOT_OPCODE},


src/assembler/lexer.rs

+use crate::{
+    assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},
+    defs::{LC3Word, RegAddr},
+    instruction::{ADD_OPCODE, AND_OPCODE, ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, ALL_LOAD_OPCODES, ALL_STORE_OPCODES, TRAP_OPCODE, NOT_OPCODE},


src/assembler/lexer.rs

+use crate::{
+    assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},
+    defs::{LC3Word, RegAddr},
+    instruction::{ADD_OPCODE, AND_OPCODE, ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, ALL_LOAD_OPCODES, ALL_STORE_OPCODES, TRAP_OPCODE, NOT_OPCODE},


src/assembler/lexer.rs

+use crate::{
+    assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},
+    defs::{LC3Word, RegAddr},
+    instruction::{ADD_OPCODE, AND_OPCODE, ALL_JUMP_OPCODES, BRANCH_OPCODE, JSR_OPCODE, ALL_LOAD_OPCODES, ALL_STORE_OPCODES, TRAP_OPCODE, NOT_OPCODE},


src/assembler/lexer.rs

+            instr: &mut MaybeUnresolvedInstr,
+        ) -> Result<(), anyhow::Error> {
+            if let Token::REGISTER(reg) = token {
+                instr.value |= (LC3Word::from(*reg) << SHIFT);


src/assembler/lexer.rs

+            instr: &mut MaybeUnresolvedInstr,
+        ) -> Result<(), anyhow::Error> {
+            if let Token::REGISTER(reg) = token {
+                instr.value |= (LC3Word::from(*reg) << SHIFT);


src/assembler/lexer.rs

+            instr: &mut MaybeUnresolvedInstr,
+        ) -> Result<(), anyhow::Error> {
+            if let Token::REGISTER(reg) = token {
+                instr.value |= (LC3Word::from(*reg) << SHIFT);


src/assembler/lexer.rs

+            instr: &mut MaybeUnresolvedInstr,
+        ) -> Result<(), anyhow::Error> {
+            if let Token::REGISTER(reg) = token {
+                instr.value |= (LC3Word::from(*reg) << SHIFT);


wpkelso and others added 24 commits December 20, 2024 13:26

add foundational structures for code->machine translation

fde1fe4

Merge branch 'main' into assembler

68bc5a3

lay tokenizer groundwork

c09d59f

Switch to using regex

8fba3da

add all regex strings

f175c9e

add some matching and a failing test

a625f0e

cleaned up register matching logic

85d5f51

add tests for comments and unclean register strings

f26d5ff

instruction tokenization and test for add

9312b4b

tokenize pseudoops

166b73f

return vector of tokens

921f5dd

split into multiple files

217288b

num tokens and tests

0dae0e3

add prefix label pass for lexer

1d9ce2c

Add test for stripping prefix labels

bfae0de

Add wrapper function for lexer interface

469ab05

token validation fn shell & fn inlining

06e7699

Implementation up to instruction formation, then stuck

c0007c8

use strum macros instead of shell tokens

0ba9d4e

Initial lexer skeleton

e5cd945

Merge branch 'main' into assembler

c28b353

add load lexing

c00c7e8

add all multi-field instructions to lexer

e418601

Add tests for currently implemented instructions

b4c9da9

wpkelso requested a review from Bennett-Petzold January 14, 2025 19:15

wpkelso temporarily deployed to testing January 14, 2025 19:15 — with GitHub Actions Inactive

wpkelso had a problem deploying to testing January 14, 2025 19:15 — with GitHub Actions Error

wpkelso temporarily deployed to testing January 14, 2025 19:15 — with GitHub Actions Inactive

wpkelso had a problem deploying to testing January 14, 2025 19:15 — with GitHub Actions Failure

wpkelso had a problem deploying to testing January 14, 2025 19:15 — with GitHub Actions Error

wpkelso had a problem deploying to testing January 14, 2025 19:15 — with GitHub Actions Failure

github-advanced-security bot found potential problems Jan 14, 2025

View reviewed changes

Bennett-Petzold approved these changes Jan 14, 2025

View reviewed changes

wpkelso merged commit bb90fa6 into main Jan 14, 2025
4 of 9 checks passed

wpkelso deleted the assembler branch January 14, 2025 20:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Partial lexer implementation #8

Partial lexer implementation #8

wpkelso commented Jan 14, 2025

		@@ -0,0 +1,327 @@
		use crate::{
		assembler::{MaybeUnresolvedInstr, Op, PseudoOp, Token},

Partial lexer implementation #8

Partial lexer implementation #8

Conversation

wpkelso commented Jan 14, 2025