8 migrate tokenization

[Jonstep101010]

• replaced almost all unsafe raw ptr usage
• a few tests for tokenization
• impl for shell and token
• reduce manual iteration
• use more rusty types
• rustyline
• remove custom unsafe code libs/interface to gnu_readline_sys
• restructured modules

[Copilot]

Copilot reviewed 26 out of 41 changed files in this pull request and generated 2 comments.

Files not reviewed (15)

• src/execution.rs: Evaluated as low risk
• src/environment/expander.rs: Evaluated as low risk
• src/builtins/exit.rs: Evaluated as low risk
• src/builtins/export.rs: Evaluated as low risk
• src/builtins/unset.rs: Evaluated as low risk
• src/environment.rs: Evaluated as low risk
• src/environment/check_key.rs: Evaluated as low risk
• src/builtins/echo.rs: Evaluated as low risk
• src/execution/builtins/pwd.rs: Evaluated as low risk
• src/execution/builtins/env.rs: Evaluated as low risk
• src/execution/execute_pipes.rs: Evaluated as low risk
• src/execution/builtins/cd.rs: Evaluated as low risk
• Cargo.toml: Evaluated as low risk
• src/environment/mod.rs: Evaluated as low risk
• src/execution/exec_bin.rs: Evaluated as low risk

Comments suppressed due to low confidence (3)

src/execution/builtins/exit.rs:5

• The error message 'exit: numeric argument required' should be printed when the argument is not a valid number, not when it is empty.

eprint_msh!("exit: numeric argument required");

src/execution/builtins/export.rs:16

• [nitpick] The error message could be more specific. Consider including the invalid key in the error message.

crate::eprint_msh!("export: `{}': not a valid identifier", command[i].to_str().unwrap());

src/execution/builtins/unset.rs:3

• The loop condition str[i] != b'\0' can lead to out-of-bounds access if the input slice does not contain a null terminator. Consider adding a bounds check.

while str[i] != b'\0' {