managed to get library version on par with the reference implementation by using memmap2. I'm understanding scoped threads a little better now... I think

The fixed line length solution is still just as slow, even using memmap

src/main/rust/Cargo.toml

@@ -9,7 +9,7 @@ edition = "2021"
 bstr = "1.9.1"
 fast-float = "0.2.0"
 memchr = "2.7.4"
-memmap = "0.7.0"
+memmap2 = "0.9.4"
 polars = { version = "0.36.2", features = ["csv", "lazy", "nightly", "streaming"]}
 rayon = "1.10.0"
 rustc-hash = "2.0.0"