* Linear probe for city indexing. Beats current leader spullara 2.2 vs 3.8 elapsed time.
* Straightforward impl using bytebuffers. Turns out memorysegments were slower than used mappedbytebuffers.
* A initial submit-worthy entry
Comparison to select entries (averaged over 3 runs)
* spullara 1.66s [5th on leaderboard currently]
* vemana (this submission) 1.65s
* artsiomkorzun 1.64s [4th on leaderboard currently]
Tests: PASS
Impl Class: dev.morling.onebrc.CalculateAverage_vemana
Machine specs
* 16 core Ryzen 7950X
* 128GB RAM
Description
* Decompose the full file into Shards of memory mapped files and process
each independently, outputting a TreeMap: City -> Statistics
* Compose the final answer by merging the individual TreeMap outputs
* Select 1 Thread per available processor as reported by the JVM
* Size to fit all datastructure in 0.5x L3 cache (4MB/core on the
evaluation machines)
* Use linear probing hash table, with identity of city name = byte[] and
hash code computed inline
* Avoid all allocation in the hot path and instead use method
parameters. So, instead of passing a single Object param called Point(x, y, z),
pass 3 parameters for each of its components. It is ugly, but this
challenge is so far from Java's idioms anyway
* G1GC seems to want to interfere; use ParallelGC instead (just a quick
and dirty hack)
Things tried that did not work
* MemorySegments are actually slower than MappedByteBuffers
* Trying to inline everything: not needed; the JIT compiler is pretty
good
* Playing with JIT compiler flags didn't yield clear wins. In
particular, was surprised that using a max level of 3 and reducing
compilation threshold did nothing.. when the jit logs print that none
of the methods reach level 4 and stay there for long
* Hand-coded implementation of Array.equals(..) using
readLong(..) & bitmask_based_on_length from a bytebuffer instead of byte by byte
* Further tuning to compile loop methods: timings are now consistenctly ahead of artsiomkorzun in 4th place.
There are methods on the data path that were being interpreted for far
too long. For example, the method that takes a byte range and simply
calls one method per line was taking a disproportionate amount of time.
Using `-XX:+AlwaysCompileLoopMethods` option improved completion time by 4%.
============= vemana ===============
[20:55:22] [lsv@vemana]$ for i in 1 2 3 4 5; do ./runTheir.sh vemana;
done;
Using java version 21.0.1-graal in this shell.
real 0m1.581s
user 0m34.166s
sys 0m1.435s
Using java version 21.0.1-graal in this shell.
real 0m1.593s
user 0m34.629s
sys 0m1.470s
Using java version 21.0.1-graal in this shell.
real 0m1.632s
user 0m35.893s
sys 0m1.340s
Using java version 21.0.1-graal in this shell.
real 0m1.596s
user 0m33.074s
sys 0m1.386s
Using java version 21.0.1-graal in this shell.
real 0m1.611s
user 0m35.516s
sys 0m1.438s
============= artsiomkorzun ===============
[20:56:12] [lsv@vemana]$ for i in 1 2 3 4 5; do ./runTheir.sh
artsiomkorzun; done;
Using java version 21.0.1-graal in this shell.
real 0m1.669s
user 0m38.043s
sys 0m1.287s
Using java version 21.0.1-graal in this shell.
real 0m1.679s
user 0m37.840s
sys 0m1.400s
Using java version 21.0.1-graal in this shell.
real 0m1.657s
user 0m37.607s
sys 0m1.298s
Using java version 21.0.1-graal in this shell.
real 0m1.643s
user 0m36.852s
sys 0m1.392s
Using java version 21.0.1-graal in this shell.
real 0m1.644s
user 0m36.951s
sys 0m1.279s
============= spullara ===============
[20:57:55] [lsv@vemana]$ for i in 1 2 3 4 5; do ./runTheir.sh spullara;
done;
Using java version 21.0.1-graal in this shell.
real 0m1.676s
user 0m37.404s
sys 0m1.386s
Using java version 21.0.1-graal in this shell.
real 0m1.652s
user 0m36.509s
sys 0m1.486s
Using java version 21.0.1-graal in this shell.
real 0m1.665s
user 0m36.451s
sys 0m1.506s
Using java version 21.0.1-graal in this shell.
real 0m1.671s
user 0m36.917s
sys 0m1.371s
Using java version 21.0.1-graal in this shell.
real 0m1.634s
user 0m35.624s
sys 0m1.573s
========================== Running Tests ======================
[21:17:57] [lsv@vemana]$ ./runTests.sh vemana
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-10000-unique-keys.txt
Using java version 21.0.1-graal in this shell.
real 0m0.150s
user 0m1.035s
sys 0m0.117s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-10.txt
Using java version 21.0.1-graal in this shell.
real 0m0.114s
user 0m0.789s
sys 0m0.116s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-1.txt
Using java version 21.0.1-graal in this shell.
real 0m0.115s
user 0m0.948s
sys 0m0.075s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-20.txt
Using java version 21.0.1-graal in this shell.
real 0m0.113s
user 0m0.926s
sys 0m0.066s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-2.txt
Using java version 21.0.1-graal in this shell.
real 0m0.110s
user 0m0.734s
sys 0m0.078s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-3.txt
Using java version 21.0.1-graal in this shell.
real 0m0.114s
user 0m0.870s
sys 0m0.095s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-boundaries.txt
Using java version 21.0.1-graal in this shell.
real 0m0.113s
user 0m0.843s
sys 0m0.084s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-complex-utf8.txt
Using java version 21.0.1-graal in this shell.
real 0m0.121s
user 0m0.852s
sys 0m0.171s
* Improve by a few % more; now, convincingly faster than 6th place
submission. So far, only algorithms and tuning; no bitwise tricks yet.
Improve chunking implementation to avoid allocation and allow
finegrained chunking for the last X% of work. Work now proceeds in two
stages: big chunk stage and small chunk stage. This is to avoid
straggler threads holding up result merging.
Tests pass
[07:14:49] [lsv@vemana]$ ./test.sh vemana
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-10000-unique-keys.txt
Using java version 21.0.1-graal in this shell.
real 0m0.152s
user 0m0.973s
sys 0m0.107s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-10.txt
Using java version 21.0.1-graal in this shell.
real 0m0.113s
user 0m0.840s
sys 0m0.060s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-1.txt
Using java version 21.0.1-graal in this shell.
real 0m0.107s
user 0m0.681s
sys 0m0.085s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-20.txt
Using java version 21.0.1-graal in this shell.
real 0m0.105s
user 0m0.894s
sys 0m0.068s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-2.txt
Using java version 21.0.1-graal in this shell.
real 0m0.099s
user 0m0.895s
sys 0m0.068s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-3.txt
Using java version 21.0.1-graal in this shell.
real 0m0.098s
user 0m0.813s
sys 0m0.050s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-boundaries.txt
Using java version 21.0.1-graal in this shell.
real 0m0.095s
user 0m0.777s
sys 0m0.087s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-complex-utf8.txt
Using java version 21.0.1-graal in this shell.
real 0m0.112s
user 0m0.904s
sys 0m0.069s
* Merge results from finished threads instead of waiting for all threads
to finish.
Not a huge difference overall but no reason to wait.
Also experiment with a few other compiler flags and attempt to use
jitwatch to understand what the jit is doing.
* Move to prepare_*.sh format and run evaluate2.sh locally.
Shows 7th place in leaderboard
| # | Result (m:s.ms) | Implementation | JDK | Submitter | Notes
|
|---|-----------------|--------------------|-----|---------------|-----------|
| 1 | 00:01.588 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_thomaswue.java)|
21.0.1-graal | [Thomas Wuerthinger](https://github.com/thomaswue) |
GraalVM native binary |
| 2 | 00:01.866 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_merykitty.java)|
21.0.1-open | [Quan Anh Mai](https://github.com/merykitty) | |
| 3 | 00:01.904 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_royvanrijn.java)|
21.0.1-graal | [Roy van Rijn](https://github.com/royvanrijn) | |
| | 00:02.398 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_ebarlas.java)|
21.0.1-graal | [Elliot Barlas](https://github.com/ebarlas) | |
| | 00:02.724 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_obourgain.java)|
21.0.1-open | [Olivier Bourgain](https://github.com/obourgain) | |
| | 00:02.771 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_algirdasrascius.java)|
21.0.1-open | [Algirdas Ra__ius](https://github.com/algirdasrascius) |
|
| | 00:02.842 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_vemana.java)|
21.0.1-graal | [Vemana](https://github.com/vemana) | |
| | 00:02.902 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_spullara.java)|
21.0.1-graal | [Sam Pullara](https://github.com/spullara) | |
| | 00:02.906 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_artsiomkorzun.java)|
21.0.1-graal | [artsiomkorzun](https://github.com/artsiomkorzun) | |
| | 00:02.970 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_isolgpus.java)|
21.0.1-open | [Jamie Stansfield](https://github.com/isolgpus) | |
* Tune chunksize to get another 2% improvement for 8 processors as used by
the evaluation script.
* Read int at a time for city name length detection; speeds up by 2% in local testing.
* Improve reading temperature double by exiting loop quicker; no major
tricks (like reading an int) yet, but good for 5th place on leaderboard in local testing.
This small change has caused a surprising gain in performance by about 4%.
I didn't expect such a big change, but perhaps in combination with the
earlier change to read int by int for the city name, temperature reading
is dominating that aspect of the time. Also, perhaps the quicker exit
(as soon as you see '.' instead of reading until '\n') means you get to
simply skip reading the '\n' across each of the lines. Since the lines
are on average like 15 characters, it may be that avoiding reading the \n
is a meaningful saving. Or maybe the JIT found a clever optimization for
reading the temperature.
Or maybe it is simply the case that the number of multiplications is now
down to 2 from the previous 3 is what's causing the performance gain?
| # | Result (m:s.ms) | Implementation | JDK | Submitter | Notes
|
|---|-----------------|--------------------|-----|---------------|-----------|
| 1 | 00:01.531 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_thomaswue.java)|
21.0.1-graal | [Thomas Wuerthinger](https://github.com/thomaswue) |
GraalVM native binary |
| 2 | 00:01.794 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_royvanrijn.java)|
21.0.1-graal | [Roy van Rijn](https://github.com/royvanrijn) | |
| 3 | 00:01.956 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_merykitty.java)|
21.0.1-open | [Quan Anh Mai](https://github.com/merykitty) | |
| | 00:02.346 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_ebarlas.java)|
21.0.1-graal | [Elliot Barlas](https://github.com/ebarlas) | |
| | 00:02.673 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_vemana.java)|
21.0.1-graal | [Subrahmanyam](https://github.com/vemana) | |
| | 00:02.689 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_obourgain.java)|
21.0.1-open | [Olivier Bourgain](https://github.com/obourgain) | |
| | 00:02.785 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_algirdasrascius.java)|
21.0.1-open | [Algirdas Ra__ius](https://github.com/algirdasrascius) |
|
| | 00:02.926 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_isolgpus.java)|
21.0.1-open | [Jamie Stansfield](https://github.com/isolgpus) | |
| | 00:02.928 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_artsiomkorzun.java)|
21.0.1-graal | [Artsiom Korzun](https://github.com/artsiomkorzun) | |
| | 00:02.932 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_spullara.java)|
21.0.1-graal | [Sam Pullara](https://github.com/spullara) | |
* Reduce one multiplication when temperature is +ve.
* Linear probe for city indexing. Beats current leader spullara 2.2 vs 3.8 elapsed time.
* Straightforward impl using bytebuffers. Turns out memorysegments were slower than used mappedbytebuffers.
* A initial submit-worthy entry
Comparison to select entries (averaged over 3 runs)
* spullara 1.66s [5th on leaderboard currently]
* vemana (this submission) 1.65s
* artsiomkorzun 1.64s [4th on leaderboard currently]
Tests: PASS
Impl Class: dev.morling.onebrc.CalculateAverage_vemana
Machine specs
* 16 core Ryzen 7950X
* 128GB RAM
Description
* Decompose the full file into Shards of memory mapped files and process
each independently, outputting a TreeMap: City -> Statistics
* Compose the final answer by merging the individual TreeMap outputs
* Select 1 Thread per available processor as reported by the JVM
* Size to fit all datastructure in 0.5x L3 cache (4MB/core on the
evaluation machines)
* Use linear probing hash table, with identity of city name = byte[] and
hash code computed inline
* Avoid all allocation in the hot path and instead use method
parameters. So, instead of passing a single Object param called Point(x, y, z),
pass 3 parameters for each of its components. It is ugly, but this
challenge is so far from Java's idioms anyway
* G1GC seems to want to interfere; use ParallelGC instead (just a quick
and dirty hack)
Things tried that did not work
* MemorySegments are actually slower than MappedByteBuffers
* Trying to inline everything: not needed; the JIT compiler is pretty
good
* Playing with JIT compiler flags didn't yield clear wins. In
particular, was surprised that using a max level of 3 and reducing
compilation threshold did nothing.. when the jit logs print that none
of the methods reach level 4 and stay there for long
* Hand-coded implementation of Array.equals(..) using
readLong(..) & bitmask_based_on_length from a bytebuffer instead of byte by byte
* Further tuning to compile loop methods: timings are now consistenctly ahead of artsiomkorzun in 4th place.
There are methods on the data path that were being interpreted for far
too long. For example, the method that takes a byte range and simply
calls one method per line was taking a disproportionate amount of time.
Using `-XX:+AlwaysCompileLoopMethods` option improved completion time by 4%.
============= vemana ===============
[20:55:22] [lsv@vemana]$ for i in 1 2 3 4 5; do ./runTheir.sh vemana;
done;
Using java version 21.0.1-graal in this shell.
real 0m1.581s
user 0m34.166s
sys 0m1.435s
Using java version 21.0.1-graal in this shell.
real 0m1.593s
user 0m34.629s
sys 0m1.470s
Using java version 21.0.1-graal in this shell.
real 0m1.632s
user 0m35.893s
sys 0m1.340s
Using java version 21.0.1-graal in this shell.
real 0m1.596s
user 0m33.074s
sys 0m1.386s
Using java version 21.0.1-graal in this shell.
real 0m1.611s
user 0m35.516s
sys 0m1.438s
============= artsiomkorzun ===============
[20:56:12] [lsv@vemana]$ for i in 1 2 3 4 5; do ./runTheir.sh
artsiomkorzun; done;
Using java version 21.0.1-graal in this shell.
real 0m1.669s
user 0m38.043s
sys 0m1.287s
Using java version 21.0.1-graal in this shell.
real 0m1.679s
user 0m37.840s
sys 0m1.400s
Using java version 21.0.1-graal in this shell.
real 0m1.657s
user 0m37.607s
sys 0m1.298s
Using java version 21.0.1-graal in this shell.
real 0m1.643s
user 0m36.852s
sys 0m1.392s
Using java version 21.0.1-graal in this shell.
real 0m1.644s
user 0m36.951s
sys 0m1.279s
============= spullara ===============
[20:57:55] [lsv@vemana]$ for i in 1 2 3 4 5; do ./runTheir.sh spullara;
done;
Using java version 21.0.1-graal in this shell.
real 0m1.676s
user 0m37.404s
sys 0m1.386s
Using java version 21.0.1-graal in this shell.
real 0m1.652s
user 0m36.509s
sys 0m1.486s
Using java version 21.0.1-graal in this shell.
real 0m1.665s
user 0m36.451s
sys 0m1.506s
Using java version 21.0.1-graal in this shell.
real 0m1.671s
user 0m36.917s
sys 0m1.371s
Using java version 21.0.1-graal in this shell.
real 0m1.634s
user 0m35.624s
sys 0m1.573s
========================== Running Tests ======================
[21:17:57] [lsv@vemana]$ ./runTests.sh vemana
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-10000-unique-keys.txt
Using java version 21.0.1-graal in this shell.
real 0m0.150s
user 0m1.035s
sys 0m0.117s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-10.txt
Using java version 21.0.1-graal in this shell.
real 0m0.114s
user 0m0.789s
sys 0m0.116s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-1.txt
Using java version 21.0.1-graal in this shell.
real 0m0.115s
user 0m0.948s
sys 0m0.075s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-20.txt
Using java version 21.0.1-graal in this shell.
real 0m0.113s
user 0m0.926s
sys 0m0.066s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-2.txt
Using java version 21.0.1-graal in this shell.
real 0m0.110s
user 0m0.734s
sys 0m0.078s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-3.txt
Using java version 21.0.1-graal in this shell.
real 0m0.114s
user 0m0.870s
sys 0m0.095s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-boundaries.txt
Using java version 21.0.1-graal in this shell.
real 0m0.113s
user 0m0.843s
sys 0m0.084s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-complex-utf8.txt
Using java version 21.0.1-graal in this shell.
real 0m0.121s
user 0m0.852s
sys 0m0.171s
* Improve by a few % more; now, convincingly faster than 6th place
submission. So far, only algorithms and tuning; no bitwise tricks yet.
Improve chunking implementation to avoid allocation and allow
finegrained chunking for the last X% of work. Work now proceeds in two
stages: big chunk stage and small chunk stage. This is to avoid
straggler threads holding up result merging.
Tests pass
[07:14:49] [lsv@vemana]$ ./test.sh vemana
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-10000-unique-keys.txt
Using java version 21.0.1-graal in this shell.
real 0m0.152s
user 0m0.973s
sys 0m0.107s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-10.txt
Using java version 21.0.1-graal in this shell.
real 0m0.113s
user 0m0.840s
sys 0m0.060s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-1.txt
Using java version 21.0.1-graal in this shell.
real 0m0.107s
user 0m0.681s
sys 0m0.085s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-20.txt
Using java version 21.0.1-graal in this shell.
real 0m0.105s
user 0m0.894s
sys 0m0.068s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-2.txt
Using java version 21.0.1-graal in this shell.
real 0m0.099s
user 0m0.895s
sys 0m0.068s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-3.txt
Using java version 21.0.1-graal in this shell.
real 0m0.098s
user 0m0.813s
sys 0m0.050s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-boundaries.txt
Using java version 21.0.1-graal in this shell.
real 0m0.095s
user 0m0.777s
sys 0m0.087s
Validating calculate_average_vemana.sh --
src/test/resources/samples/measurements-complex-utf8.txt
Using java version 21.0.1-graal in this shell.
real 0m0.112s
user 0m0.904s
sys 0m0.069s
* Merge results from finished threads instead of waiting for all threads
to finish.
Not a huge difference overall but no reason to wait.
Also experiment with a few other compiler flags and attempt to use
jitwatch to understand what the jit is doing.
* Move to prepare_*.sh format and run evaluate2.sh locally.
Shows 7th place in leaderboard
| # | Result (m:s.ms) | Implementation | JDK | Submitter | Notes
|
|---|-----------------|--------------------|-----|---------------|-----------|
| 1 | 00:01.588 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_thomaswue.java)|
21.0.1-graal | [Thomas Wuerthinger](https://github.com/thomaswue) |
GraalVM native binary |
| 2 | 00:01.866 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_merykitty.java)|
21.0.1-open | [Quan Anh Mai](https://github.com/merykitty) | |
| 3 | 00:01.904 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_royvanrijn.java)|
21.0.1-graal | [Roy van Rijn](https://github.com/royvanrijn) | |
| | 00:02.398 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_ebarlas.java)|
21.0.1-graal | [Elliot Barlas](https://github.com/ebarlas) | |
| | 00:02.724 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_obourgain.java)|
21.0.1-open | [Olivier Bourgain](https://github.com/obourgain) | |
| | 00:02.771 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_algirdasrascius.java)|
21.0.1-open | [Algirdas Ra__ius](https://github.com/algirdasrascius) |
|
| | 00:02.842 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_vemana.java)|
21.0.1-graal | [Vemana](https://github.com/vemana) | |
| | 00:02.902 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_spullara.java)|
21.0.1-graal | [Sam Pullara](https://github.com/spullara) | |
| | 00:02.906 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_artsiomkorzun.java)|
21.0.1-graal | [artsiomkorzun](https://github.com/artsiomkorzun) | |
| | 00:02.970 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_isolgpus.java)|
21.0.1-open | [Jamie Stansfield](https://github.com/isolgpus) | |
* Tune chunksize to get another 2% improvement for 8 processors as used by
the evaluation script.
* Read int at a time for city name length detection; speeds up by 2% in local testing.
* Improve reading temperature double by exiting loop quicker; no major
tricks (like reading an int) yet, but good for 5th place on leaderboard in local testing.
This small change has caused a surprising gain in performance by about 4%.
I didn't expect such a big change, but perhaps in combination with the
earlier change to read int by int for the city name, temperature reading
is dominating that aspect of the time. Also, perhaps the quicker exit
(as soon as you see '.' instead of reading until '\n') means you get to
simply skip reading the '\n' across each of the lines. Since the lines
are on average like 15 characters, it may be that avoiding reading the \n
is a meaningful saving. Or maybe the JIT found a clever optimization for
reading the temperature.
Or maybe it is simply the case that the number of multiplications is now
down to 2 from the previous 3 is what's causing the performance gain?
| # | Result (m:s.ms) | Implementation | JDK | Submitter | Notes
|
|---|-----------------|--------------------|-----|---------------|-----------|
| 1 | 00:01.531 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_thomaswue.java)|
21.0.1-graal | [Thomas Wuerthinger](https://github.com/thomaswue) |
GraalVM native binary |
| 2 | 00:01.794 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_royvanrijn.java)|
21.0.1-graal | [Roy van Rijn](https://github.com/royvanrijn) | |
| 3 | 00:01.956 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_merykitty.java)|
21.0.1-open | [Quan Anh Mai](https://github.com/merykitty) | |
| | 00:02.346 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_ebarlas.java)|
21.0.1-graal | [Elliot Barlas](https://github.com/ebarlas) | |
| | 00:02.673 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_vemana.java)|
21.0.1-graal | [Subrahmanyam](https://github.com/vemana) | |
| | 00:02.689 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_obourgain.java)|
21.0.1-open | [Olivier Bourgain](https://github.com/obourgain) | |
| | 00:02.785 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_algirdasrascius.java)|
21.0.1-open | [Algirdas Ra__ius](https://github.com/algirdasrascius) |
|
| | 00:02.926 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_isolgpus.java)|
21.0.1-open | [Jamie Stansfield](https://github.com/isolgpus) | |
| | 00:02.928 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_artsiomkorzun.java)|
21.0.1-graal | [Artsiom Korzun](https://github.com/artsiomkorzun) | |
| | 00:02.932 |
[link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_spullara.java)|
21.0.1-graal | [Sam Pullara](https://github.com/spullara) | |
* Reduce one multiplication when temperature is +ve.
* Added some documentation on the approach.
---------
Co-authored-by: vemana <vemana.github@gmail.com>
* first version
* second version (0m59s)
* third version (0m46s)
* fourth version (0m39s)
* fifth version (0m18s)
* follow naming conventions from project structure
* fix rounding (see /issues/49)
* formatting changes from build
* name should case-match github username
* sixth version (14s)
* seventh version (11s)
* potential fix for other systems?
* no need for sdk install
* binary should go to ./target
* building native-image only when not existing yet
* Calculate average by vaidhy
* Calculate average by vaidhy
* More changes
* remove worker log
* Pass -Dparellelism and switch back to open
* Try out mmap
* Improve mmap solution
* no copy version
* reduce threads
* hash code computed on the fly
* Reuse the char (Do not know if it helps)
* primitive hash map
* Primite HashMap
* Micro optimizations to push for optimizations
* Revert "Micro optimizations to push for optimizations"
This reverts commit ea333e2821ebb5c1d6d71a4e87e569a8f2f8f7f0.
* Micro optimizations to get the juice
* floorMod fixes
* findSemi and findNewLine as separate functions
* Optimized parseDouble
* More micro changes
* Aligned equal check
* more small changes
* XOR instead of compare
* Reduce loop length
* Revert changes
* Loop optimization and added native build
* Hand unrolled findSemi loop.
* Remove incorrect comments
* Taking care fo PR comments
* Add prepare script
* Missing header error fix
* remove wrong comment
---------
Co-authored-by: Anita S V <anitasvasu@gmail.com>
Co-authored-by: Anita SV <anitvasu@amazon.com>
* Use graal
* Use dynamic cores computer
* Use stream API to cleanup code
* Use max processors
* Use hash to avoid init string
* optimize concurrentmap init
* Smaller hash size
* Avoid checking concurrentmap
* Optimize data type
* string dedup
* Faster write
* Change base
* Remove time
* Use mul instead of div
* remove unneeded check
* slightly improved hash code perf
* Use unsafe to access memory + untangle the code a bit
* Adhoc cache that works a bit better
* Store station names as offset into the memory segment + length; slightly change how the hash is calculated
Squashed commit of the following:
commit 44d3736de87834b41118d45831e59fc2b052117c
Merge: fcf795f 3127962
Author: Andrew Sun <as-com@users.noreply.github.com>
Date: Thu Jan 11 20:01:13 2024 -0500
Merge branch 'gunnarmorling:main' into as-com
commit fcf795fbabacbd91891d11d21450ee4b1c479dc5
Author: Andrew Sun <me@andrewsun.com>
Date: Wed Jan 10 21:14:01 2024 -0500
Optimizations to Andrew Sun's entry
commit 4203924711bab5252ff3cbb50a90f4ce4e8e67c2
Merge: 9aed05a 085168a
Author: Andrew Sun <me@andrewsun.com>
Date: Wed Jan 10 19:40:19 2024 -0500
Merge remote-tracking branch 'upstream/main' into as-com
commit 9aed05a04bd27fe7323e66c347b1011c77da322c
Merge: 3f8df58 c2d120f
Author: Andrew Sun <me@andrewsun.com>
Date: Sun Jan 7 16:45:27 2024 -0500
Merge remote-tracking branch 'origin/as-com' into as-com
# Conflicts:
# calculate_average_asun.sh
# src/main/java/dev/morling/onebrc/CalculateAverage_asun.java
commit c2d120f0cb7f18c720a81a7f898102b310f9ecb9
Author: Andrew Sun <me@andrewsun.com>
Date: Sat Jan 6 00:45:47 2024 -0500
Add entry by Andrew Sun
commit 3f8df5803bcc8f3e29ed8bfff3077eb0e8cdab15
Author: Andrew Sun <me@andrewsun.com>
Date: Sat Jan 6 00:45:47 2024 -0500
Add entry by Andrew Sun
* Graal Native
* I need a GC :(
* Fix slash lolz
* Fix god damn output lol
I forgot java :D
* Custom hash, custom key
* More optimisations
* I don't need "optimize-build"
I don't care about image size! :D
- Eliminate redundant object creations in between
- Custom HashMap on purpose - Inspired by @spullara
- More performant temperature parsing - Inspired by @yemreinci
- JVM tweaks, decreased heap memory, and remove AlwaysPreTouch
Co-authored-by: Yavuz Tas <yavuz.tas@ing.com>
* Create clones
* Cleanup code and add memory mapping to read file
* Fix chunks reading logic to fit linebreak
* Remove unused
* Sequential
* Multi thread process chunks
* Add new line in output
* Remove unnecessary operation with map & reducer memory
* Reduce mem usage by using only 1 map
* formatting
* Remove unnecessary length check
* Remove trycatch
* Optimize double parsing
* check full hash before comparing
* implement merykitty suggestions to simplify temperature masking; required refactoring to little-endian
* standalone script for offline Perfect Hash seed searching
* stop using an oversized hash table
---------
Co-authored-by: Jason Nochlin <hundredwatt@users.noreply.github.com>
* divide the reading of the file by parts
* fix format
* add number of core partition
* fix format
* implement strToDouble
* fix strtodouble
* add locale, fix read file, tests pass
* delete unnecessary method clean
* First Version
First draft; stole chunking but it's bad
Forgot my changes
No regex building
Clean & optim
I was not benchmarking myself T_T
Faaaster
First Version
* Update calculate_average_samuelyvon.sh
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
* Add prepare script
* Fix rounding
* Fix format
* Fixing casing
* Formats of sorts?
* Rename
---------
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
* calculate_average_mtopolnik
* short hash (just first 8 bytes of name)
* Remove unneeded checks
* Remove archiving classes
* 2x larger hashtable
* Add "set" to setters
* Simplify parsing temperature, remove newline search
* Reduce the size of the name slot
* Store name length and use to detect collision
* Reduce memory loads in parseTemperature
* Use short for min/max
* Extract constant for semicolon
* Fix script header
* Explicit bash shell in shebang
* Inline usage of broadcast semicolon
* Try vectorization
* Remove vectorization
* Go Unsafe
* Use SWAR temperature parsing by merykitty
* Inline some things
* Remove commented-out MemorySegment usage
* Inline namesMem.asSlice() invocation
* Try out JVM JIT flags
* Implement strcmp
* Remove unused instance variables
* Optimize hashing
* Put station name into hashtable
* Reorder method
* Remove usage of MemorySegment.getUtf8String
Replace with UNSAFE.copyMemory() and new String()
* Fix hashing bug
* Remove outdated comments
* Fix informative constants
* Use broadcastByte() more
* Improve method naming
* More hashing
* Revert more hashing
* Add commented-out code to hash 16 bytes
* Slight cleanup
* Align hashtable at cacheline boundary
* Add Graal Native image
* Revert Graal Native image
This reverts commit d916a42326d89bd1a841bbbecfae185adb8679d7.
* Simplify shell script (no SDK selection)
* Move a constant, zero out hashtable on start
* Better name comparison
* Add prepare_mtopolnik.sh
* Cleaner idiom in name comparison
* AND instead of MOD for hashtable indexing
* Improve word masking code
* Fix formatting
* Reduce memory loads
* Remove endianness checks
* Avoid hash == 0 problem
* Fix subtle bug
* MergeSort of parellel results
* Touch up perf
* Touch up perf
* Remove -Xmx256m
* Extract result printing method
* Print allocation details on OOME
* Single mmap
* Use global allocation arena
* initial commit
* - use loop
- use mutable object to store results
* get rid of regex
* Do not allocate measurement objects
* MMap + custom double parsing ~ 1:30 (down from ~ 2:05)
* HashMap for accumulation and only sort at the end - 1:05
* MMap the whole file
* Use graal
* no GC
* Store results in an array list to avoid double map lookup
* Adjust max buf size
* Manual parsing number to long
* Add --enable-preview
* remove buffer size check (has no effect on performance)
* fix min & max initialization
* do not check for \r
* Revert "do not check for \r"
This reverts commit 9da1f574bf6261ea49c353488d3b4673cad3ce6e.
* Optimise parsing. Now completes in 31 sec down from ~43
* trying to parse numbers faster
* use open address hash table instead of the standard HashMap
* formatting
* Rename the script to match github username (change underscores to slashes)
Enable transparent huge pages, seems to improve by ~2 sec
* Revert "formatting"
This reverts commit 4e90797d2a729ed7385c9000c85cc7e87d935f96.
* Revert "use open address hash table instead of the standard HashMap"
This reverts commit c784b55f61e48f548b2623e5c8958c9b283cae14.
* add prepare_roman-r-m.sh
* SWAR tricks to find semicolon (-2 seconds ro run time)
* remove time call
* fix test
* Parallel version (~6.5 seconds)
* Add multithreaded variant to generate measurements
* Add removing existing measurements.txt file in case exists (for usability reasons)
Fix bug for number of lines generated
* Fix also for less than assumed chunk size (10M entries) per thread
#### Check List:
- [x] Tests pass (`./test.sh MeanderingProgrammer` shows no differences between expected and actual outputs)
- [x] All formatting changes by the build are committed
- [x] Your launch script is named `calculate_average_MeanderingProgrammer.sh` (make sure to match casing of your GH user name) and is executable
- [x] Output matches that of `calculate_average_baseline.sh`
* Execution time: `00:04.668`
* Execution time of reference implementation: `02:40.597`
* System: Apple M2 Max, 12 cores, 64 GB
* Implementation CalculateAverage_japplis of 1BRC from Anthony Goubard (japplis).
Local performance (7 years old desktop i7-6700K - 8 cores - 16GB) 26 seconds. For reference, Jamie Stansfield (isolgpus) is 23 seconds on my machine and 11s in your results.
I've added the nbactions.xml to the .gitignore file. When you add in NetBeans options like --enable-preview to actions like debug file or run file, it creates this file.
* Implementation CalculateAverage_japplis of 1BRC from Anthony Goubard (japplis).
Local performance (7 years old desktop i7-6700K - 8 cores - 16GB) 26 seconds. For reference, Jamie Stansfield (isolgpus) is 23 seconds on my machine and 11s in your results.
I've added the nbactions.xml to the .gitignore file. When you add in NetBeans options like --enable-preview to actions like debug file or run file, it creates this file.
second commit: Removed BufferedInputStream and replaced Measurement with IntSummaryStatistics (thanks davecom): still 23" but cleaner code
* Initial solution by raipc
* Implemented custom hash map with open addressing
* Small optimizations to task splitting and range check disabling
* Fixed off-by-one error in merge
* Run with EpsilonGC. Borrowed VM params from Shipilev
* Make script executable
* Add a license
* First working version.
* Small adjustments.
* Correct number of threads.
* Sync
* Some fixes. To LF instead of CRLF.
* Parallel reading and processing.
* Update CreateMeasurements.java
* Update CalculateAverage.java
* Small fix for bug in switching buffers.
* Update calculate_average_arjenvaneerde.sh
---------
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
* initial commit
* first attempt: segment the file and process it in parallel
* remove commented stuff
* custom parseDouble for this simple case
* fixed some issues and improved parsing
* format
* Update calculate_average_AbstractKamen.sh
---------
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
This commit introduces a new java class, CalculateAverage_couragelee, and a shell script for calculating averages. The java class utilizes NIO's memory-mapping and parallel computing techniques to perform calculations. These changes should improve the efficiency and speed of average calculations.
* feat(flippingbits): Improve parsing of measurement and few cleanups
* feat(flippingbits): Reduce chunk size to 10MB
* feat(flippingbits): Improve parsing of station names
* chore(flippingbits): Remove obsolete import
* chore(flippingbits): Few cleanups
* Optimize checking for collisions by doing this a long at a time always.
* Use a long at a time scanning for delimiter.
* Minor tuning. Now below 0.80s on Intel i9-13900K.
* Add number parsing code from Quan Anh Mai. Fix name length issue.
* Include suggestion from Alfonso Peterssen for another 1.5%.
* Optimize hash collision check compare for ~4% gain.
* Add perf stats based on latest version.
* isolgpus: fix chunk sizing when not at 8 threads
use as many cores as are available
don't buffer the station name, only use it when we need it.
get rid of the main branch
move variables inside the loop
* isolgpus: optimistically assume we can read a whole int for the station name, but roll back if we get it wrong. This should be very beneficial on a dataset where station names are mostly over 4 chars
---------
Co-authored-by: Jamie Stansfield <jalstansfield@gmail.com>
Runs with standard JDK 21.
On my computers (i5 13500, 20 cores, 32GB ram) my best run is (file fully in page cache):
49.78user 0.69system 0:02.81elapsed 1795%CPU
A bit older version of the code on Mac pro M1 32 GB:
real 0m2.867s
user 0m23.956s
sys 0m1.329s
As I wrote in comments in the code, I have a few different roundings that the reference implementation. I have seend that there is an issue about that, but no specific rule yet.
Main points:
- use MemorySegment, it's faster than ByteBuffer
- split the work in a lot of chunks and distribute to a thread pool
- fast measurement parser by using a lot of domain knowledge
- very low allocation
- visit each byte only once
Things I tried that were in fact pessimizations:
- use some internal JDK code to vectorize the hashCode computation
- use a MemorySegment to represent the keys instead of byte[], to avoid
copying
Hope I won't have a bad surprise when running on the target server 😱
* start
* slower
* still bad
* finally faster than baseline :)
* starting to go fast
* improve
* we ball
* fix race condition an newline
* change threadpool
* ~18sec on my machine
* single thread memory mapped file reader, pool of processors
* cleanup of inner classes of MetricProcessor
* doubles are parsed without external functions, strings are lazily created from byte arrays
* remove load() MappedByteBuffer in memory
* fixed handling of newline
* fix a bug & correct locale used
* MappedByteBuffer size set to 1MB
* fixed rounding
* Do not use ArrayBlockingQueue.offer since it drops elements when queue is full
* MappedByteBuffer size = 32 MB
* Adding kgeri's solution
* parallelizing CalculatorAverage_kgeri
* fixing aggregation bugs, chunk size calc for small files
* removed GC logging
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
* fix for when there's no newline at end of input
* fix for when the final record ends on the chunk boundary
---------
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
* improve double reading by eleminating string parsing in between, make calculations over on integer instead of double, parse into double at the end only once
* more improvements, sharing a single StringBuilder to build all toStrings, minor performance gain.
* micro optimizations on reading temperature
* a small skip for redundant traverses, micro optmization
* micro optimization, eleminate some if cases, saves 0.5 seconds more
* micro optimization, calculate key hash ahead eleminates more more loop, saves 0.5 seconds more :)
* optimize key equals and handling the case when a region is larger than max integer size
---------
Co-authored-by: Yavuz Tas <yavuz.tas@ing.com>
* Initial version with multiple ideas
* Added virtual thread implementation based on certain task size
* Removed evaluate file
* Fixed test issues
* Added a custom input split
* first try
* format
* Update calculate_average_imrafaelmerino.sh
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
* Update src/main/java/dev/morling/onebrc/CalculateAverage_imrafaelmerino.java
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
---------
Co-authored-by: Rafael Merino García <imrafaelmerino@gmail.com>
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
* Implementation of 1brc - felix19350
* Added license header
* Fixed failing tests
* Replaced parsing of doubles with a custom parser and integer arithmetic
---------
Co-authored-by: Bruno Felix <bruno.felix@klarna.com>
* Initial version.
* Make PGO feature optional off-by-default. Needs PGO_MODE environment
variable to be set. Add -O3 -march=native tuning flags for better
performance.
* Adjust script to be more quiet.
* Adjust max city length. Fix an issue when accumulating results.
* Tune thomaswue submission.
mmap the entire file, use Unsafe directly instead of ByteBuffer, avoid byte[] copies.
These tricks give a ~30% speedup, over an already fast implementation.
* Optimize parsing of numbers based on specific given constraints.
* Fix for segment calculation for case of very small input.
* Minor shell script fixes.
* Separate out build step into file additional_build_step_thomaswue.sh,
simplify run script and remove PGO option for now.
* Minor corrections to the run script.
---------
Co-authored-by: Alfonso² Peterssen <alfonso.peterssen@oracle.com>
* A solution with Actor Model concurrency and MappedByteBuffer
* fix test cases
* revert back the file name to original
* cache String hashCode calculation via composing with Key object
* fix wrong key caching and eleminate duplicate String creation between actors
* update possible char count in a line, fix calculate_average.sh
* increase possible line length to 256 bytes, much safer to cover 100 chars I hope
---------
Co-authored-by: Yavuz Tas <yavuz.tas@ing.com>
* artpar's attempt
* artpar's attempt
* remove int -> Integer conversions, custom parsing for measurements
* remove allocations by caching station names
also remove Integer and use int instead to remove valueOf calls
* Fix result mismatch errors
* parse int instead of double
* reduce time spent reading the mapped buffer
* cleanup unused memory
* less is faster ? vector addition doesn't look worth it
* backout from virtual threads as well
* Fix breaking tests
Somewhat mixed collection of multiple ideas, mostly based initially
on using the new JDK Vector API for extracting offsets of newlines
and semicolons.
Runs locally in just under 11 seconds on 1B rows of input on a
2020 M1 Macbook Air.
* isolgpus: submission 1
* isolgpus: fix min value bug (breaks if a negative temperature never appears)
* isolgpus: remove unused collector
* isolgpus: fix split on chunk bug
* isolgpus: change name equality algo to a cheaper check.
* isolgpus: fix chunking state to cope with last byte of last chunk
* isolgpus: hash as we go, instead of at the end
* isolgpus: adjust thread count to core count
* isolgpus: change cores to 8 statically
---------
Co-authored-by: Jamie Stansfield <jalstansfield@gmail.com>
* First performance tweaks
* further tweaks
* collect into a treemap
* Tweak JVM options
* Inline rounding into collector
* reduce some operations
* oops, add missing braces
* tweak JVM options
* small fixes
* add min and max to processing
* fix min
* remove compact strings
* replace sumWithCompensation with naive sum implementation
* use UseShenandoahGC
* integrate mmap
* integrate mmap
* Fix messed up array logic
* Set jdk version
* Use Integer calculation instead of double, add unit-test
* Bring back StationIdent optimization
Originally, StationIdent was using byte[] to store names, so the extra
String allocation could be avoided. However, that produced incorrect
sorting.
Sorting is now moved to the result merging step. Here, names are
converted to Strings.
* Implement readStationName with SIMD 256bit
* Rebase and cleanup test code, now that it's in the project
* Fix seijikun formatting
* Fix test failure in specific jobCnt edge-cases
* Also switch to graalvm
In case of key collision broken implementation will likely attribute
measurements to the wrong key and therefore it is better to have
non-zero value to end up with a wrong average value.
When all measurements are zero then averages are also zero even
when attributed to the wrong keys.
Updates #91
* Added tests for endian-calculations (had these in a different class, perhaps handy for others to see as well)
Inlined the hash function, runs locally in 2.4sec now, hopefully endian issues fix
Added equals to support any city name up to 1024 in length, don't rely on hash
* For clarity I've updated the code so endian doesn't change the hashes, easier to debug.
* Fixing bug in array check
Simple is faster
* Also spotted the diff, not just the big exception
Fixed buffer limit issue
Input created via
```sh
bash -c 'for i in {1..10000} ; do echo "id$i;0.0" ; done' >./src/test/resources/samples/measurements-10000-unique-keys.txt
```
and output via baseline implementation.
Keys are short and very similar which improves chances for collision
and hence are good for testing.
Fixes#91
* Use open-addressing scheme to deal with hash table collisions. Reduce concurrency from 16 to 8. Use bit mask rather than mod operator to confine hash code to table range.
* Properly handle file partitions that reside entirely within a line.
* Reorder statements in doProcessBuffer.
Adds test samples that can be used for unit tests or to verify
implementations via:
```bash
for sample in $(ls src/test/resources/samples/*.txt)
do
echo "Validating $sample"
rm -f measurements.txt
ln -s $sample measurements.txt
diff <(./calculate_average.sh) ${sample%.txt}.out
done
rm measurements.txt
```
For #61
Added SWAR (SIMD Within A Register) code to increase bytebuffer processing/throughput
Delaying the creation of the String by comparing hash, segmenting like spullara, improved EOL finding
Co-authored-by: Gunnar Morling <gunnar.morling@googlemail.com>
* Initial implementation using Shenandoah GC and parallel iteration
* Use memory mapped files
* Iterate the buffer once and use BigDecimal parsing instead of Double.parseDouble
* Add information about Graal
* Add sdk use to calculate script
* Memory mapped file, single-pass parsing, custom hash map, fixed thread pool
The threading was a hasty addition and needs work
* Used arraylist instead of treemap to reduce a little overhead
We only need it sorted for output, so only construct a treemap for output
* Attempt to speed up double conversion
* Cap core count for low-core systems
* Fix wrong exponent
* Accumulate measurement value in double, seems marginally faster
Benchmark Mode Cnt Score Error Units
DoubleParsingBenchmark.ourToDouble thrpt 10 569.771 ± 7.065 ops/us
DoubleParsingBenchmark.ourToDoubleAccumulateInToDouble thrpt 10 648.026 ± 7.741 ops/us
DoubleParsingBenchmark.ourToDoubleDivideInsteadOfMultiply thrpt 10 570.412 ± 9.329 ops/us
DoubleParsingBenchmark.ourToDoubleNegative thrpt 10 512.618 ± 8.580 ops/us
DoubleParsingBenchmark.ourToDoubleNegativeAccumulateInToDouble thrpt 10 565.043 ± 18.137 ops/us
DoubleParsingBenchmark.ourToDoubleNegativeDivideInsteadOfMultiply thrpt 10 511.228 ± 13.967 ops/us
DoubleParsingBenchmark.stringToDouble thrpt 10 52.310 ± 1.351 ops/us
DoubleParsingBenchmark.stringToDoubleNegative thrpt 10 50.785 ± 1.252 ops/us
