* calculate_average_mtopolnik
* short hash (just first 8 bytes of name)
* Remove unneeded checks
* Remove archiving classes
* 2x larger hashtable
* Add "set" to setters
* Simplify parsing temperature, remove newline search
* Reduce the size of the name slot
* Store name length and use to detect collision
* Reduce memory loads in parseTemperature
* Use short for min/max
* Extract constant for semicolon
* Fix script header
* Explicit bash shell in shebang
* Inline usage of broadcast semicolon
* Try vectorization
* Remove vectorization
* Go Unsafe
* Use SWAR temperature parsing by merykitty
* Inline some things
* Remove commented-out MemorySegment usage
* Inline namesMem.asSlice() invocation
* Try out JVM JIT flags
* Implement strcmp
* Remove unused instance variables
* Optimize hashing
* Put station name into hashtable
* Reorder method
* Remove usage of MemorySegment.getUtf8String
Replace with UNSAFE.copyMemory() and new String()
* Fix hashing bug
* Remove outdated comments
* Fix informative constants
* Use broadcastByte() more
* Improve method naming
* More hashing
* Revert more hashing
* Add commented-out code to hash 16 bytes
* Slight cleanup
* Align hashtable at cacheline boundary
* Add Graal Native image
* Revert Graal Native image
This reverts commit d916a42326d89bd1a841bbbecfae185adb8679d7.
* Simplify shell script (no SDK selection)
* Move a constant, zero out hashtable on start
* Better name comparison
* Add prepare_mtopolnik.sh
* Cleaner idiom in name comparison
* AND instead of MOD for hashtable indexing
* Improve word masking code
* Fix formatting
* Reduce memory loads
* Remove endianness checks
* Avoid hash == 0 problem
* Fix subtle bug
* MergeSort of parellel results
* Touch up perf
* Touch up perf
* Remove -Xmx256m
* Extract result printing method
* Print allocation details on OOME
* Single mmap
* Use global allocation arena
* Add commented-out Xmx64m XXMaxDirectMemorySize=1g
* withinSafeZone
* Update cursor earlier
* Better assert
* Fix bug in addrOfSemicolonSafe
* Move declaration lower
* Simplify code
* Add rounding error test case
* Fix DANGER_ZONE_LEN
* Deoptimize parseTemperatureSimple()
* Inline parseTemperatureAndAdvanceCursor()
* Skip masking until the last load
* Conditionally fetch name words
* Cleanup
* Use native image
* Use supbrocess
* Simpler code
* Cleanup
* Avoid extra condition on hot path
* added code
* Fixed pointers bugs
* removed my own benchmark
* added comment on how I handle hash collisions
* executed mwvn clean verify
* made scripts executable & fixed rounding issues
* Fixed way of dealing with hash collisions
* changed method name sameNameBytes to isSameNameBytes
* changes script from sh to bash
* fixed chunking bug
* Fixed bug in chunking when file size is too small
* added Runtime.getRuntime().availableProcessors
* Solution without unsafe
* Solution without unsafe
* Solution without unsafe, remove the usage of bytebuffer, passes the create_measurements3 test
* bug fix for 10k test, update also the CreateMeasurements3.java to use '\n' as newline instead of the os value (if it runs on windows it uses crlf and "breaks" the file format )
---------
Co-authored-by: Giovanni Cuccu <gcuccu@imolainformatica.it>
- faster merge by ignoring empty entries in the map
- enable CDS for faster startup (added `prepare_serkan-ozal.sh` to generate CDS archive in advance)
- some tweaks with JVM options
- optimized result printing
* Read file with multiple virtual threads and process chunks of file data in parallel.
* Updated logic to bucket every chunk of aggs into a vector and merge them into a TreeMap for printing.
* Virtual Thread / File Channels Impl.
* Renamed files with GHUsername.
* Added statement to get vals before updating.
* Added executable permission to the files.
* Latest snapshot (#1)
preparing initial version
* Improved performance to 20seconds (-9seconds from the previous version) (#2)
improved performance a bit
* Improved performance to 14 seconds (-6 seconds) (#3)
improved performance to 14 seconds
* sync branches (#4)
* initial commit
* some refactoring of methods
* some fixes for partitioning
* some fixes for partitioning
* fixed hacky getcode for utf8 bytes
* simplified getcode for partitioning
* temp solution with syncing
* temp solution with syncing
* new stream processing
* new stream processing
* some improvements
* cleaned stuff
* run configuration
* round buffer for the stream to pages
* not using compute since it's slower than straightforward get/put. using own byte array equals.
* using parallel gc
* avoid copying bytes when creating a station object
* formatting
* Copy less arrays. Improved performance to 12.7 seconds (-2 seconds) (#5)
* initial commit
* some refactoring of methods
* some fixes for partitioning
* some fixes for partitioning
* fixed hacky getcode for utf8 bytes
* simplified getcode for partitioning
* temp solution with syncing
* temp solution with syncing
* new stream processing
* new stream processing
* some improvements
* cleaned stuff
* run configuration
* round buffer for the stream to pages
* not using compute since it's slower than straightforward get/put. using own byte array equals.
* using parallel gc
* avoid copying bytes when creating a station object
* formatting
* some tuning to increase performance
* some tuning to increase performance
* avoid copying data; fast hashCode with slightly more collisions
* avoid copying data; fast hashCode with slightly more collisions
* cleanup (#6)
* tidy up
* Initial submission for jonathan_aotearoa
* Fixing typos
* Adding hyphens to prepare and calculate shell scripts so that they're aligned with my GitHub username.
* Making chunk processing more robust in attempt to fix the cause of the build error.
* Fixing typo.
* Fixed the handling of files less than 8 bytes in length.
* Additional assertion, comment improvements.
* Refactoring to improve testability. Additional assertion and comments.
* Updating collision checking to include checking if the station name is equal.
* Minor refactoring to make param ordering consistent.
* Adding a custom toString method for the results map.
* Fixing collision checking bug
* Fixing rounding bug.
* Fixing collision bug.
---------
Co-authored-by: jonathan <jonathan@example.com>
* CalculateAverage_pdrakatos
* Rename to be valid with rules
* CalculateAverage_pdrakatos
* Rename to be valid with rules
* Changes on scripts execution
* Fixing bugs causing scripts not to be executed
* Changes on prepare make it compatible
* Fixing passing all tests
* Increase direct memory allocation buffer
* Fixing memory problem causes heap space exception
* Initial impl
* Fix bad file descriptor error in the `calculate_average_serkan-ozal.sh`
* Disable Epsilon GC and rely on default GC. Because apparently, JIT and Epsilon GC don't play well together in the eval machine for short lived Vector API's `ByteVector` objects
* Take care of byte order before processing key length with bit shift operators
* Fix key equality check for long keys
/**
* Solution based on thomaswue solution, commit:
* commit d0a28599c2
* Author: Thomas Wuerthinger <thomas.wuerthinger@oracle.com>
* Date: Sun Jan 21 20:13:48 2024 +0100
*
* Changes:
* 1) Use LinkedBlockingQueue to store partial results, that
* will then be merged into the final map later.
* As different chunks finish at different times, this allows
* to process them as they finish, instead of joining the
* threads sequentially.
* This change seems more useful for the 10k dataset, as the
* runtime difference of each chunk is greater.
* 2) Use only 4 threads if the file is >= 14GB.
* This showed much better results on my local test, but I only
* run with 200 million rows (because of limited RAM), and I have
* no idea how it will perform on the 1brc HW.
*/
* fix test rounding, pass 10K station names
* improved integer conversion, delayed string creation.
* new algorithm hash, use ConcurrentHashMap
* fix rounding test
* added the length of the string in the hash initialization.
* fix hash code collisions
