Commit Graph

6 Commits

Author SHA1 Message Date
Jason Nochlin
eff73db9fe
evaluate2.sh: Check output of warmup run and abort early if failed (#333)
* refactor: replace xtrace with "print_and_execute" function

* nit: stylize error messages

* replace out_expected.txt with measurements_1B.out

* print

* prevent errors on cleanup

* run tests and check warmup run output before running benchmark

* move "git diff" pretty diff output to test.sh

* Ensure "set -e" is re-enabled if we followed a "continue" branch

* add timeouts to test.sh invocations

* use diff with tocsv.sh to show differences on failed test

* add --quiet mode to test.sh

* move prepare_$fork.sh invocation to right below hyperfine since test.sh also invokes it

* Revert "add --quiet mode to test.sh"

This reverts commit 13e9fb7f395c1bd64a62528b8349803bc1366941.

* use tee to capture test output to a temp file and print contents on failure

---------

Co-authored-by: Jason Nochlin <hundredwatt@users.noreply.github.com>
2024-01-13 12:19:29 +01:00
Alexander Yastrebov
af269e39fc Update test*.sh to support input file pattern
This is useful for testing fork(s) against subset of test samples
2024-01-10 17:53:13 +01:00
Alexander Yastrebov
7def69eee7 Fix test.sh to use prepare script 2024-01-10 14:29:18 +01:00
Alexander Yastrebov
8124a0ca08 test: add script to test all implementations
The script tests all implementations and prints PASS or FAIL status.
In case of failure it also prints implementation output to stderr.

This will be handy for adding new test samples.

Show test statuses and omit failing output:
```sh
$ ./test_all.sh 2>/dev/null
PASS artsiomkorzun
PASS baseline
PASS bjhara
PASS criccomini
FAIL ddimtirov
FAIL ebarlas
PASS filiphr
FAIL itaske
PASS khmarbaise
FAIL kuduwa-keshavram
FAIL lawrey
PASS padreati
FAIL palmr
PASS richardstartin
FAIL royvanrijn
FAIL seijikun
PASS spullara
PASS truelive
```

Show only passing implementations:
```
$ ./test_all.sh 2>/dev/null | grep PASS | cut -d' ' -f2
artsiomkorzun
baseline
bjhara
criccomini
filiphr
khmarbaise
padreati
richardstartin
spullara
truelive
```

For #61
2024-01-04 21:16:18 +01:00
Gunnar Morling
b6d33fd8fe Expanding tests and eval infra 2024-01-04 17:22:00 +01:00
Gunnar Morling
c74dc32b35 Adding test loop 2024-01-04 15:24:27 +01:00