Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

RunTimeMilliseconds not working properly #266

Open
fota666 opened this issue Oct 9, 2019 · 3 comments
Open

RunTimeMilliseconds not working properly #266

fota666 opened this issue Oct 9, 2019 · 3 comments

Comments

@fota666
Copy link

fota666 commented Oct 9, 2019

I made a small test to understand the NBench infrastructure, but the results are quite surprising.
The SkipWarmups is set to true, the NumberOfIterations is set to 2, but the SetUp is called three times.
After the first SetUp call, the SleepTroughput method is called 80 times - I assume these calls are the estimation calls.
The second and third SetUp are the actual benchmark calls, but there are only 10 SleepTroughput calls after them. And the counter's actual value is 9.59 operations
In my opinion the expected outcome should be around 20 operations (sleep for 100 ms in 2000 ms time frame).
If RunTimeMilliseconds is 1000, then the only change is that the first SetUp call is continued by 40 SleepTroughput calls instead of the previous 80, but the number of operations during the benchmark test is the same (~10).

[PerfSetup]
public void SetUp(BenchmarkContext context)
{
    stopWatchTotal.Start();
    addCounter = context.GetCounter(AddCounterName);
}

[PerfBenchmark(NumberOfIterations = 2, RunMode = RunMode.Throughput, RunTimeMilliseconds = 2000, TestMode = TestMode.Test, SkipWarmups = true)]
[CounterThroughputAssertion(AddCounterName, MustBe.GreaterThan, 1.2)]
public void SleepTroughput(BenchmarkContext context)
{
    stopWatchMessage.Restart();
    Thread.Sleep(100);
    addCounter.Increment();
}
@NicolaAtorino
Copy link

NicolaAtorino commented Feb 15, 2021

i'm running into the same problem more than one year after .Any luck finding the issue, or you abandoned the usage of the library ?

@fota666
Copy link
Author

fota666 commented Feb 16, 2021

Sorry, this was in my previous life :) Unfortunately I do not remember the solution.

@NicolaAtorino
Copy link

Actually i found the solution ! It's in my opinion a bug in the code. I am opening a pull request for this. Thanks !

NicolaAtorino pushed a commit to NicolaAtorino/NBench that referenced this issue Mar 3, 2021
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants