Add JStachio to HtmlFlow benchmark

[agentgt]

Benchmark                Mode  Cnt        Score       Error  Units
HtmlFlow.presentations  thrpt    6   445525.472 ±  2674.491  ops/s
HtmlFlow.stocks         thrpt    6   108397.828 ±  1109.594  ops/s
JStachio.presentations  thrpt    6  1260078.083 ± 18070.685  ops/s
JStachio.stocks         thrpt    6   235090.054 ±  6368.848  ops/s

Benchmarks are lies but I have a hard time believing the readme that HtmlFlow is the fastest. I also think Rocker is probably not setup correctly.

The default branch on my repo also does UTF-8 output which IMO is more important especially if we are talking about HTML output: https://github.com/agentgt/template-benchmark

Does HtmlFlow do pre-encoding.

[fmcarvalho]

Benchmarks are lies

1. Your statement appears biased.

2. This benchmark is publicly available for anyone to check themselves. We have moved past the era where claims of being the fastest lacked source code for verification. I have personally experienced this, receiving responses like, "You can develop your own version and check your results." Fortunately, this is no longer the case, and the community is now able to verify the accuracy of such claims.

3. The benchmark in question is a fork from the upstream repository at https://github.com/mbosecke/template-benchmark. I have only added HtmlFlow while leaving the rest unchanged. Rocker was already present. Unfortunately, the upstream repository is outdated, and its author no longer maintains it.

4. My experimental results from 2018 are published on GitHub and Dzone at https://dzone.com/articles/modern-type-safe-template-engines. Anyone can check and comment here as well.

5. Things change. I was not aware of JStachio. It appears to have been released in 2022? I am excited to study it further and understand its internals. I appreciate the Mustache idiom.

6. HtmlFlow employs a different approach. Instead of an external DSL (like JSP, Thymeleaf, Mustache, and others), HtmlFlow uses an internal DSL similar to KotlinX.html, J2html, ScalaTags, React JSX, and others. I prefer an internal DSL over an external one. However, others may prefer to keep templates with an external DSL. Fortunately, we have many options.

7. Benchmarks are NOT the SOLE metric. They are just one of many. HtmlFlow distinguishes itself from the competition through several key differentiators:

• Type safety
• Progressive SSR
• Unopinionated

I'm not claiming that these characteristics are superior to those of other templates. I am merely highlighting them as unique features, and those who appreciate them can make their choice accordingly.

As a bonus, in 2019, HtmlFlow was the fastest in those available benchmarks. Maybe, now is different. Let’s check and fix.

BTW as of my knowledge in 2019, benchmarks for SSR were limited to two:

1. https://github.com/mbosecke/template-benchmark
2. https://github.com/jreijn/spring-comparing-template-engines

The second one is still the most popular and up-to-date. Unfortunately, it relies on AB rather than JMH, and I am not convinced that it is the best method for benchmarking. I have submitted a pull request to upgrade that benchmark to use JMH for more accurate results, but it has remained open for over a year, which is disheartening. jreijn/spring-comparing-template-engines#89
Regrettably, I am the sole contributor who has made an effort to transition that benchmark to use JMH.

A lengthy discussion took place on this matter at jreijn/spring-comparing-template-engines#51. One of the main contributors, Vest, was not receptive to the proposed change.

For those of you seeking accurate benchmarks, perhaps you could review spring-comparing-template-engines and contribute to its enhancement. I believe the community would benefit from a widely accepted benchmark rather than multiple forks with only a handful of stars. Notably, spring-comparing-template-engines has the most stars, totaling 418.

Finally, in the next few weeks, I may not have time to thoroughly study everything you've stated. Rest assured, I will pay close attention to your work as soon as I can.

Until then, I will keep this PR open.

Thank you,
Miguel

[agentgt]

I am sorry I should have been more diplomatic about it. Benchmarks the code themselves may not be lies but damn do they bring around a lot of contention and I think can mislead the inexperienced.

Benchmarks are lies

Your statement appears biased.

That was a tongue and cheek reference that most of the industry knows when talking about benchmarks: "There are three kinds of lies: Lies, Damned Lies, and Statistics". The joke is to replace statistics with benchmarks. There is probably an XCD comic somewhere on this.

BTW I don't think they are all lies otherwise why would I jokingly ask you to update the readme?

Also benchmarks vary greatly obviously by environment. I actually run this benchmark on multiple environments via github action: https://github.com/agentgt/template-benchmark/actions/runs/5431999133

The results vary wildly between the top contenders. And unlike a lot of others that cloned the JMH I spent time trying to optimize the others.

HtmlFlow employs a different approach. Instead of an external DSL (like JSP, Thymeleaf, Mustache, and others), HtmlFlow uses an internal DSL similar to KotlinX.html, J2html, ScalaTags, React JSX, and others. I prefer an internal DSL over an external one. However, others may prefer to keep templates with an external DSL. Fortunately, we have many options.

I am aware of the internal DSL approach. I had one of the first libraries that does that approach: https://github.com/agentgt/jatl (notice how ancient it is). However mine I didn't bother with trying to enforce the schema of the HTML. So I like your library because it is a way better version than my ancient garbage! I will say in the long run I found internal DSL to be as bad as JSP so I ended up embracing Mustache/Handlebars.

A lengthy discussion took place on this matter at jreijn/spring-comparing-template-engines#51. One of the main contributors, Vest, was not receptive to the proposed change.

Yes I had similar experience on just trying to improve a library reported performance that is not my own: jreijn/spring-comparing-template-engines#65

For those of you seeking accurate benchmarks, perhaps you could review spring-comparing-template-engines and contribute to its enhancement. I believe the community would benefit from a widely accepted benchmark rather than multiple forks with only a handful of stars. Notably, spring-comparing-template-engines has the most stars, totaling 418.

With Vest apparently making decisions I'm not sure if its worth my time trying to contribute (again)... even though I did submit a PR yesterday: jreijn/spring-comparing-template-engines#127 (like check the response...) . I would be happy to collaborate with you on a fork or perhaps the project will eventually be more welcoming.

Anyway I apologize again if my "lies" comment came out as accusatory or something.

Cheers
-Adam

[agentgt]

If we did come up with a better benchmark I suggest using some of the scaffolding / harness of TechEmpower and then have a separate one with JMH. That is I think there is some benefit to a non JMH full stack test.

BTW I am a massive fan of this library. It is how I found out that you were updating the benchmarks (this fork. I follow the project.) and is why I gave you the PR because I thought you might find it interesting. If you weren't updating it I probably would not have submitted the PR.

[fmcarvalho]

I really appreciated to find JStachio.

Your proposal to utilize some of the scaffolding from TechEmpower is intriguing. I haven't used it before, and I need to take the time to study it.

As mentioned, I am currently swamped with deadlines for the next two weeks. However, I assure you that I will follow up on this afterward.

Thank you,
Miguel

[agentgt]

As mentioned, I am currently swamped with deadlines for the next two weeks. However, I assure you that I will follow up on this afterward.

Also as a reminder to me for later is I want to talk to you about potential optimization for HtmlFlow doing pre-encoding. Pre-encoding is basically taking the static parts of the template and encoding them into bytes apriori. This is to save on the cost of doing that on each request (converting Java Unicode into actual bytes). Almost all the templating benchmarks do not test for non ASCII characters but the Techempower benchmark does and it makes a difference doing the pre-encoding. Currently only JStachio, JTE and Rocker do pre-encodig.

In theory it might be possible for HtmlFlow to do this. That is detect multiple tag calls and coalesce as a byte array (or string at first).

Using String as an example instead of bytes. body().div() would do writer.append("<body><div>") instead of what I assume it does of writer.append("<body>").append("<div>").

Again no need to reply but just a reminder to myself.

[fmcarvalho]

Hi @agentgt,

I've reviewed your pull request, and I noticed similar throughput in comparison to your results. It's truly remarkable how JStachio outperforms HtmlFlow and Rocker by 2 or 3 times.

HtmlFlow also performs pre-encoding; for instance, body().div() results in writer.append("<body><div>"). In our case, we manage two types of pages: HtmlDoc and HtmlView, and the latter undergoes pre-encoding, which we are using in this benchmark.

Based on my observations, the only optimisation we haven't implemented is encoding static parts into bytes. However, since I noticed that you can't take advantage of this optimisation in this benchmark due to its focus on string handling.

Therefore, I'm having difficulty understanding why JStachio exhibits much better performance than HtmlFlow and Rocker.

The only difference I found is in the way static HTML strings are managed. While HtmlFlow performs pre-encoding at runtime, JStachio does it at compile time. Thus, HtmlFlow will load that string from an instance final field staticHtmlBlock, whereas JStachio loads it as a string literal, which we may find on bytecode ldc such as for JStachioPresentationsTemplate.class:

1: ldc #143 // String <!DOCTYPE html>\n<html>\n<head>\n\t<meta charset=\"utf-8\">\n\t<meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n\t<meta http-equiv=\"content-language\" content=\"IE=Edge\">\n\t<title>\n\t\tJFall 2013 Presentations - htmlApi\n\t</title>\n\t<link rel=\"Stylesheet\" href=\"/webjars/bootstrap/3.3.7-1/css/bootstrap.min.css\" media=\"screen\">\n</head>\n<body>\n<div class=\"container\">\n\t<div class=\"page-header\">\n\t\t<h1>\n\t\t\tJFall 2013 Presentations - htmlApi\n\t\t</h1>\n\t</div>\n

However, for the HtmlFlow scenario, I was considering whether the JVM runtime would optimize the way it fetches that String. I will conduct a test by interleaving String literals in HtmlFlow templates' definition only to check if performance increases.

Rocker also has string literals computed at compile time like JStachio. However, they pass those strings through a PlainTextUnloadedClassLoader, and this might lead to some loss of JVM optimization.

Regarding your point:

I also think Rocker is probably not set up correctly

I'm not an expert in Rocker, but I'm using it as provided by the original template-benchmark. Is there anything else to configure in Rocker for better performance?

Regarding a new benchmark, I would like to work on it and publish an article that captures the community's attention. Do you have any suggestions on how we could collaborate on this? I don't have any knowledge about TechEmpower, and perhaps you could provide some guidance or suggestions.

Cheers,
Miguel

[fmcarvalho]

@agentgt why don't you include the ThreadLocal<StringBuilder> buffer internally in JStachio, rather than using it only in the benchmark?

[agentgt]

@agentgt why don't you include the ThreadLocal<StringBuilder> buffer internally in JStachio, rather than using it only in the benchmark?

Because of Loom and or the reality that folks will want to probably use some other buffer (e.g. netty's pool of buffers). I might just remove it altogether from the benchmark as I think its only like 10% difference last time I checked.

Anyway buffer output is discussed in the Javadoc here: https://jstach.io/jstachio/io.jstach.jstachio/io/jstach/jstachio/output/package-summary.html

With tons of techniques. I think most frameworks don't offer that many output options so I one point wanted to separate that out as a decoupled module so that other templating engines could use it.

[fmcarvalho]

I might just remove it altogether from the benchmark as I think its only like 10% difference last time I checked.

Indeed, for the stocks workload, JStachio performs well. However, for presentations (copied from jreijn/spring-comparing-template-engines), JStachio experiences a slowdown of half the throughput when using the TLS buffer.

I have already merged your pull request and removed the TLS buffer. Additionally, I have included in the GitHub Actions workflow the execution of both workloads on Ubuntu, Windows, and Mac.

Note that Mac runs on only 3 cores (https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories).

I will utilize the last results from Ubuntu to update the charts on the README.