Is spring mvc (spring boot + tomcat) too slow?

[damir78]

Hi guys,

the last few days i have tested the following benchmarks intensively:

• https://github.com/xmlet/template-benchmark (by @mbosecke and @fmcarvalho )
• https://github.com/xmlet/spring-comparing-template-engines (by @jreijn )

the results highly surprised me. Yes htmlFlow and Rocker are the champions.

But look the thymeleaf results: in template-benchmark is very slow but in spring-comparing-template-engines good enough. Why make it possibly?

I think now that spring (or the layer apache tomcat + spring boot) make everything slow and the end thoughtput IS TOO SLOW as well, and makes Thymeleaf "not bad"??

What do you think about?

[Vest]

Hi Damir,

I don’t have an answer for you now. I have to study the projects you gave and compare them with the current one.

In my tests, when I analyzed two slowest engines, I found out that the disabling of caching makes them really slow because of internal locks (synchronization of a shared object). Other engines look more efficient.

Other engines I didn’t analyze too deeply. Maybe they are slow because of I/O operations. E.g. repeating loading from the classpath or file system.

I will update the issue if I have more information. But I will do this next week probably.

[Vest]

By the way, have your tried testing this project from your PC? What are the results? We use Apache Benchmark, and we measured Total time.

[damir78]

Hi Vest,
good points. Thank you. I think it mus be "battle ready" or "production like" configuration.
Do you remember we had "precompiled JSPs" ))). It looks like we need to differentiate "first time rendering" and "production execution".

[damir78]

But it is really interesting that the layer ATS (apache+tomcat+spring) can slow down the template engine (workload). TE can render 100 views but the layer ATS allows only 40 views to be executed.

[fmcarvalho]

I share the same doubts of @damir78. In this benchmark we are evaluating not only the template engine, but also the spring HTTP handling, spring routing, reading and writing from and to HTTP (this includes IO), etc... It seems that all these together has much higher overhead than the engine itself rendering a template.

In fact, testing this benchmark in my desktop (Intel i7) with Apache Benchmark, I never observed consistent results. The most performant engine is different on every experimental test.

Also I did not give deep attention trying to understand the internals and reasons for this inconsistency.

Nevertheless, I accepted the observations released in this repository and I think the proposed measurement approach has a good foundation for those results.

[lcduarte]

In my masters thesis I used these two benchmarks to evaluate the performance of HtmlFlow, i.e:

• https://github.com/xmlet/template-benchmark (by @mbosecke and @fmcarvalho )
• https://github.com/xmlet/spring-comparing-template-engines (by @jreijn )

These two take a very different approach.

In my opinion the template-benchmark takes a more direct approach to actually measuring the performance of each template engine as it doesn't introduce much overhead other than method calls for methods which render the template and the overhead of JMH, which is very lightweight compared to Spring.

On the other hand spring-comparing-template-engines introduces a lot more overhead with the whole Spring framework and having another tool issuing a massive number of HTTP requests. One could argue that this approach reflects the real use case of template engines but since template engines render their templates so effectively the orders of magnitude between template render times and the benchmark overhead somewhat "blur" the results.

So, if you are looking for raw performance comparison between template engines you should use template-benchmark. Otherwise for a more realistic use of template-engines you should use this benchmark and observe the results acknowledging the overhead.

PS: One possible solution to improve the quality of your results using this benchmark is creating a dummy route and benchmarking it. This way you can measure the overhead of Spring and remove it from the actual benchmarks you want to perform.

[Vest]

Hello everyone,

I could never imagine that this repository that nobody touched for years will become so popular this week. I see that you are discussing the different results that our (not mine, I became the maintainer, because I was more active than nobody) repository and template-benchmark produce. Since the performance is a broad topic, I have decided to answer your question "why thymeleaf is fast in the current repository, but it is slow in someone's else?".

I tried to analyze Thymeleaf internals using a Java profiler. Before I start, I want to tell you that the repository template-benchmark is using JMH, and there is no multi-threaded testing within the single JVM. Every time the new JVM process is started, where Thymeleaf was tested in a single thread:

BaseBenchmark.java

@Fork(5)
@Warmup(iterations = 5)
@Measurement(iterations = 10)

I had to modify it a little:

@Fork(5)
@Warmup(iterations = 5)
@Measurement(iterations = 10000)

Otherwise it is impossible to attach the profiler to JVM for the... hm... profiling. Because JVM shuts down quickly.

The first thing that you might probably notice is that both tests produce different results. The image below shows results of template-benchmark (left), and this repository (right):

The reason for this is in the line 578 of TemplateManager.java. The test template-benchmark uses cache property of Thymeleaf. However, our repository doesn't use cache.
In our repository, Thymeleaf calls the line 666 of TemplateManager.java, that doesn't involve any additional processing.

So, the answer to the question is cache. E.g. after I enabled the property Cacheable for Thymeleaf in our repository:

@Bean
public SpringResourceTemplateResolver thymeleafTemplateResolver() {
    SpringResourceTemplateResolver templateResolver = new SpringResourceTemplateResolver();
    ...
    templateResolver.setCacheable(true);

I got approximately 49.020 seconds for Thymeleaf instead of 10.27 seconds.

But, it is not a single answer. After I analyze the new profiling result, I found out that Thymeleaf for Spring uses SpringEL (see Issue #95) instead of OGNL. And again, as I said two different tests produce two different results.

Just in case, I will provide the screenshot of our project when cache was enabled. What is interesting, that the limited step was the data transfer, not the template processing.

I guess that's everything what I would like to tell you about the issue that is currently open. I think it is not a real issue, but a "question", and I will change its label.
I don't have any experience in Thymeleaf, so I don't know how to use it efficiently. As I said in the question #18, I wanted to test the engines with Spring, but not how your CPU and RAM work, when cache is enabled (yes, I emphasized this word, because I think you will not be the last one, who will be surprised with the project's configuration). If you have questions why other engines work differently, I would suggest you to profile their code, or ask the engine's owners.

I am open to your comments, but I don't see a bug here, so I am asking the issue owner @damir78 to close it. I don't want to be rude, but I will close this issue in few months without any activity.
By the way, in few recently closed issues I described why some engines are slow. I did a brief study to satisfy my curiosity :-)

p.s. @lcduarte I was thinking earlier about adding the empty root to this repository. But I decided not to add it, because without automated testing every new engine makes my life worseharder and harder. Many engines are very old and the Spring integration wasn't simple for me either :(

Kind regards,
Vest

[damir78]

Hi @Vest ,
thank for you research. There are very interesting informations.
I think it will help another developers who use and who wanna write own template engine.

Regargs

[Vest]

Thank you @damir78, I see you liked my comment. And I hope you found it helpful.

I appreciate this.