Develop a Laika CLI for Node.js and Native #360
Comments
|
Many thanks for authoring this proposal. While I generally think this presents an interesting and exciting opportunity, I also have quite a number of doubts, and they are less about the feasibility of this effort (I really cannot judge feasibility of anything related to Scala Native), but more about a) the scope of work that I would see as mandatory to make this an attractive offering for which I'll try to provide some context below and b) the fact that a CLI application is not a frequently requested feature at the moment. Since this comment turned out to become rather long, I've split it into three main section that address A) the target audience you intend to reach with the proposal, B) the scope of work I'd see as necessary to produce an attractive offering and C) the question of development and maintenance, specifically in the context of the other work happening in the coming months. Target AudienceThe proposal so far discusses the question whether it is feasible and what advantages CLI applications would have (I think they are pretty obvious), but an even more critical question for me is: who would actually benefit, and how many? And by that expand the question "is it feasible" to "is it worth the effort"? I personally expect the effort to be high, and I'll explain why in the next section below about scope. Regarding the target audience I see primarily two variants, which each come with their own downsides. The first option would be to still mostly target the Scala community. I don't feel like this would be attractive though. This would mean we target a subset of an audience that is already not that large to begin with. Usage, feedback and participation have somewhat increased since the 0.18 release and the Typelevel adoption, but given that Laika has a fairly unique position as one of very few pure-JVM options, I still feel it's not as high as it could be. And within the user base there is a clear dominance of JVM users vs. Scala.js users and within the former a clear dominance of plugin usage. I'll give some Sonatype download statistics for November as an example, it's for 0.18.2 and 0.19.0 combined to ignore aspects like migration speed: it's around 11,000 for the plugin, around just 150 for This leaves us with the second option, which is to target a wider audience and not just the Scala community. Existing usage patterns and download statistics are less relevant for this discussion as we would seek to attract a completely new group of users. This feels generally more attractive to me, but has its own downsides: We would enter a large and crowded arena, with tools like Jekyll, Hugo, Docusaurus and many more. All of which have a much larger user base, larger number of maintainers and 3rd party extensions and in some cases also a larger feature set. We would need to really think about how and where we would announce and market such a solution and what the selling points would be that could potentially tempt users to switch. When I started with Laika I was very well aware of the crowded nature of this arena and deliberately chose not to compete in it. If others feel like it is worth it, I am happy to chime in and contribute, but I currently do not consider to lead such an effort (more on that under Development & Maintenance). ScopeThe second important discussion is the scope of the work. An option that would be very unattractive in my view is to simply support all functionality we get mostly for free simply by adding those new target platforms and a bit of bootstrap logic for the CLI app itself while stripping off all modules that don't work on Native out of the box. The huge downsides with that would be twofold: the documentation which already needs to distinguish between library API and sbt plugin would receive further fragmentation by many features not being supported on Native or Node or in a very different way. Secondly competing with any comparable tool becomes even harder when we only offer a small subset of functionality. I would want to avoid increasing the maintenance burden by cross publishing when we realistically would only reach a small audience. For this reason I feel like it's appropriate to hijack this proposal a bit for a wider discussion of the Laika road map and I hope you don't mind the wall of text that will follow. In my view aiming for feature parity with what the sbt plugin offers plus filling functional gaps compared to competing native apps in this space would be the only viable approach. Which turns this idea into a fairly ambitious endeavour. I would actually love to change the title of this proposal from "Publish..." to "Develop..." if you guys don't mind as that would more accurately reflect the scope of the work. I haven't thought about every module and feature in detail, but I'll list a few things that spontaneously come to mind just as an example for the challenge that would lie ahead. Helium ConfigurationIn contrast to the configuration in Laika Core which supports both, HOCON and programmatic configuration for (I think) all features out of the box, that is currently not the case for Helium. I think it is feasible to add that support, but it's a non-trivial size of work. The DX of the syntax for this vast set of options would need to be carefully considered and HOCON decoders would need to be written (which is a lot of boilerplate as Laika does not support macros like some JSON libraries do - this might change when everyone is on Scala 3, but until I'd prefer not to implement those twice for Scala 2 and 3) PDF SupportAs you might be aware, PDF support in Laika is based on the Apache FOP project which in turn is based on the XSL-FO format which appears to be a legacy format right now. I would have preferred to avoid the format, but there are no comparable libraries for the JVM and not requiring any external tools was a primary goal from the beginning. Apache FOP will never work on Scala.js nor Scala Native, so PDF support would need to be re-implemented from scratch based on tooling available for the relevant platforms. One option would be to add a LaTeX renderer to Laika (a non-trivial effort on its own) and then rely on all the existing tooling around that format (the result would probably look even better than what Apache FOP produces). Another option would be to use HTML as the interim format, but that could not simply use Laika's existing renderer as it would need to deal with features only supported for PDF (manual page breaks, footnotes, configurable bookmarks and typographic controls like preventing headlines on the bottom of the page) and I am not sure whether PDF tooling accepting HTML as input format would support all these. You might feel tempted to say: let's skip PDF support then, but that brings the other downsides listed above: an inconsistent feature set and offering less than competitors. EPUB SupportThis might be significantly easier than PDF support, but I ultimately don't know. The big advantage here is that everything non-portable is concentrated in a single Laika class that depends on the JDKs ExtensibilityTo my knowledge, most alternatives in this space support a way of installing 3rd-party extensions. On top of that, the large set of extension points of Laika is an aspect that distinguishes Laika from some other solutions, so the idea of producing a CLI app that closes this door feels like stripping off a large portion of what makes Laika attractive. To achieve feature parity with the JVM version and enable community plugins, there would need to be a way to compile and load extensions. The downside, apart from the associated complexity is that the language would be Scala. When trying to target communities outside of Scala, this might be a showstopper for some. If a CLI application would see any significant adoption at all, I can already sense that some might start requesting Java APIs. But given that most extension points center around writing partial functions for AST nodes which are case classes, I'm not sure this could be added in any elegant way. HOCONThis is another piece in the puzzle where we offer something alien when targeting developers beyond Scala. I'm fairly sure that at some point people would ask for YAML support and that would be very difficult to integrate in Laika. Wrapping a YAML parser is the easy step, but we would need to translate the internal format of the result to Laika's internal HOCON format to preserve the hierarchical nature of Laika's configuration with its cross-reference support. YAML is also not a good fit for some use cases like directive attributes (a format with significant whitespace does not work well in an inline scenario). CommonMarkAnother gap to competitors that most users and contributors are probably not aware of is that Laika still builds on the original, classic Markdown definition. But today, in most cases where people talk about Markdown they actually mean CommonMark. To provide some context for those who did not follow these developments, the CommonMark spec was a joint effort to add detail to many of the vague or incomplete syntax descriptions of the original Markdown definition as well as a comprehensive test suite. Unfortunately the original author of Markdown did not agree to the name Markdown being used, hence the new name CommonMark. But in practice everyone still calls this Markdown. For Laika the plan is to integrate the CommonMark test suite of 600+ tests to match what other tools offer by now. Most users won't notice the difference, as using Markdown with Syntax HighlightingThis is another area where some catch up would be required when targeting a wider audience. Laika's built-in highlighter still lacks support for some of the major languages like C++, C#, Ruby, Rust, Go. Options here would be, a) adding those highlighters to the integrated support, which is the ideal solution as it makes the syntax available as AST nodes and thus opens it for customizations and makes it work for all output formats, b) offer some JavaScript solution out of the box, which would have the downside of not working for PDF or C) integrate with some native solution for which I have not done any research yet. SearchThis really hasn't been on my radar so far, and it's also not getting requested frequently, but many alternative solutions have some support for searching. This is relevant for HTML output only, as EPUB and PDF come with search functionality in the respective readers. I did some brief research a few years ago, but did not find any attractive solution. Most JavaScript search engines I could find where either very limited in functionality or no longer actively maintained or, in most cases, both. Parallel ExecutionThe transformation time for Laika benefits from the fact that there is only a very brief bottleneck in the middle of the transformation where the AST processing happens in a single thread (to resolve cross-references, produce tables of contents, etc.), the first and last phases of the process (parsing and rendering together with I/O) happen for all documents in parallel. I don't know whether parallel execution can be kept for Node or Native, but if not I could imagine that sequential execution might result in performance that does not match that of other native tools. Development & MaintenanceThe third and final topic I'd like to discuss is how such a solution would be developed and maintained. I hope that I could convince you with my thoughts above that providing some sort of minimal POC would most likely just increase the maintenance burden for little gain and probably little adoption in the community. But if we agree to aim a bit higher the next question is who would be doing this work and when. At this point in time I personally have no capacity myself to increase the scope of the work I do with Laika. The coming months my focus will be the 1.0 milestone series and a bit of 0.19 maintenance. For 1.0 my sole focus is improving binary compatibility. There will be a series of PRs for each package focusing around two areas: reducing the surface of the public API which is currently way too large and replacing all case classes by other constructs (apart from those forming the AST which can stay for a number of reasons). Given all these constraints and the fact that at this point in time we don't know how much interest in the community we can generate for this effort and that the amount of work is significant for producing a viable product, I'm somewhat sceptical about the idea at least for the coming months. But this view could change in the future if several criteria around the Laika project and this proposal change over time:
Finally, while the proposal addresses both, Native and Node, I think the criteria for committing to this effort have to be met for both individually, and doing one of them should not automatically imply doing the other. I'm happy to keep this open for a long time, to allow for further discussion, collecting ideas, and as a place where users can provide feedback. So if anyone has interest in a CLI application or would like to contribute to the development, please chime in and describe your use case. What would you do with it and why would you prefer it over any of the other native or Node.js toolkits which are available today? |
armanbilge
changed the title
|
Firstly, thank you for taking the time to consider this and write such a detailed response ❤️ besides the specific points of this proposal, it is really great to get insight into where things are, and where things are headed. Overall, I agree with your analysis above, both on the audience and effort required (and changed "publish" to "develop" :) I'm going to read through a second time and respond to a few points below.
Ideally this would lean on
Well, we are getting plenty of requests downstream, for searchable project documentation :) @valencik is currently exploring this arena.
Parallel execution can definitely be kept on all platforms. Applications/libraries using Cats Effect etc. combinators for concurrency and parallelism Just Work:tm: with identical semantics. On Node.js, I/O tasks can run in parallel, but there is only one thread for compute tasks. Scala Native 0.4 is currently single-threaded, but the 0.5 series in development already supports multi-threading. This is expected to land sometime in the next few months-ish. Suffice to say, it will match the JVM on all counts.
I'm always happy to review platform- and build-related PRs. I know it doesn't really address the issue you raise here, just saying :)
Along these lines, I'd love to see Laika become a Typelevel-org project, if this is something that interests you. This would secure multiple admins, who can maintain (or delegate).
In my experience cross-building libraries, it is generally much better to figure out the cross-building before 1.0 and compatibility constraints kick in, since dealing with (even subtle) platform differences becomes a lot more painful. In fact, knowing that you are working towards 1.0 was one of my motivations for opening this discussion now, before it was too late :) The intent is not to invalidate your point but better illustrate the interactions of constraints. Basically, establishing an API with long-term binary-compatibility is somewhat coupled to cross-platforming, so if both are goals they should not be done separately. |
|
Thank you for providing all these pointers. I'll comment on two of them here.
Happy to discuss this if there is interest. I just have no idea what this means in practical terms, so someone would need to brief me a bit on this... 🙂
EDIT: I rephrased the last bullet and added another one, I think I wasn't clear enough. Sorry for the confusion. |
@valencik can expand more, but by "org level" project I mean moving the project under the github.com/typelevel org (and possibly publishing under the
Yup, we are very much in agreement on this :) without a CLI I just don't see a compelling usecase for |
I think that might be a good step to consider. Changing the group ID for 1.0 would also be ok, that's a step where users will not be caught by surprise when major changes are made. Shall we move the conversation somewhere else? It does not really belong into this ticket. |
|
Hey @jenshalm, yeah, let's move this discussion elsewhere. The right place is likely an issue on the governance repo: https://github.com/typelevel/governance. So feel free to create an issue on the governance repo. You can create a blank issue, you don't have to follow the project submission template for this discussion. Or, once I have some more feedback from the team I'll create one and tag you :) |
|
Hey @jenshalm just wanted briefly followup to say that we have some description of the practical points on joining Typelevel available here: https://github.com/typelevel/governance/#why-join-typelevel And either on that repo or in Discord would be a fine place to start a further discussion if you have anymore questions :) |
|
Thank you for the pointers, I'll probably write up a little proposal sometime in the coming weeks when I return to spending a bit more time on this project. |
This issue is spun out of #281 (comment).
The goals of this issue are to propose the broader idea, evaluate if it is even within Laika's scope, and see if there is community interest.
Proposal
The idea is really two separate but (IMO) related things.
The motivation for a CLI is so that Laika can be used as a standalone site generator à la Jekyll. @valencik and I recently discussed this in context of the typelevel.org website. I'm not entirely sure if this is possible, since I've really only used it via sbt: specifically, if there is enough configuration flexibility via HOCON/templates, without resorting to Scala code.
Assuming a CLI is a feasible and attractive feature, then cross-building laika-io to Scala.js and Native becomes very interesting.
If publishing a CLI application is out-of-scope for Laika, then I think cross-building laika-io for JS/Native is less valuable. The core is already cross-published for JS (and maybe Native someday) which, if desired, can be used directly with fs2-io without going through laika-io.
The text was updated successfully, but these errors were encountered: