dotnet scripting / fsc compilation of .fsx scripts should package all the runtime dependencies along the compiled .exe

[smoothdeveloper]

fsc support for compiling .fsx files could be improved by packaging all the runtime dependencies in the output folder, next to the compiled .exe.

Describe the solution you'd like

I'd like this to be the default behaviour, but since it would be a change of behaviour, maybe it would require a flag (--copy-runtime-dependencies-to-output-folder ?)

It should take care of those constructs:

• DependencyManager that are pulled through #r "extension: " mechanism
• vanilla #r "Assembly" directives
• #load "otherscript.fsx"
• more generally, all the HashDirective constructs that F# supports in scripts

Describe alternatives you've considered

Alternative is to make a new compiler out of FCS, that would do that, but it seems more cohesive to expand the support for compiling .fsx that is already implemented in fsc

Additional context

Related to #13236 (comment)
Related to https://github.com/dotnet/interactive

cc: @KevinRansom @jonsequitur

[dsyme]

I don't think this is in scope for fsc. To properly package an application is a job for dotnet build.

So we can't realistically do this I think. You have to create a project and use dotnet build and dotnet publish to get this right. If you wish to create a dotnet fsscript tool that does this kind of thing implicitly that would be fine. But it just seems out of scope for F# tooling from this repo.

[dsyme]

I note that @KevinRansom thinks this could be in scope, see #13236 (comment) :)

So I'll leave this open as a feature request :)

[KevinRansom]

@dsyme , if I remember correctly and I may have this wrong, but I believe the package manager APIs are invoked during fsx compilation, if that is so then we could in theory move dll's to the output directory. However, we would need to specify the target Rid to the compiler instead of using the ambient rid and tell the package manager to do this extra work. It's not a high priority scenario for us, and there is likely a big bug tail in compiling and packaging scripts.

I chatted with @jonsequitur about it for notebooks and they wouldn't use this mechanism for notebook publishing. Instead they would pre-scan the code to collect all of the #r nugets and embed them in a project file they would build for the publish purpose.

I think the ability to compile and publish scripts is a reasonable feature request. But it will be a much better one when we have AOT compilation and tree shaking to reduce the size, from 10's of megabytes down to small. because deploying a script file and dotnet .fsx it will really only cost the size of the script file.

[smoothdeveloper]

they would pre-scan the code to collect all of the #r nugets

It wouldn't work with the #r "notnuget: " which currently works in dotnet interactive (and fsi, and FCS, and VS Code, and if you like to compile extension yourself, on the for pay IDEs with F# support).

there is likely a big bug tail in compiling and packaging scripts.

There is already one bug that it doesn't work on non .net framework version of compiler to compile a script.

But it will be a much better one when we have AOT compilation and tree shaking

Is this going to be part of F# compiler or at a lower level in the stack? If it is in F# compiler, and given the compiler anyways need to deduct all the assemblies for compiling a script, I see more long tail of bugs in the AOT and tree shaking, than a convenience flag to put the whole output in the output directory before such feature lands.

It also allow to conveniently serve customers that rely on .fsx but pay the cost of compiling script at every execution right now, or that need to migrate from script to projects.

Trying to strike a balance here, but since fsc already infers all the runtime assemblies, copying them out, and not relying on msbuild black magic would be really nice experience.

Improving the compiler featureset improves the life of all consumers, and technically it is already broken and not super useful to compile a script, if one needs to spend time copying dll around, and this breaks on any change of dependencies in the script.

In any case, thanks all for mulling over this.

[KevinRansom]

@smoothdeveloper --- notebooks has their own set of concerns. And an even more extensive at runtime nuget extension model.

I think compiling a script into a global tool is actually a really compelling script publishing scenario. A do nothing script compiles to 1 mb so that is the publishing overhead, scripts can then be versioned and shared using nuget. It's actually pretty durned neat. The coreclr compiler compiles the script with #r nuget fine. It's just the dependencies that are missing. But there is likely a bug farm in compiling scripts since this scenario has never been aggressively tested and used.

How do you think we could qualify whether the feature would generate enough interest to spend the effort on?

[KevinRansom]

@smoothdeveloper one more thing ...
msbuild black magic is almost always preferable, because of binary logs and the MSBuild Structured Log Viewer. The more I think about this, having the package manager drop a file in the obj\output directory and a build task which adds the results into the build copytask, is the ideal approach ... then all of the magic is visible rather than executable.

And thanks for helping me think this through.

[smoothdeveloper]

@KevinRansom, from my standpoint, I liked the idea of fsc compiling scripts, but couldn't put it in practice due to the missing assemblies and manual overhead for packaging, hence I resorted to scripts that remain scripts.

From my perspective, it is either fsc keeping the feature as is (or even deprecating but that would be an issue for those that managed to use it), or it supporting it in a way which makes it useful.

I understand the tension between msbuild, dependency manager mechanism and compiler covering more things that may evolve as dotnet itself evolves, but the KISS to me, would be fsc/fsi already knows all assemblies and a file copy under a flag, wouldn't put everything at stake and make the feature actually useful.

I am not expecting a comital, and we should try to get feedback from other parties, that like .fsx more than msbuild black magic and project files.

Looking from standpoint of having F# adopted as scripting environment, like say python (#13271), offering more gradation between "you can script even more complex stuff easily" and "you need to learn everything about dotnet projects" seems like a good ground for F#, given that no other dotnet language is really trying, and dotnet interactive is more geared toward "python notebook" scenarios.

It makes "F# stands on its own" without being detrimental to "F# is core part of donet" IMO.

[KevinRansom]

I think the experience will be along the lines of:

1. dotnet new scriptglobaltool -langfsx
2. add your script text to the project.
3. dotnet pack

And you are done. Most everything would be automagic. I can even imagine a global tool that automates step 1 and 2.

Regardless of that, a project file is required to build on the coreclr. You can do it without msbuild, but figuring out which framework references to reference is a non trivial task, indeed just typing them all is pretty grotesque too.

The coreclr is our focus, and supporting dotnet sdk scenarios is the best route forward, because of that, we can leverage the work of the wider dotnet team. I think we can fairly easily support a compiled script as global tool scenario without breaking into any sort of future purgatory. I think it would be a decent mechanism for developers to build, deploy and test their scripts without requiring any specialized fsi specific knowledge.

[smoothdeveloper]

@KevinRansom I like your plot, but it probably opens things to consider, with regard to support in IDEs and FCS?

Do you also see that DependencyManager infrastructure and extensions would need to change to support what you are envisioning?

An option to compile a given .fsx as "top level" in .fsproj, with a specific output dir, maybe that would not require new language flag and just be .fsproj targets infrastructure instead?

[MecuSorin]

@smoothdeveloper Can you describe the real usage scenario for your proposal? Beside notebooks, I cannot understand why do you want it, when we already have the fsproj concept in place.
I can only guess that you want to have multiple scripts without the extra work for transforming them in separate projects. If that is the case, then Don suggestion to make a tool outside fsc that is generating those projects and builds them is the best approach - isolates complexity to the proper place and is not bloating fsc

[KevinRansom]

@MecuSorin ,
I believe that what @smoothdeveloper would like to see is a developer experience for sompiling scripts that is as simple as the old desktop compile experience

fsc program.fsx

On the desktop framework this built an .exe that ran. In the cross platform multi-assembly world we now live in, the project file is pretty much essential. However, because #r "nuget: blah" which is valid .fsx syntax and thus can be compiled with F# compiler; the dependencies retrieved using it are not visible to the dotnet build and pack processes. They can of course be easily added to the project file for compile, but it is an extra manual step. What I believe @smoothdeveloper is really asking for is a mechanism that simplifies compiling and deploying F# scripts that have complex dependencies.

This isn't a scenario we have spent much time on thinking through, but is certainly quite valuable to script developers. Any change we make, would most likely involve teaching dotnet build to understand how to access the results of the #r nuget action.

[smoothdeveloper]

@MecuSorin, fair points, albeit, all scripts, in their environment, may have some issues, to turn into full fledged projects, many scripts are IT tasks, and migrating those into full fledged project is akin to ask someone who has bunch of powershell for IT admin, to move them into C# projects, such move generally doesn't happen.

F# Interactive and fsc.exe (the .net framework / desktop one) sit in an odd spot, I can't really use the primer as a reliable scripting environment for IT tasks, mainly because to deploy it, it requires much more than the F# standalone package of old times, and fsc can't provide what it needs to easily compile those same scripts and deploy them with all the runtime dependencies into a place that has dotnet of some form.

I am fine if the request gets abandoned but somehow, if F# is going to have some credence as a scripting language (not just serving as scratchpad to F# developers), something has to happen (#12895 is also a concern).

If we are talking about a tool, which is not the compiler or part of dotnet delivery, I guess it would be an up for grab, but requiring much arcane knowledge of dotnet, msbuild and FCS.

I don't see extending fsc in such way as bloating it, given:

• currently, dotnet fsi handles complex cases of pulling runtime dependencies
• adding some of that logic in the "compile a script for packaging" story can rely on this
• some test cases of this repository become part of the CI

But I appreciate and respect there are forces / opinions feeling the opposite.

[kant2002]

Most probably I want to have dotnet publish myscript.fsx, because this is ultimately publishing with all parameters which dotnet publish can provide. I imaging if we can teach dotnet-publish to handle .fsx files as projects. If think this through, IronPython can benefit from this too, as other languages. but this way may require some form of support from dotnet SDK for scripting environments which maybe accepted or not.

As a practical way to do it, I create prototype tool which I share in #13961 . I collect all references and handle other directives, similar to what @jonsequitur seems to be doing and pass additional arguments to dotnet publish. Would be curious to know how they do it.

seems to be whole scope of work is relatively small, but packaging solution and property land it inside existing tooling is complicated.

[kant2002]

It may be interesting to know, that C# people also want something similar as discussed here dotnet/designs#213, and maybe this will lead to cooperation which give enough support in the .NET SDK to make this happens in more consistent way for .NET ecosystem, so changes to FSC/FSI would look organic.

[roboz0r]

I have a project which I believe achieves a lot of what is required here. Currently I am compiling fsx scripts via FCS and copying the resulting dll and the dependencies into a temporary folder so that I can load it dynamically into my software. My solution is quite different to @kant2002 as it doesn't process the fsx at all and just relies on the FCS output.

I would be happy to open source this portion of my code but I'm not experienced with creating NuGet packages or dotnet tools. Assuming that a suitable solution to #14289 is implemented would this approach be useful to publish?

[jkone27]

not sure if this has been mentioned, but seems there is a global dotnet tool which can add framework reference, can this be used/referenced and just exploited in F# scripts, but this will not solve the dotnet build issue ofc, maybe there is another open issue for framework reference with makes more sense for this comment, was trying to find it

• https://github.com/dotnet-script/dotnet-script

#r "sdk:Microsoft.NET.Sdk.Web"

or is this a dependency we dont want, probably having the same format would be good to share most as .NET community in synthax?