diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b25c15b --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +*~ diff --git a/assets/outputs.png b/assets/outputs.png new file mode 100644 index 0000000..0cc25b9 Binary files /dev/null and b/assets/outputs.png differ diff --git a/assets/store-folder.png b/assets/store-folder.png new file mode 100644 index 0000000..bdf182f Binary files /dev/null and b/assets/store-folder.png differ diff --git a/user/mgarai/20210915-making-booster-cross-platform.md b/user/mgarai/20210915-making-booster-cross-platform.md new file mode 100644 index 0000000..b758ce4 --- /dev/null +++ b/user/mgarai/20210915-making-booster-cross-platform.md @@ -0,0 +1,201 @@ +--- +title: "Making Booster compilation service cross-platform" +categories: "websharper,fsharp,compilation,cross-platform,linux" +abstract: "Change a console application used as a service to support cross-platform. RIDs. What they are good for. Executables across platforms. Detached child process. NamedPipe in Linux. IO behavior when using nohup." +identity: "-1,-1" +--- + +## Intro + +Make use of experience from making WebSharper Booster console application (used as service) cross-platform. From Windows only to Linux compatible. + +## RIDs + +RIDs ([Runtime Identifier][runtime-identifier]) are specifiers for compiler to output operating system, version, architecture, "distro kind" specific code. + +That is more information that can be inferred from `RuntimeInformation.IsOSPlatform(OSPlatform.Windows)` or `RuntimeInformation.IsOSPlatform(OSPlatform.Linux)`. For further information check [Platform compatibility analyzer][platform-compatibility-analyzer]. For example it's a valid RID to use `linux-musl-x64`, but the `musl` part is not really version specific. It's not easy to infer it from runtime. + +### Why specify RID? + +Deploying the application itself doesn't need to be RID qualified ([.NET application publishing overview][dotnet-application-publishing-overview]). Simple `dotnet publish` produces the right `.dll` (cross-platform) and an executable, which is current platform specific. + +> Publishing an app as framework-dependent produces a cross-platform binary as a `.dll` file, and a platform-specific executable that targets your current platform + +When cross-platform executable is needed, than the `dotnet publish` command needs a `-r` flag. Which is the RID. The `` tag in the project file specifies RID. + +If the executable expected to be of the right name: + +Windows specific +```shell +wsfscservice.exe +``` + +Linux specific +```bash +./wsfscservice +``` + +I need to specify RID. This helps finding the running service from the `Task Manager` or `ps`. + +### Current platform's RID? + +As mentioned before the RID have more information coded in itself than `RuntimeInformation` namespace gives us. If the execution of the application dispatched through `.targets` file definition, we can use the RID that's available there for .NET Core SDK Runtime Identifier. I found that variable while checking through SDK's code reverse engineering how `_UsingDefaultPlatformTarget` is set. For `_UsingDefaultPlatformTarget` = `true` something is used as [default RID][default-runtime-id]. + +```xml +$(NETCoreSdkRuntimeIdentifier) +``` + +Some [information][netcoresdkruntimeidentifier-information] about the `MSBuild` variable: + +> The NETCoreSdkRuntimeIdentifier MSBuild property determines the bitness of *.comhost.dll + +This is a good default value to find out the folder where platform specific `.dll` is located: + +```xml + +$(MSBuildThisFileDirectory)/../tools/net5.0/$(NETCoreSdkRuntimeIdentifier)/ +``` + +In the two cases tested `$(NETCoreSdkRuntimeIdentifier)` is substituted for `win-x64` and `linux-x64`. + +### Keeping the process's name across different platforms + +Alternative would be using the cross-platform `.dll` with `dotnet wsfscservice.dll` but this makes hard to find the running process through all running `dotnet` instances. [Rename the running process is not possible][rename-the-running-process-is-not-possible] or require `start "wsfscservice" dotnet wsfscservice.dll` "hack" which is Windows specific. Also finding the process programmatically is simpler: + +```fsharp +let runningServers = + try + Process.GetProcessesByName("wsfscservice") + |> Array.filter (fun x -> System.String.Equals(x.MainModule.FileName, fileNameOfService, System.StringComparison.OrdinalIgnoreCase)) + with + | e -> + // If the processes cannot be queried, because of insufficient rights, the Mutex in service will handle + // not running 2 instances + nLogger.Error(e, "Could not read running processes of wsfscservice.") + [||] +``` + +Attach/reattach debugger is [easier][reattach-debugger]. + +So the choice is [framework-dependent][framework-dependent] but non `--self-contained`. `--self-contained` would make the executable bigger, and makes harder to patch the runtime around the executable. + +### `` vs `` + +Expecting that using `` instead of `` (mind the plural) changes the output structure of published nuget is wrong. When used correctly the RID specific output should be scattered around in `Debug`/`Release` folder (inside for example `win-x64`, `linux-x64`, `linux-musl-x64`). But `dotnet publish` doesn't publish for all different RIDs specified in the ``. Instead uses directly the `Debug`/`Release` folder. Just like the `dotnet build` command doesn't. + +![](/assets/outputs.png) + +For that to work we needed RIDs count `publish` command for all executables. Libraries doesn't need extra care. In our build process it looks like this (Fake specific. `DotNet.publish` calls `dotnet publish`) + +```fsharp +let publishExe (mode: BuildMode) fw input output = + for rid in [ "win-x64"; "linux-x64"; "linux-musl-x64" ] do + let outputPath = + __SOURCE_DIRECTORY__ "build" mode.ToString() output fw rid "deploy" + DotNet.publish (fun p -> + { p with + Framework = Some fw + OutputPath = Some outputPath + NoRestore = true + SelfContained = false |> Some // add --self-contained false + Runtime = rid |> Some // add -r from the list above. + Configuration = DotNet.BuildConfiguration.fromString (mode.ToString()) + }) input +... +BuildAction.Custom <| fun mode -> + publishExe mode "net5.0" "src/compiler/WebSharper.FSharp/WebSharper.FSharp.fsproj" "FSharp" + publishExe mode "net5.0" "src/compiler/WebSharper.FSharp.Service/WebSharper.FSharp.Service.fsproj" "FSharp" + publishExe mode "net5.0" "src/compiler/WebSharper.CSharp/WebSharper.CSharp.fsproj" "CSharp" +``` + +This also can be achieved by simple shell script, which iterates on the RIDs specified in the ``. If the singular `-r` parameter is not from the listed `` RIDs from the project file the `dotnet publish` will fail! + +Important tags to the project file to be added: + +```xml +Exe +net5.0 +wsfscservice +wsfscservice +win-x64;linux-x64;linux-musl-x64 +``` + +### Self contained + +It was a surprise that `--self-contained`'s default value is variable. If you specify RID, it's `true` if you just leave `dotnet publish` it's `false`. So to avoid [Publish self-contained][publish-self-contained], `--self-contained false` had to be added. + + +## Running detached child processes + +When spinning off an executable the process will become the child process of the main executable. That means however someone starts a process to be background it will be closing with the parent executable. Our Console application can't function as a service. + +Starting the executable can be: + +```fsharp +// start a detached wsfscservice.exe. Platform specific. +let cmdName = if System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform(System.Runtime.InteropServices.OSPlatform.Windows) then + "wsfscservice.exe" else "wsfscservice" +let cmdFullPath = (location, cmdName) |> System.IO.Path.Combine +let startInfo = ProcessStartInfo(cmdFullPath) +startInfo.CreateNoWindow <- true +startInfo.UseShellExecute <- false +startInfo.WindowStyle <- ProcessWindowStyle.Hidden +proc <- Process.Start(startInfo) +``` + +But it will be still child process. The solution is to start the process operation system specifically. Create a `.cmd` for Windows using `start` with `/b` parameter to be [background process][windows-background]. + +```shell +@echo off + +start /d %~dp0 /b wsfscservice.exe +``` + +And a `.sh` for Linux. Here [`nohup`][nohup] makes `wsfscservice` immune to hangups and `&` at the end makes it [background process][linux-background]. + +> If a command is terminated by the control operator &, the shell executes the command in the background in a subshell. The shell does not wait for the command to finish, and the return status is 0 + +```bash +#!/usr/bin/env bash + +nohup "$(dirname "$BASH_SOURCE")/wsfscservice" >/dev/null 2>&1 & +``` + +`Process.Start` can start the platform specific `.cmd`/`.sh`. No matter it returned immediately with `0` code it's sure that the background process is running. `Task Manager` shows `wsfscservice` and `ps -al` show `wsfscservice`. "Start new process, without being a child of the spawning process" question on `Stackoverflow.com` top voted answer also does [something similar][so-use-stub-exe]. + +```fsharp +let cmdName = if System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform(System.Runtime.InteropServices.OSPlatform.Windows) then + "wsfscservice_start.cmd" else "wsfscservice_start.sh" +``` + +## Caveat around `nohup` + +The service processes commands in an infinite loop. This can be achieved by startup a mailbox processor with `Async.Start`, then on the main thread block the execution. At first blocking the main thread was done by `Console.ReadLine() |> ignore`. Avoiding busy wait. But `nohup`'s IO piping makes this line of code just continue. + +Another possible solution for non-busy wait is using: + +```fsharp +use locker = new AutoResetEvent(false) +locker.WaitOne() |> ignore +``` + +That solved that a code working on Windows now keeps the service running on Linux as well. + +## NamedPipes on Linux + +One thing it's worth mentioning is `NamedPipe`'s `PipeTransmissionMode.Message` is [not supported][not-supported] in the Linux runtime. Had to reimplement the `IsMessageComplete` behavior on `PipeTransmissionMode.Byte` either by pre-sending bytes count (how much byte will a full message be independent of buffer size), or sending separator byte sequence between messages. + +[not-supported]: +[so-use-stub-exe]: +[linux-background]: +[nohup]: +[windows-background]: +[netcoresdkruntimeidentifier-information]: +[default-runtime-id]: +[publish-self-contained]: +[framework-dependent]: +[reattach-debugger]: +[rename-the-running-process-is-not-possible]: +[dotnet-application-publishing-overview]: +[platform-compatibility-analyzer]: +[runtime-identifier]: diff --git a/user/mgarai/20210921-writing-a-dotnet-tool.md b/user/mgarai/20210921-writing-a-dotnet-tool.md new file mode 100644 index 0000000..a7c70a8 --- /dev/null +++ b/user/mgarai/20210921-writing-a-dotnet-tool.md @@ -0,0 +1,181 @@ +--- +title: "Writing a dotnet tool" +categories: "dotnet,tool,cli,fsharp" +abstract: "Writing a dotnet tool with help of the dotnet CLI" +identity: "-1,-1" +--- + +## Intro + +Writing a dotnet tool from scratch have a lot of support from the dotnet CLI. With the following simple steps you can have a tool from the dotnet infrastructure. + +## Steps + +Create the console application for the tool: +```ps1 +> dotnet new console -lang F# -n +> cd +``` + +For the `tool-name` check the next section + +(Optional) Add a solution file for the project: +```ps1 +> dotnet new sln +> dotnet sln add .fsproj +``` + +Check build +```ps1 +> dotnet build +``` + +Edit Program.fs. A dotnet tool is basically a console application, so implement what you like. + +Add necessary variables for the `.fsproj`. You should see something like this: +```xml + + + + Exe + net5.0 + + 3390;$(WarnOn) + + true + + + + + + + + +``` + +Add `tool-manifest`: +```ps1 +> dotnet new tool-manifest +``` + +Edit `.config\dotnet-tools.json` (the tool manifest). Specify the version and available commands like this: +```json +{ + "version": 1, + "isRoot": true, + "tools": { + "": { + "version": "1.0.0", + "commands": [ ] + } + } +} +``` + +Run pack +```ps1 +> dotnet pack +> cd bin/Debug +``` + +Install the new tool globally. Here the `tool-name` is not the `.nupkg` file name. It's just the `` given at `dotnet new -n `. +```ps1 +> dotnet tool install -g --add-source . +``` + +Verify installation: +```ps1 +> cd %USERPROFILE%\.dotnet\tools +or +> cd $HOME/.dotnet/tools +--- +> dir +or +> ls +``` + +You have a dotnet tool installed. + +There is also a [tutorial][tutorial] for that in the `Microsoft` documentation. + +## Name of the executable. \ + +Before just creating the console application with the come up name for the tool, consider if `dotnet-` prefix should be added or not. + +It's important to understand if you issue command +```ps1 +> dotnet xy +``` +in the dotnet ecosystem it's searching for a tool named `xy` prefixed by `dotnet-` and runs it. If anywhere in your `PATH` variable's directories there is a `dotnet-xy` it will run it. It's just a nice to have to install them where `dotnet install` puts `dotnet-xy`. + +The installation is versioned. At the installation destination there will be subdirectories for the installed versions like that: + +![](/assets/store-folder.png) + +The important question is the usage of the tool. For `dotnet xy` go with `dotnet-xy`. For `xy` go with `xy`. + +Choosing between the 2 is first step, so the main directory at `dotnet new console -n xy` can be in par with the final executable's name. + +For more information go to [this tutorial][custom-location]. + +### ToolCommandName + +The `` variable in the `.csproj`/`.fsproj` file doesn't set the executable's name when installed into tools. `` does. `dotnet pack` will use `` which goes into +`%USERPROFILE%\.dotnet\tools` or `$HOME/.dotnet/tools` or where `--tool-path` points to in `dotnet tool install --tool-path ...`. All should be in the `PATH` environment variable (be careful, `--tool-path` is not by default). + +## FAQ +**Question:** I changed the code, but no change is visible in the installed tool. Why? + +**Answer:** Have you repacked the changes? +```ps1 +> dotnet pack +> cd bin/Debug +> dotnet tool install -g --add-source . +``` + +Have you changed version, and forgot to bump it accidentally? +```ps1 +> dotnet tool install -g --add-source . --version 1.0.0 +instead of +> dotnet tool install -g --add-source . --version 1.1.0 +``` + +### +**Q:** I have the following error: +``` +error NU1101: Unable to find package . No packages exist with this id in source(s): dotnet-websharper-GitHub, fsc feed, Microsoft Visual Studio Offline Packages, nuget.org +The tool package could not be restored. +Tool '' failed to install. This failure may have been caused by: + +* You are attempting to install a preview release and did not use the --version option to specify the version. +* A package by this name was found, but it was not a .NET tool. +* The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem. +* You mistyped the name of the tool. + +For more reasons, including package naming enforcement, visit https://aka.ms/failure-installing-tool +``` +How can I solve it? + +**A:** This can mean anything. Usually the problem is missing `--add-source `, or using the file name instead of `[PACKAGE_NAME]`. +```ps1 +> dotnet tool install -g --version 1.0.0 .1.0.0.nupkg +instead of +> dotnet tool install -g --add-source . --version 1.0.0 +``` +### +**Q:** Can't find the executable in `%USERPROFILE%\.dotnet\tools`. Why? + +**A:** Have you provided `-g` flag? +```ps1 +> dotnet tool install -g ... +``` +### +**Q:** `dotnet tool list` doesn't list what I have provided in the tools manifest. Why? + +**A:** `dotnet tool list` also differentiate between global and local tools. Have you added `-g` flag for your installed global tool? +```ps1 +> dotnet tool list -g +``` + +[tutorial]: +[custom-location]: