Standard output buffer size

[jmalczak]

Hi

Thanks for the great library. I have specific case where I use CliWrap in integration test suite. I start asp.net core server as separate process in test setup method and then in test execute requests and check responses. To be able to see output logs from asp.net core I use ListenAsync and print output from process standard output to test console. When I debug test asp.net core server process still works in background so I can put breakpoint in my test and even fire requests from some other tool like postman. This works just fine. But there is specific case which is related to process output buffer size. When I pause on breakpoint, ListenAsync is paused as well, so it doesn't pick up messages from buffer. So if my asp.net core app prints too many messages to standard output process stops and wait till buffer will be available. So after couple requests from postman my asp.net core process stops serving requests until i fire ListenAsync in debugger to free buffer.

So I could ignore standard output from process, but it's useful for some test debugging scenarios. Do you know any way how to handle this issue? I saw that process buffer size is hardcoded, if I would be able to change it to larger value this would allow my asp.net core app to run longer since buffer wouldn't be full so quickly. But I believe this is not possible.

Thanks

[Tyrrrz]

Hi. If I understand your problem correctly (which I might not), enlarging the buffer on CliWrap's side might not help. There is another buffer on your application's side as well, whose size needs to be changed too. However, even then, I'm not sure if it's the right solution.

One thing you can try is use Observe() instead of ListenAsync(). The former doesn't have any back pressure, so the buffers shouldn't get "stuck". On the other hand, if you are not handling the incoming events fast enough, you might get flooded. You can also use pretty much the same code as with ListenAsync() if you install System.Reactive and use ForEachAsync() extension for IObservable<CommandEvent>. Not sure if it will solve your issue, but you could try.

[jmalczak]

Sorry, I am not the best in explaining problems. I have exactly this issue https://r00t.dk/post/2021/07/06/process-redirectstandardoutput-warning/ and since CliWrap uses process class underneath, I end up having lock while using ListenAsync. When I put breakpoint in debugger it acts the same as WaitForExit in example above, it stops execution of my current process. Since execution is stopped it can't empty standard output buffer. This cause process which I have executed to stop as well, since buffer is full and it cannot write to the buffer.

Program.exe

-> start Program2.exe using CliWrap which is chatty to standard output
-> await ListenAsync in a loop and print messages from Program2.exe standard output to Program.exe standard output
-> put a breakpoint during debugging, ListenAsync stops executing
-> Program2.exe still works in background
-> time pass and Program2.exe has outputted more than 4096 characters to standard output and filled in whole buffer
-> since we are stopped in debugger ListenAsync cant empty buffer
-> this makes Program2.exe to stop execution until buffer is free
-> if we allow Program.exe to run and invoke ListenAsync couple of times, this will empty buffer
-> this will allow Program2.exe to continue execution until buffer is filled in again

I know this is not an issue of CliWrap, but I was hoping that you may know some way of either increasing c# process class standard output buffer. Or maybe you have any idea how to solve this issue. I was thinking about running my app using CliWrap and redirect standard output to file, for example:

Program2.exe > program2.exe.log

I could then run CliWrap with redirect standard output disabled since it would end up in file anyway

Then I could read in Program.exe contents from program2.exe.log and output them to console. This would avoid putting anything in process start standard output buffer. This way I could hang on breakpoint in Program.exe, and Program2.exe would be executing without any issues in background. This seems a bit hacky to me, that is why I was hoping that maybe you may have better way of solving it :)

Btw. Slava Ukraini!

[Tyrrrz]

Have you tried something simple like just Cli.Wrap(...) | Console.WriteLine (or whatever other delegate)? If you just need to route logs, it might be enough.