Distributed: "From worker" message missing or showing wrong worker id

[yha]

Log message (@info, @warn, @error) on workers do not show the "from worker" message (for local workers and first remote worker), or show the wrong worker id (for other remote workers):

julia> using Distributed

julia> local_procs = addprocs(4)
4-element Array{Int64,1}:
 2
 3
 4
 5

julia> remote_procs = addprocs([(remote_machine_spec, 4)]; remote_args...)
4-element Array{Int64,1}:
 6
 7
 8
 9

julia> @everywhere @info myid()
[ Info: [ 1
Info: 5
[ [ Info: Info: 2
4
[ Info: 3
      From worker 6:    [ Info: 8
      From worker 6:    [ Info: 7
[ Info: 6
      From worker 6:    [ Info: 9

Local and remote versioninfo():

julia> versioninfo()
Julia Version 1.5.2
Commit 539f3ce943 (2020-09-23 23:17 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-9800X CPU @ 3.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake-avx512)

julia> @sync remotecall(()->InteractiveUtils.versioninfo(), 6);

julia>       From worker 6:     Julia Version 1.5.3
      From worker 6:    Commit 788b2c77c1 (2020-11-09 13:37 UTC)
      From worker 6:    Platform Info:
      From worker 6:      OS: Linux (x86_64-pc-linux-gnu)
      From worker 6:      CPU: Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
      From worker 6:      WORD_SIZE: 64
      From worker 6:      LIBM: libopenlibm
      From worker 6:      LLVM: libLLVM-9.0.1 (ORCJIT, cascadelake)

[JeffBezanson]

This seems to be because Logging captures stderr before the worker startup process gets a chance to redirect it.

[JeffBezanson]

Thinking about how to address this. One thing I tried that works is to launch the worker julia processes with the same pipe for stdout and stderr, instead of sharing the parent process stderr. However, because of #31036 it seems we need to preserve the option for the worker to share the parent process stderr. It also doesn't seem ideal that calling redirect_stderr doesn't redirect default logging. I suspect we need an IO object that means "whatever std* is currently".

[JeffBezanson]

Aha! #26798

[simonbyrne]

See also JuliaLang/Distributed.jl#51 for similar issues.