Improving startup time

[PallHaraldsson]

I tried to apply: https://discourse.julialang.org/t/ann-new-package-snoopprecompile/84778

to your package (just the package for now, not at dependencies, which might do the trick). It's really simple, or maybe I misapplied it since the timing still (similarly) slow, or because it's needed for maybe that one important dependency?

I'm not sure what's taking the most time for using the package. I tried to profile it, for the init function, but it hanged. Strange...

I did time your dependencies separately, and could some or all of these be dropped? I like that you just handle Python dependencies, but could that top one be an optional dependency, only leaded on demand somehow? It seems not needed always.

julia> @time using CondaPkg
  0.285375 seconds (345.91 k allocations: 27.317 MiB, 5.59% compilation time)

julia> @time using MacroTools
  0.076592 seconds (92.97 k allocations: 7.899 MiB)

julia> @time using Tables
  0.054301 seconds (52.75 k allocations: 3.678 MiB)

julia> @time using UnsafePointers
  0.026749 seconds (33.21 k allocations: 2.017 MiB)

julia> @time using REPL
  0.010808 seconds (663 allocations: 56.875 KiB)

I wanted to make a PR for julialang.org to promote your package, replacing PyCall there. But I rather want the startup faster first, and doubt PR will be accepted otherwise. Also a question is v1.0 coming? That's for braking changes, so if needed, drop some of the optional dependencies? Both packages have pros and cons, and I want yours to have the same pros (and none of the cons, I think it's so already, except different API) of PyCall, i.e. as good startup. If you do too much at startup, could it be an opt-in function, or in a different package? Though ideally (pre)compile would just work fast... Why is REPL dep used?

[cjdoris]

I have also tried and failed to use SnoopPrecompile to any effect. But I have discovered 1-2 seconds of savings I can make in PythonCall.

I don't know why CondaPkg takes that long to start, it does practically nothing until you ask it.

I'm thinking hard about the whole PythonCall API with a view to getting to v1. There are some significant changes I have in mind to various parts, such as dependency management (I'm considering replacing CondaPkg with something more general) and conversion (which is currently over complicated). Maybe at that point it can go on the Julia website.

Feel free to reopen if I didn't answer something.

[miguelbiron]

I tried to profile it, for the init function, but it hanged. Strange...

Sorry for the off-topic comment on a long-closed issue, but I'm currently trying to profile a PythonCall-based bridge, and the profiler seems to just hang. This message is the only issue related to profiling PythonCall I've found so far. Could you point me to any resources on how to do this? Thanks.