> when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it?
No.
> How does the Nim code yield the thread for other goroutines, does it have to register a callback?
There are no callbacks. Yielding happens automatically when launching another goroutine, when sending, receiving or selecting on a channel. You can also yield explicitly with go_yield() - the better named equivalent of Go's runtime.Gosched().
It's easier to understand if you realize that all those operations with goroutines and channels end up being done in the Go runtime.
> > when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it?
>No.
Are you sure? Once a thread enters cgo it is considered blocked and is removed from the thread pool according to these sources [1][2]. I previously found a thread where Ian Lance Taylor explained it more explicitly but I couldn't find that now. Is that not what is happening though when __go_go invokes your function pointer?
I do not understand how the Nim code can live in the segmented stack of a goroutine, nor how the Go runtime could know it is time to grow that stack.
Yes. See the chinese whispers benchmark with 500000 goroutines and a maximum resident set size of 5.4 GB on amd64. It has the same memory usage (and run time) as the Go version compiled with gccgo.
> Once a thread enters cgo
This has nothing to do with cgo. It's a different mechanism specific to gccgo.
> I do not understand how the Nim code can live in the segmented stack of a goroutine
Good thing you asked. I just ported to Nim the peano.go benchmark described as a "torture test for segmented stacks" and... it failed. The fix was to pass -fsplit-stack to gcc when it compiles the C code generated by nim.
> nor how the Go runtime could know it is time to grow that stack
From what I can tell it's done in __splitstack_makecontext() and friends from GCC's libgo.
No.
> How does the Nim code yield the thread for other goroutines, does it have to register a callback?
There are no callbacks. Yielding happens automatically when launching another goroutine, when sending, receiving or selecting on a channel. You can also yield explicitly with go_yield() - the better named equivalent of Go's runtime.Gosched().
It's easier to understand if you realize that all those operations with goroutines and channels end up being done in the Go runtime.