[_rlh]

You use runtime.GC() when you did your benchmark. This tells the Go runtime to do an aggressive stop the world GC, which it does. The normal GC is concurrent and if you use it latency should not be a problem. For your use case I'd remove runtime.GC() and try timing the RPC and look for latency spikes. Report back if you can, if you are still seeing high latency we can file a bug report with the Go team.

[teh_cmc]

Yes, these benchmarks use forced GC calls (i.e. all phases are STW) because it's the only (good) way I can think of to make theses tests deterministic (maybe you know of a better way? I'd definitely be interested).

Of course, I don't have such calls on production systems; and while concurrent collections greatly reduce the time spent in STW phases, latency spikes are still a very real issue, as I explained here [1]. The monitoring on my production systems leaves absolutly no doubt that those latency spikes are due to garbage collections: once the GC was out of the picture, every thing was flat and fast again.

[1] https://news.ycombinator.com/item?id=10650885

[_rlh]

Some folks export GODEBUG=gctrace=1 which will result in the GC dumping out a lot of interesting information including stop the world latency. It doesn't increase the determinism and the GC cycle is started when the runtime decides but it does provide visibility into the concurrent GC latency in a benchmark or actual application. Perhaps you already use it and that is how you noticed the latency problems to start with.

I do know that you need version 1.5 of Go that was released last August to get the low latency GC. If throughput is an issue some folks adjust GOGC to use as much RAM as they can. If none of this helps file an issue report with the Go team. You seem to have a nice well thought out work around but a reproducible gctrace showing hundreds of millisecond of GC latency on heaps with millions of objects will be of interest to the Go team and might really help them.

I hope this helps.