We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

[synthetigram]

This problem is not going to go away so easily. Numerous core Java classes (like BufferedInputStream) use synchronized. I count 1600+ usages in java.base. The blocking issue means it's _much_ easier to accidentally run into this, rather than waving it away as an unlikely edge case.

I personally ran into this Using the built in com.sun webserver, with a virtual thread executor. My VPS only has two CPUs which means the FJP that virtual threads run on only have 2 active threads at a time. I ran into this hang when some of the connection hung, blocking any further requests from being processed.

[pron]

As the JEP states, pinning due to synchronized is a temporary issue. We didn't want to hold off releasing virtual threads until that matter is resolved (because users can resolve it themselves with additional work), but a fix already exists in the Loom repository, EA builds will be offered shortly for testing, and it will be delivered in a GA release soon.

Those who run into this issue and are unable or unwilling to do the work to avoid it (replacing synchronized with j.u.c locks) as explained in the adoption guide [1] may want to wait until the issue is resolved in the JDK.

I would strongly recommend that anyone adopting virtual threads read the adoption guide.

[1]: https://docs.oracle.com/en/java/javase/21/core/virtual-threa...

[jillesvangurp]

People always forget that things that only happen every few million times, can happen fairly frequently on a busy server. This has bitten me numerous times. The nature of a lot of these types of issues is that they are hard to detect and hard to reproduce.

Virtual threads are nice for unblocking legacy code but they aren't without issues. There are better options for new code with less trade offs on the jvm as well. I've recently been experimenting with jasync-postgresql (there's a mysql variant as well) as an alternative to JDBC in Kotlin. It's a nice library. It does have some limitations and is a bit on the primitive side. But it appears to be somewhat widely used in various database frameworks for Scala, Java, and Kotlin.

Databases and database frameworks are an area on the JVM where there just is a huge amount of legacy code built on threads and blocking IO. It's probably one of the reasons Oracle worked on virtual threads as migrating away from these frameworks is unlikely to ever happen in a lot of code bases. So, waving a magic wand and making all that code non blocking is very attractive. But of course that magic has some hard limitations and synchronize blocks are one of those. I imagine they are working on improving that further.

[he0001]

BufferdInputStream is rewritten and is only using synchronized if subclassed. In fact there has been a lot of work removing the synchronized keyword.

[mrintegrity]

Totally off topic but I am getting tired of the AI generated images used on nearly all blog posts nowadays. They are instantly recognisable, it just seems low effort and lowers the feeling of quality one might otherwise have

[sdedovic]

To me, its more about the style than the use of an AI. But I agree.

I enjoyed this writeup by Michael Lynch on finding an illustrator [1], for their blog. In doing some of my own writing, I've really found it enlightening how much secondary work goes into publishing your own work. I often think its so nice to be able to _just_ plug in what I want on a site and get a (more or less) free illustration. But as someone selling their own work / time, it feels wrong. I'd rather pay a real human and build a relationship and have something more quality. On the other hand, though, it can be expensive, time consuming, and I've been screwed over. Often it seems like a bigger risk than its worth.

So idk, you're trading some hardship and risk for an ethical dilemma but ease of use.

[1] https://mtlynch.io/how-to-hire-a-cartoonist/

[cpeterso]

Worse yet, the dining philosophers in the image have too many hands. No wonder they’re deadlocking! :)

[mort96]

Between the shitty obviously-AI-generated square header image with floating hands everywhere, the equally shitty obviously-AI-generated image in the middle and the "Please pay for Medium" banner which takes up literally half the page, this blog post does its utmost to make a truly terrible impression.

[blindriver]

I prefer AI generated images over stock photos though. You can tell that both are phony, but at least the AI can be a bit more creative.

[minimaxir]

There's a quality spectrum of AI-generated header images. Some are just random DALL-E output which aren't intrinsically relevant to the article (like the one used in this article), but you can have a little fun with it and do something distinct. This may require more control than just using Bing Image Creator.

Also, a thumbnail tip: square thumbnails are bad. If you have to use a square 1024x1024 AI generation, crop it to something like 1024x575, which incidentally can make things difficult if using AI generation since figuring out what to crop requires human intervention.

[gcau]

It cheapens it, making it look like AI-generated seo-blog-spam. I'd rather a technical diagram or some plain icons, at least that would look tasteful.

[sureglymop]

I just don't really understand the value they're supposed to bring. If everyone uses the same looking generates images it just makes all the blogs look the same again. Then the thing is, they usually have nothing to do with the actual article. So why not just leave them out and not waste space.

[kelnos]

I dislike the style this particular author chose, but don't object in general. Assuming the images are actually somewhat relevant (or at least funny), I think I'd prefer an AI-generated image over a big wall of text.

To each their own, though, of course.

[azinman2]

Typically it’ll be in lieu of nothing or stock photography. Doesn’t it seem better than that?

[DeathArrow]

Whole Medium thing seems low effort. I hardly remember reading a well written article there.

[nottorp]

Interesting. Now that you mention it, there are illustrations there. But I'm pretty sure I subconsciously scrolled past them to get to the rest of the article without consciously noticing them on the first read.

Generated or hand drawn, they're kinda a wasted effort on a technical post.

[Zetobal]

At least you now know which of your peers have no taste and strange beauty standards. Some images posted by my colleagues for everyone to see on LinkedIn look like sexist propaganda cartoons.

Stock images used to hide this "quality" better than I thought.

[ta8645]

Honestly, the images attached to the article seemed great to me; they were colorful and fun. I don't see any reason to care who or what created them.

[Rapzid]

I like the image. It's cute and it's fun picking out the defects.

[davidgerard]

exactly. It says "I can't be bothered producing this" and I feel like, so why should I be bothered reading it?

[danielovichdk]

Kinda like when someone pulls in FOSS code or a package without contributing or at least email the authors.

[MrBuddyCasino]

Someone (not me) put it like this:

"to the trained eye you can already see that every single ai generated image is a picture of the same thing"

[NovaX]

It is a known caveat that virtual threads do not work well with long running synchronization by pinning the thread. That unfortunately means that for many applications it may be premature to adopt them, but it is mature enough for broader evaluation by the libraries and frameworks. The Java team provided a status of their efforts recently [1].

https://www.youtube.com/watch?v=WoQJnnMIlFY&t=421s

[avodonosov]

Sorry, the first sentence is a mis-informing wording.

The `synchronized` pins the thread only when from within of the `synchronized` the program calls a blocking operation that would normally unmount the virtual thread, like blockingQueue.take() or similar. (Which is not a sane coding practice). It's because the unmounting, as it's implemented today, does not work well with synchronized.

It's better if people read JEP 444 than rely on forum comments, to avoid being misinformed.

Speaking of long-running - even without synchronized, a long running code keeps the native thread occupied, until some blocking operation is called. So an endless loop that does not call a virtual-thread-ready blocking operation will occupy the native thread forever.

Java virtual threads are a kind of cooperative multithreading - another virtual thread only gets chance to kick-in when some current virtual thread reaches specific blocking operations. In contrast to preemptive multi-threading with native threads.

So I agree with your conclusion. Virtual threads can not (yet?) be blindly used as a drop-in replacement of native threads for existing code. And the new code needs to take their specifics into account.

BTW, another method I discovered to block the native carrier thread that executes a virtual thread is to call blocking reading through FileInputStream, for example reading from the console. The FileInputStream does not implement virtual thread parking at all (yet?).

[spintin]

https://www.youtube.com/watch?v=WoQJnnMIlFY&t=260s

To get the whole context, so virtual threads are unusable?

What holds a monitor by default and is there a workaround?

Found more:

    A virtual thread cannot be unmounted during blocking operations when it is pinned to its carrier. A virtual thread is pinned in the following situations:

    The virtual thread runs code inside a synchronized block or method

    The virtual thread runs a native method or a foreign function (see Foreign Function and Memory API)

For those those that don't know what this means: Blocking network TCP IO needs a sychronized block to work = you can't use virtual threads for networking. I wish they formulated it like that from the start!

Atleast now we know what they meant with don't use virtual threads for anything but tasks <- not blocking IO with synchronization!

So for now manual NIO is still the king of the hill.

We are reaching peak humanity levels of complexity!

[saagarjha]

The problem is not “long running synchronization” but synchronization that relies on stuff running outside of virtual threads to unblock itself. There is no issue beyond performance if you perform filesystem operations in your mounted state.

[papercrane]

Curious if you considered switching to a different connection pooling library. These days I usually use HikariCP which is fast an actively maintained. c3p0 hasn't had any activity for years, I'm not sure if it's still maintained.

[samus]

Crucially, c3p0 will probably never see the `synchronized` blocks being replaced by reentrant locks. Since LTS offers exist for Java 21, many libraries might actually do that. But I actually hope that the ecosystem resists, which would force virtual thread users suffering from this problem to upgrade soon.

[rickette]

Indeed, Hikari is the go to connection pool for some years now. It's even the default when running Spring Boot.

[eivanov89]

Perhaps we'll give HikariCP a chance. However, please keep in mind that the goal of the YDB team is to enhance database performance. We needed virtual threads to make TPC-C efficient enough to generate a reasonable load on a modest amount of hardware.

[tveita]

Looks like HikariCP is also awaiting fixes for this https://github.com/brettwooldridge/HikariCP/pull/2055

[whalesalad]

Been ages since I’ve touched it but back in 2017-2018 I had some fun integrating HikariCP in place of c3p0 in some Clojure projects and it was more performant.

[vvern]

Go has a mechanism to spawn a new thread (m in ho runtime parlance) if it thinks one of its threads might be blocked in a cgo (go’s “native function” equivalent). That prevents stuff like this.

[avodonosov]

Java does the same for Object.wait(), only the number of such compensating threads is limited by default, but can be extended via config option. They have exhausted the default number of compensating threads, I think.

And they are mistaken to call this situation a "pinning"

JEP 444:

> The vast majority of blocking operations in the JDK will unmount the virtual thread, freeing its carrier and the underlying OS thread to take on new work. However, some blocking operations in the JDK do not unmount the virtual thread, and thus block both its carrier and the underlying OS thread. This is because of limitations at either the OS level (e.g., many filesystem operations) or the JDK level (e.g., Object.wait()). The implementations of these blocking operations compensate for the capture of the OS thread by temporarily expanding the parallelism of the scheduler. Consequently, the number of platform threads in the scheduler's ForkJoinPool may temporarily exceed the number of available processors. The maximum number of platform threads available to the scheduler can be tuned with the system property jdk.virtualThreadScheduler.maxPoolSize.

(In my testing the default ForkJoinPool limit was 256)

So theoretically they could have extended the jdk.virtualThreadScheduler.maxPoolSize to a number sufficient for the use case. Although their workaround with semaphores is probably more reliable - no need to guess the sufficient number.

The situation with Object.wait() is not what JEP 444 calls "pinning". The "pinning" happens, for example, when one calls `syncronized(....) {blockingQueue.take()}`, which is not sane coding, BTW. In this case the native thread is blocked and is not compensated by another thread - much worse than the Object.wait(). The number of native threads that run virtual threads is equal to the number of CPUs by default, so "pinning" immediately makes one CPU unavailable to the virtual threads of the application.

All those issues are temporarily, as I understand. The JDK team works for fix Object.wait(), synchronized, etc.

[neonsunset]

So does C# with active blocking detection (which injects threads to counteract this) and hill climbing algorithm to scale threadpool threads automatically.

[spintin]

The warning shots across the bow where heard with this statement from the devs:

"Don't replace platform/native threads with virtual ones, replace tasks (without further explanation) instead"?!

Combine that with the fact that they chose to implement the scheduler in Java instead of C(++) and you're set for performance problems.

Remember that NIO took from 1.5 to 1.7 to be usable/performant and that was native!

Edit: Finally figured out why: https://news.ycombinator.com/item?id=39010648

[saagarjha]

This is a common problem when migrating a system from threads to virtual threads. In general, using primitives which block the current thread and prevent forward progress can quickly lead to deadlocks. It’s a hard issue to catch because in the past usually this would get “solved” by spawning a new thread to complete the task but in a world with virtual threads the runtime is usually reluctant to spawn more threads, so there’s nothing that can service more work if you’ve blocked all the threads.

[moonchild]

I am extremely confused.

> There are two scenarios in which a virtual thread cannot be unmounted during blocking operations because it is pinned to its carrier:

> When it executes code inside a synchronized block or method

Isn't 'synchronized' effectively sugar for taking a kind of lock? Why can't it be treated uniformly by the scheduler?

[taspeotis]

Did they get a deadlock again? https://news.ycombinator.com/item?id=38939165

[dikei]

I wonder if HikariCP, currently the best Java DB Connection Pooling library, suffer the same issue as c3p0.

[jshowalter]

Why not treat this as a bug, and fix it in Java 21? For compliance reasons we can only use LTS versions, and the next one isn't until September 2025, according to https://www.oracle.com/java/technologies/java-se-support-roa....

[torrent]

Seems to be a similiar problem field as writing blocking functions that call async functions in C# and co-existence of synchronous and asynchronous code.

There are numerous recommendations such as

https://learn.microsoft.com/en-us/archive/msdn-magazine/2015...

Final phase is "I hope these techniques will help you adopt async into your existing applications in a way that works best for you."

[pierrebai]

I thought the blog was great but "in summary" conclusion bad.

The summary merely stated that Java virtual thread are great. I expected a summary of the problem and solution, for example something like:

When using Java 21 virtual threads, you can end-up starved of carrier threads due to all carrier threads waiting on a pool exhausted resources with no thread available to free such resources. The solution is to wrap those resources in a virtual-thread aware object. In our case, we solved our problem by wrapping connections in semaphores.

[xyst]

Concurrency, parallelism. These are among the most misunderstood concepts in programming/software development.

TLS (especially mutual TLS) and Oauth also join this club.

[skyde]

Why synchronized block are not preemptible? When compiling

public void syncMethod() {

        synchronized(lock) {
            // some code
        }

}

they could translate to

public void syncMethod() {

        await reentrantLockAsync.lockAsync();
        try {
            await somecodeAsync();
        } finally {
            await lock.unlockAsync()
        }
    }

[dxxvi]

Why was c3p0 used (its latest version was released in Dec 2019)? Those tests existed for a while and people were too lazy to replace c3p0 with something newer? I guess that they spent all their time to use virtual threads in those tests and had no time left to look at c3p0.

[bheadmaster]

Why couldn't JVM detect when all carrier threads are blocked, and just spawn more of them?

[bob1029]

Sounds similar to the quirks you get with TPL in .NET under some circumstances. For library code, a ConfigureAwait(false) invoke should be considered to signify that the execution does not need to resume on the original thread.

[nottorp]

Can someone explain what these virtual threads are in 2 sentences please?