WingNews logo WingNews GitHub [2]
top | item 34971579

Show HN: Scribble Diffusion – Turn your sketch into a refined image using AI

373 points| zsikelianos | 3 years ago |scribblediffusion.com

89 comments

order
[+] dragonwriter|3 years ago|reply
This looks like its just a quick front-end to the ControlNet scribble model (perhaps with a customized underlying model instead of base Stable Diffusion), with canned settings, presumable a canned negative prompt, and maybe some common stuff added beyond the user input to the positive prompt. Which is not to be dismissive, its a very nice demo of what SD+ControlNet scribble can do.

But for people who like it, the ControlNet scribble model (and the other ControlNet models, depth-map based, pose control, edge detection, etc.) [0] are supported in the ControlNet extension [1] to the A1111 Stable Diffusion Web UI [2], and probably similar extensions for other popular stable diffusion UIs. Should work in any current browser, and at least the A1111 UI, with ControlNet models, works on machines with as little as 4GB VRAM.

[0] home repo: https://huggingface.co/lllyasviel/ControlNet but for WebUI you probably want the ones linked from the readme of the WebUI ControlNet extensions [1]

[1] https://github.com/Mikubill/sd-webui-controlnet (EDIT: even if you aren’t using the A1111 WebUI, this repo has a nice set of examples of what each of the ControlNet models does, so it may be worth checking out.)

[2] https://github.com/AUTOMATIC1111/stable-diffusion-webui

[+] davely|3 years ago|reply
I recently added support for it on ArtBot[1] and it supports additional options like models, steps, control type, etc, as well drawing on a canvas[2] and generating input from that. It’s pretty wild!

Free and open source [3].

Bonus: it works on Firefox (unless you’re using private mode — because Firefox doesn’t make IndexedDb available to web apps in private mode and things end up breaking)

[1] https://tinybots.net/artbot/controlnet

[2] https://tinybots.net/artbot/draw

[3] https://github.com/daveschumaker/artbot-for-stable-diffusion

[+] liuliu|3 years ago|reply
You probably don't need canned positive prompt. ControlNet has a "guess mode" and in reality, basically just send the control to positive part with empty prompt while not sending control to negative part with empty prompt.

Edit: nvm, this particular demo does require you to type in positive prompt.

[+] zsikelianos|3 years ago|reply
Hey, Scribble Diffusion author here.

Thanks for your kind words and feedback. I love seeing all these links to your scribbles.

I'm an engineer at Replicate, which is a place to run ML models in the cloud. [0] We built Scribble Diffusion as an open-source app [1] to demonstrate how to use Replicate.

This is all built on ControlNet [2], a brilliant technique by Lvmin Zhang and Maneesh Agrawala [3] for conditioning diffusion models on additional sources, in this case human scribbles. It also allows for controlling Stable Diffusion using other inputs like pose estimation, edge detection, and depth maps.

ControlNet has only existed for three weeks, but people are already doing all kinds of cool stuff with it, like an app [4] that lets you pose a stick figure and generate DreamBooth images that match the pose. There are already a bunch of models [5] on Replicate that build on it.

I see a few bits of feedback here about issues with the Scribble Diffusion UI, and I'm tracking them on the GitHub repo. If you want to help out, please feel free to open an issue or pull request.

[0] https://replicate.com

[1] https://github.com/replicate/scribble-diffusion

[2] https://github.com/lllyasviel/ControlNet

[3] https://arxiv.org/abs/2302.05543

[4] https://twitter.com/dannypostmaa/status/1630442372206133248

[5] https://replicate.com/explore?query=controlnet

[+] trees101|3 years ago|reply
What a simple, but excellent concept! I kind of expected it to crash after being posted to HN, but surprisingly it is still going strong. What does it cost to host something like that? What does it cost to generate each scribble image?
[+] dragonwriter|3 years ago|reply
If you haven’t played with Stable Diffusion based stuff before, note that as well as telling it what (in terms of subject) you want in the prompt, you can also tell it style things. Compare:

“a goofy owl”: https://scribblediffusion.com/scribbles/oymg4kadgvezxppvwkf5...

“a goofy owl, realistic photograph, depth of field, 4k, HDR”: https://scribblediffusion.com/scribbles/va5l24amjnb55g62renz...

“a goody owl, pointillism”: https://scribblediffusion.com/scribbles/5dfnru4f6zguphjvvdrl...

[+] sinistersnare|3 years ago|reply
Looks awesome, I think this is a great mix with prompt engineering.

It seems this does not work on Firefox? I could only draw on about half the canvas and it was pretty buggy. Dont support the chrome monoculture!

[+] zsikelianos|3 years ago|reply
Hey, Scribble Diffusion author here.

Sorry about the trouble. The Firefox incompatibility is the result of a bug in the underlying npm package we're using to render the drawing tool and canvas.

The issue is being tracked here: https://github.com/replicate/scribble-diffusion/issues/17#is...

We may need to wait for a fix to that, or consider swapping out the package we use for scribbling on a canvas.

[+] citizenkeen|3 years ago|reply
Agreed. I nearly just closed the tab when it didn't work on Firefox.
[+] dylan604|3 years ago|reply
wow, yeah, very frustrating experience
[+] 1minusp|3 years ago|reply
Will be nice to be able to upload a picture/sketch instead of having to sketch something out in the scratchpad. I tend to sketch block diagrams from time to time and hate having to use some tool to draw/drag/align things. Very cool idea though!
[+] zsikelianos|3 years ago|reply
Hey there. Scribble Diffusion author here.

If you already have your own images, you can use the Replicate model directly: https://replicate.com/jagilley/controlnet-scribble -- you can upload your image using the Replicate web UI or do it programmatically using the model's HTTP API.

[+] slhomme|3 years ago|reply
Uploading my own images was one of the reasons I created my own version of this. It works great with kids pictures! This is so fun to see my kids smile on their face every time they see the AI version of their drawings!

Uploaded photos Examples: https://imgur.com/a/jeWgRvH Website: https://crappydrawings.ai

[+] amflare|3 years ago|reply
This is cool. At first I was skeptical that the sketch had nothing to do with anything and it was just feeding the prompt into an AI. But using the example "sun behind mountains"[0] it seemed to work alright. What really blew my mind was when I paired the sketch with a prompt that made no sense at all[1]. Somehow the AI cobbled together a picture that used the prompt, but also has all the same geometry of the sketch.

[0]: https://scribblediffusion.com/scribbles/elun6gwkxrcr7eqy5jao... [1]: https://scribblediffusion.com/scribbles/tpsty6qcxjfrxbxz3n6b...

[+] bitwize|3 years ago|reply
I love how it synthesized Hide-the-Pain Harold to satisfy the prompt.
[+] janosdebugs|3 years ago|reply
The drawing doesn't work for me on Firefox.
[+] pamelafox|3 years ago|reply
I'm surprised, it actually did pretty well with a prompt I've been trying on all the generative AIs: "princess volcano kitty cat"

https://imgur.com/hVfCIwU

I use that prompt because it's something my preschooler draws on a daily basis with no qualms at all, but can often be hard for generative AI to imagine. So it's getting close to the imaginative abilities of a 3 year old! Halfway there.

[+] latexr|3 years ago|reply
I opened the page and did not change anything of the given example. After multiple tries, I continued to fail to produce the prompted image:

https://i.imgur.com/08o9zkG.jpg

https://i.imgur.com/KkBtTyd.jpg

Eventually I did get something which looked like a partial success, but with low resolution and not something I’d consider appealing:

https://i.imgur.com/SJWkwBp.jpg

https://i.imgur.com/BpSUD4j.jpg

This is not a dig on the author. I have yet to see a simple prompt give a good result with Stable Diffusion. Are there examples of it?

[+] nilslindemann|3 years ago|reply
I drew three dots and wrote "spinning symbol", and it even animated the result.
[+] ajjenkins|3 years ago|reply
This would be so much fun as a feature in Drawful.