Midjourney's "ControlNet" (Patreon)
Content
Before you get excited: no, MJ doesn't have a ControlNet extension. If you don't know what is ControlNet, it is a methodology available for Stable Diffusion (another image generator system) that provides a lot of control over the generated images. For example:
image by Ng Wai Foong
One of the options is the "LineArt":
So I tried to force Midjourney to produce images based on my own lineart, which is good enough to be understood by MJ, but is bad enough to lack shadows, movement, style, person.... ok, too much bad self-talk. Anyway, this is the image I am using in this post:
I did some tests, but to avoid a long post, those are my (not final) conclusions:
The image doesn't work like ControlNet (as expected). MJ interprets image prompts not as templates, but the way /describe would do. So my image doesn't mean "short girl in a strange way of running" (or something like that), but "happy little girl running clipboard drawing, in the style of traincore, heavy inking, shiny/glossy" etc.
Yet MJ was able to copy the girl's pose in an accurate way, sometimes in different angles:
prompt: [ image prompt ] color pencil --s 120, 600, 1000
Style Raw interpreted by prompt too literally but it went better when I changed to "watercolor"
prompt: [ image prompt ] color pencil --style raw --s 10, 600, 1000
prompt: [ image prompt ] watercolor --style raw --s 10, 800, 1000
Niji 5 also brought something to the table. Vanilla Niji, Expressive, and Cute focused on closer shots, while Scenic provided a background.
prompt: [ image ] digital painting --niji 5 --s 600 --style {expressive, cute, scenic }
Of course, this is just the tip of the iceberg! I didn't test a different style, with thinner lines or pencils, for example. Also, we are using very simple text prompts, because it is easier to see the balance between image and text. Finally, image weight will probably give us different results. In other words, there is a world of experimentation and creativity in this small feature of Midjourney.
I invite you to join us in our Discord server, and join our experiments!
See you soon,
Library Coach (Jos)
Files