Conditioning Vector Math for ComfyUI?

Let me first state for the record I could be wildly wrong here.

I am meddling with stuff that I'm not exactly understanding. This also may already be achievable within the existing comfyui tools, but I didn't understand it and wanted to play with nodes in an extension, so this was my excuse to get involved.

I kept reading the idea of the text encoder as a multi-dimensional vector of some form.

if you encode man and king and then king - man, you now have a directional vector that embeds the idea of making a man into a king. so If I start with a constant seed and combine the encoded text like "A man on a horse" + (("king" - "man") * scale) in some way, I could walk this seed towards a kingly person on a horse.... right? I don't expect this to be perfectly cohesive, I understand composition could change, etc.. I get that it's kinda stochastic.

But how can I see this inside of comfyui? I approached this with some of the conditioning tools and things I could find, but they didn't make sense to me in that something was off. My assumption was that my material understanding of what I wanted to do wasn't exactly right in this context (a high level understanding vs an actual implementation)

So I set out to make some nodes. To better grasp where my understanding is broken. Is this even... right, this was an amalgamation of my understanding and claude.ai helping me out, so ... ehem lots of assumptions, that I had a hard time finding the documentation, it seems "the code is the documentation" is the name of the game here.

I have a few overarching questions: * Am I even capable of doing this... is this materially right in that the concept is compatible with stable diffusion/comfy ui, * What do I need to do to make it work, or is it just silly and abandon the idea altogether? * Can this already be done in ComfyUI

Thanks for the help, here's my "Concept Direction Calculator" code, I may make a repo for this later, my intent was to publish if it was useful.

class ConceptDirectionCalculator:
    @classmethod
    def INPUT_TYPES(s):
        return {"required": {
            "conditioning_a": ("CONDITIONING",),
            "conditioning_b": ("CONDITIONING",),
        }}

    RETURN_TYPES = ("CONDITIONING",)
    FUNCTION = "calculate_direction"
    CATEGORY = "conditioning"

    def calculate_direction(self, conditioning_a, conditioning_b):
        result_conditioning = []

        for (cond_a, meta_a), (cond_b, meta_b) in zip(conditioning_a, conditioning_b):
            # Ensure embeddings are on the same device and have the same dtype
            cond_b = cond_b.to(device=cond_a.device, dtype=cond_a.dtype)

            # Calculate direction vector
            direction_vector = cond_b - cond_a

            # Normalize the direction vector
            norm = torch.norm(direction_vector, dim=-1, keepdim=True)
            normalized_direction = torch.where(norm != 0, direction_vector / norm, direction_vector)

            # Use small values for the direction instead of unit vector
            scaled_direction = normalized_direction * 1e-3

            # Handle pooled_output if present
            result_meta = meta_a.copy()
            if "pooled_output" in meta_a and "pooled_output" in meta_b:
                pooled_a = meta_a["pooled_output"]
                pooled_b = meta_b["pooled_output"].to(device=pooled_a.device, dtype=pooled_a.dtype)
                pooled_direction = pooled_b - pooled_a
                pooled_norm = torch.norm(pooled_direction)
                normalized_pooled_direction = pooled_direction / pooled_norm if pooled_norm != 0 else pooled_direction
                result_meta["pooled_output"] = normalized_pooled_direction * 1e-3

            result_conditioning.append([scaled_direction, result_meta])

        return (result_conditioning,)

6 comments · 75% upvoted · other discussions

comments

Patient-Librarian-33

Im bumping because it looks cool, but this is too esoteric for me

Dismal-Rich-7469

Wrote a reply. You might be intrested.

Patient-Librarian-33

Thanks, reading it!

RevolutionaryLime758

It's definitely worth making a node for.  I'm actually surprised no one has already, as this concept has been understood for a long time now. Here's another implementation that is a bit further along than yours if you need more inspiration (no comfy node, just code): https://github.com/linoytsaban/semantic-sliders

Also implemented with a UI on huggingface space:  https://huggingface.co/spaces/latentexplorers/latentnavigation-flux

Dismal-Rich-7469

It works but you have to go further up the pipeline to make it happen.

pipeline for SD 1.5 is "text prompt" => //tokenizer//=> ids corresponding to Nx768 token_vectors => //CLIP// => 1x768 text_encoding => //SD Model// => image

This happens in SDXL and FLUX as well , but they use an additional text_encoder in paralell to the one above.

Note the that token_vector != text_encoding ,

even though they are both 1x768 tensors.

"Conditioning" in ComfyUI seems to refer to text_encoding.

At least according to these docs: https://blenderneko.github.io/ComfyUI-docs/Core%20Nodes/Conditioning/#:~:text=Conditioning%20In%20ComfyUI%20Conditionings%20are%20used%20to%20guide,nodes%20that%20can%20be%20found%20in%20this%20segment.

Check out these two notebooks:

token_vectors_math.ipynb : https://huggingface.co/datasets/codeShare/text-to-image-prompts/blob/main/Google%20Colab%20Notebooks/token_vectors_math.ipynb

For text encodings , Similiarity_calculator.ipynb : https://huggingface.co/datasets/codeShare/text-to-image-prompts/blob/main/Google%20Colab%20Notebooks/sd_token_similarity_calculator.ipynb

I'm planning to use the latter as an improved CLIP interrogator.

That's my main project , but you can go exploring the tokens if you like.

An easy and quick way to create cool embeddings out if the vocab.safetensors is to do torch.roll on them : https://pytorch.org/docs/stable/generated/torch.roll.html

This will rotate the elements in the 1x768 tensors vectors. You won't get any similarity at all to the original tokens , but they will "represent things" , if that makes sense.

I'm currently building this repo , so expect changes to it.

//---//

To make the calculation work

, you'll have to calculate the "King" - "Man" token_vector. Just vector math subtraction.

Lets call this vector X

Then tokenize "A man on a horse" into their ids [ 49406 , 320 , 786 , 525 , 320 , 4558 , 49407]

Remove the <|start-of-text|> and <|end-of-text|> at either end so you have

ids = [ 320 , 786 , 525 , 320 , 4558 ]

, and fetch the corresponding token_vectors from the vocab.safetensors I've prepared in the notebook.

for id in ids : vocab[f'{id}']

And then join these 5 tensors , with the 6th tensor X (which has no ID) , to create a 6x768 tensor

Normally , you send ids to CLIP to get a text_encoding , but you can't do it here since X has no ID

So you will have to search huggingface on ways to get a text_encoding from the tensors directly : https://huggingface.co/docs/transformers/model_doc/clip

, as you would do when loading a textual_inversion for example , which is also a Nx768 embedding with no IDs

prozacgod [OP]

Welp this will take a week to digest... HAHA! Thanks! This was a really high quality response. I'm really wondering how this idea will react with other conditioning tools, like control nets and such. Or even unsampler and then ksample that back but with some narrow and specific influence removed or added.