Conditioning Vector Math for ComfyUI?
Let me first state for the record I could be wildly wrong here.
I am meddling with stuff that I'm not exactly understanding. This also may already be achievable within the existing comfyui tools, but I didn't understand it and wanted to play with nodes in an extension, so this was my excuse to get involved.
I kept reading the idea of the text encoder as a multi-dimensional vector of some form.
if you encode man and king and then king - man, you now have a directional vector that embeds the idea of making a man into a king. so If I start with a constant seed and combine the encoded text like "A man on a horse" + (("king" - "man") * scale) in some way, I could walk this seed towards a kingly person on a horse.... right? I don't expect this to be perfectly cohesive, I understand composition could change, etc.. I get that it's kinda stochastic.
But how can I see this inside of comfyui? I approached this with some of the conditioning tools and things I could find, but they didn't make sense to me in that something was off. My assumption was that my material understanding of what I wanted to do wasn't exactly right in this context (a high level understanding vs an actual implementation)
So I set out to make some nodes. To better grasp where my understanding is broken. Is this even... right, this was an amalgamation of my understanding and claude.ai helping me out, so ... ehem lots of assumptions, that I had a hard time finding the documentation, it seems "the code is the documentation" is the name of the game here.
I have a few overarching questions: * Am I even capable of doing this... is this materially right in that the concept is compatible with stable diffusion/comfy ui, * What do I need to do to make it work, or is it just silly and abandon the idea altogether? * Can this already be done in ComfyUI
Thanks for the help, here's my "Concept Direction Calculator" code, I may make a repo for this later, my intent was to publish if it was useful.
class ConceptDirectionCalculator:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"conditioning_a": ("CONDITIONING",),
"conditioning_b": ("CONDITIONING",),
}}
RETURN_TYPES = ("CONDITIONING",)
FUNCTION = "calculate_direction"
CATEGORY = "conditioning"
def calculate_direction(self, conditioning_a, conditioning_b):
result_conditioning = []
for (cond_a, meta_a), (cond_b, meta_b) in zip(conditioning_a, conditioning_b):
# Ensure embeddings are on the same device and have the same dtype
cond_b = cond_b.to(device=cond_a.device, dtype=cond_a.dtype)
# Calculate direction vector
direction_vector = cond_b - cond_a
# Normalize the direction vector
norm = torch.norm(direction_vector, dim=-1, keepdim=True)
normalized_direction = torch.where(norm != 0, direction_vector / norm, direction_vector)
# Use small values for the direction instead of unit vector
scaled_direction = normalized_direction * 1e-3
# Handle pooled_output if present
result_meta = meta_a.copy()
if "pooled_output" in meta_a and "pooled_output" in meta_b:
pooled_a = meta_a["pooled_output"]
pooled_b = meta_b["pooled_output"].to(device=pooled_a.device, dtype=pooled_a.dtype)
pooled_direction = pooled_b - pooled_a
pooled_norm = torch.norm(pooled_direction)
normalized_pooled_direction = pooled_direction / pooled_norm if pooled_norm != 0 else pooled_direction
result_meta["pooled_output"] = normalized_pooled_direction * 1e-3
result_conditioning.append([scaled_direction, result_meta])
return (result_conditioning,)
6 comments · 75% upvoted · other discussions
Im bumping because it looks cool, but this is too esoteric for me
Dismal-Rich-7469 2 points · Sep 2024
Wrote a reply. You might be intrested.
Patient-Librarian-33 2 points · Sep 2024
Thanks, reading it!