∞ OpenAI Model Options¶

Documentation¶

Class name: LLMOpenAIModelOpts
Category: SALT/Language Toolkit/Loaders/Options
Output node: False

This node provides a comprehensive interface for configuring options for various language models, including OpenAI's models. It allows users to fine-tune model parameters such as temperature, token limits, and API retry limits, as well as embedding model settings like batch size and dimensions. The node aims to offer a flexible setup for language model utilization, catering to specific needs for text generation and embedding tasks.

Input types¶

Required¶

llm_model
- unknown
- Comfy dtype: LLM_MODEL
- Python dtype: unknown

Optional¶

model_temperature
- Specifies the temperature for the language model, affecting the randomness of the output. Lower values result in more deterministic outputs, while higher values increase creativity.
- Comfy dtype: FLOAT
- Python dtype: float
model_max_tokens
- Defines the maximum number of tokens the language model can generate in a single response, setting a limit on the output length.
- Comfy dtype: INT
- Python dtype: int
model_api_max_retries
- Determines the maximum number of times to retry the API call in case of failure, ensuring robustness in network conditions.
- Comfy dtype: INT
- Python dtype: int
model_api_timeout
- Sets the timeout duration for the API call, ensuring the process does not hang indefinitely.
- Comfy dtype: INT
- Python dtype: int
model_reuse_anyscale_client
- Indicates whether to reuse the Anyscale client across multiple requests, optimizing resource utilization.
- Comfy dtype: BOOLEAN
- Python dtype: bool
multimodal_max_new_tokens
- Specifies the maximum number of new tokens for multimodal inputs, applicable when the model supports multimodal data.
- Comfy dtype: INT
- Python dtype: int
multimodal_image_detail
- Determines the level of detail for images in multimodal inputs, with possible values being 'low', 'high', or 'auto', affecting the model's processing of visual data.
- Comfy dtype: COMBO[STRING]
- Python dtype: str
embed_batch_size
- Sets the batch size for processing embeddings, impacting performance and resource usage.
- Comfy dtype: INT
- Python dtype: int
embed_dimensions
- Specifies the dimensionality of the embeddings, affecting the vector representation's granularity.
- Comfy dtype: INT
- Python dtype: int
embed_api_max_retries
- Determines the maximum number of retries for embedding API calls, ensuring reliability in network conditions.
- Comfy dtype: INT
- Python dtype: int
embed_api_timeout
- Sets the timeout duration for embedding API calls, preventing indefinite hanging.
- Comfy dtype: INT
- Python dtype: int
embed_reuse_anyscale_client
- Indicates whether to reuse the Anyscale client for embedding operations, optimizing for resource efficiency.
- Comfy dtype: BOOLEAN
- Python dtype: bool
model_additional_kwargs
- Allows for additional keyword arguments to be passed to the model, offering extended customization.
- Comfy dtype: STRING
- Python dtype: Dict[str, Any]
embed_additional_kwargs
- Enables additional keyword arguments for embedding operations, providing further customization options.
- Comfy dtype: STRING
- Python dtype: Dict[str, Any]
model_system_prompt
- Defines a system-level prompt that can be used to guide the model's generation process, offering a way to influence the context or direction of the output.
- Comfy dtype: STRING
- Python dtype: str

Output types¶

model
- Comfy dtype: LLM_MODEL
- A dictionary containing the updated language model and embedding model objects, reflecting the applied configuration options.
- Python dtype: Dict[str, Any]

Usage tips¶

Infra type: CPU
Common nodes: unknown

Source code¶

class LLMOpenAIModelOpts:
    """
    Sets various options for the model, and embedding.
    """
    def __init__(self):
        pass

    @classmethod
    def INPUT_TYPES(cls):
        return {
            "required": {
                "llm_model": ("LLM_MODEL",),
            },
            "optional": {
                "model_temperature": ("FLOAT", {"min": 0, "max": 1.0, "default": DEFAULT_TEMPERATURE, "step": 0.001}),
                "model_max_tokens": ("INT", {"min": 8, "default": 4096}),
                "model_api_max_retries": ("INT", {"min": 1, "max": 12, "default": 3}),
                "model_api_timeout": ("INT", {"min": 8, "max": 120, "default": 60}),
                "model_reuse_anyscale_client": ("BOOLEAN", {"default": True}),

                "multimodal_max_new_tokens": ("INT", {"min": 8, "default": 300}),
                "multimodal_image_detail": (["low", "high", "auto"],),

                "embed_batch_size": ("INT", {"min": 8, "default": DEFAULT_EMBED_BATCH_SIZE}),
                "embed_dimensions": ("INT", {"min": 1, "default": DEFAULT_EMBEDDING_DIM}),
                "embed_api_max_retries": ("INT", {"min": 1, "max": 12, "default": 3}),
                "embed_api_timeout": ("INT", {"min": 8, "max": 120, "default": 60}),
                "embed_reuse_anyscale_client": ("BOOLEAN", {"default": True}),

                "model_additional_kwargs": ("STRING", {"multiline": True, "dynamicPrompts": False, "default": "", "placeholder": "Additional model kwargs JSON"}),
                "embed_additional_kwargs": ("STRING", {"multiline": True, "dynamicPrompts": False, "default": "", "placeholder": "Additional embed kwargs JSON"}),
                "model_system_prompt": ("STRING", {"multiline": True, "dynamicPrompts": False, "placeholder": "System directions, or rules to follow globally across nodes."}),
            }
        }

    RETURN_TYPES = ("LLM_MODEL", )
    RETURN_NAMES = ("model", )

    FUNCTION = "set_options"
    CATEGORY = f"{MENU_NAME}/{SUB_MENU_NAME}/Loaders/Options"

    def set_options(self, llm_model:Dict[str, Any], **kwargs) -> Dict[str, Any]:
        llm = llm_model['llm']
        embed = llm_model['embed_model']

        # LLM Options
        llm.temperature = kwargs.get("model_temperature", DEFAULT_TEMPERATURE)
        llm.max_retries = kwargs.get("model_api_max_retries", 3)
        llm.reuse_client = kwargs.get("model_reuse_anyscale_client", True)
        llm.additional_kwargs = json.loads(kwargs.get("model_additional_kwargs", {}).strip()) if kwargs.get("model_additional_kwargs", {}).strip() != "" else {} # Default to `None` if empty string
        llm.system_prompt = kwargs.get("model_system_prompt", None)

        # Embed Options
        embed.embed_batch_size = kwargs.get("embed_batch_size", DEFAULT_EMBED_BATCH_SIZE)
        embed.dimensions = kwargs.get("embed_dimensions", DEFAULT_EMBEDDING_DIM) if kwargs.get("embed_dimensions", DEFAULT_EMBEDDING_DIM) > 0 else None # Default to `None` if not above 0
        embed.additional_kwargs = json.loads(kwargs.get("embed_additional_kwargs", {}).strip()) if kwargs.get("embed_additional_kwargs", "").strip() != "" else {} # Default to `None` if empty string
        embed.max_retries = kwargs.get("embed_api_max_retries", 3)
        embed.timeout = kwargs.get("embed_api_timeout", 60)
        embed.reuse_client = kwargs.get("embed_reuse_anyscale_client", True)

        if isinstance(llm, OpenAIMultiModal):
            llm.max_new_tokens = kwargs.get("multimodal_max_new_tokens", 300)
            llm.image_detail = kwargs.get("multimodal_image_detail", "low")
        else:
            llm.max_tokens = kwargs.get("model_max_tokens", 4096)

        llm_model['llm'] = llm
        llm_model['embed_model'] = embed

        return (llm_model,)