Model Configuration

ByteBuddy supports multiple AI models, allowing you to choose the most suitable model according to your needs.

Configuration File Location

ByteBuddy configuration files are located at:

• ~/.bytebuddy/config.yaml
• config.yaml in the project root directory

Supported Provider Types

🚀 Popular Providers

These providers have extensive user bases and mature ecosystems:

• OpenAI - GPT series models
• Anthropic - Claude series models
• Google - Gemini models
• Azure - Azure OpenAI service
• Amazon Bedrock - AWS managed models

🔧 Self-hosted Solutions

Run models in local or private environments:

• Ollama - Local model runner
• LM Studio - Desktop model application
• LLaMA.cpp - C++ inference engine
• LLaMA Stack - Distributed inference

🌐 API Aggregation Services

Access multiple models through a single interface:

• OpenRouter - Multi-model routing
• Together - Open source model platform
• HuggingFace - Inference API

🏢 Enterprise Services

Professional services suitable for enterprise scenarios:

• Vertex AI - Google Cloud AI
• Tetrate - Service mesh integration
• NVIDIA - GPU optimized models

🧪 Experimental and Emerging Providers

The latest models and services:

• xAI - Elon Musk's AI company
• Mistral - European AI company
• DeepSeek - Chinese AI company
• Groq - Fast inference chips
• Inception - Emerging model service

Model Roles

ByteBuddy supports configuring different models for different roles:

• chat: For conversation interaction and complex task processing
• edit: For code editing tasks
• apply: For code application operations
• autocomplete: For real-time code completion
• embed: For embedding vector generation
• rerank: For search result reranking
• summarize: For text summarization

Basic Configuration Structure

ByteBuddy uses YAML configuration format. Here is the basic structure for model configuration:

Single Model Configuration

models:
  - name: "gpt-4o" # Model name
    provider: "openai" # Provider
    model: "gpt-4o" # Specific model
    apiKey: "${OPENAI_API_KEY}" # API key (environment variable)
    roles: ["chat"] # Role types: chat, edit, apply, autocomplete, embed, rerank, summarize
    capabilities: ["tool_use"] # Capabilities: tool_use, image_input, next_edit
    defaultCompletionOptions: # Default completion options
      temperature: 0.7
      maxTokens: 4000
    requestOptions: # Request options
      timeout: 30000
      verifySsl: true

Multi-model Configuration

models:
  - name: "chat-model"
    provider: "openai"
    model: "gpt-4o"
    apiKey: "${OPENAI_API_KEY}"
    roles: ["chat", "edit"]

  - name: "local-model"
    provider: "ollama"
    model: "llama3"
    apiBase: "http://localhost:11434"
    roles: ["chat"]

  - name: "embedding-model"
    provider: "openai"
    model: "text-embedding-3-small"
    apiKey: "${OPENAI_API_KEY}"
    roles: ["embed"]

Configuration Field Description

Required Fields

• name: Unique identifier for the model configuration
• provider: Model provider (such as openai, anthropic, ollama, etc.)
• model: Specific model name

Optional Fields

• apiKey: API key, supports environment variable format ${VARIABLE_NAME}
• apiBase: Custom API endpoint URL
• roles: Array of model roles, optional values include:
  • chat: Chat conversation
  • edit: Code editing
  • apply: Apply changes
  • autocomplete: Code autocompletion
  • embed: Embedding vector generation
  • rerank: Reranking
  • summarize: Text summarization
• capabilities: Array of model capabilities, optional values include:
  • tool_use: Tool usage capability
  • image_input: Image input support
  • next_edit: Next edit mode
• defaultCompletionOptions: Default completion options
  • temperature: Generation randomness (0-2)
  • maxTokens: Maximum generation token count
  • topP: Nucleus sampling parameter
  • stream: Whether to stream output
• autocompleteOptions: Autocomplete options
  • maxPromptTokens: Maximum prompt token count
  • useCache: Whether to use cache
• requestOptions: Request options
  • `timeout**: Request timeout (milliseconds)
  • verifySsl: Whether to verify SSL certificates
  • headers: Custom request headers
  • proxy: Proxy settings

Environment Variable Configuration

For security, it's recommended to use environment variables to store API keys:

# ~/.bashrc or ~/.zshrc
export OPENAI_API_KEY="your-openai-api-key"
export ANTHROPIC_API_KEY="your-anthropic-api-key"

Common environment variables:

• ${OPENAI_API_KEY} - OpenAI API key
• ${ANTHROPIC_API_KEY} - Anthropic API key
• ${GOOGLE_API_KEY} - Google AI API key
• ${AZURE_OPENAI_API_KEY} - Azure OpenAI API key

Common Provider Configuration Examples

OpenAI Configuration

models:
  - name: "gpt-4o"
    provider: "openai"
    model: "gpt-4o"
    apiKey: "${OPENAI_API_KEY}"
    roles: ["chat", "edit"]
    defaultCompletionOptions:
      temperature: 0.7
      maxTokens: 4000

Anthropic Configuration

models:
  - name: "claude-3-5-sonnet"
    provider: "anthropic"
    model: "claude-3-5-sonnet-20241022"
    apiKey: "${ANTHROPIC_API_KEY}"
    roles: ["chat"]
    capabilities: ["tool_use"]
    defaultCompletionOptions:
      temperature: 0.7
      maxTokens: 8000

Google Gemini Configuration

models:
  - name: "gemini-2.0-flash"
    provider: "google"
    model: "gemini-2.0-flash"
    apiKey: "${GOOGLE_API_KEY}"
    roles: ["chat"]
    capabilities: ["image_input"]

Ollama Local Model Configuration

models:
  - name: "local-llama"
    provider: "ollama"
    model: "llama3"
    apiBase: "http://localhost:11434"
    roles: ["chat"]
    requestOptions:
      timeout: 60000

Azure OpenAI Configuration

models:
  - name: "azure-gpt-4"
    provider: "azure"
    model: "gpt-4"
    apiKey: "${AZURE_OPENAI_API_KEY}"
    apiBase: "https://your-resource.openai.azure.com/"
    env:
      resourceName: "your-resource-name"
      deploymentName: "gpt-4-deployment"
      apiVersion: "2024-02-15-preview"

Usage

After configuration, models will automatically take effect in corresponding chat sessions:

1. Configure models: Add model configurations in config.yaml
2. Set environment variables: Configure API keys and other sensitive information
3. Restart application: Let configuration take effect
4. Start using: Models will be automatically available in the chat interface

Troubleshooting

Common Issues

Q: Model cannot be loaded?

• Check if configuration file syntax is correct
• Verify environment variables are set correctly
• Confirm API key is valid and has sufficient quota

Q: Local model connection failed?

• Confirm Ollama service is running
• Check if apiBase URL is correct
• Verify local model is downloaded

Q: Response time too long?

• Adjust timeout parameter
• Choose faster model or provider
• Consider using streaming output

Q: Configuration not taking effect?

• Check if configuration file location is correct
• Verify YAML syntax
• Restart ByteBuddy application

Verify Configuration

Create test configuration to verify settings:

# Validate configuration file syntax
bytebuddy config validate

# Test model connection
bytebuddy models test --name your-model-name

Best Practices

Security

1. Environment Variables: Always use environment variables to store API keys
2. Access Control: Set minimum necessary permissions for API keys
3. Regular Rotation: Regularly update API keys

Performance Optimization

1. Model Selection: Choose appropriate models based on task complexity
2. Parameter Adjustment: Optimize temperature, maxTokens and other parameters
3. Cache Settings: Enable appropriate caching strategies

Cost Control

1. Model Choice: Use more economical models for simple tasks
2. Token Limits: Set reasonable maxTokens limits
3. Local Deployment: Use local models for sensitive data

Next Steps

1. Select Provider: Choose suitable model provider based on your needs
2. Create Configuration: Add model configuration in ~/.bytebuddy/config.yaml
3. Set Keys: Configure corresponding environment variables
4. Test Verification: Verify configuration works correctly
5. Start Using: Use configured models in chat sessions