Kubernetes 部署
Kubernetes 部署
相关源文件
本章引用的主要源码文件:
ct.yamldeployment/helm/charts/onyx/Chart.lockdeployment/helm/charts/onyx/Chart.yamldeployment/helm/charts/onyx/templates/api-deployment.yamldeployment/helm/charts/onyx/templates/celery-beat.yamldeployment/helm/charts/onyx/templates/celery-worker-docfetching.yamldeployment/helm/charts/onyx/templates/celery-worker-docprocessing.yamldeployment/helm/charts/onyx/templates/celery-worker-heavy.yamldeployment/helm/charts/onyx/templates/celery-worker-light.yamldeployment/helm/charts/onyx/templates/celery-worker-monitoring.yamldeployment/helm/charts/onyx/templates/celery-worker-primary.yamldeployment/helm/charts/onyx/templates/configmap.yamldeployment/helm/charts/onyx/templates/indexing-model-deployment.yamldeployment/helm/charts/onyx/templates/inference-model-deployment.yamldeployment/helm/charts/onyx/templates/webserver-deployment.yamldeployment/helm/charts/onyx/values.yamldeployment/terraform/modules/aws/README.mddeployment/terraform/modules/aws/eks/main.tfdeployment/terraform/modules/aws/eks/outputs.tfdeployment/terraform/modules/aws/eks/variables.tfdeployment/terraform/modules/aws/onyx/main.tfdeployment/terraform/modules/aws/onyx/outputs.tfdeployment/terraform/modules/aws/onyx/variables.tfdeployment/terraform/modules/aws/onyx/versions.tfdeployment/terraform/modules/aws/opensearch/main.tfdeployment/terraform/modules/aws/opensearch/outputs.tfdeployment/terraform/modules/aws/opensearch/variables.tfdeployment/terraform/modules/aws/postgres/main.tfdeployment/terraform/modules/aws/postgres/outputs.tfdeployment/terraform/modules/aws/postgres/variables.tfdeployment/terraform/modules/aws/s3/main.tfdeployment/terraform/modules/aws/s3/variables.tfdeployment/terraform/modules/aws/vpc/main.tfdeployment/terraform/modules/aws/vpc/outputs.tfdeployment/terraform/modules/aws/waf/main.tfdeployment/terraform/modules/aws/waf/outputs.tfdeployment/terraform/modules/aws/waf/variables.tf
本文档记录了 Onyx 的 Kubernetes 部署架构。内容涵盖使用 Helm Chart 实现的系统部署,包括所有核心组件的 Deployment、StatefulSet、Service、ConfigMap 和持久化卷的配置。
有关 Docker Compose 部署配置,请参见 Docker Compose 部署。有关环境变量配置的详细信息,请参见 环境配置。有关 Kubernetes 环境中的数据库模式管理和迁移,请参见 数据库迁移。
部署架构总览
Onyx 的 Kubernetes 部署主要通过位于 deployment/helm/charts/onyx 的综合性 Helm Chart 进行管理 deployment/helm/charts/onyx/Chart.yaml:1-50。该架构采用微服务模式,将无状态应用逻辑与有状态数据存储分离。
核心组件映射
| Docker Compose 服务 | Kubernetes 实体 | Helm 模板/子 Chart | 镜像 |
|---|---|---|---|
api_server | Deployment | api-deployment.yaml | onyxdotapp/onyx-backend |
web_server | Deployment | webserver-deployment.yaml | onyxdotapp/onyx-web-server |
relational_db | 集群(CNPG) | postgresql 子 Chart | cloudnative-pg 操作器 |
index | StatefulSet | vespa 子 Chart | vespaengine/vespa |
cache | Deployment | redis 子 Chart | redis |
inference_model | Deployment | inference-model-deployment.yaml | onyxdotapp/onyx-model-server |
indexing_model | Deployment | indexing-model-deployment.yaml | onyxdotapp/onyx-model-server |
系统数据流
下图将高层系统名称与 Helm 模板中定义的特定 Kubernetes 服务和部署实体关联起来。
来源:deployment/helm/charts/onyx/Chart.yaml:1-50,deployment/helm/charts/onyx/values.yaml:1-239,deployment/helm/charts/onyx/templates/api-deployment.yaml:1-111
Helm Chart 配置
Onyx 的 Helm Chart 使用 values.yaml 文件来驱动所有子 Chart 和本地模板的配置。
全局设置
Chart 定义了镜像版本和主机配置的全局变量,以确保整个技术栈的一致性 deployment/helm/charts/onyx/values.yaml:5-11。
组件开关
主开关 vectorDB.enabled(默认值:true)控制索引基础设施的部署 deployment/helm/charts/onyx/values.yaml:36-37。当禁用时:
- 后端 Pod 上会设置
DISABLE_VECTOR_DB为true。 - 索引模型服务器和所有 Celery 工作器部署(primary、light、heavy、docfetching 等)会被跳过
deployment/helm/charts/onyx/values.yaml:28-35。
API 服务器初始化
API 服务器部署在启动过程中会自动处理数据库迁移,在启动 uvicorn 进程之前执行 alembic upgrade head deployment/helm/charts/onyx/templates/api-deployment.yaml:73-75。
来源:deployment/helm/charts/onyx/values.yaml:1-239,deployment/helm/charts/onyx/templates/api-deployment.yaml:1-111
后台工作器专业化
Kubernetes 部署将后台工作拆分为专门的工作器池,以实现独立的扩缩容和资源分配。
工作器池定义
每种工作器类型都在各自的模板文件中定义,并监听特定的 Celery 队列。
| 工作器部署 | 队列/角色 | 配置要点 |
|---|---|---|
light | vespa_metadata_sync,connector_deletion,doc_permissions_upsert,opensearch_migration | 处理元数据同步和清理任务 deployment/helm/charts/onyx/templates/celery-worker-light.yaml:70-72 |
heavy | connector_pruning,connector_doc_permissions_sync,csv_generation,sandbox | 处理资源密集型的修剪和权限同步 deployment/helm/charts/onyx/templates/celery-worker-heavy.yaml:64-72 |
primary | celery,periodic_tasks | 通用任务处理和 Celery Beat 任务消费 deployment/helm/charts/onyx/templates/celery-worker-primary.yaml:64-72 |
monitoring | monitoring | 专门用于系统健康监控任务的池 deployment/helm/charts/onyx/templates/celery-worker-monitoring.yaml:64-72 |
beat | 不适用 | 触发周期性任务的调度器 deployment/helm/charts/onyx/templates/celery-beat.yaml:62-67 |
健康探针
工作器使用专门的 Python 探针脚本 onyx/background/celery/celery_k8s_probe.py 进行就绪探针和存活探针检查,以确保 Celery 消费者实际可用 deployment/helm/charts/onyx/templates/celery-worker-light.yaml:100-112。
来源:deployment/helm/charts/onyx/templates/celery-worker-light.yaml:1-118,deployment/helm/charts/onyx/templates/celery-worker-heavy.yaml:1-118,deployment/helm/charts/onyx/templates/celery-worker-primary.yaml:1-118,deployment/helm/charts/onyx/templates/celery-worker-monitoring.yaml:1-118
AI 模型服务器
Onyx 区分了推理(实时聊天/搜索)和索引(批量文档处理)模型服务器。
推理模型服务器
- 用途: 为用户查询和聊天交互提供嵌入向量服务。
- 服务: 通过
inference-model-service在端口9000上暴露deployment/helm/charts/onyx/values.yaml:141-147。 - 资源: 默认请求 2 个 CPU 和 3Gi 内存
deployment/helm/charts/onyx/values.yaml:162-164。
索引模型服务器
- 用途: 批量处理文档以构建向量索引。
- 配置: 设置
INDEXING_ONLY="True"以优化批量吞吐量deployment/helm/charts/onyx/templates/indexing-model-deployment.yaml:68-69。 - 优化: 启动命令中包含
--limit-concurrency标志(默认值:10),以防止在大量索引期间发生内存溢出(OOM)deployment/helm/charts/onyx/templates/indexing-model-deployment.yaml:59。
来源:deployment/helm/charts/onyx/values.yaml:140-221,deployment/helm/charts/onyx/templates/indexing-model-deployment.yaml:1-95
数据层与持久化
PostgreSQL(CloudNativePG)
Onyx 使用 CloudNativePG 操作器来管理 PostgreSQL。
- 名称覆盖: 必须设置为
cloudnative-pg以便操作器发现deployment/helm/charts/onyx/values.yaml:15-18。 - 存储: 默认值为
10Gi,并启用enableSuperuserAccess: truedeployment/helm/charts/onyx/values.yaml:19-24。
Vespa
- 存储: 部署时使用
30Gi的 PersistentVolumeClaim 用于索引deployment/helm/charts/onyx/values.yaml:43-51。 - 安全: 以 root 身份运行并具有特权上下文,以允许 Vespa 的内部内存管理
deployment/helm/charts/onyx/values.yaml:63-65。
OpenSearch
- 初始化: 对于 2.12 及以上版本,需要设置
OPENSEARCH_INITIAL_ADMIN_PASSWORDdeployment/helm/charts/onyx/values.yaml:99-103。 - Java 选项: 堆大小通过
opensearchJavaOpts自动配置(默认值:4g)deployment/helm/charts/onyx/values.yaml:122-125。
来源:deployment/helm/charts/onyx/values.yaml:13-126,deployment/helm/charts/onyx/Chart.yaml:20-50
基础设施即代码(Terraform)
对于 AWS 部署,Onyx 提供了 Terraform 模块来配置底层的 EKS 集群和托管服务。
EKS 集群配置
eks 模块负责创建 Kubernetes 控制平面和节点组,包括用于 S3 和 RDS 访问的 IRSA(服务账户的 IAM 角色)deployment/terraform/modules/aws/onyx/main.tf:69-94。
托管数据存储
生产环境通常使用托管的 AWS 服务,而不是集群内部署:
- RDS PostgreSQL: 通过
postgres模块配置,包含自动备份和 CloudWatch 告警deployment/terraform/modules/aws/onyx/main.tf:46-60。 - Elasticache Redis: 配置了
auth_token以实现传输加密deployment/terraform/modules/aws/onyx/main.tf:33-44。 - OpenSearch 服务: 支持多可用区部署,包含备用节点和专用主节点
deployment/terraform/modules/aws/onyx/main.tf:112-145。
来源:deployment/terraform/modules/aws/onyx/main.tf:1-145,deployment/terraform/modules/aws/onyx/variables.tf:1-256,deployment/terraform/modules/aws/postgres/main.tf:1-60