Tech Blog
Go back

LiteLLM with Huawei Cloud MaaS: FinOps, ECS Stability, and Cache Strategy

Edit page

Exploring LiteLLM with Huawei Cloud MaaS: FinOps, ECS Stability, and Cache Strategy

How to Use the LiteLLM Huawei MaaS Single ECS Skill

This article is backed by the reusable skill package:

https://github.com/binrogithub/1-3-Cloud-Adoption-Skills/tree/docs/general-skill-framework/AI/AI-Development/LiteLLM-Huawei-MaaS-Single-ECS

Use this skill when you want an AI coding agent to help deploy, configure, validate, or repair a single-ECS LiteLLM gateway in front of Huawei Cloud MaaS.

Typical prompts:

Recommended skill files:

The recommended workflow is:

  1. Review SKILL.md to understand the operating model.
  2. Copy and adapt the config templates for the target ECS host.
  3. Put the Huawei MaaS API key only in the LiteLLM environment file, not in client tools.
  4. Expose explicit LiteLLM model names such as huawei-glm-5.1.
  5. Configure non-zero token costs before testing budgets.
  6. Create LiteLLM virtual keys for users, services, Cline, or VS Code extensions.
  7. Validate direct MaaS access, LiteLLM proxy access, budget enforcement, and optional cache behavior.

Executive Summary

Huawei Cloud MaaS exposes large language models through an OpenAI-compatible API, which makes it straightforward to connect tools such as Cline, VS Code extensions, internal applications, and AI agents. The missing enterprise layer is not basic API compatibility. The harder problems are financial governance, key isolation, stable connectivity, and predictable behavior under repeated workloads.

LiteLLM can fill this gap by acting as a gateway in front of Huawei Cloud MaaS. In this pattern, LiteLLM runs on a Huawei Cloud ECS instance, stores proxy metadata in PostgreSQL, uses Redis for transient state and optional response caching, and exposes controlled virtual keys to downstream users. The Huawei MaaS API key stays on the server. Applications receive LiteLLM virtual keys instead.

This article summarizes an implementation-oriented exploration of:

Reference Architecture

The recommended single-ECS layout is intentionally simple:

Client / Cline / Internal App
        |
        | OpenAI-compatible request
        v
LiteLLM Proxy on ECS :4000
        |
        | OpenAI-compatible upstream request
        v
Huawei Cloud MaaS

Local ECS services:
- PostgreSQL: LiteLLM keys, teams, budgets, spend logs
- Redis: router state, auth-key cache, optional response cache
- systemd: process supervision for LiteLLM, Redis, PostgreSQL

Control-plane separation:

Data-plane separation:

This design is useful when the organization wants one controlled gateway rather than many developers, IDEs, agents, and services calling MaaS directly.

Huawei MaaS Model Mapping

Huawei MaaS can be exposed to LiteLLM as an OpenAI-compatible provider. A practical model mapping for glm-5.1 looks like this:

model_list:
  - model_name: "huawei-glm-5.1"
    litellm_params:
      model: "openai/glm-5.1"
      api_base: os.environ/HUAWEI_MAAS_API_BASE
      api_key: os.environ/HUAWEI_MAAS_API_KEY
      timeout: 120
      input_cost_per_token: 1.078e-06
      output_cost_per_token: 3.774e-06

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true

router_settings:
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD
  enable_pre_call_checks: true

For the Southeast Asia MaaS endpoint, the base URL format is:

https://api-ap-southeast-1.modelarts-maas.com/openai/v1

The important implementation detail is that the public LiteLLM model name does not need to equal the upstream MaaS model ID. For example, downstream clients can call huawei-glm-5.1, while LiteLLM forwards to upstream glm-5.1.

FinOps Problem

Direct MaaS usage is easy to start but difficult to govern. Common problems include:

LiteLLM addresses these problems by turning MaaS access into a managed gateway model.

FinOps Controls with LiteLLM

LiteLLM supports virtual keys with model allow-lists, budgets, budget durations, rate limits, and metadata. A typical virtual key can be created with:

curl -X POST "$LITELLM_BASE/key/generate" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key_alias": "huawei-glm-5.1-monthly",
    "models": ["huawei-glm-5.1"],
    "max_budget": 0.1,
    "budget_duration": "1mo"
  }'

Budget updates use the same control plane:

curl -X POST "$LITELLM_BASE/key/update" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "<virtual_key>",
    "max_budget": 0.00001,
    "budget_duration": "1mo"
  }'

Current spend can be checked with:

curl "$LITELLM_BASE/key/info?key=<virtual_key>" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

When a budget is exceeded, LiteLLM blocks the request before sending it upstream. The expected error type is:

{
  "error": {
    "type": "budget_exceeded",
    "code": "400"
  }
}

This behavior is useful for testing. A very small budget, such as 0.00001 USD, can verify that budget enforcement is active.

Unit Cost Configuration

Budget enforcement depends on correct token pricing. If LiteLLM sees a model cost of zero, successful calls may not consume budget, and max_budget will not behave as expected.

For conservative glm-5.1 accounting, use the highest unit prices from the observed range:

Input:  $1.078 / 1M tokens -> 1.078e-06 USD/token
Output: $3.774 / 1M tokens -> 3.774e-06 USD/token

These values can be configured directly in litellm_params, which is usually the clearest option for OpenAI-compatible MaaS deployments:

input_cost_per_token: 1.078e-06
output_cost_per_token: 3.774e-06

Alternatively, they can be configured in model_prices_and_context_window.json, but the key must match the provider path that LiteLLM uses. If the live LiteLLM config uses model: openai/glm-5.1, setting only custom_openai/glm-5.1 may not affect the running model group. In operational deployments, putting costs directly on each MaaS model group is easier to validate.

Always verify with:

curl "$LITELLM_BASE/model/info" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

The model should show non-zero input and output token costs.

External VS Code and Cline Usage

For external VS Code plus Cline, the client should point to LiteLLM, not MaaS directly:

Provider: OpenAI Compatible
Base URL: http://<litellm-host>:4000/v1
API Key: <LiteLLM virtual key>
Model: huawei-glm-5.1

This preserves central governance:

For first validation, use a deterministic prompt such as:

Reply with exactly: ok

Only enable tool-heavy or agentic Cline workflows after a simple /v1/chat/completions request succeeds.

ECS as a Stability Layer

Running LiteLLM on ECS is not only a governance choice. It also improves operational stability.

Without a gateway, every client environment must solve the same problems:

With an ECS-hosted LiteLLM gateway, clients only need to reach one internal endpoint. The ECS instance becomes the stable egress point to Huawei MaaS.

Practical stability benefits:

Recommended ECS service responsibilities:

This architecture is still a single-host design. It improves manageability and connection consistency, but it is not high availability. For HA, move Redis and PostgreSQL off-box first, then run multiple LiteLLM instances behind a load balancer.

Cache Exploration

Huawei MaaS does not currently provide the same provider-side prompt cache behavior that OpenAI or Claude expose for some models. The natural question is whether LiteLLM can compensate.

The answer is: LiteLLM can compensate at the response-cache layer, but it cannot reproduce provider-side prompt/KV cache.

There are three separate concepts:

1. LiteLLM Response Cache

LiteLLM response cache stores the full model response. If the next request has the same cache key, LiteLLM returns the stored response directly and does not call Huawei MaaS.

This can reduce:

A Redis-backed response cache can be enabled with:

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD
    ttl: 3600

Cache health can be checked with:

curl "$LITELLM_BASE/cache/ping" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

If response cache is not initialized, LiteLLM returns an error similar to:

Cache not initialized. litellm.cache is None

Redis configuration alone is not enough. router_settings.redis_host can support router or proxy state, but response caching requires litellm_settings.cache: true.

2. LiteLLM Semantic Cache

LiteLLM can also use semantic cache backends such as Redis semantic cache or Qdrant semantic cache. This allows similar prompts to hit the same cached answer based on embedding similarity.

This is useful for:

It is risky for:

Semantic cache is not equivalent to prompt cache. It reuses an old answer when the prompt is similar enough. That can be useful, but it can also be wrong.

3. Provider-Side Prompt Cache

OpenAI and Claude prompt cache is different. It does not return a previous answer. Instead, the model provider reuses internal computation for a repeated prompt prefix and still generates a fresh response.

That matters because it allows this pattern:

Fixed long system prompt + fixed tools + fixed context + new user question

The prefix can be cached by the provider, while the answer is still newly generated. LiteLLM cannot implement this if the upstream model service does not expose or support internal KV/prompt cache. The KV state lives inside the model serving layer, not in LiteLLM or Redis.

Therefore, LiteLLM can reduce repeated full-response calls, but it cannot make Huawei MaaS behave exactly like OpenAI or Claude prompt cache unless Huawei MaaS supports that feature upstream.

Use different cache strategies for different workloads:

Exact repeated prompts:
  Use LiteLLM Redis response cache.

FAQ-style similar questions:
  Consider semantic cache with strict similarity thresholds and monitoring.

Agentic coding tools such as Cline:
  Do not expect high response-cache hit rates.
  Stabilize prompts and reduce repeated context instead.

Long fixed system prompts:
  Keep the prefix stable.
  This helps if the provider later supports prompt cache, but LiteLLM alone cannot reuse KV state.

For Cline and coding agents, the better optimization is usually not response caching. It is prompt hygiene:

Validation Checklist

Before calling the deployment ready, validate:

  1. Direct Huawei MaaS request succeeds.
  2. LiteLLM /health succeeds with the master key.
  3. LiteLLM /v1/chat/completions succeeds with the public model name.
  4. /model/info shows non-zero input and output costs.
  5. A virtual key can be generated with a model allow-list.
  6. A low-budget key is blocked with budget_exceeded.
  7. Cline can call LiteLLM with an OpenAI-compatible provider configuration.
  8. /cache/ping confirms whether response cache is enabled.
  9. Redis and PostgreSQL are bound to localhost unless external access is explicitly required.
  10. The Huawei MaaS API key is not present in client configurations.

Lessons Learned

LiteLLM is most valuable here as a governance and stability layer. It turns Huawei MaaS from a raw model endpoint into a managed internal AI gateway.

For FinOps, the critical detail is unit-cost configuration. Budgets are only meaningful when LiteLLM can calculate spend. For glm-5.1, putting conservative token prices directly into litellm_params makes the behavior explicit and easy to verify.

For connection stability, ECS provides a controlled network and runtime boundary. Instead of troubleshooting MaaS connectivity from many developer machines and tools, operators troubleshoot one gateway.

For cache, LiteLLM can reduce repeated full-response calls, but it cannot replace true provider-side prompt cache. This distinction matters. Response cache returns an old answer; provider prompt cache accelerates and discounts input processing while still generating a new answer.

The practical recommendation is to deploy LiteLLM on ECS for MaaS access control and FinOps first, enable Redis response cache only for workloads where stale answer reuse is acceptable, and treat semantic cache as a specialized optimization rather than a default behavior.

Edit page
Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email
Previous Post
Building an Enterprise-Grade AI Coding Platform on Huawei Cloud MaaS
Next Post
Building a LangGraph-Based CSS Elasticity AIOps Agent