Local AI

Develop against the Vertex AI generation API, offline.

A tested subset of generateContent and streamGenerateContent through the official genai SDK, and Google's custom prediction contract on CloudBurrow's own runtime. See which model it runs, which options it honours and which requests it refuses.

Get started
Local AI

Check the supported path before enabling it.

Read the generation surface, the model’s provenance and the runtime before choosing a test.

  • See which SDK paths were checked for generation.
  • Read where the runnable model came from.
  • Check the release and runtime limits before choosing a test.
CloudBurrow Local AI
At a glance

The generation path

Generation
One text turn, CPU
Embeddings
Text, images, audio, video, PDF
Prediction
Local SDK endpoint
Inside local ai

Go straight to the detail you need.

Generation

A subset of generateContent and streamGenerateContent (server-sent events), verified with google.golang.org/genai v1.71.0 on both of the SDK's backend paths: Vertex AI v1beta1 and v1beta.

Honoured

  • temperature
  • topP
  • topK
  • seed
  • maxOutputTokens
  • stopSequences
  • responseSchema

responseSchema with a JSON response is applied by constrained decoding, so the answer matches the schema. The console playground offers the same options.

Refused, not ignored

  • other generation options
  • tools
  • safety settings
  • multimodal input
  • multi-turn
  • countTokens

Each returns an error naming the field.

No usageMetadata. No silent model substitution. Off by default; enable it with --local-ai-model.

The exact surface, and why each field is refused, in docs/generation.md

Model evidence

Model provenance

Provenance and the audit behind it, in docs/local-ai.md

The tested generation model is litert-community/gemma-4-E2B-it (2.59 GB), a community conversion, labelled as one everywhere it appears.

The console playground says so on screen:

This model is a COMMUNITY conversion, not published by Google. Its output is not a Gemini result and is not labelled as one.

Output quality is not claimed.

Runtime

LiteRT-LM is built from Google's source and runs on CPU. GPU and Metal are unsupported. Inference stays local after the runtime and model are on your machine. Prepare those artifacts before working offline.

The runtime image is published with each release, ghcr.io/cloudburrow/litert-lm:<tag>, and a release CLI uses it by digest, so local generation no longer needs a checkout. v0.1.0 built it from a checkout with make litert-lm.

Custom prediction

Google's serving contract (AIP_HTTP_PORT, the health and predict routes, instances in and predictions out) on CloudBurrow's own runtime.

Not supported

  • Model Registry
  • Endpoint and DeployedModel management
  • Model Garden
  • tuning
  • batch prediction
  • Artifact fetching from AIP_STORAGE_URI

The startup deadline is 240 seconds. A failed start reports the container’s output.

Custom prediction containers, in docs/prediction.md

Local PredictionService

With Cloud Run enabled, the local AI address serves PredictionService over gRPC and its JSON routes. The official client can call a custom prediction container using its Cloud Run service ID as the endpoint ID. The service also routes the supported publisher-model operations to the configured local generation or embedding model.

For a container with an h2c port, the raw gRPC methods relay bytes to its own gRPC server. The local data path does not create a Vertex model registry, deployed model, or managed endpoint resource. v0.1.0 has no PredictionService.

See the tested methods and their limits.

AI from BigQuery SQL

In v0.3.0, BigQuery remote models call the instance’s own local Vertex AI endpoint, never Google: ML.GENERATE_TEXT and ML.GENERATE_EMBEDDING, and the AI functions AI.GENERATE with output_schema, AI.GENERATE_BOOL, _INT, _DOUBLE and _TABLE through constrained decoding, AI.GENERATE_EMBEDDING, AI.EMBED, AI.SIMILARITY, AI.CLASSIFY, AI.IF and AI.SCORE. Remote models honour max_output_tokens and the sampling options.

Over object tables, a CLOUD_AI_VISION_V1 remote model runs ML.ANNOTATE_IMAGE text detection, read locally with Tesseract; other remote service types answer 501. ML.DECODE_IMAGE, ML.RESIZE_IMAGE, ML.CONVERT_COLOR_SPACE and ML.CONVERT_IMAGE_TYPE feed ML.PREDICT of an imported TensorFlow Lite or TensorFlow image model.

BigQuery: what is served · the BigQuery ML rows in docs/compatibility.md

Local embeddings

Build with make build-onnx, provide ONNX Runtime, and enable --local-ai-embeddings. The default build has no ONNX Runtime. This path uses pinned community conversions, with each downloaded model file checked against its SHA-256.

The opt-in path starts with embeddinggemma-300m-onnx-community for text. Download and load embeddinggemma-2-onnx-community from Local models to embed, with EmbedContent, text; PNG, JPEG, WebP (animated too), BMP, HEIC, HEIF and AVIF images; WAV and MP3 audio; H.264, H.265, VP9 and AV1 video, optionally with its audio track; and PDFs of up to six pages. Supported media may also be read from the same instance’s Cloud Storage, with that service enabled.

With document_ocr, a scanned PDF page with no text layer is read with Tesseract OCR, compiled into CloudBurrow, so its text joins the embedding. Nothing is sent to a hosted OCR or model service.

The official LiteRT embedding route remains blocked on the model export. Real-model results are from the dated developer-machine runs in the compatibility notes, the latest on 2026-11-17. Routine CI builds the optional runtime without rerunning those model measurements.

Read the model, build and media requirements.

Develop locally

Build against Google Cloud APIs, locally

Free and open source under Apache-2.0. No account, no sign-up, no Google Cloud bill.

Get started