Develop against the Vertex AI generation API, offline.
A tested subset of generateContent and streamGenerateContent through the official genai SDK, and Google's custom prediction contract on CloudBurrow's own runtime. See which model it runs, which options it honours and which requests it refuses.
Get startedCheck the supported path before enabling it.
Read the generation surface, the model’s provenance and the runtime before choosing a test.
- See which SDK paths were checked for generation.
- Read where the runnable model came from.
- Check the release and runtime limits before choosing a test.
The generation path
- Generation
- One text turn, CPU
- Embeddings
- Text, images, audio, video, PDF
- Prediction
- Local SDK endpoint
Go straight to the detail you need.
Generation
A subset of generateContent and streamGenerateContent (server-sent events), verified
with google.golang.org/genai v1.71.0 on both of the
SDK's backend paths: Vertex AI v1beta1 and v1beta.
Honoured
temperaturetopPtopKseedmaxOutputTokensstopSequencesresponseSchema
responseSchema with a JSON response is applied by constrained decoding, so the answer matches the schema.
The console playground offers the same options.
Refused, not ignored
- other generation options
- tools
- safety settings
- multimodal input
- multi-turn
- countTokens
Each returns an error naming the field.
No usageMetadata. No silent model substitution. Off by default; enable it with
--local-ai-model.
The exact surface, and why each field is refused, in docs/generation.md
The tested generation model is litert-community/gemma-4-E2B-it
(2.59 GB), a community conversion, labelled as one
everywhere it appears.
The console playground says so on screen:
This model is a COMMUNITY conversion, not published by Google. Its output is not a Gemini result and is not labelled as one.
Output quality is not claimed.
Runtime
LiteRT-LM is built from Google's source and runs on CPU. GPU and Metal are unsupported. Inference stays local after the runtime and model are on your machine. Prepare those artifacts before working offline.
The runtime image is published with each release, ghcr.io/cloudburrow/litert-lm:<tag>, and a
release CLI uses it by digest, so local generation no longer needs a checkout. v0.1.0 built it from a checkout
with make litert-lm.
Custom prediction
Google's serving contract (AIP_HTTP_PORT, the health and predict routes, instances in
and predictions out) on CloudBurrow's own runtime.
Not supported
- Model Registry
- Endpoint and DeployedModel management
- Model Garden
- tuning
- batch prediction
- Artifact fetching from AIP_STORAGE_URI
The startup deadline is 240 seconds. A failed start reports the container’s output.
Local PredictionService
With Cloud Run enabled, the local AI address serves PredictionService over gRPC and its JSON routes. The official client can call a custom prediction container using its Cloud Run service ID as the endpoint ID. The service also routes the supported publisher-model operations to the configured local generation or embedding model.
For a container with an h2c port, the raw gRPC methods relay bytes to its own gRPC server. The local
data path does not create a Vertex model registry, deployed model, or managed endpoint resource. v0.1.0 has no
PredictionService.
AI from BigQuery SQL
In v0.3.0, BigQuery remote models call the instance’s own local Vertex AI endpoint, never Google:
ML.GENERATE_TEXT and ML.GENERATE_EMBEDDING, and the AI functions
AI.GENERATE with output_schema, AI.GENERATE_BOOL, _INT,
_DOUBLE and _TABLE through constrained decoding, AI.GENERATE_EMBEDDING,
AI.EMBED, AI.SIMILARITY, AI.CLASSIFY, AI.IF and
AI.SCORE. Remote models honour max_output_tokens and the sampling options.
Over object tables, a CLOUD_AI_VISION_V1 remote model runs ML.ANNOTATE_IMAGE text
detection, read locally with Tesseract; other remote service types answer 501. ML.DECODE_IMAGE,
ML.RESIZE_IMAGE, ML.CONVERT_COLOR_SPACE and ML.CONVERT_IMAGE_TYPE feed
ML.PREDICT of an imported TensorFlow Lite or TensorFlow image model.
BigQuery: what is served · the BigQuery ML rows in docs/compatibility.md
Local embeddings
Build with make build-onnx, provide ONNX Runtime,
and enable --local-ai-embeddings. The default build has no ONNX Runtime. This path uses pinned
community conversions, with each downloaded model file checked against its SHA-256.
The opt-in path starts with embeddinggemma-300m-onnx-community for text. Download and load
embeddinggemma-2-onnx-community from Local models to embed, with EmbedContent,
text; PNG, JPEG, WebP (animated too), BMP, HEIC, HEIF and AVIF images; WAV and MP3 audio; H.264, H.265, VP9
and AV1 video, optionally with its audio track; and PDFs of up to six pages. Supported media may also be
read from the same instance’s Cloud Storage, with that service enabled.
With document_ocr, a scanned PDF page with no text layer is read with Tesseract OCR, compiled
into CloudBurrow, so its text joins the embedding. Nothing is sent to a hosted OCR or model service.
The official LiteRT embedding route remains blocked on the model export. Real-model results are from the dated developer-machine runs in the compatibility notes, the latest on 2026-11-17. Routine CI builds the optional runtime without rerunning those model measurements.
Build against Google Cloud APIs, locally
Free and open source under Apache-2.0. No account, no sign-up, no Google Cloud bill.