Skip to main content

ChatVertexAI

LangChain.js supports Google Vertex AI chat models as an integration. It supports two different methods of authentication based on whether you're running in a Node environment or a web environment.

Setup​

Node​

To call Vertex AI models in Node, you'll need to install the @langchain/google-vertexai package:

tip

See this section for general instructions on installing integration packages.

npm install @langchain/google-vertexai
tip

We're unifying model params across all packages. We now suggest using model instead of modelName, and apiKey for API keys.

You should make sure the Vertex AI API is enabled for the relevant project and that you've authenticated to Google Cloud using one of these methods:

  • You are logged into an account (using gcloud auth application-default login) permitted to that project.
  • You are running on a machine using a service account that is permitted to the project.
  • You have downloaded the credentials for a service account that is permitted to the project and set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of this file.
tip

See this section for general instructions on installing integration packages.

npm install @langchain/google-vertexai

Web​

To call Vertex AI models in web environments (like Edge functions), you'll need to install the @langchain/google-vertexai-web package:

npm install @langchain/google-vertexai-web

Then, you'll need to add your service account credentials directly as a GOOGLE_VERTEX_AI_WEB_CREDENTIALS environment variable:

GOOGLE_VERTEX_AI_WEB_CREDENTIALS={"type":"service_account","project_id":"YOUR_PROJECT-12345",...}

Lastly, you may also pass your credentials directly in code like this:

import { ChatVertexAI } from "@langchain/google-vertexai-web";

const model = new ChatVertexAI({
  authOptions: {
    credentials: {"type":"service_account","project_id":"YOUR_PROJECT-12345",...},
  },
});

Usage​

The entire family of gemini models are available by specifying the modelName parameter.

For example:

import { ChatVertexAI } from "@langchain/google-vertexai";
// Or, if using the web entrypoint:
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const model = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.5-flash-001",
});

const response = await model.invoke("Why is the ocean blue?");
console.log(response);
/*
AIMessageChunk {
  "content": "The ocean appears blue due to a combination of factors:\n\n**1. Rayleigh Scattering:**\n\n* This is the primary reason.  Sunlight is made up of all colors of the rainbow.  When sunlight enters the ocean, blue wavelengths are scattered more than other colors by water molecules. \n* This scattered blue light is what we see.  Think of it like sunlight being scattered by the atmosphere, making the sky appear blue.\n\n**2. Absorption of Other Colors:**\n\n* Water absorbs red, orange, yellow, and green wavelengths of light more readily than blue.  This means less of those colors reach our eyes.\n* The deeper the water, the more red light is absorbed, making the ocean appear even bluer.\n\n**3. Other Factors:**\n\n* **Depth:**  The deeper the water, the bluer it appears.\n* **Turbidity:**  The presence of particles like sediment or plankton can affect the color.  A cloudy ocean might appear more greenish or brown.\n* **Time of Day:**  The ocean can appear different colors depending on the angle of the sun.\n\n**In Summary:**\n\nThe ocean appears blue primarily due to Rayleigh scattering, where blue wavelengths of light are scattered more effectively by water molecules. This, combined with the absorption of other colors by water, results in the blue hue we perceive.\n",
  "usage_metadata": {
    "input_tokens": 6,
    "output_tokens": 276,
    "total_tokens": 282
  }
}
 */

API Reference:

tip

See the LangSmith trace for the example above here.

Multimodal​

The Gemini API can process multimodal inputs. The example below demonstrates how to do this:

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatVertexAI } from "@langchain/google-vertexai";
import fs from "node:fs";

const model = new ChatVertexAI({
  model: "gemini-pro-vision",
  temperature: 0.7,
});

const image = fs.readFileSync("./hotdog.jpg").toString("base64");
const prompt = ChatPromptTemplate.fromMessages([
  [
    "human",
    [
      {
        type: "text",
        text: "Describe the following image.",
      },
      {
        type: "image_url",
        image_url: "data:image/png;base64,{image_base64}",
      },
    ],
  ],
]);

const response = await prompt.pipe(model).invoke({
  image_base64: image,
});

console.log(response.content);
/*
This is an image of a hot dog. The hot dog is on a white background. The hot dog is a grilled sausage in a bun.
*/

API Reference:

tip

See the LangSmith trace for the example above here.

Streaming​

ChatVertexAI also supports streaming in multiple chunks for faster responses:

import { ChatVertexAI } from "@langchain/google-vertexai";
// Or, if using the web entrypoint:
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const model = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.5-flash-001",
});
const stream = await model.stream([
  ["system", "You are a funny assistant that answers in pirate language."],
  ["human", "What is your favorite food?"],
]);

for await (const chunk of stream) {
  console.log(chunk.content);
}

/*
A
hoy, matey! Me favorite food be a hearty plate o' grub,
 with a side o' scurvy dogs and a tankard o' grog
. Argh!


*/

API Reference:

tip

See the LangSmith trace for the example above here.

Tool calling​

ChatVertexAI also supports calling the model with a tool:

import { ChatVertexAI } from "@langchain/google-vertexai";
import { type GeminiTool } from "@langchain/google-vertexai/types";
import { zodToGeminiParameters } from "@langchain/google-vertexai/utils";
import { z } from "zod";
// Or, if using the web entrypoint:
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const calculatorSchema = z.object({
  operation: z
    .enum(["add", "subtract", "multiply", "divide"])
    .describe("The type of operation to execute"),
  number1: z.number().describe("The first number to operate on."),
  number2: z.number().describe("The second number to operate on."),
});

const geminiCalculatorTool: GeminiTool = {
  functionDeclarations: [
    {
      name: "calculator",
      description: "A simple calculator tool",
      parameters: zodToGeminiParameters(calculatorSchema),
    },
  ],
};

const model = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.5-flash-001",
}).bind({
  tools: [geminiCalculatorTool],
});

const response = await model.invoke("What is 1628253239 times 81623836?");
console.log(JSON.stringify(response.additional_kwargs, null, 2));
/*
{
  "tool_calls": [
    {
      "id": "a20075d3b0e34f7ca60cc135916e620d",
      "type": "function",
      "function": {
        "name": "calculator",
        "arguments": "{\"number1\":1628253239,\"operation\":\"multiply\",\"number2\":81623836}"
      }
    }
  ]
}
 */

API Reference:

tip

See the LangSmith trace for the example above here.

withStructuredOutput​

Alternatively, you can also use the withStructuredOutput method:

import { ChatVertexAI } from "@langchain/google-vertexai";
import { z } from "zod";
// Or, if using the web entrypoint:
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const calculatorSchema = z.object({
  operation: z
    .enum(["add", "subtract", "multiply", "divide"])
    .describe("The type of operation to execute"),
  number1: z.number().describe("The first number to operate on."),
  number2: z.number().describe("The second number to operate on."),
});

const model = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.5-flash-001",
}).withStructuredOutput(calculatorSchema);

const response = await model.invoke("What is 1628253239 times 81623836?");
console.log(response);
/*
{ operation: 'multiply', number1: 1628253239, number2: 81623836 }
 */

API Reference:

tip

See the LangSmith trace for the example above here.

VertexAI tools agent​

The Gemini family of models not only support tool calling, but can also be used in the Tool Calling agent. Here's an example:

import { z } from "zod";

import { tool } from "@langchain/core/tools";
import { AgentExecutor, createToolCallingAgent } from "langchain/agents";

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatVertexAI } from "@langchain/google-vertexai";
// Uncomment this if you're running inside a web/edge environment.
// import { ChatVertexAI } from "@langchain/google-vertexai-web";

const llm: any = new ChatVertexAI({
  temperature: 0,
});

// Prompt template must have "input" and "agent_scratchpad input variables"
const prompt = ChatPromptTemplate.fromMessages([
  ["system", "You are a helpful assistant"],
  ["placeholder", "{chat_history}"],
  ["human", "{input}"],
  ["placeholder", "{agent_scratchpad}"],
]);

// Mocked tool
const currentWeatherTool = tool(async () => "28 °C", {
  name: "get_current_weather",
  description: "Get the current weather in a given location",
  schema: z.object({
    location: z.string().describe("The city and state, e.g. San Francisco, CA"),
  }),
});

const agent = await createToolCallingAgent({
  llm,
  tools: [currentWeatherTool],
  prompt,
});

const agentExecutor = new AgentExecutor({
  agent,
  tools: [currentWeatherTool],
});

const input = "What's the weather like in Paris?";
const { output } = await agentExecutor.invoke({ input });

console.log(output);

/* 
It's 28 degrees Celsius in Paris.
*/

API Reference:

tip

See the LangSmith trace for the agent example above here.

Was this page helpful?


You can also leave detailed feedback on GitHub.