LangChain Integration

Added in version 2.9.1.

LangChain compatible models/interfaces are needed for LangChain applications to invoke OCI generative AI service or LLMs deployed on OCI data science model deployment service.

Preview Feature

While the official integration of OCI and LangChain will be added to the LangChain library, ADS provides a preview version of the integration. It it important to note that the APIs of the preview version may change in the future.

Integration with Generative AI

The OCI Generative AI service provide text generation, summarization and embedding models.

To use the text generation model as LLM in LangChain:

from ads.llm import GenerativeAI

llm = GenerativeAI(
    compartment_id="<compartment_ocid>",
    # Optionally you can specify keyword arguments for the OCI client, e.g. service_endpoint.
    client_kwargs={
        "service_endpoint": "https://inference.generativeai.us-chicago-1.oci.oraclecloud.com"
    },
)

llm.invoke("Translate the following sentence into French:\nHow are you?\n")

Here is an example of using prompt template and OCI generative AI LLM to build a translation app:

from langchain.prompts import PromptTemplate
from langchain.schema.runnable import RunnableParallel, RunnablePassthrough
from ads.llm import GenerativeAI

# Map the input into a dictionary
map_input = RunnableParallel(text=RunnablePassthrough())
# Template for the input text.
template = PromptTemplate.from_template(
    "Translate English into French. Do not ask any questions.\nEnglish: Hello!\nFrench: "
)
llm = GenerativeAI(
    compartment_id="<compartment_ocid>",
    # Optionally you can specify keyword arguments for the OCI client, e.g. service_endpoint.
    client_kwargs={
        "service_endpoint": "https://inference.generativeai.us-chicago-1.oci.oraclecloud.com"
    },
)

# Build the app as a chain
translation_app = map_input | template | llm

# Now you have a translation app.
translation_app.invoke("Hello!")
# "Bonjour!"

Similarly, you can use the embedding model:

from ads.llm import GenerativeAIEmbeddings

embed = GenerativeAIEmbeddings(
    compartment_id="<compartment_ocid>",
    # Optionally you can specify keyword arguments for the OCI client, e.g. service_endpoint.
    client_kwargs={
        "service_endpoint": "https://inference.generativeai.us-chicago-1.oci.oraclecloud.com"
    },
)

embed.embed_query("How are you?")

Integration with Model Deployment

Available in LangChain

The same OCIModelDeploymentVLLM and ModelDeploymentTGI classes are also available from LangChain.

If you deploy open-source or your own LLM on OCI model deployment service using vLLM or HuggingFace TGI , you can use the ModelDeploymentVLLM or ModelDeploymentTGI to integrate your model with LangChain.

from ads.llm import ModelDeploymentVLLM

llm = ModelDeploymentVLLM(
    endpoint="https://<your_model_deployment_endpoint>/predict",
    model="<model_name>"
)

from ads.llm import ModelDeploymentTGI

llm = ModelDeploymentTGI(
    endpoint="https://<your_model_deployment_endpoint>/predict",
)

Authentication

By default, the integration uses the same authentication method configured with ads.set_auth(). Optionally, you can also pass the auth keyword argument when initializing the model to use specific authentication method for the model. For example, to use resource principal for all OCI authentication:

import ads
from ads.llm import GenerativeAI

ads.set_auth(auth="resource_principal")

llm = GenerativeAI(
    compartment_id="<compartment_ocid>",
    # Optionally you can specify keyword arguments for the OCI client, e.g. service_endpoint.
    client_kwargs={
        "service_endpoint": "https://inference.generativeai.us-chicago-1.oci.oraclecloud.com"
    },
)

Alternatively, you may use specific authentication for the model:

import ads
from ads.llm import GenerativeAI

llm = GenerativeAI(
    # Use security token authentication for the model
    auth=ads.auth.security_token(profile="my_profile"),
    compartment_id="<compartment_ocid>",
    # Optionally you can specify keyword arguments for the OCI client, e.g. service_endpoint.
    client_kwargs={
        "service_endpoint": "https://inference.generativeai.us-chicago-1.oci.oraclecloud.com"
    },
)