Deploy LangChain Application#

Oracle ADS SDK now supports the deployment of LangChain application to OCI data science model deployment and you can easily do so just by writing a couple lines of code.

New in version 2.9.1.

Configuration#

Ensure that you have created the necessary policies, authentication, and authorization for model deployments. Here we’re using the resource_principal as auth type and you can configure the policy as below.

allow dynamic-group <dynamic-group-name> to manage data-science-model-deployments in compartment <compartment-name>

Create LangChain Application#

Create a simple LangChain application that links prompt and cohere model as below. Remember to replace the <cohere_api_key> with the actual cohere api key.

import os
from langchain.llms import Cohere
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
os.environ["COHERE_API_KEY"] = "<cohere_api_key>"

cohere = Cohere()
prompt = PromptTemplate.from_template("Tell me a joke about {subject}")
llm_chain = LLMChain(prompt=prompt, llm=cohere, verbose=True)

Now you have a LangChain object llm_chain. Try running it with the prompt {"subject": "animals"} and it should give you the corresponding result.

llm_chain.run({"subject": "animals"})

Initialize the ChainDeployment#

Initialize class ChainDeployment from ADS SDK and pass the LangChain object llm_chain from previous step as parameter. The artifact_dir is an optional parameter which points to the folder where the model artifacts will be put locally. In this example, we’re using a temporary folder generated by tempfile.

import tempfile
from ads.llm.deploy import ChainDeployment

artifact_dir = tempfile.mkdtemp()

chain_deployment = ChainDeployment(
    chain=llm_chain,
    artifact_dir=artifact_dir
)

Prepare the Model Artifacts#

Call prepare from ChainDeployment to generate the score.py and serialize the LangChain application to chain.yaml file under artifact_dir folder. Parameters inference_conda_env and inference_python_version are passed to define the conda environment where your LangChain application will be running on OCI cloud. Here, replace custom_conda_environment_uri with your conda environment uri that has the latest ADS 2.9.1 and replace python_version with your conda environment python version.

Note

For how to customize and publish conda environment, take reference to Publishing a Conda Environment to an Object Storage Bucket

chain_deployment.prepare(
    inference_conda_env="<custom_conda_environment_uri>",
    inference_python_version="<python_version>",
)

Below is the chain.yaml file that was saved from llm_chain object. For more information regarding LLMs model serialization, see here.

_type: llm_chain
llm:
  _type: cohere
  frequency_penalty: 0.0
  k: 0
  max_tokens: 256
  model: null
  p: 1
  presence_penalty: 0.0
  temperature: 0.75
  truncate: null
llm_kwargs: {}
memory: null
metadata: null
output_key: text
output_parser:
  _type: default
prompt:
  _type: prompt
  input_types: {}
  input_variables:
  - subject
  output_parser: null
  partial_variables: {}
  template: Tell me a joke about {subject}
  template_format: f-string
  validate_template: false
return_final_only: true
tags: null
verbose: true

Save Artifacts to OCI Model Catalog#

Call save to pack and upload the artifacts under artifact_dir to OCI data science model catalog. Once the artifacts are successfully uploaded, you should be able to see the id of the model.

chain_deployment.save(display_name="LangChain Model")

Deploy the Model#

Deploy the LangChain model from previous step by calling deploy. Remember to replace the <cohere_api_key> with the actual cohere api key in the environment_variables. It usually takes a couple of minutes to deploy the model and you should see the model deployment in the output once the process completes.

chain_deployment.deploy(
    display_name="LangChain Model Deployment",
    environment_variables={"COHERE_API_KEY":"<cohere_api_key>"},
)

Invoke the Deployed Model#

Now the OCI data science model deployment endpoint is ready and you can invoke it to tell a joke about animals.

chain_deployment.predict(data={"subject": "animals"})["output"]
../../_images/prediction.png

Alternatively, you can use OCI CLI to invoke the model deployment. Remember to replace the langchain_application_model_deployment_url with the actual model deployment url which you can find in the output from deploy step.

oci raw-request --http-method POST --target-uri <langchain_application_model_deployment_url>/predict --request-body '{"subject": "animals"}' --auth resource_principal
../../_images/cli_prediction.png