Add code interpreter into your LLM apps with llm-sandbox

LLM Sandbox is a lightweight, portable environment built to run code generated by Large Language Models (LLMs) in a secure, isolated setting using Docker containers. It is a great fit for machine learning infrastructure running on Kubernetes. With its intuitive interface, you can easily set up, manage, and execute code within a controlled environment, simplifying the process of testing and running LLM-generated scripts.

Key Features

  • Effortless Setup: Create sandbox environments quickly with minimal configuration.
  • Complete Isolation: Run code in isolated Docker containers to safeguard your host system.
  • Versatile: Supports multiple programming languages, including Python, Java, JavaScript, C++, Go, and Ruby.
  • Highly Portable: Use predefined Docker images or custom Dockerfiles for ultimate flexibility.
  • Scalable: Seamlessly integrate with Kubernetes and remote Docker hosts for larger deployments.

Using llm-sandbox with Cloudfleet

Set up CLoudfleet CLI

Use the Cloudfleet console to get instructions for the CLI installation and setup (Clusters -> Connect). If you don’t have a cluster yet, follow the getting started guide. Your commands will look similar to the following:

cloudfleet auth add-profile user default 12312312-abcd-1234-5678-90abcdef1234

cloudfleet clusters kubeconfig abcdefgh-1234-5678-90ab-cdef01234567

kubectl config set-context abcdefgh-1234-5678-90ab-cdef01234567/default

Install llm-sandbox 

pip install llm-sandbox

Create script

Create a new file named sandbox_demo.py and add the following code:

from llm_sandbox import SandboxSession

with SandboxSession(
    lang="python", keep_template=True, verbose=True, use_kubernetes=True
) as session:
    output = session.run("print('Hello, World!')")
    print(output.text)

    output = session.run(
        "import numpy as np\nprint(np.random.rand())", libraries=["numpy"]
    )
    print(output.text)

    session.execute_command("pip install pandas")
    output = session.run("import pandas as pd\nprint(pd.__version__)")
    print(output.text)

Execute the script

Run the script using the following command: python3 sandbox_demo.py. The output will be similar to the following:

Using local Kubernetes context since client is not provided..
Copying /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py..
Executing command: mkdir -p /tmp
Output: code.py

Copied /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py in 3.52 seconds
Executing command: python /tmp/code.py
Output: Hello, World!
Hello, World!

Executing command: pip install numpy
Output: Collecting numpy
  Downloading numpy-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.5/19.5 MB 39.4 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.0.2

[notice] A new release of pip is available: 23.0.1 -> 24.2
[notice] To update, run: pip install --upgrade pip
Copying /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py..
Executing command: mkdir -p /tmp
Output: code.py

Copied /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py in 3.36 seconds
Executing command: python /tmp/code.py
Output: 0.24481829174507796
0.24481829174507796

Executing command: pip install pandas
Output: Collecting pandas
  Downloading pandas-2.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.1/13.1 MB 65.0 MB/s eta 0:00:00
Collecting tzdata>=2022.7
  Downloading tzdata-2024.1-py2.py3-none-any.whl (345 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 345.4/345.4 kB 40.1 MB/s eta 0:00:00
Collecting python-dateutil>=2.8.2
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 kB 36.1 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.22.4 in /usr/local/lib/python3.9/site-packages (from pandas) (2.0.2)
Collecting pytz>=2020.1
  Downloading pytz-2024.2-py2.py3-none-any.whl (508 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 508.0/508.0 kB 47.7 MB/s eta 0:00:00
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: pytz, tzdata, six, python-dateutil, pandas
Successfully installed pandas-2.2.3 python-dateutil-2.9.0.post0 pytz-2024.2 six-1.16.0 tzdata-2024.1

[notice] A new release of pip is available: 23.0.1 -> 24.2
[notice] To update, run: pip install --upgrade pip
Copying /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py..
Executing command: mkdir -p /tmp
Output: code.py

Copied /tmp/code.py to sandbox-python-2afe03b697ed4432af7209edeea84b87:/tmp/code.py in 3.20 seconds
Executing command: python /tmp/code.py
Output: 2.2.3
2.2.3

Monitor the execution

Cloudfleet will automatically create a new Kubernetes pod for each script execution. For workloads that require GPU resources, Cloudfleet can provision GPU nodes on demand. You can monitor the pods, nodes, and node claims using kubectl. The output will be similar to the following:

$  kubectl get nodes -o wide
NAME                  STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
apt-bull-3752625927   Ready    <none>   1m    v1.29.3   100.71.182.46   52.24.15.49   Ubuntu 22.04.4 LTS   6.5.0-1024-aws   containerd://1.7.19

$  kubectl get nodeclaims -o wide
NAME            TYPE                         CAPACITY   ZONE   NODE                  READY   AGE   ID                                           NODEPOOL   NODECLASS
managed-x79s7   AWS-USW2-AZ2-T3-MICRO-SPOT   spot       west   apt-bull-3752625927   True    1m    AWS://managed/usw2-az2/apt-bull-3752625927   managed    managed

$  kubectl get pods -o wide
NAME                                              READY   STATUS        RESTARTS   AGE   IP             NODE                  NOMINATED NODE   READINESS GATES
sandbox-python-2afe03b697ed4432af7209edeea84b87   1/1     Running       0          59s   10.244.1.56    apt-bull-3752625927   <none>           <none>
On this page