OpenAI Adapter

If you're building an app that will make LLM requests from the backend, then we recommend using OAICA, our OpenAI-compatible Adapter! This tool will help you set up your Spark wallet, connect to the Inference Grid, and provide an API endpoint that's compatible with OpenAI's SDKs.

git clone https://github.com/buildonspark/inference-grid-examples
cd inference-grid-examples/adapter
npm install
npm run dev

Initial Setup

First, you'll need to create a configuration file. This file will contain your Inference Grid credentials as well as your Spark wallet mnemonic. You can create this file by running:

npm run dev init

This will save your credentials to oaica.json. Don't lose it!

{
    "network": "MAINNET",
    "port": 3031,
    "publicKey": "e407b0aec1ebb57dea8e35dc0d4ed954e4d8f3192d82b53c4c42b660af6728a3",
    "privateKey": "5edb56ce65ae267ceda9e8162dbd2dcfd4a20faa7c2d64c2744df61173afeb10",
    "spark": {
        "mnemonic": "typical stereo dose party penalty decline neglect feel harvest abstract stage winter"
    }
}

The privateKey is used to identify your application to the Inference Grid. The mnemonic is used to initialize a Spark wallet for paying invoices.

Spark Wallet

If you created a new Spark Wallet (or if you run out of funds in an existing one), you can top it up by running:

npm run dev wallet

Which will show your current balance and help you generate a Lightning invoice. Ask a member of the event staff to pay your invoice to top up your wallet!

LLM Usage

Next, you can serve the chat completion API by running:

npm run dev serve

This listens on port 3031 by default. For example, to use it with the OpenAI Python SDK, you can do the following:

import json
from openai import OpenAI

model_spec = {
    "tier_selector": ">3",
    "flags": [
        "uncensored", # Support open source, uncensored models.
    ],
}

client = OpenAI(
    base_url="http://localhost:3031",
)

client.chat.completions.create(
    model=json.dumps(model_spec),
    messages=[
        {
            "role": "user",
            "content": "Tell me a joke!",
        },
    ],
)

This request will be routed to providers on the Inference Grid and paid for from your Spark wallet in real-time!