Using AI providers

Golem comes with an extensive set of libraries (opens in a new tab) for connecting to various AI and AI related third party providers. These libraries are providing abstractions for various areas, and pluggable implementations for them. This way the actual third party provider can be switched any time without modifying the Golem agent's code.

The currently supported areas are:

Large Language Models (LLM)
Graph Databases
Search Engines
Web Search Engines
Speech to Text
Video Generation
Code Snippet Execution

All these interfaces are ready to use from any Golem agent through the Golem SDK. If one or more of the above interfaces are used in an agent, then an implementation needs to be selected for each by adding them to the golem.yaml file of the component.

The golem.yaml files coming from the agent templates are coming with commented-out entries for each possible choice of implementations, so just uncomment the desired ones and provide any necessary configuration (usually in form of environment variables).

The rest of the page shows detailed information about these libraries, the available implementations and their basic interface (without a full documentation of every type involved).

LLMs

The list of supported LLM providers is the following:

The core interface for LLMs:

declare module 'golem:llm/llm@1.0.0' {
   /**
      * --- Core Functions ---
      * Make a single call to the LLM.
      * To continue the conversation:
      * - append tool responses and new messages to the events and use send again
      * - or use the chat-session wrapper, which help in maintaining the chat events
      * @throws Error
      */
   export function send(events: Event[], config: Config): Response;
   /**
      * Makes a single call to the LLM and gets back a streaming API to receive the response in chunks.
      */
   export function stream(events: Event[], config: Config): ChatStream;
   // ...
}

Graph Databases

The list of supported graph databases is the following:

The core interface for graph databases:

declare module 'golem:graph/transactions@1.0.0' {
  export class Transaction {
      /**
       * === QUERY OPERATIONS ===
       * Execute a database-specific query string
       * @throws GraphError
       */
      executeQuery(options: ExecuteQueryOptions): QueryExecutionResult;
      /**
       * == TRAVESAL OPERATIONS ==
       * Find shortest path between two vertices
       * @throws GraphError
       */
      findShortestPath(options: FindShortestPathOptions): Path | undefined;
      /**
       * Find all paths between two vertices (up to limit)
       * @throws GraphError
       */
      findAllPaths(options: FindAllPathsOptions): Path[];
      /**
       * Get k-hop neighborhood around a vertex
       * @throws GraphError
       */
      getNeighborhood(options: GetNeighborhoodOptions): Subgraph;
      /**
       * Check if path exists between vertices
       * @throws GraphError
       */
      pathExists(options: PathExistsOptions): boolean;
      /**
       * Get vertices at specific distance from source
       * @throws GraphError
       */
      getVerticesAtDistance(options: GetVerticesAtDistanceOptions): Vertex[];
      /**
       * Get adjacent vertices through specified edge types
       * @throws GraphError
       */
      getAdjacentVertices(options: GetAdjacentVerticesOptions): Vertex[];
      /**
       * Get edges connected to a vertex
       * @throws GraphError
       */
      getConnectedEdges(options: GetConnectedEdgesOptions): Edge[];
      /**
       * === VERTEX OPERATIONS ===
       * Create a new vertex
       * @throws GraphError
       */
      createVertex(options: CreateVertexOptions): Vertex;
      /**
       * Create multiple vertices in a single operation
       * @throws GraphError
       */
      createVertices(vertices: CreateVertexOptions[]): Vertex[];
      /**
       * Get vertex by ID
       * @throws GraphError
       */
      getVertex(id: ElementId): Vertex | undefined;
      /**
       * Update vertex properties (replaces all properties)
       * @throws GraphError
       */
      updateVertex(options: UpdateVertexOptions): Vertex;
      /**
       * Delete vertex (and optionally its edges)
       * @throws GraphError
       */
      deleteVertex(id: ElementId, deleteEdges: boolean): void;
      /**
       * Find vertices by type and optional filters
       * @throws GraphError
       */
      findVertices(options: FindVerticesOptions): Vertex[];
      /**
       * === EDGE OPERATIONS ===
       * Create a new edge
       * @throws GraphError
       */
      createEdge(options: CreateEdgeOptions): Edge;
      /**
       * Create multiple edges in a single operation
       * @throws GraphError
       */
      createEdges(edges: CreateEdgeOptions[]): Edge[];
      /**
       * Get edge by ID
       * @throws GraphError
       */
      getEdge(id: ElementId): Edge | undefined;
      /**
       * Update edge properties
       * @throws GraphError
       */
      updateEdge(options: UpdateEdgeOptions): Edge;
      /**
       * Delete edge
       * @throws GraphError
       */
      deleteEdge(id: ElementId): void;
      /**
       * Find edges by type and optional filters
       * @throws GraphError
       */
      findEdges(options: FindEdgesOptions): Edge[];
      /**
       * === TRANSACTION CONTROL ===
       * Commit the transaction
       * @throws GraphError
       */
      commit(): void;
      /**
       * Rollback the transaction
       * @throws GraphError
       */
      rollback(): void;
      /**
       * Check if transaction is still active
       */
      isActive(): boolean;
    }
  // ...
}

Search Engines

The list of supported search engines is the following:

The core interface for search engines:

declare module 'golem:search/core@1.0.0' {
  /**
     * Index lifecycle
     * @throws SearchError
     */
    export function createIndex(options: CreateIndexOptions): void;
    /**
     * @throws SearchError
     */
    export function deleteIndex(name: IndexName): void;
    /**
     * @throws SearchError
     */
    export function listIndexes(): IndexName[];
    /**
     * Document operations
     * @throws SearchError
     */
    export function upsert(index: IndexName, doc: Doc): void;
    /**
     * @throws SearchError
     */
    export function upsertMany(index: IndexName, docs: Doc[]): void;
    /**
     * @throws SearchError
     */
    export function delete_(index: IndexName, id: DocumentId): void;
    /**
     * @throws SearchError
     */
    export function deleteMany(index: IndexName, ids: DocumentId[]): void;
    /**
     * @throws SearchError
     */
    export function get(index: IndexName, id: DocumentId): Doc | undefined;
    /**
     * Query
     * @throws SearchError
     */
    export function search(index: IndexName, query: SearchQuery): SearchResults;
    /**
     * @throws SearchError
     */
    export function streamSearch(index: IndexName, query: SearchQuery): SearchStream;
    /**
     * Schema inspection
     * @throws SearchError
     */
    export function getSchema(index: IndexName): Schema;
    /**
     * @throws SearchError
     */
    export function updateSchema(index: IndexName, schema: Schema): void;
  // ...
}

Web Search Engines

The list of supported web search engines is the following:

The core interface for web search engines:

declare module 'golem:web-search/web-search@1.0.0' {
    /**
     * Start a search session, returning a search context
     * @throws SearchError
     */
    export function startSearch(params: SearchParams): SearchSession;
    /**
     * One-shot search that returns results immediately (limited result count)
     * @throws SearchError
     */
    export function searchOnce(params: SearchParams): [SearchResult[], SearchMetadata | undefined];
    export class SearchSession {
      /**
       * Get the next page of results
       * @throws SearchError
       */
      nextPage(): SearchResult[];
      /**
       * Retrieve session metadata (after any query)
       */
      getMetadata(): SearchMetadata | undefined;
    }
  // ...
}

Speech to Text

The list of supported speech to text providers is the following:

The core interface for speech to text providers:

declare module 'golem:stt/transcription@1.0.0' {
    /**
     * @throws SttError
     */
    export function transcribe(request: TranscriptionRequest): TranscriptionResult;
    /**
     * @throws SttError
     */
    export function transcribeMany(requests: TranscriptionRequest[]): MultiTranscriptionResult;
  // ...
}

Video Generation

The list of supported video generation providers is the following:

The core interface for video generation providers:

declare module 'golem:video-generation/video-generation@1.0.0' {
    /**
     * @throws VideoError
     */
    export function generate(input: MediaInput, config: GenerationConfig): string;
    /**
     * @throws VideoError
     */
    export function poll(jobId: string): VideoResult;
    /**
     * @throws VideoError
     */
    export function cancel(jobId: string): string;
  // ...
}
declare module 'golem:video-generation/lip-sync@1.0.0' {
    /**
     * @throws VideoError
     */
    export function generateLipSync(video: LipSyncVideo, audio: AudioSource): string;
    /**
     * @throws VideoError
     */
    export function listVoices(language: string | undefined): VoiceInfo[];
  // ...
}
declare module 'golem:video-generation/advanced@1.0.0' {
    /**
     * @throws VideoError
     */
    export function extendVideo(options: ExtendVideoOptions): string;
    /**
     * @throws VideoError
     */
    export function upscaleVideo(input: BaseVideo): string;
    /**
     * @throws VideoError
     */
    export function generateVideoEffects(options: GenerateVideoEffectsOptions): string;
    /**
     * @throws VideoError
     */
    export function multiImageGeneration(options: MultImageGenerationOptions): string;
  // ...
}

Code Snippet Execution

The list of supported languages is the following:

JavaScript
Python

The core interface for code snippet execution:

declare module 'golem:exec/executor@1.0.0' {
    /**
     * Blocking, non-streaming execution
     * - `lang` specifies the programming language and version.
     * - `modules` are additional code files to include in the execution context. these can be imported in `snippet` in a language-specific way.
     * - `snippet` is the top level code to execute.
     * - `options` is controlling the script runner environment, see the run-options record for more details
     * The returned value captures the stdout and stderr of the executed snippet.
     * @throws Error
     */
    export function run(lang: Language, modules: File[], snippet: string, options: RunOptions): ExecResult;
 
    export class Session {
      /**
       * Create a new session for executing code snippets in the specified language, with a set of additional
       * code files that can be imported in the executed snippets.
       */
      constructor(lang: Language, modules: File[]);
      /**
       * Upload a data file to the session, which can be accessed in the executed snippets through standard file system APIs.
       * @throws Error
       */
      upload(file: File): void;
      /**
       * Execute a code snippet in the session in a blocking way
       * - `snippet` is the top level code to execute.
       * - `options` is controlling the script runner environment, see the run-options record for more details
       * The returned value captures the stdout and stderr of the executed snippet.
       * @throws Error
       */
      run(snippet: string, options: RunOptions): ExecResult;
      /**
       * Downloads a data file from the session.
       * @throws Error
       */
      download(path: string): Uint8Array;
      /**
       * Lists all the data files available in the session. These will include the ones that were `upload`ed and also
       * any other file created by the executed snippets.
       * @throws Error
       */
      listFiles(dir: string): string[];
      /**
       * Sets the current working directory within the session.
       * @throws Error
       */
      setWorkingDir(path: string): void;
    }
  // ...
}

Agent Filesystem Using Relational Databases