Quick Start: Knowledge Table

1. Introduction

This guide demonstrates how to use JamAI Base TypeScript SDK to upload and embed files into Knowledge Tables for AI-powered document processing and retrieval.

What are Knowledge Tables?

Knowledge Tables are specialized tables in JamAI Base that provide hybrid-search capabilities through both full-text search (FTS) and vector embeddings:

• Search Capabilities:

  • Full-Text Search (FTS): Traditional keyword-based search for exact and partial matches

  • Semantic Search: Vector embedding-based search for meaning and context

• Document Processing:

  • Automatically chunks documents into manageable segments

  • Generates vector embeddings for semantic understanding

  • Indexes content for full-text search

  • Preserves document structure (tables, layouts, etc) and metadata

• Use Cases:

  • Document retrieval using both keywords and semantic meaning

  • Question-answering agent

  • Content recommendation

  • Knowledge base search and discovery

Supported File Types

The following file formats are supported:

• Text files: .txt, .md, .csv, .tsv

• Documents: .doc, .docx, .pdf

• Presentations: .ppt, .pptx

• Spreadsheets: .xls, .xlsx

• Markup/Data: .xml, .html, .json, .jsonl

Prerequisites

Before starting, you'll need:

• Node.js 16.x or higher

• Project ID and Personal Access Token (PAT)

• Documents to process

2. Installation and Setup

Installing Required Packages

npm install jamaibase dotenv

Basic Configuration

import JamAI from "jamaibase";
import * as dotenv from "dotenv";

// Load environment variables
dotenv.config();

const PROJECT_ID = "your_project_id";
const PAT = process.env.PAT || "your_PAT";

const client = new JamAI({
  projectId: PROJECT_ID,
  token: PAT,
});

3. Creating Your Knowledge Table

1

Create a new table

Navigate to your JamAI Base knowledge tables tab and create a new knowledge table.

2

Note table ID

After creating the table, note down the table ID for later use.

4. Implementation

4.1 Complete Document Uploader Class

import JamAI from "jamaibase";
import { File } from "formdata-node";
import * as fs from "fs";
import * as path from "path";

class DocumentUploader {
  private client: JamAI;

  constructor(projectId: string, pat: string) {
    // Initialize the document uploader
    this.client = new JamAI({
      projectId: projectId,
      token: pat,
    });
  }

  validateFile(filePath: string): boolean {
    // Validate if file exists and has supported format
    if (!fs.existsSync(filePath)) {
      throw new Error(`File not found: ${filePath}`);
    }

    const supportedTypes = [
      ".csv",
      ".tsv",
      ".txt",
      ".md",
      ".doc",
      ".docx",
      ".pdf",
      ".ppt",
      ".pptx",
      ".xls",
      ".xlsx",
      ".xml",
      ".html",
      ".json",
      ".jsonl",
    ];
    const fileExt = path.extname(filePath).toLowerCase();

    if (!supportedTypes.includes(fileExt)) {
      throw new Error(
        `Unsupported file format. Supported formats: ${supportedTypes.join(
          ", "
        )}`
      );
    }

    return true;
  }

  getMimeType(filePath: string): string {
    // Get MIME type of the file
    const mimeTypes: { [key: string]: string } = {
      ".csv": "text/csv",
      ".tsv": "text/tab-separated-values",
      ".txt": "text/plain",
      ".md": "text/markdown",
      ".doc": "application/msword",
      ".docx":
        "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
      ".pdf": "application/pdf",
      ".ppt": "application/vnd.ms-powerpoint",
      ".pptx":
        "application/vnd.openxmlformats-officedocument.presentationml.presentation",
      ".xls": "application/vnd.ms-excel",
      ".xlsx":
        "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      ".xml": "application/xml",
      ".html": "text/html",
      ".json": "application/json",
      ".jsonl": "application/jsonl",
    };

    const fileExt = path.extname(filePath).toLowerCase();
    return mimeTypes[fileExt] || "application/octet-stream";
  }

  getOptimalChunkSettings(filePath: string): { size: number; overlap: number } {
    // Determine optimal chunk settings based on file type
    const fileExt = path.extname(filePath).toLowerCase();

    const settings: Record<string, { size: number; overlap: number }> = {
      // Text-based documents
      ".txt": { size: 1000, overlap: 200 },
      ".md": { size: 1000, overlap: 200 },
      ".csv": { size: 800, overlap: 150 },
      ".tsv": { size: 800, overlap: 150 },

      // Rich text documents
      ".doc": { size: 1200, overlap: 250 },
      ".docx": { size: 1200, overlap: 250 },
      ".pdf": { size: 1500, overlap: 300 },

      // Presentations
      ".ppt": { size: 1000, overlap: 200 },
      ".pptx": { size: 1000, overlap: 200 },

      // Spreadsheets
      ".xls": { size: 800, overlap: 150 },
      ".xlsx": { size: 800, overlap: 150 },

      // Markup/structured documents
      ".xml": { size: 1000, overlap: 200 },
      ".html": { size: 1000, overlap: 200 },
      ".json": { size: 800, overlap: 150 },
      ".jsonl": { size: 800, overlap: 150 },
    };

    return settings[fileExt] || { size: 1000, overlap: 200 };
  }

  async uploadDocument(
    filePath: string,
    tableId: string,
    customChunkSize?: number,
    customChunkOverlap?: number
  ): Promise<boolean> {
    try {
      // Validate file
      this.validateFile(filePath);

      // Get file information
      const fileName = path.basename(filePath);
      const fileStats = fs.statSync(filePath);
      const fileSize = fileStats.size;
      const mimeType = this.getMimeType(filePath);

      // Get chunk settings
      const settings = this.getOptimalChunkSettings(filePath);
      const chunkSize = customChunkSize || settings.size;
      const chunkOverlap = customChunkOverlap || settings.overlap;

      console.log(`Uploading: ${fileName}`);
      console.log(`File type: ${mimeType}`);
      console.log(`File size: ${(fileSize / 1024).toFixed(2)} KB`);
      console.log(`Chunk size: ${chunkSize}, Overlap: ${chunkOverlap}`);

      // Read file and create File object
      const fileBuffer = fs.readFileSync(filePath);
      const file = new File([fileBuffer], fileName, { type: mimeType });

      // Upload and embed file
      const response = await this.client.table.embedFile({
        file: file,
        table_id: tableId,
        chunk_size: chunkSize,
        chunk_overlap: chunkOverlap,
      });

      console.log(`Upload successful: ${fileName}`);
      return true;
    } catch (error) {
      console.error(`Error uploading ${filePath}: ${error}`);
      return false;
    }
  }
}

5. Complete Standalone Script

Save this as knowledge_uploader.ts:

import JamAI from "jamaibase";
import { File } from "formdata-node";
import * as fs from "fs";
import * as path from "path";
import * as dotenv from "dotenv";

class DocumentUploader {
  private client: JamAI;

  constructor(projectId: string, pat: string) {
    this.client = new JamAI({
      projectId: projectId,
      token: pat,
    });
  }

  validateFile(filePath: string): boolean {
    if (!fs.existsSync(filePath)) {
      throw new Error(`File not found: ${filePath}`);
    }

    const supportedTypes = [
      ".csv",
      ".tsv",
      ".txt",
      ".md",
      ".doc",
      ".docx",
      ".pdf",
      ".ppt",
      ".pptx",
      ".xls",
      ".xlsx",
      ".xml",
      ".html",
      ".json",
      ".jsonl",
    ];
    const fileExt = path.extname(filePath).toLowerCase();

    if (!supportedTypes.includes(fileExt)) {
      throw new Error(
        `Unsupported file format. Supported formats: ${supportedTypes.join(
          ", "
        )}`
      );
    }

    return true;
  }

  getMimeType(filePath: string): string {
    const mimeTypes: { [key: string]: string } = {
      ".csv": "text/csv",
      ".tsv": "text/tab-separated-values",
      ".txt": "text/plain",
      ".md": "text/markdown",
      ".doc": "application/msword",
      ".docx":
        "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
      ".pdf": "application/pdf",
      ".ppt": "application/vnd.ms-powerpoint",
      ".pptx":
        "application/vnd.openxmlformats-officedocument.presentationml.presentation",
      ".xls": "application/vnd.ms-excel",
      ".xlsx":
        "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      ".xml": "application/xml",
      ".html": "text/html",
      ".json": "application/json",
      ".jsonl": "application/jsonl",
    };

    const fileExt = path.extname(filePath).toLowerCase();
    return mimeTypes[fileExt] || "application/octet-stream";
  }

  getOptimalChunkSettings(filePath: string): { size: number; overlap: number } {
    const fileExt = path.extname(filePath).toLowerCase();

    const settings: { [key: string]: { size: number; overlap: number } } = {
      ".txt": { size: 1000, overlap: 200 },
      ".md": { size: 1000, overlap: 200 },
      ".csv": { size: 800, overlap: 150 },
      ".tsv": { size: 800, overlap: 150 },
      ".doc": { size: 1200, overlap: 250 },
      ".docx": { size: 1200, overlap: 250 },
      ".pdf": { size: 1500, overlap: 300 },
      ".ppt": { size: 1000, overlap: 200 },
      ".pptx": { size: 1000, overlap: 200 },
      ".xls": { size: 800, overlap: 150 },
      ".xlsx": { size: 800, overlap: 150 },
      ".xml": { size: 1000, overlap: 200 },
      ".html": { size: 1000, overlap: 200 },
      ".json": { size: 800, overlap: 150 },
      ".jsonl": { size: 800, overlap: 150 },
    };

    return settings[fileExt] || { size: 1000, overlap: 200 };
  }

  async uploadDocument(
    filePath: string,
    tableId: string,
    customChunkSize?: number,
    customChunkOverlap?: number
  ): Promise<boolean> {
    try {
      this.validateFile(filePath);

      const fileName = path.basename(filePath);
      const fileStats = fs.statSync(filePath);
      const fileSize = fileStats.size;
      const mimeType = this.getMimeType(filePath);

      const settings = this.getOptimalChunkSettings(filePath);
      const chunkSize = customChunkSize || settings.size;
      const chunkOverlap = customChunkOverlap || settings.overlap;

      console.log(`Uploading: ${fileName}`);
      console.log(`File type: ${mimeType}`);
      console.log(`File size: ${(fileSize / 1024).toFixed(2)} KB`);
      console.log(`Chunk size: ${chunkSize}, Overlap: ${chunkOverlap}`);

      const fileBuffer = fs.readFileSync(filePath);
      const file = new File([fileBuffer], fileName, { type: mimeType });

      const response = await this.client.table.embedFile({
        file: file,
        table_id: tableId,
        chunk_size: chunkSize,
        chunk_overlap: chunkOverlap,
      });

      console.log(`Upload successful: ${fileName}`);
      return true;
    } catch (error) {
      console.error(`Error uploading ${filePath}: ${error}`);
      return false;
    }
  }
}

async function processFolder(
  folderPath: string,
  uploader: DocumentUploader,
  tableId: string,
  chunkSize?: number,
  chunkOverlap?: number
): Promise<{ successful: string[]; failed: string[] }> {
  const successful: string[] = [];
  const failed: string[] = [];

  const files = fs.readdirSync(folderPath);

  for (const filename of files) {
    const filePath = path.join(folderPath, filename);
    if (fs.statSync(filePath).isFile()) {
      const success = await uploader.uploadDocument(
        filePath,
        tableId,
        chunkSize,
        chunkOverlap
      );
      if (success) {
        successful.push(filename);
      } else {
        failed.push(filename);
      }
    }
  }

  return { successful, failed };
}

// Main execution
async function main() {
  // Load environment variables
  dotenv.config();

  // Configuration (use environment variables or command-line args)
  const PROJECT_ID = process.env.JAMAI_PROJECT_ID || "your_project_id";
  const PAT = process.env.JAMAI_API_KEY || "your_PAT";
  const TABLE_ID = process.env.TABLE_ID || "your_table_id";
  const INPUT_PATH = process.env.INPUT_PATH || "path/to/file_or_folder";

  // Initialize uploader
  const uploader = new DocumentUploader(PROJECT_ID, PAT);

  // Process input
  if (fs.statSync(INPUT_PATH).isFile()) {
    // Single file upload
    console.log("Processing single file...\n");
    const success = await uploader.uploadDocument(INPUT_PATH, TABLE_ID);

    if (success) {
      console.log("\nFile uploaded successfully!");
    } else {
      console.log("\nFile upload failed.");
    }
  } else if (fs.statSync(INPUT_PATH).isDirectory()) {
    // Batch folder upload
    console.log("Processing folder...\n");
    const { successful, failed } = await processFolder(
      INPUT_PATH,
      uploader,
      TABLE_ID
    );

    console.log("\n=== Upload Summary ===");
    console.log(`Successful: ${successful.length}`);
    console.log(`Failed: ${failed.length}`);

    if (successful.length > 0) {
      console.log("\nSuccessful uploads:");
      successful.forEach((file) => console.log(`  ✓ ${file}`));
    }

    if (failed.length > 0) {
      console.log("\nFailed uploads:");
      failed.forEach((file) => console.log(`  ✗ ${file}`));
    }
  } else {
    console.log("Invalid input path");
  }
}

main().catch(console.error);

6. Usage Examples

6.1 Single File Upload

// Initialize uploader
const uploader = new DocumentUploader(PROJECT_ID, PAT);

// Upload single document
const success = await uploader.uploadDocument(
  "path/to/document.pdf",
  "your_table_id"
);

if (success) {
  console.log("Upload successful!");
}

6.2 Batch Folder Upload

const { successful, failed } = await processFolder(
  "path/to/documents/folder",
  uploader,
  "your_table_id"
);

console.log(`Successfully uploaded: ${successful.length} files`);
console.log(`Failed uploads: ${failed.length} files`);

6.3 Custom Chunk Settings

// Upload with custom chunk size and overlap
await uploader.uploadDocument(
  "path/to/document.pdf",
  "your_table_id",
  2000, // Custom chunk size
  400 // Custom chunk overlap
);

7. Best Practices

7.1 Chunking Strategy

Recommended Settings by Document Type:

Document Type	Chunk Size	Overlap	Reasoning
PDFs	1500	300	Preserves paragraph context
Text Files	1000	200	Balanced for general text
Spreadsheets	800	150	Smaller for structured data
JSON/JSONL	800	150	Maintains object boundaries

7.2 Error Handling

async function robustUpload(
  filePath: string,
  tableId: string,
  maxRetries: number = 3
) {
  const uploader = new DocumentUploader(PROJECT_ID, PAT);

  for (let i = 0; i < maxRetries; i++) {
    try {
      const success = await uploader.uploadDocument(filePath, tableId);
      if (success) return true;
    } catch (error) {
      console.log(`Attempt ${i + 1} failed: ${error}`);
      if (i === maxRetries - 1) throw error;
      await new Promise((resolve) => setTimeout(resolve, 1000 * (i + 1)));
    }
  }
  return false;
}

7.3 Batch Processing with Rate Limiting

async function batchUploadWithLimit(
  files: string[],
  tableId: string,
  concurrentLimit: number = 3
) {
  const uploader = new DocumentUploader(PROJECT_ID, PAT);
  const results = [];

  for (let i = 0; i < files.length; i += concurrentLimit) {
    const batch = files.slice(i, i + concurrentLimit);
    const batchResults = await Promise.all(
      batch.map((file) => uploader.uploadDocument(file, tableId))
    );
    results.push(...batchResults);

    // Rate limiting delay
    if (i + concurrentLimit < files.length) {
      await new Promise((resolve) => setTimeout(resolve, 1000));
    }
  }

  return results;
}

8. Running the Script

Using Environment Variables

1. Create .env file:

   JAMAI_PROJECT_ID=your_project_id
   JAMAI_API_KEY=your_PAT
   TABLE_ID=your_table_id
   INPUT_PATH=path/to/documents

2. Run the script:

   ts-node knowledge_uploader.ts

Using Command Line (modify script to accept args)

ts-node knowledge_uploader.ts \
    --project-id your_project_id \
    --pat your_PAT \
    --table-id your_table_id \
    --input path/to/documents

9. Troubleshooting

Common Issues

1. File not found error

   • Verify file path is correct

   • Use absolute paths for reliability

2. Unsupported file format

   • Check file extension against supported types

   • Verify file is not corrupted

3. Upload timeout

   • Check internet connection

   • For large files, consider increasing timeout settings

   • Verify file size is reasonable

4. Chunk size errors

   • Ensure chunk size > chunk overlap

   • Try default settings first

   • Adjust based on document structure

10. Next Steps

1

Query Your Knowledge Base

• Use hybrid search to find relevant documents.

• Combine with Action Tables for RAG applications.

2

Create RAG Applications

• Link Knowledge Tables to Action Tables.

• Build question-answering systems.

• Develop intelligent document assistants.

3

Monitor and Optimize

• Review search quality.

• Adjust chunk settings if needed.

• Add more documents to improve coverage.