SkillNavigator - API

Setup

You need to configure a .env file where different credentials are stored. As starting point you can copy the .env.example and rename it to .env.

Architecture Overview

The index_vector.js implements a skill recommendation API with the following key components:

Core Components

• Vector Search Engine: Uses vector embeddings to find relevant skills based on text input
• Language Detection: Supports English and German text analysis
• GPT Integration: Leverages GPT models for topic extraction and relevance scoring
• MySQL Database: Stores skill metadata and details

Main API Endpoints

• POST /recommendations/chats: Analyzes text and returns relevant skills
• GET /recommendations/keywords: Returns keywords for a given query text
• POST /skills/tree: Returns a breadcrumb path for a give skill

Processing Pipeline

1. Language Detection & Validation

   • Detects input language (EN/DE)
   • Validates text length and format

2. Topic Extraction

   • Uses GPT-4 to extract relevant topics/skills from text
   • Filters for student-focused skills
   • Returns up to 10 skills

3. Vector Search

   • Performs parallel vector similarity search for each topic
   • Uses separate vector stores for English and German
   • Returns top matching skills

4. Relevance Scoring

   • Uses GPT-3.5-turbo to evaluate skill relevance
   • Scores skills from 0-100 based on context
   • Processes skills in batches for efficiency

5. Result Processing

   • Deduplicates results
   • Filters by minimum relevance threshold
   • Formats response with skill details from database
   • Calculates distance scores

CLI

Inside the cli directory are to scripts to import ESCO csv files into the MySQL database and to vectorize the MySQL database and push it to qdrant vector database.

Techstack

• NodeJS
• Langchain JS for A.I. functionallity
• Qdrant as vector database
• MySQL as main database

Build Mockserver Docker Image for Deployment (On Apple Silicon)

Build the image docker build --platform linux/amd64 --no-cache -t skillnavigator-api . Run the container docker run --platform linux/amd64 -dp 3000:3000 skillnavigator-api Export the image for Portainer docker save skillnavigator-api:latest | gzip > skillnavigator-api.tar.gz

Build Production API Docker Image for Deployment (On Apple Silicon)

Build the image docker build --platform linux/amd64 --no-cache -t skillnavigator-api-prod . Run the container docker run --platform linux/amd64 -dp 3001:3001 skillnavigator-api-prod Export the image for Portainer docker save skillnavigator-api-prod:latest | gzip > skillnavigator-api-prod.tar.gz

Build Development API Docker Image for Deployment (On Apple Silicon)

Build the image docker build --platform linux/amd64 --no-cache -t skillnavigator-api-dev . Run the container docker run --platform linux/amd64 -dp 3001:3001 skillnavigator-api-dev Export the image for Portainer docker save skillnavigator-api-dev:latest | gzip > skillnavigator-api-dev.tar.gz