Sorry for being away for so long. I've been head-down strengthening my software engineering foundation—consuming rather than producing. After a while, I started feeling this pull to balance things out by building again.

I just finished Computer Systems: A Programmer's Perspective (CS:APP), and I can't recommend it enough. If you've ever felt gaps in your foundational knowledge, this book fills them. It's one of the best technical books I've read. Now I'm working through AI Engineering by Chip Huyen, and it felt like the right time to pick up where we left off.

Why Read This

My background is software engineering, not machine learning. Over the past year I've shifted into what Chip calls "AI engineering": building applications on top of foundation models. I want to share what I've learned with those who haven't yet gotten the chance to experience building in this space.

Life is demanding and the field keeps moving. Not everyone has time to keep up—reading papers, watching tutorials, building side projects. That's exactly why I'm building this series: real projects, real source code, walked through step by step. A way to stay current with what's actually possible.

What We're Building

A 10GB model. A 67-line prompt. 7 commands to set up. 2 config values.

That's what turns a crumpled receipt photo into structured JSON—line items, taxes, discounts, multiple currencies. All running locally on your laptop.

From this:

To this:

{
  "currency": "USD",
  "receipt_type": "service",
  "tax_format": "added",
  "receipt_subtotal": 145.00,
  "receipt_total_tax_amount": 9.06,
  "receipt_total_tax_percentage": 6.25,
  "receipt_total": 154.06,
  "items": [
    {
      "item_name": "Front and rear brake cables",
      "item_quantity": 1,
      "item_unit_price": 100.00,
      "item_total_price": 100.00,
      "confidence_score": 0.95
    },
    {
      "item_name": "New set of pedal arms",
      "item_quantity": 2,
      "item_unit_price": 15.00,
      "item_total_price": 30.00,
      "confidence_score": 0.95
    },
    {
      "item_name": "Labor 3hrs",
      "item_quantity": 3,
      "item_unit_price": 5.00,
      "item_total_price": 15.00,
      "confidence_score": 0.95
    }
  ]
}

Note: The model correctly handles quantity math (2 × $15 = $30), extracts tax separately (6.25% = $9.06), and assigns confidence scores to each extraction.

Let me show you how.

Did You Know: How VLMs "See" Images

Vision Language Models don't process images like humans. They divide images into patches (typically 14×14 or 16×16 pixels), encode them through a vision transformer, then project into the language model's embedding space. Each patch becomes a "token" the model reasons about—similar to how words become tokens in text. A single receipt image might become hundreds or even thousands of visual tokens depending on resolution.

Before We Start

You'll need:

Hardware: 16GB RAM minimum, 24GB+ recommended for smooth VLM inference.

No GPU or limited RAM? You can use Ollama's cloud models instead of running locally. See the Cloud Alternative section.

Model: We'll use qwen3-vl:8b-thinking-q8_0 (~10GB). Pull it now:

ollama pull qwen3-vl:8b-thinking-q8_0

Everything else runs in Docker.

Why Vision Language Models

Traditional OCR reads text without understanding context. Vision Language Models understand what they're looking at.

When processing a restaurant receipt:

• OCR sees: "Total: $47.83"

• VLM understands: This is the final amount, separate from subtotal and tax

When processing an invoice:

• OCR sees: "Net 30"

• VLM understands: This is payment terms, not an amount

VLMs read documents like humans do—understanding layout, context, and meaning.

The Three Pillars

1. Vision Language Model (qwen3-vl)

The "eyes" that read and understand receipts and invoices. It processes images, understands document layouts (headers, line items, totals), and outputs structured JSON data.

2. n8n Workflow Engine

The "brain" that orchestrates the entire process. It handles file uploads and validation, manages job queuing and processing, and coordinates data extraction and export through visual workflows.

Also our backend. n8n's webhooks serve as our API layer—no separate backend needed. It handles job creation, result fetching, and CSV exports directly.

3. Web Upload Interface

The "front door" for easy document submission. Drag-and-drop interface with progress tracking and status updates, plus the ability to download processed results.

Did You Know: Why Visual Workflow Tools Matter

n8n workflows are JSON under the hood. Each node has inputs, outputs, and configuration. The visual builder prevents syntax errors and makes debugging visible—you can see exactly where data flows and where it breaks.

Pro tip I learned the hard way: Build workflows in the UI first, export them, then modify the JSON if needed. Trying to write n8n workflow JSON from scratch (even with AI assistance) often leads to syntax issues that take forever to debug.

System Architecture

The Five Services

Our Docker Compose stack orchestrates five services:

graph TB
    subgraph Docker Network
        WEB[Nginx<br/>Web Interface<br/>Port 8080]
        n8n[n8n<br/>Workflow Engine<br/>Port 5678]
        PG[(PostgreSQL<br/>Database)]
        MIG[DB Migrator<br/>Schema Setup]
        PGA[pgAdmin<br/>Optional Admin UI]
    end

    USER((User)) --> WEB
    WEB --> n8n
    n8n --> PG
    n8n --> OLLAMA
    MIG --> PG
    PGA --> PG

    OLLAMA[Ollama<br/>VLM on Host<br/>Port 11434]

• postgres: PostgreSQL 15 Alpine for job tracking and extracted data

• db-migrator: Runs once on startup to apply SQL migrations

• n8n: Workflow engine (v2.4.8) with 6 workflows orchestrating the pipeline

• web: Nginx Alpine serving the upload interface

• pgadmin: Optional database admin panel

Data Flow

sequenceDiagram
    participant U as User
    participant W as Web Interface
    participant N as n8n Webhooks
    participant Q as Queue Monitor
    participant V as VLM Processor
    participant O as Ollama (Host)
    participant D as PostgreSQL

    U->>W: Upload receipt image
    W->>N: POST /webhook/upload-receipt
    N->>D: Create receipt record (status: pending)
    N->>D: Create processing job
    N-->>W: Return job ID

    loop Every 30 seconds
        Q->>D: Poll for pending jobs
        Q->>V: Trigger VLM Processor
    end

    V->>D: Fetch receipt file path
    V->>O: Send image + prompt
    O-->>V: Return extracted JSON
    V->>D: Update receipt with items
    V->>D: Mark job completed

    U->>W: View processed receipts
    W->>N: GET /webhook/get-receipts
    N->>D: Query completed receipts
    N-->>W: Return receipt list with items

The Philosophy: Setup Done Right

7 Commands. 2 Config Values.

I believe automation should respect your time. If you can't get from zero to working in under 10 minutes, something's wrong with the design.

Here's the complete setup:

git clone https://github.com/FarzamMohammadi/self-hosted-ai-stack.git
cd self-hosted-ai-stack/part-3-receipt-processing-with-vlm
cp .env.example .env
ollama pull qwen3-vl:8b-thinking-q8_0
# Edit .env: set OLLAMA_VLM_MODEL
docker-compose up -d
# Open http://localhost:5678 → Settings → n8n API → Create API Key
# Add N8N_API_KEY=<your-key> to .env
python scripts/setup-n8n.py
docker compose restart n8n

Done. The system is running:

• Web Interface: http://localhost:8080

• n8n Workflows: http://localhost:5678

• pgAdmin: http://localhost:5050

What Happens Behind the Scenes

When you run docker-compose up, a carefully orchestrated sequence unfolds:

1. PostgreSQL starts and reports healthy (not just "running")

2. Migrations run—in dependency order, wrapped in transactions

3. n8n waits for migrations to succeed, not just complete

4. Web server starts only after n8n is ready

This isn't accidental. The compose file uses service_healthy and service_completed_successfully conditions—the strictest dependency types Docker offers. If migrations fail, n8n doesn't start. No silent failures. No corrupt state.

When you run setup-n8n.py, a second orchestration happens:

1. Validates your API key and database connection

2. Creates PostgreSQL credentials inside n8n

3. Uploads all 6 workflows from JSON files

4. Activates each workflow

The restart at the end? A workaround for a known n8n bug where webhooks created via API don't register until the container restarts.

The Small Things That Matter

Sensible defaults. Only 2 required config values. Database credentials, ports, timeouts—all have working defaults.

Helpful errors. When something fails, the error tells you what to do, not just what went wrong:

Invalid N8N_API_KEY
Go to Settings → n8n API in n8n UI to create a valid API key

Idempotent operations. Run the setup script twice? It skips what's already done. Migrations already applied? Tracked and skipped. Safe to re-run, safe to experiment.

I have a deep love for automation—there's something satisfying about simplifying processes until they just work. Every automated step is one less thing to remember, one less way to fail. Great developer experience isn't just for others; it's for future you at 2am, six months from now, debugging something you've completely forgotten.

Processing a Receipt Together

You have the repo cloned. You have the stack running. Let's use it.

Step 1: Upload a Receipt

Try it: Navigate to http://localhost:8080 in your browser. You'll see the upload interface with a drag-and-drop zone.

Need a test receipt? There are 4 included in test-receipts/. Or use any receipt photo from your phone.

Drag your receipt onto the zone (or click to browse). You'll see a progress bar, then a success message showing the filename and "Pending" status. Two buttons appear: "Upload Another" and "View All Receipts".

What n8n does behind the scenes:

1. Validates file type (JPEG, PNG, WEBP only) and size (max 10MB)

2. Generates a UUID for the receipt

3. Creates a date-based path: /app/uploads/receipts/2026/02/02/uuid-filename.jpg

4. Saves the file to disk (persisted to volumes/uploads/ on host)

5. Creates a receipt record in PostgreSQL (status: pending)

6. Creates a processing job in the queue

7. Returns the job ID to the user

Key files:

• web/index.html - The upload page

• web/js/upload.js - Handles drag-drop, validation, and API calls

• n8n/workflows/receipt-job-creation.json - Workflow 1: Receipt Upload & Job Creation

Did You Know: Why UUIDs + Date Paths?

UUIDs prevent collisions even in distributed systems—two users uploading receipt.jpg at the same millisecond get unique filenames. Date-based paths (/2026/02/02/) make archival and cleanup trivial: delete everything older than 90 days with a simple directory operation. File hashing provides basic duplicate detection—the system can flag if you're uploading the same receipt twice.

Step 2: What Gets Stored

Here's what lands in the database when you upload:

-- receipts table (simplified)
id:                UUID
filename:          "sample-receipt.jpg"
file_path:         "/app/uploads/receipts/2026/02/02/abc123-sample-receipt.jpg"
file_size:         245000
mime_type:         "image/jpeg"
file_hash:         "a1b2c3d4..."
processing_status: "pending"
items:             NULL
created_at:        NOW()

Key file: db/migrations/00001-receipts.sql

Did You Know: Why JSONB for AI Output?

AI models return unpredictable structures. What if a receipt has 3 items? Or 30? What if some have discounts and others don't? JSONB stores flexible JSON with full indexing support (GIN indexes). You can query into it (WHERE items->>'item_name' LIKE '%coffee%'), index specific fields, and evolve the schema without migrations.

Step 3: View Your Receipts

Try it: Click "View All Receipts", or navigate directly to http://localhost:8080/receipts.html.

You'll see a table with columns: thumbnail, filename, upload date, status, items count, and confidence score. The receipt you just uploaded shows "Pending" status—the AI hasn't processed it yet.

Behind the scenes: The receipts page calls Workflow 4: Receipt List Provider (receipt-list-provider.json) to fetch receipts with filters and pagination.

Key files:

• web/receipts.html - The management page

• web/js/receipts.js - List, filter, detail, export logic

• n8n/workflows/receipt-list-provider.json - Workflow 4: Receipt List Provider

Step 4: Watch the Processing

Try it: Find the "Auto-refresh (10s)" toggle in the top right and enable it. Or click "Refresh" manually every few seconds.

Within 30-60 seconds, watch the status change: Pending → Processing → Completed. Once complete, the "Items" column shows how many line items were extracted, and a confidence score appears.

What happens behind the scenes:

Every 30 seconds, Workflow 2: Receipt VLM Processing Queue Monitor polls the database for pending jobs and triggers the VLM processor for each one.

Why 30 seconds? It's a balance between latency and resource efficiency. For a small business processing a few receipts a day, this works well. For high volume, you can reduce it.

Key file: n8n/workflows/receipt-vlm-processing-queue-monitor.json - Workflow 2: Receipt VLM Processing Queue Monitor

Step 5: The AI Magic (VLM Processing)

When Workflow 2 finds your pending job, it triggers Workflow 3: Receipt VLM Processing & Item Extraction. This is where the AI reads your receipt.

What Workflow 3 does:

1. Fetches the receipt file from disk

2. Encodes the image to base64

3. Sends the image + extraction prompt to Ollama

4. Parses the JSON response

5. Updates the receipt record with extracted items

6. Updates the job status (completed or failed)

7. If failed, retries up to 3 times

Using cloud? Same workflow works—see Cloud Alternative.

Key file: n8n/workflows/receipt-vlm-processor.json - Workflow 3: Receipt VLM Processing & Item Extraction

Step 6: The Extraction Prompt

Here's the prompt that tells the VLM how to extract receipt data (v9):

Extract items AND totals from this receipt image as JSON.

Output format:
{
  "currency": "USD",
  "receipt_type": "grocery|restaurant|retail|service|unknown",
  "tax_format": "added|inclusive|none",
  "receipt_subtotal": 0.00,
  "receipt_total_tax_amount": 0.00,
  "receipt_total_tax_percentage": 0.00,
  "receipt_total": 0.00,
  "items": [
    {
      "item_name": "Product Name",
      "item_quantity": 1,
      "item_unit_price": 0.00,
      "item_base_price": 0.00,
      "item_discount_amount": null,
      "item_tax_price": null,
      "item_total_price": 0.00,
      "item_sequence": 1,
      "confidence_score": 0.95,
      ...
    }
  ]
}

Rules:

TAX HANDLING - Identify the tax format first:

1. TAX ADDED (US/Canada style):
   - Look for separate "Subtotal", "Tax/Sales Tax", and "Total" lines
   - Total = Subtotal + Tax
   - tax_format = "added"

2. TAX INCLUSIVE (EU/Swiss style):
   - Look for "Total" with "Incl. X% MwSt/VAT/TVA: amount"
   - Tax is already part of the total, not added on top
   - tax_format = "inclusive"
   - receipt_subtotal = receipt_total - receipt_total_tax_amount

3. NO TAX SHOWN:
   - tax_format = "none", set tax fields to null

OTHER RULES:
- Remove quantity prefixes (1x, 2x) from item_name, put in item_quantity
- item_total_price = item_base_price - item_discount_amount (when discount exists)
- Math: item_quantity × item_unit_price = item_base_price
- Return JSON only

Key file: prompts/ocr/v9-system-prompt.md

67 lines. Clear output schema. Essential rules only. This simplicity is intentional—and hard-won. More on that below.

Did You Know: The LLM Dictionary and Tokenization

LLMs don't see characters—they see tokens. Each model has a "vocabulary" (dictionary) of ~32,000-100,000 tokens. Common words like "the" are single tokens, while rare words get split ("tokenization" might become "token" + "ization").

Why does this matter for prompts? The model processes tokens, not text. A well-structured JSON schema is unambiguous—the model knows exactly what format you expect. Generic placeholders like 0.00 or "Product Name" are familiar patterns from training data. This is why clear schemas work better than verbose explanations—they're precise, not just shorter.

Step 7: Results Stored in Database

Once processing completes, here's what gets saved:

-- Updated receipt record
processing_status: "completed"
items: [
  {
    "item_name": "Front and rear brake cables",
    "item_quantity": 1,
    "item_unit_price": 100.00,
    "item_total_price": 100.00,
    "confidence_score": 0.95
  },
  {
    "item_name": "New set of pedal arms",
    "item_quantity": 2,
    "item_unit_price": 15.00,
    "item_total_price": 30.00,
    "confidence_score": 0.95
  },
  ...
]
items_count: 3
total_confidence_score: 0.95
receipt_subtotal: 145.00
receipt_total_tax_amount: 9.06
receipt_total: 154.06

The JSONB column holds all extracted items with their confidence scores, while computed fields (items_count, total_confidence_score) enable fast queries without re-parsing the JSON.

Step 8: View the Extracted Data

Try it: On the receipts page (http://localhost:8080/receipts.html), click any completed receipt row to open the detail modal.

What you'll see:

• Left side: The original receipt image (scroll to zoom)

• Right side: Extracted metadata (filename, date, size, status, currency, receipt type) and a table of extracted items

Each item shows: name, quantity, unit price, base price, discount (if any), tax, total price, and a confidence score.

Behind the scenes: Workflow 5: Receipt Detail Provider (receipt-detail-provider.json) fetches the receipt record with the items array.

Did You Know: What Confidence Scores Actually Mean

Confidence isn't probability of correctness. It's the model's internal certainty about its output—how "surprised" it would be if wrong. A 0.95 confidence means the model is very sure about its answer. But high confidence + wrong answer = hallucination territory. That's why we use confidence thresholds:

• 90-100%: Auto-approve for accounting

• 70-89%: Quick manual review

• 50-69%: Detailed review required

• Below 50%: Manual entry likely faster

Step 9: Export Your Data

Try it: Click the "Export CSV" button in the header to download all completed receipts.

The CSV includes: receipt ID, filename, upload date, currency, receipt type, subtotal, tax, total, and all extracted line items flattened into columns.

What Workflows 4-6 do:

• Workflow 4: Receipt List Provider (receipt-list-provider.json): List receipts with filters and pagination

• Workflow 5: Receipt Detail Provider (receipt-detail-provider.json): Get receipt detail with items array

• Workflow 6: Receipt Export Generator (receipt-export-generator.json): Export completed receipts to CSV

Key files:

• web/receipts.html - The management page

• web/js/receipts.js - List, filter, detail, export logic

• n8n/workflows/receipt-list-provider.json

• n8n/workflows/receipt-detail-provider.json

• n8n/workflows/receipt-export-generator.json

The Art of Simplicity: A Prompt Optimization Journey

When More Is Less

This is the lesson that humbled me most during this project.

The receipt processing system needed to handle diverse formats—different currencies (USD, CHF, GBP), quantity notations (2x, x3, QTY columns), discounts, and tax calculations. My original prompt was a 400+ line behemoth, complete with:

• Detailed parsing approaches for every scenario

• Five examples with step-by-step breakdowns

• Mathematical validation checklists

• Multiple tax handling patterns

• Exhaustive field documentation

On paper, it looked thorough. Professional, even. I was proud of it.

In practice? Maybe 60-70% accuracy. Items would be missed, quantities misinterpreted, discounts incorrectly calculated. The model seemed to be... overthinking.

The Iterative Discovery

I set up a systematic testing pipeline with four diverse receipt samples:

1. A USD receipt with items and tax at bottom

2. A Swiss CHF receipt with quantity prefixes (2x Latte Macchiato) and European formatting

3. A US repair invoice with QTY columns and tabular data

4. A UK receipt with line-level discounts and manager overrides

What followed was an exercise in humility. After multiple iterations—tweaking parameters, adjusting examples, refining instructions—I had a revelation: What if the problem isn't that I'm not explaining enough, but that I'm explaining too much?

I stripped the prompt down. Out went the five detailed examples. Out went the philosophical explanations about tax patterns. Out went the validation checklists. What remained:

Extract items from this receipt image as JSON.

Output format:
{
  "currency": "USD",
  "has_total_tax_only": true,
  "items": [
    {
      "item_name": "Product Name",
      "item_quantity": 1,
      "item_unit_price": 5.00,
      "item_base_price": 5.00,
      "item_discount_amount": null,
      "item_total_price": 5.00,
      ...
    }
  ]
}

Rules:
- has_total_tax_only=true unless tax is shown on EACH item line separately
- Remove quantity prefixes (1x, 2x) from item_name, put in item_quantity
- item_total_price = item_base_price - item_discount_amount (when discount exists)
- Math: item_quantity × item_unit_price = item_base_price
- Return JSON only

About 30 lines instead of 400+.

The Results

I ran stability tests—five times per receipt with varying random seeds:

20 out of 20 runs passed. Every quantity correct. Every discount captured. Every currency identified. The math checked out perfectly.

I stared at the results, equal parts vindicated and embarrassed. All those hours crafting the perfect 400-line prompt, and the model just needed me to get out of its way.

The Lesson

The model already knew how to read receipts. It had seen millions of them during training. My elaborate prompt wasn't teaching it anything—it was confusing it.

When I bombarded it with examples, edge cases, and detailed instructions, the model tried to reconcile all that information with every image it saw. It started second-guessing obvious interpretations. It looked for complexities that weren't there.

The simple prompt worked because it:

1. Clearly defined the output format - Show, don't tell

2. Stated only essential rules - Five rules instead of fifty paragraphs

3. Trusted the model's inherent capabilities - It knows what a receipt looks like

4. Avoided overthinking triggers - No examples that might not match the current image

Broader Implications

This principle extends beyond receipt processing:

• Prompt engineering is often about subtraction, not addition

• Clear output schemas beat extensive explanations

• Trust the model's training—it's seen more examples than you can write

• Test systematically with diverse inputs to validate changes

As Antoine de Saint-Exupéry wrote: "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away."

The journey from 400 lines to 30 was a reminder that in AI engineering, simplicity isn't just elegant—it's effective.

Testing Methodology: Verify Before You Deploy

The Trap: Teaching to the Test

After discovering the power of simplicity, I needed to extend the prompt to capture receipt-level totals (subtotal, tax, total)—not just individual items. Here's how I approached it.

My v7 prompt had a has_total_tax_only boolean flag but no fields to capture the actual tax amount. When processing a US invoice with:

• Subtotal: $145.00

• Sales Tax 6.25%: $9.06

• Total: $154.06

The system would extract items correctly but lose the tax information entirely.

My first instinct was to add example values to the prompt:

"receipt_subtotal": 145.00,
"receipt_total_tax_amount": 9.06,
"receipt_total_tax_percentage": 6.25,
"receipt_total": 154.06

This "worked"—but I was essentially giving the model the answers. When testing on the same receipt that matched my example values, of course it succeeded. Training data that matches test data tells you nothing about real-world performance.

I've made this mistake before. You'd think I'd learn.

The Fix: Generic Placeholders

I revised to use zeros as placeholders, forcing the model to read the receipt:

"receipt_subtotal": 0.00,
"receipt_total_tax_amount": 0.00,
"receipt_total_tax_percentage": 0.00,
"receipt_total": 0.00

The Testing Protocol

Rather than running one test and calling it done, I ran 10 tests with different random seeds on the same receipt:

for seed in 43 44 45 46 47 48 49 50 51 52; do
  curl -s http://localhost:11434/api/generate \
    -d '{"model":"qwen3-vl:8b-thinking-q8_0", "seed":'$seed', ...}' \
    | jq '{tax:.receipt_total_tax_amount, pct:.receipt_total_tax_percentage}'
done

US Invoice Results (Sample 3):

10/10 PASSED - Consistent extraction across all seeds.

Testing Across Receipt Types

One receipt type isn't enough. I tested against different formats:

Swiss Receipt (European inclusive tax):

• Format: "Incl. 7.6% MwSt 54.50 CHF: 3.85"

• Result: 1/5 passed—the prompt struggled with tax-inclusive formats

This revealed a gap: the v8 prompt handled US-style "tax added" receipts perfectly but needed refinement for European "tax included" formats. This led directly to the v9 prompt with explicit multi-region tax handling.

Key Takeaways

1. Never use real values as examples - Use generic placeholders (0.00, "Product Name") to test actual extraction ability

2. Run multiple seeds - A single successful test means nothing. Run 5-10 tests with different random seeds to verify consistency

3. Test diverse inputs - Different receipt formats (US tax-added, EU tax-inclusive, no-tax) expose edge cases

4. Document failures - A 1/5 pass rate tells you exactly where to focus next

5. Iterate systematically - Don't guess. Test, measure, refine, repeat

This transforms prompt engineering from art to science. You stop hoping your prompt works and start knowing when and how it fails.

When Things Go Wrong: Built-in Safeguards

Document processing hits edge cases. The system handles them:

AI Response Failures:

• Automatic retry with up to 3 attempts

• Failed jobs marked for manual review

• Error logging for troubleshooting

Image Quality Issues:

• Pre-validation catches unsupported file types

• Confidence scores below thresholds flag items for review

• User gets clear feedback on what went wrong

Processing Timeouts:

• 60-second timeout prevents hung processes

• Failed jobs remain in queue for retry

• Queue monitor continues processing other jobs

Image Quality Tips

For best results:

• Resolution: 300-600 DPI ideal, 150 DPI minimum

• Size: 1024px max width reduces processing time

• Orientation: Right-side up (rotation affects accuracy)

• Lighting: Good contrast between text and background

• Format: JPG or PNG preferred

If you can't read it clearly, neither can the AI. Flatten creased receipts, use good lighting, fill the frame.

Did You Know: Image Preprocessing

Poor-quality images can be improved before sending to the VLM. Techniques like contrast enhancement (CLAHE), deskewing, sharpening, and binarization can make faded or noisy receipts more readable. Tools like OpenCV or PIL can automate this. We didn't implement preprocessing here, but it's worth exploring for consistently low-quality scans.

Common Issues

What You've Built

This isn't just a receipt processor. It's a demonstration of what's possible when you combine:

• Accessible AI—A 10GB model that runs on your laptop, not a cloud bill

• Thoughtful automation—7 commands to deploy, 2 values to configure

• Hard-won simplicity—A 67-line prompt that took 400+ lines to discover wasn't needed

• Engineering care—Health checks, validation chains, helpful errors

The real achievement isn't the features. It's that it works—reliably, on modest hardware, without fighting you.

The Lessons

On prompts: Simplicity beats complexity. Trust the model's training.

On setup: Great developer experience starts with the little details. Automate what can be automated. Simplify until it just works.

On building: The best code is often the code you deleted.

Coming Up in Part 4

We'll take n8n to the next level with agent orchestration—building workflows that don't just process data, but make decisions and use tools:

• AI agents that can call functions (calculators, web search, database queries)

• Multi-step reasoning workflows

• Agentic behavior patterns in n8n

The vision: Workflows that think, not just execute.

Resources

• n8n Documentation - Workflow automation platform

• Ollama - Local LLM hosting

• qwen3-vl on Ollama - Vision Language Model

• PostgreSQL JSONB - JSON operations

• GitHub Repository - Full source code

Cloud Alternative: When Local Isn't an Option

If you have less than 16GB RAM or no dedicated GPU, use Ollama's cloud models instead. Same API, same workflow—just a different model name.

Setup:

# Update to Ollama v0.12+
ollama --version
# Download latest from https://ollama.com/download if needed

# Sign in
ollama signin

# Update .env
OLLAMA_VLM_MODEL=qwen3-vl:235b-cloud

Available vision models: qwen3-vl:235b-cloud (recommended), deepseek-v3.1:671b-cloud

Cost: Ollama Turbo is $20/month. Ollama states they don't retain your data.

Other providers: You can modify the n8n workflow to use OpenAI or other APIs by changing VLM_API_URL, adding auth headers, and adjusting the request format in receipt-vlm-processor.json.

This is Part 3 of the "Complete Self-Hosted AI Infrastructure" series. We're building increasingly sophisticated AI capabilities, all running locally on your machine. Thanks for joining me on this journey.

We're adding document processing capabilities to the foundation from Part 1. This extends your chat interface to work with your files and documents.

Two key additions make this possible:

• Document Processing: Extract text from documents like PDFs, Word docs, and text files using Apache Tika (with OCR capabilities for scanned documents)

• RAG (Retrieval-Augmented Generation): Search and retrieve relevant information from your documents to answer questions

This transforms your local AI from conversation-only to document-aware. Upload a file, ask questions about its contents, and get answers grounded in your actual documents rather than the model's training data.

The bigger picture

I'm building this self-hosted AI stack and documenting everything as I go. Part 2 is where real power emerges. We're adding the intelligence layer that transforms your local AI from a chat toy into a genuine knowledge assistant.

Here's the thing: even with today's resources, implementing proper document processing and RAG together is surprisingly complex. This guide captures the lessons learned getting both capabilities production ready.

What we've built so far:

• Part 1: Building the foundation with Open WebUI, Ollama, and Postgres: Your foundational chat interface with local LLM hosting

What's still coming:

• Part 3: Visual Data Extraction: Web upload interface with N8N workflows and Vision Language Models for extracting structured data from images

• Part 4: Model Superpowers: Advanced WebUI configuration with tools and knowledge integration

• Part 5: Intelligent Automation: WebUI filters and N8N workflows for content processing

Prerequisites

• Part 1 foundation stack running

• Ollama running (verify with ollama serve and if you see "address already in use", it's already running)

• 24GB+ RAM recommended (16GB minimum)

• 50GB+ storage for documents and vectors

• Docker and Docker Compose

Important: If Part 1 is running, stop it first with cd ../part-1-building-the-foundation && docker compose down before starting Part 2.

Credits where credits are due

Massive thanks to the creators and contributors of these open source projects that make this possible:

• Apache Tika - Repository: apache/tika

• Qdrant - Repository: qdrant/qdrant

• Ollama - Repository: ollama/ollama

• Open WebUI - Repository: open-webui/open-webui

• Docker - Repositories: moby/moby & docker/compose

Quick start (skip explanations, just get it running)

1. Download the embedding model (~275MB):

   ollama pull nomic-embed-text
   

2. Clone the repository or pull latest:

   git clone https://github.com/FarzamMohammadi/self-hosted-ai-stack
   # Or if you already have it: git pull origin main
   

3. Navigate to part 2:

   cd part-2-rag-with-tika-and-qdrant
   

4. Start the enhanced stack:

   docker compose up -d
   

Done. Open http://localhost:3000, sign in, and you'll see document upload button (+) in the bottom left corner of the chat. Upload a PDF and start asking questions about it.

Important: Open WebUI handles files through two distinct pathways:

Document Processing (what we built): Extracts text from documents like PDFs, Word docs, and text files through Tika. Does NOT work with standalone image files (PNG, JPEG, etc.).

Image Analysis (not covered here): For standalone image files, you'll need to download and use a vision model.

Understanding what you built

Let's break down what we just built and why each component matters. Understanding the architecture transforms you from someone following steps into someone who owns the system.

What are document processing and RAG, and why you need both

Your AI model was trained on data from months or years ago. It doesn't know about your company's latest reports, personal documents, or today's news. Ask about quarterly financials, and it apologizes for lacking access.

Document Processing uses Apache Tika to extract machine-readable text directly from documents like PDFs, Word docs, and other formats by parsing their internal structure. For scanned documents or image-based content within PDFs, Tika automatically falls back to OCR using Tesseract when needed.

RAG (Retrieval-Augmented Generation) solves the knowledge gap by giving your AI instant access to search through your documents for relevant information.

Together, they create a document intelligence system that processes documents and provides grounded answers.

The transformation:

Real examples:

Document processing: "What's the total on this invoice?"

• Without document processing: "I can't read documents or extract text from files."

• With document processing + RAG: "Based on your invoice PDF, the total amount due is $2,847, with a due date of March 15th."

Document analysis: "What were our biggest challenges last quarter?"

• Without RAG: "I don't have access to your quarterly reports."

• With document processing + RAG: "Based on your Q3 report, the biggest challenges were supply chain delays (mentioned 8 times) and staffing shortages in the manufacturing division (pages 12-14)."

This transforms your AI from a chat interface into a document intelligence system that reads, understands, and answers questions about your documents.

How vector similarity works

Traditional search looks for exact word matches. Search for "machine learning performance" and miss documents about "AI optimization" or "model efficiency" entirely.

Embeddings solve this: mathematical representations that capture semantic meaning.

Think of embeddings as GPS coordinates for concepts:

• "Machine learning" → [0.2, 0.8, 0.1, ...] (768 dimensions)

• "AI optimization" → [0.3, 0.7, 0.2, ...]

• "Car repair" → [0.9, 0.1, 0.3, ...]

Closer coordinates mean more similar meaning. When nomic-embed-text converts "reduce costs" into a 768-dimensional vector, it positions near related concepts:

• "Budget optimization" (distance: 0.23)

• "Expense management" (distance: 0.31)

• "Financial efficiency" (distance: 0.28)

This mathematical precision ensures RAG retrieves conceptually relevant information beyond keyword matches.

Vector search visualization showing how concepts cluster in high-dimensional space. Related concepts like "Kitten" and "Cat" appear close together in mathematical space, enabling semantic search that finds meaning beyond exact keyword matches.

Source: https://odsc.medium.com/a-gentle-introduction-to-vector-search-3c0511bc6771

Vector databases store and search embeddings

Once you have embeddings, you need somewhere to store and search them efficiently. Regular databases can't handle similarity search.

Traditional database (SQL):

SELECT * FROM documents WHERE title CONTAINS 'machine learning'

Finds: Documents with exact phrase "machine learning" in title

Vector database:

SEARCH FOR documents SIMILAR TO embedding([0.2, 0.8, 0.1, ...])

Finds: Documents about AI, neural networks, model training, performance optimization, etc.

Vector databases use specialized algorithms to find similar vectors among millions quickly. Semantic search understands intent:

• Search: "reduce costs" → Finds: "budget optimization", "expense management", "financial efficiency"

• Search: "team problems" → Finds: "communication challenges", "staff conflicts", "collaboration issues"

How the complete RAG flow works

Document Processing (Setup Phase): Upload document → Apache Tika server extracts machine-readable text from documents like PDFs, Word docs, and text files (using OCR via Tesseract for scanned content when needed) → Open WebUI splits text into 1000-character chunks → nomic-embed-text model (via Ollama) converts each chunk into 768-dimensional vectors → Qdrant vector database stores vectors with original text for fast similarity search

Query Processing (When You Ask Questions): Question → nomic-embed-text model converts question to 768-dimensional vector → Qdrant vector database searches and finds 5 most similar document chunks → Open WebUI compiles relevant text excerpts → Enhanced prompt (question + retrieved excerpts) sent to Ollama LLM → AI generates grounded response with source attribution

The complete document processing + RAG orchestration: Open WebUI handles this entire workflow behind the scenes:

• Apache Tika extracts text from documents like PDFs, Word docs, and text files (with automatic OCR fallback for scanned content)

• nomic-embed-text converts extracted content into 768-dimensional vectors

• Qdrant stores and indexes vectors for fast similarity search

• Open WebUI orchestrates the complete pipeline from upload to intelligent response

What typically requires complex document processing and RAG engineering becomes simple upload and query. Your documents transform into a searchable knowledge base within seconds.

Setting up the document processing and RAG components

Now that you understand the concepts, let's examine the technical implementation that makes everything work.

System requirements

Performance estimates are rough approximations based on system understanding and real-world usage patterns.

Putting it all together

Here's our enhanced docker-compose.yml that adds complete document processing and RAG capabilities to our foundation:

version: '3.8'

services:
  postgres:
    image: postgres:15-alpine
    container_name: postgres
    ports:
      - '5432:5432'
    environment:
      - POSTGRES_DB=openwebui
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=securepassword123
    volumes:
      - ./volumes/postgres/data:/var/lib/postgresql/data
    restart: unless-stopped
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U postgres -d openwebui']
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - local-ai-network

  pgadmin:
    image: dpage/pgadmin4:latest
    container_name: pgadmin
    ports:
      - '5050:80'
    environment:
      - PGADMIN_DEFAULT_EMAIL=admin@local.ai
      - PGADMIN_DEFAULT_PASSWORD=admin123
      - PGADMIN_CONFIG_SERVER_MODE=False
      - PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED=False
    volumes:
      - ./volumes/pgadmin:/var/lib/pgadmin
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
    networks:
      - local-ai-network

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: webui
    ports:
      - '3000:8080'
    volumes:
      - ./volumes/open-webui/data:/app/backend/data
    environment:
      # Ollama connection
      - OLLAMA_BASE_URL=http://host.docker.internal:11434

      # Database connection
      - DATABASE_URL=postgresql://postgres:securepassword123@postgres:5432/openwebui

      # Basic settings
      - WEBUI_SECRET_KEY=your-secret-key-here
      - WEBUI_AUTH=true
      - ENABLE_SIGNUP=true
      - DEFAULT_MODELS=qwen3:8b
      
      # Document Processing + RAG Configuration (NEW)
      - UPLOAD_DIR=/app/backend/data/uploads
      - ENABLE_PERSISTENT_CONFIG=false
      - CONTENT_EXTRACTION_ENGINE=tika
      - TIKA_SERVER_URL=http://tika:9998
      - VECTOR_DB=qdrant
      - QDRANT_URI=http://qdrant:6333/
      - QDRANT_COLLECTION_PREFIX=document_chunks
      - RAG_EMBEDDING_MODEL=nomic-embed-text
      - RAG_EMBEDDING_ENGINE=ollama
      - ENABLE_RAG_HYBRID_SEARCH=false
      - RAG_RELEVANCE_THRESHOLD=0.75
      - CHUNK_SIZE=1000
      - CHUNK_OVERLAP=100

    extra_hosts:
      - 'host.docker.internal:host-gateway'
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
    networks:
      - local-ai-network

  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant
    ports:
      - '6333:6333'
      - '6334:6334'
    environment:
      - QDRANT__SERVICE__HTTP_PORT=6333
      - QDRANT__SERVICE__GRPC_PORT=6334
    volumes:
      - ./volumes/qdrant/storage:/qdrant/storage
    restart: unless-stopped
    networks:
      - local-ai-network

  tika:
    image: apache/tika:latest-full
    container_name: tika-server
    ports:
      - '9998:9998'
    environment:
      - TIKA_CONFIG=/opt/tika/config/tika-config.xml
      - JAVA_OPTS=-Xmx2g -Xms512m -Dfile.encoding=UTF-8 -Djava.awt.headless=true
      - TIKA_OCR_LANGUAGE=eng
      - TIKA_PDF_OCR_STRATEGY=OCR_AND_TEXT_EXTRACTION
    volumes:
      - ../shared/tika/config:/opt/tika/config
    command:
      - --host=0.0.0.0
      - --port=9998
      - --config=/opt/tika/config/tika-config.xml
    restart: unless-stopped
    networks:
      - local-ai-network

networks:
  local-ai-network:
    driver: bridge

What's new from Part 1:

This Docker Compose builds on your existing Part 1 setup by adding two key containers:

• Qdrant: Vector database for storing document embeddings

• Apache Tika: Text extraction engine for documents like PDFs, Word docs, and text files

Key configuration changes:

• ENABLE_PERSISTENT_CONFIG=false: Lets these new document processing + RAG settings override Part 1 configs

• CONTENT_EXTRACTION_ENGINE=tika: Routes document processing through the new Tika container

• RAG_RELEVANCE_THRESHOLD=0.75: Sets search quality threshold

• QDRANT_COLLECTION_PREFIX=document_chunks: Names your vector storage collections

• TIKA_PDF_OCR_STRATEGY=OCR_AND_TEXT_EXTRACTION: Extracts machine-readable text first, then uses OCR for scanned content within documents

• CHUNK_SIZE=1000 & CHUNK_OVERLAP=100: Document splitting settings for better search

Tika Configuration

The repository includes a pre-configured tika-config.xml file mounted into the Tika container via the volume mapping ../shared/tika/config:/opt/tika/config (shown on line 305 of the Docker Compose file). This configuration contains simple yet optimized settings for document processing, including PDF text extraction and OCR fallback capabilities, plus resource limits to prevent excessive usage during document processing.

Testing your document processing + RAG setup

Time to verify everything works as expected. Let's systematically test each component:

1. Container health check

docker compose ps

All five containers should show as running:

• postgres (healthy)

• pgadmin

• webui

• qdrant

• tika-server

2. Verify service endpoints

Qdrant Database:

• Health check: Open http://localhost:6333/healthz - should show "healthz check successful"

• Dashboard: Browse http://localhost:6333/dashboard - explore your document collections and vector data

Tika server:

• Quick version check: Open http://localhost:9998/version - shows Tika version info

• All available routes: Browse http://localhost:9998/ - lists all Tika API endpoints

Ollama with both models:

ollama list

Should list both qwen3:8b and nomic-embed-text.

3. Upload and test different document types

1. Navigate to http://localhost:3000

2. Sign in with your existing account

3. Look for the document upload icon in the bottom left corner of the chat

4. Upload documents: PDFs, Word docs, or text files

5. Wait for processing to complete

6. Ask specific questions about the document content

Test questions for document processing:

• "What's the total amount on this invoice?"

• "Extract the key information from this document"

• "What are the main points in this document?"

Test questions for RAG analysis:

• "What are the main topics covered in this document?"

• "Summarize the key findings"

• "What does the document say about [specific topic]?"

4. Verify vector storage

Check that Qdrant is storing your document vectors: http://localhost:6333/dashboard#/collections

You should see collections created for your uploaded documents.

5. View extracted text (optional)

Want to see exactly what text was extracted from your uploaded documents? This helps verify text extraction accuracy (including OCR when used) and understand what content the AI uses.

1. Visit http://localhost:5050

2. Login with admin@local.ai / admin123

3. If you haven't already connected to the database, add a server connection:

   1. Right click on Servers → Register → Server

   2. General tab → Name: local-ai

   3. Connection tab:

      • Host name/address: postgres

      • Port: 5432

      • Maintenance database: postgres

      • Username: postgres

      • Password: securepassword123

   4. Click Save

4. Navigate to local-ai → Databases → openwebui → Schemas → public → Tables → file

5. Right-click on the file table and select View/Edit Data → All Rows

6. Look at the data column to see the extracted text from your uploaded documents

What you'll see:

• The filename column shows your original file names

• The data column contains the full extracted text from each document

• This is exactly what the AI uses to answer your questions about the documents

Troubleshooting tip: If the data column is empty or contains gibberish, the document may be corrupted or contain primarily images. This setup works best with standard document formats and doesn't process standalone image files.

Troubleshooting common issues

What's next

You've successfully built a sophisticated document processing and RAG-enabled document intelligence system! Your self-hosted stack now includes:

✅ Ollama serving your LLM and embedding models

✅ Open WebUI with complete document intelligence

✅ PostgreSQL storing conversations and configurations

✅ Qdrant powering semantic document search and retrieval

✅ Apache Tika providing text extraction from documents like PDFs, Word docs, and text files (with OCR capabilities for scanned content)

Coming up in Part 3

We'll build a visual data extraction pipeline with:

• Web upload interface for batch image processing

• N8N workflows orchestrating the extraction pipeline

• Vision Language Models (VLM) via Ollama for analyzing images

• PostgreSQL storage for tracking jobs and extracted structured data

• Automated job processing with retry logic and error handling

This adds the ability to extract structured data from images at scale, perfect for processing receipts, invoices, forms, or any visual documents.

Homework before Part 3

Explore your new document processing and RAG capabilities:

• Test document processing: Upload PDFs, Word docs, and text files

• Test RAG: Upload various documents and ask complex questions

• Multi-language: Review the tika-config.xml file in self-hosted-ai-stack/shared/ directory to understand language settings, add new languages to it, then test documents in different languages (hands-on way to learn Tika configuration)

• Multi-format: Try Word docs, PowerPoint slides, Excel sheets (remember: this processes standard document formats, but not standalone image files)

• Advanced queries: Try complex multi-document queries across different formats

• Monitor processing: Browse vector collections in Qdrant's web interface at http://localhost:6333/dashboard

Helpful resources

• Open WebUI RAG Documentation

• Qdrant Documentation

• Apache Tika Supported Formats

• Ollama Embedding Models

This is part of my "Complete Self-Hosted AI Infrastructure" series. Follow along as we build increasingly sophisticated AI capabilities, all running self-hosted on your machine.

By the end of this post, you'll have a fully operational chat interface connected to the LLM of your choice, running completely locally. No external APIs, no data leaving your machine. Just pure local AI power.

We'll implement and orchestrate the foundational self-hosted LLM inference tools that form the backbone for everything we'll build in this series.

The bigger picture

I'm currently building a self-hosted AI stack from scratch, and I'm documenting everything as I go. I want to capture the knowledge while it's fresh and share the real-world lessons I've learned along the way.

Here's the thing: even with today's resources, piecing together a solid self-hosted AI stack is surprisingly challenging. The more complex your needs get, the scarcer the guidance becomes. This series is the comprehensive guide I wish I had when I started this journey. At least this way, people like you and me who are totally new to it can get up and start quickly, then evolve and expand from there.

What's coming up (outline may evolve as I discover better integration approaches while writing):

• Part 2: Document Processing and RAG with Apache Tika and Qdrant Vector Database

• Part 3: Visual Data Extraction: Web upload interface with N8N workflows and Vision Language Models for extracting structured data from images

• Part 4: Model Superpowers: Advanced WebUI configuration with tools and knowledge integration

• Part 5: Intelligent Automation: WebUI filters and N8N workflows for content processing

Prerequisites

• A machine with at least 16GB of RAM and 50GB of disk space

• Docker installed

• GPU recommended for better performance. Without a GPU, expect slower responses (10-30 seconds vs 1-3 seconds) as models run on CPU. Still totally usable, just requires patience. If performance is too slow, consider downloading a smaller model.

Credits where credits are due

Massive thanks to the creators and contributors of these incredible open source projects that make all this possible:

• Ollama - Repository: ollama/ollama

• Open WebUI - Repository: open-webui/open-webui

• PostgreSQL - Repository: postgres/postgres

• pgadmin - Repository: pgadmin-org/pgadmin4

• Docker - Repositories: moby/moby & docker/compose

Our foundation stack

We're building with battle-tested open source tools that work beautifully together. This foundation stays simple and robust, perfect for expanding on in later parts.

Here's what we're working with:

• Ollama - Self-hosted LLM hosting made simple

• Open WebUI - Beautiful and feature-rich chat UI for LLMs

• PostgreSQL - Reliable data storage for conversations and configs

• pgadmin - Web-based administration UI for PostgreSQL

• Docker & Docker Compose - Container orchestration to tie it all together

Why these specific tools? They're what I'm actually running in production, and I know their quirks inside and out. Feel free to swap any component for your preferred alternatives. The architecture stays the same.

Quick start (skip explanations, just get it running)

1. Install Ollama: Head to https://ollama.com/download

   Mac users: Use the app instead of Docker. My experience with Docker on an M4 MacBook was rough. The containerized version couldn't access my GPU, making everything painfully slow.

2. Download a model (this pulls and runs it):

   ollama pull qwen3:8b
   

   Why qwen3:8b? It hits the sweet spot between performance and compatibility. Since LLM usage demands significant computing power, this model is small enough to run on most machines without specialized hardware. Check out other options at https://ollama.com/search.

3. Verify your model downloaded:

   ollama list
   

   You should see qwen3:8b listed in the output:

   NAME                                   ID              SIZE      MODIFIED    
   qwen3:8b                               bdbd181c33f2    5.3 GB    1 hour ago
   

   Starting Ollama: If you ever need to get Ollama running, you can either start the app from your Applications folder or run ollama serve in your terminal. If you see "Error: listen tcp 127.0.0.1:11434: bind: address already in use", that means Ollama is already running.

4. Clone the repository:

   git clone https://github.com/FarzamMohammadi/self-hosted-ai-stack
   

5. Navigate to the foundation:

   cd part-1-building-the-foundation
   

6. Fire it up (make sure Docker is running):

   docker compose up -d
   

That's it! Open http://localhost:3000, sign up, select qwen3:8b in the top-left corner, and start chatting with your self-hosted AI.

Understanding what you built

Ollama

Ollama makes self-hosted LLM hosting dead simple. It's built on top of the excellent llama.cpp project, trading some customization for incredible ease of use. Installation takes minutes, and Ollama automatically detects the optimal configuration for your hardware. No tweaking required. Just download and run.

Installation options

You can run Ollama via Docker or as a native app, but this guide uses the native app since that's what our docker-compose setup expects.

Download the native app from https://ollama.com/download and install it following the standard process for your operating system.

Why I go with the native app

My experience with Docker on an M4 MacBook was pretty disappointing. The containerized version couldn't tap into my machine's GPU, turning what should be snappy responses into sluggish waits. The native app, however, plays nicely with macOS's Metal framework and delivers the performance you'd expect.

Choosing your model

The model library at https://ollama.com/search is extensive. You can pull any listed model with a simple command, or even create custom models using Modelfiles.

For this tutorial, I'm using qwen3:8b. It strikes a great balance between capability and resource requirements:

ollama pull qwen3:8b

Docker setup alternative

If you prefer setting it up via Docker, here's an example setup:

services:
  ollama:
    image: docker.io/ollama/ollama:latest
    ports:
      - 7869:11434
    volumes:
      - ./ollama/ollama:/root/.ollama
    container_name: ollama
    pull_policy: always
    tty: true
    restart: always
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_HOST=0.0.0.0
    networks:
      - ollama-docker

networks:
  ollama-docker:
    external: false

Source: mythrantic/ollama-docker

PostgreSQL & pgadmin

PostgreSQL becomes our data backbone, storing conversations, configurations, and eventually (in later parts) data for additional services we'll integrate. I went with PostgreSQL because it's bulletproof reliable and Open WebUI supports it excellently. pgadmin gives us a clean web interface for database exploration.

The configuration

Keeping it straightforward with just the essentials:

services:
  postgres:
    image: postgres:15-alpine
    container_name: postgres
    ports:
      - '5432:5432'
    environment:
      - POSTGRES_DB=openwebui
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=securepassword123
    volumes:
      - ./volumes/postgres/data:/var/lib/postgresql/data
    restart: unless-stopped
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U postgres -d openwebui']
      interval: 10s
      timeout: 5s
      retries: 5

  pgadmin:
    image: dpage/pgadmin4:latest
    container_name: pgadmin
    ports:
      - '5050:80'
    environment:
      - PGADMIN_DEFAULT_EMAIL=admin@local.ai
      - PGADMIN_DEFAULT_PASSWORD=admin123
      - PGADMIN_CONFIG_SERVER_MODE=False
      - PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED=False
    volumes:
      - ./volumes/pgadmin:/var/lib/pgadmin
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy

Configuration notes:

• Using postgres:15-alpine for the right balance of size and features

• The healthcheck prevents connection race conditions

• pgadmin runs in desktop mode since this is local-only

• Simple credentials for localhost. Change them if you expose this externally

Open WebUI

Open WebUI delivers the ChatGPT experience, basically a nice chat interface for streamlined interaction with LLMs, but running entirely on your machine. After testing various interfaces (text-generation-webui, SillyTavern, and others), Open WebUI won me over with its clean design and incredible customization options.

What makes it special

• Clean interface that actually works without fuss

• Built-in RAG support (we'll dig into this in Part 2)

• Seamless Ollama integration

• Excellent APIs for integration with your own projects

• Active development with a supportive community

Setting it up

Here's the configuration that ties everything together:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: webui
    ports:
      - '3000:8080'
    volumes:
      - ./volumes/open-webui/data:/app/backend/data
    environment:
      # Ollama connection
      - OLLAMA_BASE_URL=http://host.docker.internal:11434
      
      # Database connection
      - DATABASE_URL=postgresql://postgres:securepassword123@postgres:5432/openwebui
      
      # Basic settings
      - WEBUI_SECRET_KEY=your-secret-key-here
      - WEBUI_AUTH=true
      - ENABLE_SIGNUP=true
      - DEFAULT_MODELS=qwen3:8b
      
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy

Note on extra_hosts: This setting is required because we're running Ollama as a native app (not in Docker). If you choose to run Ollama in Docker instead, remove the extra_hosts section and update OLLAMA_BASE_URL to use the container name (e.g., http://ollama:11434).

Key settings explained:

• OLLAMA_BASE_URL points to our native Ollama app via host.docker.internal

• Database URL connects to our PostgreSQL container

• Auth is enabled. Disable for single-user setups if preferred

• DEFAULT_MODELS should match your downloaded model

Putting it all together

Time to wire everything up. Here's the complete docker-compose.yml that orchestrates our entire self-hosted AI stack.

The complete configuration

Create a new directory for your project and drop in this docker-compose.yml:

version: '3.8'

services:
  postgres:
    image: postgres:15-alpine
    container_name: postgres
    ports:
      - '5432:5432'
    environment:
      - POSTGRES_DB=openwebui
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=securepassword123
    volumes:
      - ./volumes/postgres/data:/var/lib/postgresql/data
    restart: unless-stopped
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U postgres -d openwebui']
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - local-ai-network

  pgadmin:
    image: dpage/pgadmin4:latest
    container_name: pgadmin
    ports:
      - '5050:80'
    environment:
      - PGADMIN_DEFAULT_EMAIL=admin@local.ai
      - PGADMIN_DEFAULT_PASSWORD=admin123
      - PGADMIN_CONFIG_SERVER_MODE=False
      - PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED=False
    volumes:
      - ./volumes/pgadmin:/var/lib/pgadmin
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
    networks:
      - local-ai-network

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: webui
    ports:
      - '3000:8080'
    volumes:
      - ./volumes/open-webui/data:/app/backend/data
    environment:
      # Ollama connection
      - OLLAMA_BASE_URL=http://host.docker.internal:11434

      # Database connection
      - DATABASE_URL=postgresql://postgres:securepassword123@postgres:5432/openwebui

      # Basic settings
      - WEBUI_SECRET_KEY=your-secret-key-here
      - WEBUI_AUTH=true
      - ENABLE_SIGNUP=true
      - DEFAULT_MODELS=qwen3:8b

    extra_hosts:
      - 'host.docker.internal:host-gateway'
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
    networks:
      - local-ai-network

networks:
  local-ai-network:
    driver: bridge

Starting the stack

Three simple steps:

1. Make sure Docker is running

2. Confirm Ollama is active: Run ollama serve in your terminal. If it starts, great! If you see "Error: listen tcp 127.0.0.1:11434: bind: address already in use", that means Ollama is already running

3. Fire it up from your project directory:

docker compose up -d

The -d flag runs everything in the background. First startup takes a few minutes while Docker downloads the images.

Once everything is up and running, head to http://localhost:3000 and you'll need to create an account before you can start using the interface. The first user to sign up automatically becomes the admin.

Testing your setup

Let's verify everything works. Here's my quick validation routine:

1. Container health check

docker compose ps

All three containers should show as running. If any show "Exited", investigate with:

docker compose logs [service-name]

2. Verify Ollama connection

Quick test to ensure Ollama responds:

Mac/Linux/Windows Command Prompt:

curl http://localhost:11434/api/tags

Windows PowerShell:

Invoke-RestMethod http://localhost:11434/api/tags

Browser fallback: Open http://localhost:11434/api/tags in your browser

You should see your downloaded models in the response.

3. Access the web interface

1. Navigate to http://localhost:3000

2. Create an account (first user becomes admin automatically)

3. You should land on the clean chat interface

4. First conversation

1. The qwen3:8b model should be selected by default (since we set it as DEFAULT_MODELS in our docker-compose). If it isn't, you can find the model selector in the top-left corner

2. Choose qwen3:8b if not already selected

3. Send a test message and wait for the response

Slow responses? Check that Ollama is running and your model downloaded completely.

5. Peek at the database (optional)

Curious about what's happening under the hood?

1. Visit http://localhost:5050

2. Login with admin@local.ai / admin123

3. Add a server connection:

   1. Right click on Servers (in the left side menu) → Register → Server

   2. In the General tab → Name: local-ai

   3. Switch to the Connection tab and enter:

      • Host name/address: postgres

      • Port: 5432

      • Maintenance database: postgres

      • Username: postgres

      • Password: securepassword123

   4. Click Save

   5. Once connected, navigate to local-ai → Databases → openwebui → Schemas → public → Tables to see the OpenWebUI tables that were created automatically when it connected

Troubleshooting common issues

"Cannot connect to Ollama"

• Verify Ollama is actually running (check system tray)

• If using Docker Ollama, double-check port mappings

"Database connection failed"

• Give PostgreSQL more initialization time

• Confirm postgres container health: docker compose ps

"Port already in use"

• Modify port mappings in docker-compose.yml

• Stop whatever service is occupying those ports

"Performance is awful"

• Usually means Ollama can't access your GPU

• On Mac, stick with the native app over Docker

• Try a smaller model if resources are tight

What's next

Congratulations! You've built a solid foundation for your self-hosted AI stack. You're now running:

✅ Ollama serving your self-hosted LLM
✅ Open WebUI providing a beautiful chat interface
✅ PostgreSQL storing conversations and configurations
✅ pgadmin for database management

This foundation gives us a rock-solid base for the advanced capabilities we'll add in upcoming parts.

Coming up in Part 2

We'll supercharge our setup with RAG (Retrieval-Augmented Generation) by adding:

• Apache Tika for document processing (PDFs, Word docs, images, etc.)

• Qdrant vector database for semantic search

• Document upload and intelligent retrieval through Open WebUI

Everything we've built today will integrate seamlessly with these new components.

Homework before Part 2

Take some time to explore what we've built:

• Try different models (llama3.2, codellama, mistral)

• Experiment with Open WebUI's settings and themes

• Upload some documents and see how basic file handling works

• Poke around the conversation history in pgadmin

Helpful resources

• Open WebUI Documentation

• Ollama Model Library

• PostgreSQL Basics

This is part of my "Complete Self-Hosted AI Infrastructure" series. Follow along as we build increasingly sophisticated AI capabilities, all running self-hosted on your machine.

Howdy! I'm back with some fresh insights and experiences to share. This time, we're diving deep into the world of release deployments, with a special focus on a strategy that might just revolutionize your approach: Blue-Green deployments.

Blue-green deployment is a release strategy used in software development to reduce downtime and risk during updates.

Source: Blue-Green Deployment

But wait, there's more! I'm not flying solo on this mission. My trusty wingman and team lead extraordinaire, Jaime Valencia, has joined forces with me to bring you this piece. Why, you ask? Well, we didn't just theorize about this strategy – we rolled up our sleeves, designed it, and implemented it in our own organization. And let me tell you, the benefits were so mind-blowing that we couldn't keep this knowledge to ourselves!

By going through the entire lifecycle of developing and implementing this strategy, we gained invaluable insights into its advantages, challenges, and important considerations. In the end, we completely transformed our deployment process, making life easier not just for ourselves, but for all teams involved in deployments. Oh, and did I mention we also supercharged the overall quality of our deliveries?

Now, I know what you're thinking: "Blue-Green deployments? That's not exactly breaking news!" And you're right. If you Google it right now, you'll find a smorgasbord of examples and implementations. But here's the kicker – most of them feel like they're missing something. They're often tailored for individual projects, showing you how to leverage specific containerization features or orchestration mechanisms to achieve a so-called Blue-Green strategy.

For us, the real challenge wasn't implementing this strategy for individual projects. No, no, no. Our Mount Everest was implementing a strategy that could cover our entire project ecosystem. We're talking about deploying multiple interconnected services together and creating a complete isolated testing environment – all without disturbing a single hair on your real users' heads.

So, buckle up! We're about to show you how to set up a Blue-Green release deployment strategy for your entire project suite. We'll walk you through deploying multiple interconnected services together, creating a completely isolated testing environment, and doing it all without your real users even noticing. This approach doesn't just let you test and validate your upcoming production release – it gives you the luxury of time to address any issues that pop up in testing, well ahead of your production release due date.

Is This for Me? Why Should I Keep Reading?

Let's play a quick game of "Does This Sound Familiar?":

• Do your software releases sometimes feel like you're defusing a bomb while blindfolded?

• Have you ever wished for just a little more time to test your changes in staging, convinced it would save you a ton of stress and trouble?

• Does it seem like no matter how hard you try, you just can't create an environment that truly mimics production?

• Do you break out in a cold sweat every time you hear the words "deployment day"?

If you nodded your head to even one of these, then congratulations (or condolences?) – this article is definitely for you! And even if you're sitting there thinking, "Nah, my deployments are smooth as butter," I'd still encourage you to read on. Who knows? You might pick up a new trick or two, or at the very least, learn some fancy new deployment lingo to impress your colleagues at the next standup.

Benefits: What's In It for Me?

Alright, let's cut to the chase. Here's what you stand to gain by implementing a Blue-Green deployment strategy:

1. Zero User Impact: Your users won't even know you're updating. It's like changing the tires on a moving car, but way less dangerous.

2. Zero Downtime: As soon as your new version is ready, traffic starts flowing to it immediately. It's like teleportation, but for your software.

3. Multi-Release Flexibility: If your new deployment turns out to be a dud, no worries! You can redeploy a new version or wipe the slate clean without affecting a single user in your live environment. It's like having a time machine for your deployments.

4. Stable and Secure Releases: By keeping your Beta environment separate from Live services, you can test thoroughly and securely. It's like having a safety net, a bulletproof vest, and a lucky charm all rolled into one.

5. Efficient Management: You're in the driver's seat when it comes to how and when changes are promoted from Beta to Live. It's like being the air traffic controller for your software releases.

6. Flexible Deployment Scheduling: Deploy to your Beta environment anytime, independent of your Live service schedule. Test and verify to your heart's content, well ahead of your live deployment date. It's like rehearsing for a play with an infinitely patient audience.

Challenge: If It Was That Easy, Everyone Would Be Doing It

Now, I know what you're thinking: "This sounds too good to be true!" And you're right to be skeptical. Implementing a Blue-Green deployment strategy isn't without its challenges. Let's take a look at some of the hurdles you might face:

Shared Services: The Isolation Conundrum

One of the main challenges in preparing an isolated beta environment is, well, isolating all the services in the environment. The fewer shared services you have, the more isolated your beta environment becomes.

Take your system's message broker, for example. To isolate its communication solely to beta components, you need to allocate one exclusively for your beta resources. The same process applies to other shared services. Create separate ones for beta where you can and set your beta resources to use them instead of the production ones.

Once your beta environment is ready for promotion to live, you can bid farewell to these temporary services. We'll discuss the creation and removal process further under 'Ensuring consistent configuration across both environments' below.

It's important to note that some resources can't be easily created just for the sake of beta environment isolation. Even if they could be, integrating them into the beta resources isn't always straightforward. A common example is a database. This is where you need to consider your options carefully. If you decide to share them between both live and beta, do so with proper planning and precautions in mind.

Near-Zero Downtime: The Quest for Seamlessness

While zero-downtime is the holy grail, the reality is that there's potential for brief service interruptions during the transition, depending on your configuration. You'll need to customize your setup based on your specific needs and the tools you're using.

If you take away one thing from this entire blog regarding Blue-Green deployments, let it be this: implementing the Blue-Green strategy isn't the toughest part – it's refining it to the point of seamless deployment that's the real challenge.

Forward/Backward Compatibility: The Time-Traveler's Dilemma

Managing database schema changes and migrations is another significant challenge. You and your team must always keep forward and backward compatibility in mind when making changes. This is particularly crucial when it comes to database changes.

Forward compatibility: A system's ability to accept input intended for a future version of itself without breaking.

Backward compatibility: A system's ability to work with input from an older version of itself.

In database contexts:

• Forward: Schemas that can handle new fields added in future versions.

• Backward: New schema changes that don't break existing queries or data structures.

Maintaining both types of compatibility allows for smoother system evolution and reduces risks during updates or migrations.

Rollback Procedures: The "Oops" Button

A contingency rollback plan must be in place in case of emergency. This can pose a challenge if the Blue-Green strategy you employ is only prepared to deploy forward. It's like having an ejector seat in your car – you hope you never have to use it, but you'll be glad it's there if you need it.

Ensuring Consistent Configuration: The Clone Wars

This is mainly an issue if you're not already using Infrastructure as Code (IaC). There's no doubt it's a learning curve, but to implement a Blue-Green strategy, I'd strongly advise you to invest the time. Once you set up one environment for deployment via IaC, you can mirror it by refactoring it into a template equivalent and use it for both Beta and Live deployment.

The best part? Your releases will always be uniform! It's a huge gain! Plus, it's great to add IaC to your toolbelt if you haven't already!

Increased Infrastructure and Hosting Costs: The Price of Peace of Mind

You will have to maintain two full production environments simultaneously when both beta and live are up, there's no doubt about that. But given the benefits, I can promise you that any manager or higher-up who needs to approve your proposal will dive in headfirst when you show them the advantages. Nothing beats peace of mind, especially for the bigwigs. (Just kidding... sort of.)

Implementation: Enough, You've Sold It, Show Me How!

If you've read this far, then buckle up – you're in for a ride! First, we'll show you a few different methods of implementation. Then, we'll walk you through our approach. If you're in a hurry, you can check out the repository here: blue-green-with-k8s. The README there contains all the commands needed to quickly set up this deployment strategy.

Implementation Methods: How Many Ways Are There?

We personally implemented this strategy in two different methods before creating this third one for you.

Services Based Approach - As a Unit

Our first implementation, which was really our introduction to Blue-Green deployments, revolved around our application's search functionality. We revamped our entire release deployment process and reduced its outage and preparation time by implementing a Blue-Green strategy for our Search Engine and its related services. We focused solely on three interconnected services, creating those 3 services as one unit (aka color) together.

In this approach, we would deploy an Azure Search Service (Azure's Search Engine service), a custom event-driven, container-based Search Engine indexer, and a Job Scheduler service that managed both the Azure Search Service and the custom indexer. Each new deployment would create this unit of resources as a new color.

This was our introduction to the potential of Blue-Green deployments!

Entire Project Suite - Individual Yet Interconnected with Azure Container Apps

Leveraging the knowledge gained in our first implementation, we implemented a similar approach for an entire new project suite that we got to work on and experiment with.

This time, we leveraged Azure Container Apps for Blue/Green version management. We would deploy new beta revisions to existing containers, to co-exist alongside stable revisions in use by users. This leverages the multi-revision feature of Azure Container Apps to allow us to have two revisions simultaneously in action. However, by always redirecting traffic solely to the Live aka Stable revision, live users only interacted with the Stable version of our applications. At the same time, we were able to test our Beta version!

This implementation was more intricate, and if you're planning to implement blue-green for your project suite in production, chances are something like this will be more appropriate for you, especially if you use or plan on using Azure Container Apps. A basic implementation can be found in Microsoft Docs here: Blue-Green Deployment in Azure Container Apps. I'd strongly suggest you use that solely for introductory purposes only. For a real implementation, make sure to test thoroughly and customize based on your needs.

It's worth noting that this implementation is on my todo list as well, which I'll be working on later.

Entire Project Suite - Individual Yet Interconnected with Kubernetes

This is the new implementation that we'll be walking you through today. A step-by-step implementation of a Blue-Green strategy via Kubernetes with three services: a database, a frontend UI service, and a backend API service.

The next section describes the step-by-step implementation.

Step-By-Step Implementation: Show Me the Money!

We'll be deploying a suite of projects using Blue-Green deployment. In our example, we will first deploy a stable version, which will be our initial live deployment. Then, we'll make some small changes in a separate branch and deploy that as our Beta version. Once we see them side by side, both in use and action, we'll then promote our Beta to Live, emulating a real production deployment. In our example, the database will remain shared between both live and beta services, while the frontend and backend services will be created anew for each version.

To help visualize this process, let's look at a simplified illustration of our Blue-Green deployment strategy:

Source: Blue-Green Deployment

This diagram shows how user traffic is initially directed to the "blue" (stable) environment. Once we're confident in our beta version, we can seamlessly switch traffic to the "green" (beta) environment, which then becomes our new stable version.

We've prepared all the steps for you and consolidated the deployment into one script. First, I'll walk you through our services. Then I'll show our deployment Kubernetes configuration files to see what we're deploying. Finally, I'll walk you through the actual deployment process. This is where we'll examine our implementation of Blue-Green and see for ourselves on our local machine how easy it is to get started!

The database has two tables:

1. Greetings

2. Farewells

First, we'll deploy our stable (starting) version of the services which retrieve and display the first record it finds in our Greetings table.

Then, emulating a feature version in a separate branch, we revise our backend and frontend services to also retrieve and display the first record it finds in the Farewells table, alongside the Greetings table.

After deploying our feature branch, we'll see the simultaneous deployment results first hand, and together, promote the feature branch (aka Beta environment) to our Live environment, overriding the initial version that solely displayed the first Greetings record.

Assumptions

I'm assuming you're familiar with the following technologies. If not, don't worry! A quick Google search should get you up to speed. If I can learn it, it'll be a piece of cake for you!

• Bash Scripting

• Kubernetes

• Docker

  • Make sure to enable the Docker Kubernetes cluster

• YAML

Step 1: Creation of Dockerfiles

Containerization of your services is a vital step that cannot be bypassed. If containerization is new to you, you're in for a treat. Get busy learning it right now because it's a must-have in your toolbelt.

We created two Dockerfiles for our frontend and backend services. Nothing fancy about them – they containerize our backend and frontend services in the simplest manner possible.

Backend: Dockerfile

Frontend: Dockerfile

Our database is MariaDB, an open-source database that's already available on Docker Hub, where we can simply pull and deploy it from. Shoutout to all the open-source contributors out there. You rock!

Step 1.1: Automating Image Preparation with build-images Bash Script

For MariaDB, since the image is already available on Docker Hub, all we need to do is include the image path in our Kubernetes configuration file, which you'll see in the next step. However, for our own backend and frontend services, we must prepare and build them ourselves.

For building our backend and frontend images, we've created an automated bash script: build-images.sh. This script will be utilized in our deployment-manager script later on, to build, tag, and push our images to our local Docker instance using random tags.

Step 2: Creation of Kubernetes Config Files

Our standard method of operation is to first create default deployment files. These will allow us to test our default non-Blue-Green deployments well before creating a template file to use for Blue-Green deployments.

Here are the default non-template deployment files:

Backend: backend.yaml

Frontend: frontend.yaml

Database: db.yaml

Explanation

Backend & Frontend

For both our backend and frontend services, we create two Kubernetes resources:

1. Deployment: This resource manages the creation and scaling of pods running our application.

2. Service: This resource exposes our deployment, making it accessible both within the cluster and externally.

These are configured as follows:

• Backend: Internally accessible at port 8080, externally at port 30001

• Frontend: Internally accessible at port 3000, externally at port 30002

This setup allows our services to communicate internally and be accessible from the outside world.

By creating these non-template files first, we can thoroughly test our basic setup before moving on to the more complex Blue-Green deployment configuration.

Database

For our database, we've created a more complex setup using MariaDB. Here's a breakdown of the resources:

1. ConfigMap (mariadb-init): This contains our SQL initialization script. It creates our blue_green database and sets up two tables: greetings and farewells, each populated with one record.

2. PersistentVolumeClaim: This requests 300MB of storage for our database, ensuring data persistence across pod restarts.

3. Service: This exposes our MariaDB internally and externally:

   • Internal port: 3306

   • NodePort (external access): 30306

4. Deployment: This deploys our MariaDB container:

   • Uses MariaDB version 11.4

   • Sets up environment variables for the root password

   • Mounts volumes for data persistence and initialization

The initialization script (in the ConfigMap) runs on database creation, setting up our required schema:

CREATE DATABASE IF NOT EXISTS blue_green;

USE blue_green;

CREATE TABLE IF NOT EXISTS greetings (
  id INT PRIMARY KEY NOT NULL AUTO_INCREMENT, 
  greeting VARCHAR(100)
);

INSERT INTO greetings (greeting) VALUES ('hello, hi, how are yah?');

CREATE TABLE IF NOT EXISTS farewells (
  id INT PRIMARY KEY NOT NULL AUTO_INCREMENT, 
  farewell VARCHAR(100)
);

INSERT INTO farewells (farewell) VALUES ('adios, good bye, see yah?');

In short, we take care of the entire database creation, preparation, and setup in this deployment file. Absolutely no future changes are needed, allowing us to focus on learning the deployment processes instead of getting bogged down with database setup that's not relevant to our main goal.

Step 3: Creation of Kubernetes Config Templates from Config Files

Since only one version of the database will be deployed, there's no need to create a database Kubernetes configuration template file. However, since we are deploying the backend and the frontend dynamically, we will need to leverage the benefits a template provides for the corresponding Kubernetes configuration files.

Backend: backend.yaml.template

Frontend: frontend.yaml.template

Explanation

Backend

We add a {{STATUS}} variable that gets overwritten by our deployment type, i.e., beta or stable.

We replace the image tag with {{IMAGE_TAG}}, which will be overwritten by the randomly generated image tags we'll be creating for our beta and live services.

Frontend

Similar to the backend, we add {{STATUS}} and {{IMAGE_TAG}}, that override the deployment type and the image tag respectively.

For the frontend, however, we also added a ConfigMap that overrides our frontend's runtime environment variable. This overrides the {{BACKEND_NODE_PORT}} and dynamically sets the API URL that the frontend will be pointing to.

apiVersion: v1
kind: ConfigMap
metadata:
  name: frontend-config-{{STATUS}}
  labels:
    app: b-g
    component: frontend
    status: {{STATUS}}
data:
  config.js: |
    window.RUNTIME_CONFIG = {
      API_URL: "http://localhost:{{BACKEND_NODE_PORT}}",
    };

Step 4: Deployment of Live Services (Starting Point - First Stable Deployment)

Execution

From this step onward, we'll be using the deploy-manager bash script that automates our entire deployment process. We'll explain every single step to make sure you grasp its processes in depth as we go along.

To get started, make sure to have Docker running, with the Kubernetes cluster enabled. You must also have the ability to run bash scripts, for which plenty of documentation already exists.

At the root of our blue-green-with-k8s repo, we run the following command to deploy our Live services:

./deploy-manager.sh deploy stable

Result

The output will look something like this, with randomly allocated URLs where we can access our services on our local machine:

==============================
  Current Deployment Status
==============================

NAME                READY   UP-TO-DATE   AVAILABLE   AGE
b-g-backend-stable   1/1     1            1           2m13s
b-g-frontend-stable  1/1     1            1           2m11s
b-g-mariadb          1/1     1            1           2m14s

--- Current Services ---
NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)
AGE
b-g-backend-service-stable   NodePort    10.104.19.168   <none>        8080:30
951/TCP   2m13s
b-g-frontend-service-stable  NodePort    10.100.205.149  <none>        3000:30
183/TCP   2m11s
kubernetes                   ClusterIP   10.96.0.1       <none>        443/TCP
7m27s
mariadb-service              NodePort    10.109.117.1    <none>        3306:30
306/TCP   2m14s

--- Access URLs ---
Stable Backend: localhost:31543
Stable Frontend: localhost:32693
MariaDB: localhost:30306

Stable Environment Visualization

When we access the stable frontend URL (in this case, localhost:32693), we see the following:

• A "Greeting" section displaying: "hello, hi, how are yah?"

If we access the stable backend URLs directly, we see:

• At the /greeting endpoint (localhost:31543/greeting): "hello, hi, how are yah?"

This demonstrates that our stable version is correctly retrieving and displaying the greeting message from the database.

Explanation

The ./deploy-manager.sh deploy stable command will trigger the following flow in the script:

deploy)
        if [ "$#" -ne 2 ] || [[ ! "$2" =~ ^(stable|beta)$ ]]; then
            print_error "Usage: $0 deploy <stable|beta>"

            show_exit_prompt
        fi

        handle_deployment "$2"
        ;;

This calls the handle_deployment function to deploy our live services. The function looks like this:

handle_deployment() {
    local deployment_type=$1

    local tag=$(generate_random_tag)

    build_images "$tag"

    deploy_db

    deploy "$deployment_type" "$tag"

    check_env "$deployment_type"
}

First, it creates two variables:

1. deployment_type, which is being set to stable

2. tag, a randomly generated string to be used as our image tag

Then, we build, tag, and push our backend and frontend images using the build_images "$tag" command, which uses our automated build-images bash script. This overrides the tag for both images to this new randomly generated one.

readonly BUILD_SCRIPT="./build-images.sh"

build_images() {
    local tag=$1

    if [[ ! -x "$BUILD_SCRIPT" ]]; then
        chmod +x "$BUILD_SCRIPT"
    fi

    print_header "Building Images"

    print_info "Building images with tag: $tag"

    "$BUILD_SCRIPT" true "$tag"

    print_success "Images successfully built with tag: $tag"
}

Next, we deploy the database using the deploy_db function, which deploys our db.yaml via the Kubernetes CLI. It waits until the database status becomes ready.

readonly DB_DEPLOYMENT="./deployments/local/db.yaml"

deploy_db() {
    print_header "Deploying Database"
    kubectl apply -f "$DB_DEPLOYMENT"

    print_info "Waiting for Database to be fully ready..."

    if kubectl wait --for=condition=ready --timeout=60s pod -l app=b-g,component=mariadb; then
        print_success "Database is ready!"
    else
        print_error "Database is not ready. Please check the deployment."
        show_exit_prompt
    fi
}

Finally, we deploy our backend and frontend services, using backend.yaml.template and frontend.yaml.template using the deploy() function. We pass our deployment type (stable) and a randomly generated image tag. This is where the magic happens! It's a bit verbose for debugging and UX sake, but still easy to follow.

readonly BACKEND_TEMPLATE="./deployments/local/backend.yaml.template"
readonly FRONTEND_TEMPLATE="./deployments/local/frontend.yaml.template"

Deploy() {
    local status=$1
    local image_tag=$2

    print_header "Deploying $status Environment"

    print_subheader "Deploying $status backend"

    sed -e "s/{{STATUS}}/$status/g" -e "s/{{IMAGE_TAG}}/$image_tag/g" "$BACKEND_TEMPLATE" | kubectl apply -f -

    print_info "Waiting for $status backend deployment to be available..."

    if ! kubectl wait --for=condition=available --timeout=60s deployment/b-g-backend-$status; then
        print_error "Backend deployment failed. Check the logs for more information."

        kubectl get pods -l app=b-g,component=backend,status=$status

        kubectl describe deployment b-g-backend-$status

        show_exit_prompt
    fi

    local service_name="b-g-backend-service-$status"

    print_info "Checking if $status backend service exists..."

    if kubectl get service "$service_name" &>/dev/null; then
        print_success "$status backend service exists."
    else
        print_error "Backend service $service_name does not exist. Check the configuration."

        show_exit_prompt
    fi

    local backend_node_port=$(kubectl get service "$service_name" -o jsonpath='{.spec.ports[0].nodePort}')

    print_subheader "Deploying $status frontend"

    sed -e "s/{{STATUS}}/$status/g" -e "s/{{IMAGE_TAG}}/$image_tag/g" -e "s/{{BACKEND_NODE_PORT}}/$backend_node_port/g" "$FRONTEND_TEMPLATE" | kubectl apply -f -

    print_info "Waiting for $status frontend deployment to be available..."

    if ! kubectl wait --for=condition=available --timeout=60s deployment/b-g-frontend-$status; then
        print_error "Frontend deployment failed. Check the logs for more information."

        kubectl get pods -l app=b-g,component=frontend,status=$status

        kubectl describe deployment b-g-frontend-$status

        show_exit_prompt
    fi

    print_success "$status deployment updated with image tag: $image_tag"

    print_info "Frontend configured to use backend at http://localhost:$backend_node_port"
}

In this function, we first deploy the backend template with stable status using the following line:

sed -e "s/{{STATUS}}/$status/g" -e "s/{{IMAGE_TAG}}/$image_tag/g" "$BACKEND_TEMPLATE" | kubectl apply -f -

The usage of status and image tag can be seen in the template.

Backend: backend.yaml.template

In short, the status is being used as an identifier to indicate which type of environment the backend service will be. In this case, it's stable, which denotes our Live environment. The image tag is of course being used to pull the version of backend service we want to deploy.

We then await the backend service's deployment until it becomes available. Once available, we retrieve its access port. We will use this value to override our frontend service's environment settings to ensure it's pointing to the correct service.

Next, we repeat the same steps, this time for our frontend service via the following line:

sed -e "s/{{STATUS}}/$status/g" -e "s/{{IMAGE_TAG}}/$image_tag/g" -e "s/{{BACKEND_NODE_PORT}}/$backend_node_port/g" "$FRONTEND_TEMPLATE" | kubectl apply -f -

This time, we not only pass the deployment status and frontend service image tag but also pass the backend node port so that our frontend service can access our backend.

We once again wait until the deployment of frontend services is complete and becomes available.

Once all services are successfully created, the script ends by running a status check using the show_status function. The function uses the Kubernetes CLI (kubectl) to get and display the status of the resources we deployed, as well as their ports.

show_status() {
    print_header "Current Deployment Status"
    kubectl get deployments

    print_subheader "Current Services"
    kubectl get services

    print_subheader "Access URLs"

    local services=("b-g-backend-service-stable" "b-g-backend-service-beta" "b-g-frontend-service-stable" "b-g-frontend-service-beta" "mariadb-service")

    local names=("Stable Backend" "Beta Backend" "Stable Frontend" "Beta Frontend" "MariaDB")

    for i in "${!services[@]}"; do
        local port=$(kubectl get service ${services[$i]} -o jsonpath='{.spec.ports[0].nodePort}' 2>/dev/null)

        if [ -n "$port" ]; then
            print_info "${names[$i]}: http://localhost:$port"
        fi
    done
}

And there you have it! That's how we deploy our Live services. In the next part, we'll look at deploying our Beta services and promoting them to Live.

Step 5: Deployment of Beta Services (New Feature)

Execution

To deploy a new version of our services for a better demonstration of the promotion process, we've created a separate branch with additional changes on top of our Live services to emulate a new version deployment.

First, switch from the main branch to feat/display-farewell, then we once again repeat the steps we did in the previous Deployment of Live services step. The only difference is that we have to pass the beta argument instead of stable when running the deploy-manager script.

./deploy-manager.sh deploy beta

Result

The output this time consists of both stable as well as beta services.

==============================
  Current Deployment Status
==============================

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
b-g-backend-beta      1/1     1            1           5s
b-g-backend-stable    1/1     1            1           2m12s
b-g-frontend-beta     1/1     1            1           2s
b-g-frontend-stable   1/1     1            1           2m9s
b-g-mariadb           1/1     1            1           2m14s

--- Current Services ---
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)
          AGE
b-g-backend-service-beta      NodePort    10.100.39.121    <none>        8080:31
209/TCP   6s
b-g-backend-service-stable    NodePort    10.104.191.229   <none>        8080:31
543/TCP   2m13s
b-g-frontend-service-beta     NodePort    10.103.147.219   <none>        3000:30
984/TCP   3s
b-g-frontend-service-stable   NodePort    10.109.144.9     <none>        3000:32
693/TCP   2m10s
kubernetes                    ClusterIP   10.96.0.1        <none>        443/TCP
          5h3m
mariadb-service               NodePort    10.109.29.175    <none>        3306:30
306/TCP   2m15s

--- Access URLs ---
Stable Backend: http://localhost:31543
Beta Backend: http://localhost:31209

Stable Frontend: http://localhost:32693
Beta Frontend: http://localhost:30984

MariaDB: http://localhost:30306

We now have two environments running simultaneously in parallel. Each is fully accessible and isolated to its own domain.

Beta Environment Visualization

When we access the beta frontend URL (in this case, localhost:30984), we see the following:

1. A "Greeting" section displaying: "hello, hi, how are yah?"

2. A "Farewell" section displaying: "adios, good bye, see yah?"

This demonstrates that our new feature branch has successfully added the farewell message to the frontend.

If we access the beta backend URLs directly, we see:

1. At the /greeting endpoint (localhost:31209/greeting):

    "hello, hi, how are yah?"
   

2. At the /farewell endpoint (localhost:31209/farewell):

    "adios, good bye, see yah?"
   

These backend responses confirm that our beta version is correctly fetching both the greeting and farewell messages from the database.

Explanation

The exact same processes take place as in the Deployment of Live services step, except a new image is created from our backend and frontend services, which contains the changes we made for our new feature - to retrieve and display not only the greeting but also the farewell from the database.

The services then become available on separate randomly generated URLs for accessing. This isolation allows us to thoroughly test our new feature in the beta environment without affecting the live services that our users are currently accessing.

By comparing the beta and stable versions side by side, we can easily verify that our new feature (adding the farewell message) is working as expected in the beta environment before we decide to promote it to live.

Step 6: Promotion of Beta to Live (Overriding Live Services)

In the real world, this would be where you'd test and validate the deployment of your new changes. At this point, we have two possible paths:

Issues Found In Beta - Redeployment Plan

One of the great perks of the Blue-Green strategy (or at least the way we've implemented it, and you should try to do as well) is that we can deploy new beta services as many times as we want. We can continuously override our Beta environment with new changes without affecting our Live services. All that's needed is to run the same beta deployment command again. It's that easy!

./deploy-manager.sh deploy beta

Validation Successful - Promotion Plan

Execution

If all has gone well in beta validation and you're ready to move forward, we promote our beta services to the Live environment and override it. The deploy-manager script is once again easily capable of handling this via the following command:

./deploy-manager.sh promote

Result

The script once again displays the status of the promotion, as well as the resources we've deployed. This time, however, both the beta and live URLs will display the same content, because at this point, the initial Live services have been overwritten with Beta changes.

Explanation

The promote argument triggers the following flow:

promote)
        print_header "Promoting Beta to Stable"

        beta_image=$(kubectl get deployment b-g-backend-beta -o jsonpath='{.spec.template.spec.containers[0].image}')

        deploy "stable" "${beta_image##*:}" # Extract tag from image

        print_success "Beta promoted to stable. New stable image tag: ${beta_image##*:}"
        ;;

Here, we leverage the Kubernetes CLI (kubectl) to retrieve the image tag we deployed our beta services with. This tag is shared between both the backend and frontend. We then use this tag to call the deploy function using the stable tag, which overwrites the existing Live services with our beta ones. Essentially, we're repeating the Deployment of Live services step but with the beta image tag.

Source: AB Tasty Blog

Step 7: Teardown Beta Services (Finalize Live Deployment & Remove Unused Resources)

Execution

Once again emulating a practical example, this is where we would terminate our unused beta environment to save costs. The deploy-manager script can again be used to manage the teardown.

./deploy-manager.sh teardown beta

Result

Once the teardown of beta resources is complete, all beta resources are removed.

==============================
  Tearing down beta resources
==============================
deployment.apps "b-g-backend-beta" deleted
deployment.apps "b-g-frontend-beta" deleted
service "b-g-backend-service-beta" deleted
service "b-g-frontend-service-beta" deleted
configmap "frontend-config-beta" deleted

✔ Beta resources teardown complete.

Explanation

Our deploy-manager script command triggers the following flow:

teardown)
        if [ "$#" -ne 2 ] || [[ ! "$2" =~ ^(all|beta)$ ]]; then
            print_error "Usage: $0 teardown <all|beta>"

            show_exit_prompt
        fi

        teardown_resources "$2"
        ;;

This calls the teardown_resources function to remove beta resources.

teardown_resources() {
    local scope=$1

    if [ "$scope" == "all" ]; then
        print_header "Tearing down all resources"

        kubectl delete deployment,service,configmap,pvc -l app=b-g

        print_success "All resources teardown complete."
    elif [ "$scope" == "beta" ]; then
        print_header "Tearing down beta resources"

        kubectl delete deployment,service,configmap -l app=b-g,status=beta

        print_success "Beta resources teardown complete."
}

Step 8: Teardown Live Services (Cleanup)

Execution

And to finalize your local machine cleanup, we run the teardown again, this time terminating Live services as well.

./deploy-manager.sh teardown all

Result

==============================
  Tearing down all resources
==============================
deployment.apps "b-g-backend-stable" deleted
deployment.apps "b-g-frontend-stable" deleted
deployment.apps "b-g-mariadb" deleted
service "b-g-backend-service-stable" deleted
service "b-g-frontend-service-stable" deleted
service "mariadb-service" deleted
configmap "frontend-config-stable" deleted
configmap "mariadb-init" deleted
persistentvolumeclaim "mariadb-pv-claim" deleted

✔ All resources teardown complete.

Explanation

Our deploy-manager script command triggers the following flow:

teardown)
        if [ "$#" -ne 2 ] || [[ ! "$2" =~ ^(all|beta)$ ]]; then
            print_error "Usage: $0 teardown <all|beta>"

            show_exit_prompt
        fi

        teardown_resources "$2"
        ;;

Except this time, it enters the first condition in our if statement, which removes all blue-green related resources (regardless of environment type), based on resource tagging we did in our Kubernetes configuration files.

teardown_resources() {
    local scope=$1

    if [ "$scope" == "all" ]; then
        print_header "Tearing down all resources"

        kubectl delete deployment,service,configmap,pvc -l app=b-g

        print_success "All resources teardown complete."
    elif [ "$scope" == "beta" ]; then
        print_header "Tearing down beta resources"

        kubectl delete deployment,service,configmap -l app=b-g,status=beta

        print_success "Beta resources teardown complete."
}

And there you have it!

We've successfully implemented, deployed, promoted, and cleaned up our Blue-Green deployment strategy. This approach gives you the flexibility to:

1. Test new versions of your services in isolation

2. Promote beta versions to live with minimal downtime

3. Easily redeploy new beta versions if issues are found during testing

4. Quickly roll back to the previous stable version if problems arise after promotion

This multi-faceted flexibility is the true power of the Blue-Green deployment strategy, providing a safety net that allows for more confident and agile deployments.

Final Thoughts: Painting Your Deployments Blue and Green

As we wrap up this journey through the land of Blue-Green deployments, let's take a moment to reflect on what we've accomplished. We've not just talked about a deployment strategy – we've implemented it, tested it, and seen its benefits firsthand.

Here are the key takeaways from our Blue-Green adventure:

1. Reduced Risk: By creating an isolated beta environment, we can thoroughly test new features without affecting live users.

2. Flexibility: The ability to easily rollback or redeploy gives us unprecedented control over our release process.

3. Improved User Experience: With near-zero downtime deployments, our users can enjoy uninterrupted service.

4. Better Testing: The isolated beta environment allows for more comprehensive testing in a production-like setting.

5. Cost-Effective: While there is an initial increase in resources, the long-term benefits in terms of stability and reduced downtime far outweigh the costs.

Implementing Blue-Green deployments might seem daunting at first, but as we've seen, with the right tools and approach, it's a achievable goal that can revolutionize your deployment process.

Remember, the journey doesn't end here. As you implement this strategy in your own projects, you'll undoubtedly encounter unique challenges and opportunities. Embrace them! Each obstacle is a chance to refine your process and make it even more robust.

So, are you ready to take your deployments to the next level? The blue (or green) sky's the limit!

Happy deploying, and may your releases always be smooth and your downtimes non-existent!

As I sit down to write this, I’ve been navigating the software engineering landscape for almost two years. Now, that might not sound like a lot, but the list of things I’ve picked up along the way could stretch on so long, you’d probably give up scrolling. Today, though, I’m not here to dive into the deep technical weeds. Instead, I want to talk about my journey, share the insights I’ve gathered, and the tips I’ve picked up as a Software Engineer.

Recently, I started mentoring and training new engineers. Impressively, one of them hit me with, "Farzam, what tips do you have for me?" This got me thinking back to my own early days, scouring Google, Reddit, YouTube - you name it - for any golden nuggets on how to be the best software engineer possible. That’s when it clicked. It's time to put keyboard to monitor and share a post on my journey and what I've learned so far.

Some of the points I’m planning to hit might resonate across the board, regardless of your field. Then there are those bits that might seem more specific to software engineering, but here’s my take: try to grasp the essence behind them. Even if they don’t seem immediately relevant to what you do, there might be a way to adapt and apply these insights in ways you hadn’t considered before.

Let's Get Acquainted

Before diving deeper, I think it’s crucial you get to know me a bit better. Understand what drives me and, more importantly, the type of person I am. Getting a glimpse into my world will help you grasp my professional ethos and appreciate why my insights might be worth your while.

I’m a staunch believer in the principles of Kaizen and an unabashed advocate for continuous improvement and growth. Wondering what Kaizen is? Here’s my favorite definition:

Source: LaWhimsy

"Kaizen," initially just a cool term I'd use to sound impressive in job interviews, has genuinely become a cornerstone of my life. This post is saturated with Kaizen's principles, aiming to pass a bit of that mindset onto you. Through the upcoming discussions, my goal is to not only share the spirit of continuous improvement but also to steer you clear of surface-level development. With the content that follows, I'm committed to fostering a mindset of ongoing growth and meticulousness in our work. Together, let's dive deeper into professional development, bypassing the shortcuts that lead to shallow understanding.

Finding My Long-Term Vision

There was a time when I was just coasting through life, lacking any real drive or direction. It wasn't because things were going poorly or I didn't have aspirations; I was simply short-sighted. The turning point came in my early twenties, spurred by an insatiable curiosity for reading. Books became my window to new perspectives, prompting me to scrutinize my life and the world around me. They encouraged me to start thinking with a long-term perspective, dreaming about where I wanted to be in the distant future. Once I adopted this broader outlook, reverting to my previous short-term mindset was unthinkable.

To this day, I encounter many who are still viewing life through that narrow lens. Occasionally, I'll offer a sharp word or two, hoping to prompt a broader view, but realizing the importance of long-term thinking is a journey each person must undertake on their own. I'm convinced that we all eventually arrive at this perspective, though, for some, the realization comes perilously late. By then, the opportunity to meaningfully shape their future based on these expanded horizons may have significantly diminished.

The Journey Continues: Embracing Growth

Embracing Change

I reached a point where I refused to let stagnation be part of my story, both personally and professionally. A guiding principle for me has become: "How you do anything is how you do everything." This mindset fuels my drive for excellence in all areas of life.

The Challenges

Pursuing this ideal of constant improvement, I've realized, comes with its own set of challenges. It involves a continuous search for areas to grow, not just within myself but also recognizing potential in others. This journey can be exhausting, as turning a blind eye to opportunities for enhancement is not in my nature.

In personal life, I've found that growth largely lies within our own control—our health, relationships, and self-care are ours to improve. Professional growth, however, introduces a more complex dynamic. The workplace brings together a diverse mix of individuals, each with their own backgrounds, cultures, and attitudes towards work. Striving to be the best version of oneself in such an environment can sometimes feel like a solo effort in a team sport.

Another maxim I hold dear is: "If you're going to do something, strive to do it better than anyone else. Do it all the way. If you're going to half-ass it, why bother?" This philosophy, especially when applied to one's career, doesn't always align with the outlook of everyone in the workplace. While some may be content with meeting the basic requirements, my approach has always been to push beyond these boundaries. Working alongside individuals who don’t share this commitment can be challenging.

The Reality of Team Dynamics

The essence of teamwork, for me, has always been about putting forth A-level quality effort. Witnessing anything less can be frustrating, draining one's energy. This isn’t to say that I haven’t encountered many astonishing colleagues; indeed, I've had the privilege of working with some truly impressive individuals. However, it's the inconsistency that poses the challenge.

Echoing a sentiment by Steve Jobs, the synergy of A-players working together can create an unparalleled dynamic. Jobs noted the transformative power of assembling a team of top performers who, once united, set a standard of excellence that becomes self-perpetuating. This ideal environment, where A-players thrive among their peers, is something I strive for.

The impact of diverse effort levels in a team setting is undeniable. Despite an individual’s best efforts, the collective output reflects the contribution of all members. The aspiration to maintain high standards can sometimes be diluted in a mixed-setting team, highlighting the importance of a unified drive towards excellence.

Reflections on Effort and Passion

In software engineering, recognizing the range of commitment levels in the workplace is crucial. While not everyone chooses to put in extra time to learn new skills, delve into emerging technologies, or expand their knowledge, some do. It's important to respect these differences. Yet, the real challenge is when this new knowledge isn't shared with the team. Sharing is key to avoid misunderstandings and ensure that the team grows together, not apart. It's about finding a balance that suits your lifestyle and benefits the team.

The core issue is how we integrate and share our expertise. Learning in isolation can lead to problems down the line. Sharing knowledge is essential for elevating everyone's skills and keeping the team united. It's critical that everyone is aligned on how to apply new insights to enhance our work. This not only prevents potential issues but also boosts our collective capabilities.

This approach isn't about pushing personal dedication onto others but about striking a balance that supports both individual ambitions and team growth. My goal is to emphasize my commitment to continuous learning and improvement, and to encourage a culture of sharing and collaboration. Together, we can achieve excellence as a cohesive, knowledgeable team.

Going forward, let's explore the insights and practical advice that have been key to this journey.

The Journey's Milestones: Achieving Quality and Team Synergy

Embracing Collective Success

Recognizing the importance of teamwork has been a crucial realization for me: You Are Your Team! This insight might not resonate if you're tackling projects on your own. However, in a team setting, our collective achievements are what truly matter, with trust being the cornerstone of outstanding results.

When rolling out new changes, remember, it’s a team effort. Completion isn’t just about your code landing in the main branch. It involves harmonizing every contribution to align seamlessly with the team’s vision for a successful release.

A key principle we adhere to in our discussions is clarifying what 'done' and 'ready for release' mean for us. In my perspective, a task isn't truly finished until it's received unanimous approval: development has wrapped up, QA has given its blessing, and the product owner is confident our solution meets the customer's needs. This collective agreement is vital, enabling us to thoroughly understand the development lifecycle and maintain our high standards across every project.

The Essence of Care in Development

The level of care invested in the development process directly influences the quality of the outcome. Genuine passion and dedication lead to superior results. If your commitment is waning, consider seeking a role more aligned with your passions. When you work on projects you're truly passionate about, care and quality naturally follow.

Emphasizing care in our work is crucial; without it, we risk producing subpar results, lacking the quality, passion, and effort needed for excellence. This underscores the importance of Uncle Bob's Programmer's Oath, establishing clear standards to uphold daily as a guiding beacon for our profession.

The Programmer's Oath

"In order to defend and preserve the honor of the profession of computer programmers,

I Promise that, to the best of my ability and judgement:

1. I will not produce harmful code.

2. The code that I produce will always be my best work. I will not knowingly allow code that is defective either in behavior or structure to accumulate.

3. I will produce, with each release, a quick, sure, and repeatable proof that every element of the code works as it should.

4. I will make frequent, small, releases so that I do not impede the progress of others.

5. I will fearlessly and relentlessly improve my creations at every opportunity. I will never degrade them.

6. I will do all that I can to keep the productivity of myself, and others, as high as possible. I will do nothing that decreases that productivity.

7. I will continuously ensure that others can cover for me, and that I can cover for them.

8. I will produce estimates that are honest both in magnitude and precision. I will not make promises without certainty.

9. I will never stop learning and improving my craft."

Source: The Clean Code Blog

Enforcing these aptly written standards underscores the importance of discipline, rules, and regulations in software engineering, akin to practices in law and medicine where strict guidelines govern professional conduct. The omnipresence of software in everything from daily items to complex systems highlights the critical need for robust standards and ethics in our field.

Although these guidelines may not be formally codified, passing them from mentors like Uncle Bob to me, from me to you, and from you to others through word of mouth is essential for our discipline's integrity and impact. I advocate for this tradition, aiming to inspire a future where these practices are more formally recognized and adopted.

Perfecting the Craft of Software Engineering

To me, software engineering transcends mere coding; it's an art form. My work embodies not just the interaction with a computer through words and symbols but an expression of special, intricate, and incredibly beautiful art. Each piece of software I create is a unique masterpiece, effortlessly communicating its needs and desires to other systems. Achieving this level of craftsmanship has been a journey of continuous study and practice, and I'm still on that path.

Just as artists like Picasso once started with works far from their later masterpieces, I view my early efforts in software development in a similar light. Initially, what we create may not meet our vision of beauty or function, but it's through relentless practice, study, and refinement that we inch closer to our ideals. I dedicate myself to not only improving my code output but enhancing my overall ability to engineer software solutions. This pursuit is a long-term commitment, a never-ending duel with the learning process.

Source: The Lost Sock

To continually refine our art, we must immerse ourselves in learning, studying every facet that contributes to the perfection of our craft, and persistently create, iterate, and evolve our work. It's a process of producing many iterations—some might call "failures"—but through this journey, we develop works of beauty that we can be truly proud of. Like the great artists who transformed their initial failures into revered art, we, too, can elevate software engineering to a form of art admired and respected for its beauty and impact.

Cultivating Collaboration Beyond Pair Programming

Collaboration fuels my enthusiasm. There’s a unique energy in brainstorming sessions, idea exchanges, and collaborative design efforts with my team. This collaborative spirit transcends mere teamwork; it involves mutual learning and significantly enhances our collective capabilities.

I advocate for embracing collaboration in all facets of our work as software engineers. This means not just cooperating on projects, but actively discussing ideas, topics, designs, and more in our daily tasks. Whether within your team or extending beyond, clear communication and open dialogue are key to innovation and effective problem-solving.

Pair Programming: A Highlight of Collaboration

Within this culture of collaboration, pair programming stands out as a prime example, though it's just one aspect of our collective effort. Initially, I was skeptical about its efficacy—wondering why it would take two to do one person’s job. However, the experience was eye-opening.

Pair programming isn't merely about coding together; it's a deep dive into another engineer's thought process, showcasing the direct benefits of collaborative work. It's about experiencing firsthand how others approach problems and develop solutions.

Such collaboration, especially through pair programming, has been crucial to my learning. It encourages asking questions, no matter how simple they may seem, and actively seeking understanding. While pair programming is invaluable, it's important to recognize that it's part of a broader ethos of collaboration that permeates every aspect of our work.

Promoting a culture of collaboration—whether through pair programming or daily interactions—enhances our work and leads to personal and professional growth. It's about more than just specific practices; it's a fundamental approach to software engineering that encourages openness, shared learning, and continuous improvement.

Navigating Pull Requests: Maximizing Benefits, Minimizing Pitfalls

Pull requests are a pivotal aspect of modern software development, presenting a choice: engage with them thoroughly or not at all.

Engaging with Pull Requests

Choosing pull requests necessitates a commitment to thoroughness. When executed well, they elevate the quality of your work, leveraging the insights of knowledgeable reviewers to refine your code and guide corrections. This process isn't just about critique; it's a collaborative effort to enhance project outcomes.

Enhancing Clarity with Pair Reviews

For instances where a change's context might elude a reviewer, pair reviews offer a direct approach to resolve ambiguities and hone feedback, making sure misunderstandings are minimized and the review process is as effective as possible.

Acknowledging the Drawbacks

Yet, pull requests are not without their challenges. They can introduce delays, lead to merge conflicts, and sometimes suffer from less engagement or surface-level feedback from reviewers. Especially large changes might not receive the attention they require, and there's the potential for negative dynamics if feedback is not well-received.

Alternatives to Pull Requests

Continuous Integration: A Viable Path

Opting out of pull requests often means embracing Continuous Integration (CI). This path requires a robust testing framework to detect and prevent the integration of errors, maintaining the stability of the main branch. Crucially, it also demands a high level of trust among team members and those involved in making changes, as the process relies on each individual's commitment to maintaining code quality without the immediate oversight of pull request reviews.

Post-Merge Reviews

Another strategy involves merging changes immediately and conducting reviews post-merge. Platforms like Azure DevOps support this by allowing comments on commits after merging, mitigating the issues of delays and conflicts. This approach, however, may present challenges in achieving consistency across a team.

Making the Choice

Ultimately, the decision to use or forego pull requests should be tailored to your team's dynamics, workflow, and project objectives. Each approach has its merits and challenges, and the right choice is the one that aligns with your team's needs, fostering a productive and collaborative software development environment.

"Move it Along" Mindset: Quality Over Speed

Lately, a phrase from a colleague has been echoing in my team: "Let's try to move it along. After all, it's only an MVP." I get where he's coming from—the push to speed things up, especially with all the big changes throwing us curveballs left and right. But here's the thing: I'm all for diving deep. When I'm getting my hands on a new library, I need to peek under the hood, do a bit of digging to really get what I'm dealing with. Blindly rushing through? Not my style.

This whole "move it along" vibe seems to push for speed over substance. Yet, as Uncle Bob puts it, "The only way to go fast is to go well." That hits home for me. I'm a firm believer in understanding the nitty-gritty, making sure the work I do isn't just quick, but solid. This approach has paid off big time for me, and I don't see that changing. In the end, knowing your stuff inside and out? That's the real shortcut.

Surface-Level Development: The Pitfalls of Rushing

The rush to move quickly often leads to just skimming the surface of your work, a practice I've observed in colleagues who consistently make changes without delving into the details. When it comes time to review or discuss these changes, they find themselves at a loss to explain the actual impact of their work. This situation highlights a crucial point.

Understanding the context of your work is non-negotiable. Skipping this step can lead to repetitive corrections, especially in complex systems. Moreover, this surface-level approach often results in broken testing environments, adding layers of complexity to troubleshooting and delaying progress further. Deep engagement with your tasks not only prevents these setbacks but also contributes to more durable and impactful outcomes.

Going Beyond Surface-Level Development

The essence of impactful work lies in a profound understanding of the task at hand. Start by thoroughly grasping all the requirements. Take the time to review the objectives and strategize your approach, considering both planning and design aspects. If there's an existing implementation, delve into it to understand its foundation and workings. Engage with other team members familiar with the project's context for discussions—not limited to developers. This holistic approach ensures a well-rounded understanding and paves the way for meaningful contributions.

Copy-Pasting: A Slippery Slope

In my early days of coding, just before the rise of advanced AI, mastering the art of searching for solutions was a critical skill for any software developer. This often led to discovering and repurposing solutions crafted by others. Initially, I saw copy-pasting as a standard practice, widely accepted and practiced amongst my peers. However, my perspective on this has drastically shifted.

Copy-pasting isn't limited to snagging answers off Stack Overflow. It extends to reusing code from your own projects, which I've come to view as a precarious shortcut.

The act of lifting code from external sources, whether it’s Stack Overflow, another project, code from the annals of programming history, or even the best that AI has to offer, opens the door to inadvertently introducing errors. Often, code is copied without a thorough line-by-line review, based on the assumption that if it worked elsewhere, it surely must work here as well. This mindset, however, can lead to significant oversights.

This realization hit me hard. In the beginning, I frequently borrowed from existing solutions, only to find that I missed critical details, even in projects with nearly identical requirements. Since then, I've made a concerted effort to minimize copy-pasting. On the rare occasions I do, I ensure I fully understand every aspect of the code I'm incorporating. Ultimately, the responsibility for the code's performance rests on my shoulders, regardless of its original author.

Falling into the trap of copy-pasting is yet another barrier to achieving the depth of understanding necessary for meaningful development. This follows naturally from our previous discussion on the importance of going beyond surface-level development.

Lifelong Responsibility for Your Code

Once you author a change, that piece of code is yours to back—forever. Adopting the mindset that you're accountable for your work indefinitely encourages you to tackle new tasks with the appropriate level of responsibility.

Don't fall into the trap of thinking that once QA has approved your code, the buck stops with them. The responsibility remains firmly on your shoulders. Maintaining this level of ownership ensures you avoid the pitfall of perpetually making only surface-level changes. This commitment to accountability is crucial for fostering a deeper, more meaningful approach to software development.

Gathering Better Requirements

The complexity of a task directly correlates with the necessity for thorough planning and design. Initially, writing code seems daunting, but with experience, you come to realize coding is often the easiest part. The challenge lies in ensuring the robustness and high quality of your changes, which demands meticulous planning and preparation.

Putting Your Ego Aside

Mistakes are inevitable, and the sooner you accept this, the quicker you'll learn and improve. I'm opinionated, basing my knowledge on evidence and experience, ensuring my discussions are fact-driven. Yet, no matter how solid your knowledge base is, being proven wrong is part of the journey. A quote that resonates deeply with me is, "It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so."

Strive for correctness, but don’t take it personally when corrected. Your capacity to acknowledge and learn from errors will set you apart in the long run.

Knowing When to Seek Help

My relentless nature initially seemed like an asset. However, I've learned that persistence must be balanced. Here's what I mean:

Example of Misguided Persistence: Once, I spent days trying to fix a minor issue in a feature I was developing, only to realize after two days that I needed help. With assistance, the problem was resolved in an hour.

Example of Balanced Persistence: Facing another challenging issue, I sought help after an hour of solo effort. Together with a colleague, we found a solution within the next hour.

The distinction between stubbornness and productive perseverance is recognizing when to ask for help. While dedication is valuable, knowing when you've hit a wall and seeking assistance can save time and lead to better outcomes.

Ensuring Excellence: The Vital Role of Testing

Adopting a "Trust but Verify" Approach in Software Development

Early in your career, it's tempting to take everything you hear as gospel, especially when it comes from those with more experience. This initial trust is crucial for building rapport within teams, yet in the nuanced field of software development, adopting a "trust but verify" approach is invaluable.

Consider this incident during a software release: the IT team needed to execute several database scripts in preparation for the update. As the person overseeing release support, I was informed that all scripts had run successfully, albeit with a minor anomaly - the customary confirmation message from the database was missing.

Despite assurances that everything was probably fine, I leaned on a lesson learned from past experience: always verify. We convened on a call to meticulously review the process and discovered that the script lacked the essential "commit transaction" step, preventing the system from saving the changes. This critical omission didn't prompt an error since the lack of a "commit" command isn't flagged as an error by the system, yet it meant our updates hadn't been properly applied to the database. By taking the time to verify, we rectified the issue, ensuring the release proceeded smoothly and with certainty.

The Reality Beyond "It Works on My Machine"

We’ve all heard it, or even said it: "It works on my machine!" But the reality is, the final deployment environment, whether it's an on-premises server or cloud-based services, operates under different conditions. Recognizing this is crucial for ensuring that our software performs as intended in the real world.

Leverage Testing Environments to Anticipate Real-World Scenarios

Access to testing environments that mimic the production setting is a golden opportunity. These environments are not merely features of your development toolkit; they're a necessity. By closely replicating the production environment, your deployments and updates in the testing environment let you reach production-level outcomes without the risks of directly impacting users.

This approach underscores a commitment to delivering quality software. By thoroughly testing in environments that closely replicate the production setting, you're not just preventing potential headaches for your team and users; you're actively contributing to a culture of reliability and excellence in your projects.

Shaping Excellence in Software Through Early Testing

Deploying your changes to a testing environment, particularly one designated for engineers before QA review, is your initial shield against unexpected problems. This step lets you discover and solve issues early, ensuring your work is double-checked and refined by you before it gets to the quality team. By this stage, it's polished and almost ready for production. This proactive approach demonstrates a deep commitment to the reliability and excellence of your work.

Test-Driven Development (TDD)

Test Drive Development is a powerful approach to software design and implementation. Although initially challenging to adopt, its value becomes immediately apparent once mastered. Start slow, set aside the learning curve, and focus on becoming the best engineer possible.

Beyond Unit Tests

After mastering key unit testing skills such as mocking, assertion techniques, test-driven development (TDD), and code coverage analysis, move on to more advanced testing strategies like integration, end-to-end (E2E), and acceptance tests to further enhance your capabilities. This not only improves your coding skills but also exposes you to new testing methodologies that enhance your abilities further than what you'd achieve without testing.

Continuously Verifying System Needs

Well-crafted tests do more than address immediate concerns; they continuously validate your changes, ensuring robust and reliable updates no matter who makes them. Our tests are our safety net.

The Importance of Testing

Estimates suggest that for every 1,000 lines of code delivered, there are typically 15 to 50 bugs, meaning 1.5% to 5% of all code may contain errors. These numbers can fluctuate but emphasize a constant reality: writing code always means creating bugs, as the two go hand in hand. By employing a variety of tests, emulating production environments closely, and thoroughly verifying changes, we can further reduce these numbers.

For me, the ultimate goal is receiving minimal feedback from QA. It's not that I don't value collaboration with them; on the contrary, I appreciate working together on changes. However, minimal feedback post-commit is my indicator of success, signifying no further refinements or issues were found. It's always a challenging goal, regardless of experience level. And even if QA and I find no issues, it doesn't mean there are none; it just indicates thorough anticipation of desired behaviors. Never underestimate the power of testing; it enhances not only the quality of outcomes but also your skills, making you a more well-rounded engineer.

Advancing the Journey: Strategies for Professional Growth in Software Engineering

Shifting gears from the intricacies of coding and the quest for quality, let's dive into the broader horizon of long-term career development. I've picked up a few strategies along the way that have been game-changers for climbing the career ladder.

Grab the Reins of Your Future

The best piece of advice? Don’t sit around waiting for a nudge forward. Taking the initiative is key. While it might seem like your manager or mentors should be paving your path, in reality, they're often swamped or might not fully grasp what you need. It's on you to pinpoint your improvement areas and bring them into the conversation. Remember, you're the captain of your ship.

Voice Your Needs

When you stumble upon something that sparks an idea or a potential growth avenue, bring it up with your mentors and manager. What works for me might not be the silver bullet for you, given our unique learning curves. For instance, craving more exposure to high-level discussions, I asked to shadow sessions with senior engineers, promising to blend into the background. It's about finding what accelerates your own growth and making it known.

Embrace Feedback

Make it a ritual, maybe every half year, to gather the folks you work closely with and seek their honest insights. There's always something to polish or a new angle to consider.

Chart Your Learning Journey

Identify the skills that will propel you forward and seek out resources tailored to these areas. This strategic approach to learning can dramatically shorten the path to your goals.

When in Doubt, Reach Out

Asking for help isn't just about overcoming immediate hurdles; it's about broadening your understanding of what it takes to level up. Dive into discussions about the skills and experiences you need, and don't shy away from seeking guidance.

Know and Surpass Your Role's Expectations

Understanding your role's expectations is the first step. Next, identify ways to surpass those expectations. Engage with your team and manager to clarify any uncertainties, and then aim to exceed them.

Initiative Is Your Best Friend

Whenever you spot an opportunity to add value, jump on it. Whether it's a small gesture or spearheading a major project, taking action without waiting for an invitation speaks volumes. My journey with ADO Express, born from spotting a process gap, highlights the impact of proactive problem-solving.

Find Your Side Hustle

Keep an eye out for side projects that pique your interest or passion. They're not just a break from the routine; they're a bridge to new skills and technologies. Diversify your learning to keep your passion alive and kicking.

Stay Curious

Adopt a habit of researching every new term or technology you encounter. A quick Google search can either satisfy your curiosity or lead you down a rabbit hole of fascinating discoveries. Either way, you're broadening your horizon with every search.

In weaving these strategies into your professional fabric, you're not merely preparing for the future; you're actively shaping it to fit your vision of success in software engineering. This journey is yours to command, with curiosity, initiative, and continuous learning as your guiding stars.

Tackle the Tough Stuff

The most significant growth stems from taking on the tough challenges. It's tempting to stick to familiar territory, relying on the skills that got us to where we are. Yet, from my experience, the most substantial, transformative growth, especially in software engineering, comes from embracing challenges that stretch your boundaries—the kind that ventures into the unknown. Pushing beyond comfort zones not only sharpens your technical abilities but also fosters resilience and innovation, making you a true standout in the field.

Opting for the more challenging project, delving into unfamiliar code, and navigating complex problems—these endeavors are essential for elevating our capabilities. It transcends mere technical skill enhancement, fostering resilience, honing problem-solving acumen, and developing the perseverance needed to overcome obstacles.

Why Tackling the Tough Stuff Matters

Embracing difficult tasks isn’t just good for your skill set; it's crucial for innovation and staying ahead in the fast-paced tech landscape. It forces you to think creatively, to be resourceful, and to find solutions where others might hit a wall.

So, as we move forward, remember: opting for the path with more resistance is not about making things hard for yourself. It’s about growth, learning, and setting yourself up for long-term success in your career. Let’s not shy away from the tough stuff. It’s where we find our greatest potential.

Wrapping Up The Journey

Key Insights

Throughout this journey, you've encountered the theme of continuous improvement time and again. The essence of consistently choosing the more challenging path, actively pursuing your goals, and pushing beyond expectations has been underscored. The key takeaway is the importance of taking charge of both your personal and professional life, as ultimately, no one else will. Embrace this mindset, and you'll see a transformative shift in your outlook on life, fostering an insatiable hunger to learn, improve, and strive to be the best version of yourself.

Stay Consistent & Enjoy The Journey

Adopting a lifestyle of continuous improvement means accepting that we are always a work in progress. The journey might get tough, and there may be moments when you feel utterly stuck. The crucial part is navigating through these challenges with a steadfast commitment to improvement. The image below captures the essence of maintaining consistency, even as your intensity may fluctuate. Real life is full of ups and downs, but as long as you keep the flame of consistency alive, you're on the right path.

Source: Consistency Over Intensity

Start On What You Need Right Now!

I urge you to reflect on your career aspirations and identify areas needing the most attention. Develop a pragmatic action plan using the strategies discussed and transition those plans into tangible actions. The time for procrastination is over; seize the day with unparalleled zeal, for the future is unwritten, and only you hold the pen.

Prepare To Be The Black Sheep

Consider the individuals you admire, those who have achieved greatness. A common trait among them is their uniqueness—they're the black sheep. They dare to think differently and act distinctively. Embracing this mindset is challenging but crucial for those aiming to stand out. If you blend in with the crowd, what sets you apart?

Engage Your Long-Term Vision Goggles

Imagine your future. That typical question, "Where do you see yourself in 5 years?" isn't just small talk; it's a prompt to envision your path forward. Don't stop at five years; think ten, twenty years ahead. Determine your goals and pursue them with relentless passion, knowing that the only one who can truly propel you forward is yourself.

Imagine Endless Possibilities For Yourself

Consider the power of small habits, as eloquently captured by a favorite saying: "Reading 20 pages per day equals 30 books a year. Saving $10 per day accumulates to $3650 a year. Running 1 mile a day totals 365 miles a year." These small, daily actions compound into significant achievements over time. Envision where these habits could take you in just a year, and remember, time flies.

The Journey Will Be Tough, But You'll Be Even Tougher

Embracing these principles will undoubtedly challenge you, but your resilience and consistency will see you through. The path to greatness is never easy, but it's always worth it.

Thankful For The Amazing Team I Work With

My journey has been deeply influenced by the incredible team and mentors around me. Their guidance has been instrumental in my rapid growth from a novice to a role where I'm viewed as moving towards an intermediate/senior level. This progression, from taking on side projects to initiating blogs, reflects a journey of hard work and dedication. I hope my experiences inspire and guide you in your own path to success.

It's a Never Ending Journey

Thank you for investing your time in reading this post. My aim was to inspire reflection on these topics and wish you the best of luck on your journey of career development. The road ahead is perpetual, filled with learning and growth at every turn. Keep moving forward, and may your path be ever upward.

Azure AI Search, formerly Azure Cognitive Search, is your versatile toolkit for data management. It's an AI-driven platform that optimizes data accessibility while easing database load. My first encounter with Azure AI Search was as a solution for database overload. Its capacity for handling and rapidly retrieving vast data sets, coupled with a user-friendly search process, makes it a standout choice.

But there's more to it than just easing database load. Azure AI Search acts like a digital librarian, swiftly and efficiently organizing and pulling up information. The AI-powered search not only improves accuracy but also enhances the user experience. Plus, it integrates smoothly with other Azure services, adding flexibility across various applications.

Azure AI Search revolutionizes the way we handle, search, and interact with large data sets. It has become an indispensable tool in application development, especially crucial in scenarios where enhanced search functionality plays a pivotal role.

Key Features

• Efficient Data Handling: Acts as your digital librarian, swiftly organizing and retrieving large datasets.

• AI-Enhanced Search Precision: Boosts search accuracy for an improved user experience.

• Seamless Azure Integration: Ideal for integration within the Azure ecosystem.

• Advanced Data Querying: Features like searchable, filterable, sortable, and facetable fields enable sophisticated data queries.

• Database Relief: It significantly lightens the load on your databases, particularly in search-heavy applications, by efficiently managing complex queries that would traditionally tax your database resources.

• Versatile Search Capabilities: Excels in text and vector similarity searches, ideal for pattern analysis in large datasets.

• Complex Data Type Modeling: Handles intricate data structures with ease. For instance, it can manage complex objects like a hotel's 'Room' attributes, where each room is characterized by its description, room number, and rate.

• Content Transformation: Converts text or image files into searchable content, ideal for assets stored in Azure Blob Storage or Azure Cosmos DB.

• Customizable Text Analysis: Supports various linguistic tools and customizes text analysis for multi-language and format content.

Now that we've covered the features, let's delve into understanding a Search Index in Azure AI Search.

Understanding a Search Index

A Search Index is like the index of a book but for digital data. It lists topics and keywords, guiding you to the information you need. This specialized database is tailored for large data sets, organizing data for efficient searching and retrieval. It's the go-to tool for finding that 'needle in the digital haystack'.

Azure AI Search Enhanced Features and Application Scenarios

Azure AI Search elevates search functionality with features for full-text searches (searchable fields), refining results (filterable fields), organizing results (sortable fields), and aiding in search refinement (facetable fields). It excels in various application scenarios, including full-text and vector similarity searches, making it suitable for generative AI applications. Additionally, it streamlines the implementation of search features like relevance tuning and faceted navigation and transforms extensive files into searchable formats. It also accommodates diverse linguistic and custom text analyses, ensuring versatility in content processing.

With an understanding of Azure AI Search's capabilities, it's time to turn theory into action by building our own Search Index.

What's the Plan?

I am a firm believer in learning by doing. We're diving into the process of building a Search Index in C#. I'll take you through each step, breaking down the code along the way. After we explore each part, we'll run the project and check out what we've accomplished. Here’s what we’ll be doing:

1. Creating the Azure Data Source (Via Azure Portal): We'll start by creating the data source for our Search Index, using Azure Table Storage.

2. Creating the Azure AI Search Service (Via Azure Portal): Next, we establish the Azure Search Service, the home for our Search Index and its components.

3. Cloning the Repository & Project Preparation (Optional): For hands-on experience, clone the repository and align your appsettings.json with your Azure setup.

4. Populating the Data Source (Via Code): With Azure Table Storage ready, we'll populate it with 'Books' table data, perfect for filtering, sorting, and searching.

5. Building the Search Index Infrastructure (Via Code): We then shift our focus to creating a Search Index that aligns with our 'Books' table's fields.

6. Populating the Search Index (Via Code): Now, we're at the data loading phase. Here, we'll set up an Indexer within the Azure Search Service. Think of the Indexer as a bridge: it's provided by Azure free of charge, and it connects two key points. On one end, we have our Data Source Connection, our source, where we've got our 'Books' table data. On the other end, there's our Search Index, the target, ready to receive and organize this data. Once we link the Indexer to our data source, it'll funnel data into the Search Index, transforming raw data into a structured, searchable format.

7. Querying Data from the Search Index (Via Code): Finally, with our Search Index fully stocked, we'll run various queries to explore Azure AI's querying capabilities, giving us a practical understanding of its potential.

I'm Assuming

1. You Have an Azure Subscription: To create resources, an Azure Subscription is required. If you don't have one yet, signing up is straightforward, and you'll immediately receive $200 in credits, which is more than enough for this project.

2. A Prepared Resource Group: This is where we'll host our resources. If you’re not familiar with setting up a resource group in Azure, you can follow the steps outlined here.

3. Basic Programming Knowledge (Optional): It's helpful to know some programming basics, especially how to use languages with external libraries. We'll be using C# and SDKs here. If not, don't worry, I'll guide you through everything.

4. .NET IDE Ready to Go (Optional): Ideal to have Visual Studio or Rider installed.

Note: If coding isn't your thing or you prefer to learn passively, you can skip the optional steps and still grasp the key concepts.

Setting Up the Foundations

Azure Data Source Creation

We begin by establishing a data source for our Azure Search Index. Azure supports a variety of them, including Azure Blob Storage, Cosmos DB, Data Lake Storage Gen2, and several others, with some like Azure Files and Azure MySQL currently in preview. You can dive deeper into these options here.

For simplicity, we're going with Azure Table Storage. It's straightforward to set up. Just head to your Resource Group, click '+ Create', and find 'Storage Account' in the Azure Marketplace.

Choose the same Region as your Resource Group and opt for 'Locally-redundant storage' under Redundancy to minimize costs. The rest? Just leave it as is and hit 'Create'.

Feel free to explore the official documentation for more details.

Azure AI Search Service Creation

Next, we set up the Azure AI Search Service. In your Resource Group, select '+ Create' and search for 'Azure AI Search' in the Azure Marketplace.

Again select the same Region as your Resource Group, but this time, pick the 'Free' Pricing Tier. Leave the other settings default and proceed with 'Review + Create'. This setup won't cost you a dime.

Feel free to explore the official documentation for more details.

Preparing for Coding

Cloning the Repository & Project Preparation (Optional)

For those who prefer hands-on coding, clone the azure-search-index-builder repository. Make sure to align the settings in the appsettings.json file with your newly created Storage Account and Search Service.

Here's how the appsettings.json file should look based on my previous steps:

{
  "AzureStorageAccount": {
    "Name": "mystorageaccount9999", // Your Azure Storage Account name - Can be found on the very top of the Storage Account overview
    "Key": "xxxxxxx" // Your Azure Storage Account key - Can be found in the Storage Account settings, under "Access keys"
  },
  "AzureSearchService": {
    "Url": "https://my-searchservice9999.search.windows.net/", // Your Azure Search Service URL, in the format of https://<service name>.search.windows.net
    "AdminApiKey": "xxxxxxx" // Your Azure Search Service Admin API key, can be found in the Azure Search Service settings, under "Keys"
  }
}

If you're more inclined to learn passively, feel free to skip this step.

Diving into the Code

Populating the Data Source

Now, we dive into the code. In the Program.cs file, you'll find the CreateAndPopulateBooksTable() method where the Data Source preparation happens. We set up and fill the 'Books' table in Azure Table Storage, using data from the books.json file.

Here’s a quick look at what we're doing:

string CreateAndPopulateBooksTable(string azureStorageAccountName, string azureStorageAccountKey)
{
    // Create Books table in Azure Storage
    const string books = "Books";

    var tableServiceManager = new TableServiceManager(azureStorageAccountName, azureStorageAccountKey);

    tableServiceManager.CreateTable(books);

    // Insert data from JSON file into Books table
    tableServiceManager.InsertRecordsIntoTable(books);

    return books;
}

We kick things off by creating an instance of the TableServiceManager class, which sets up our connection to Azure Table Storage. This connection is crucial as it's our gateway to interacting with the storage service.

Here's how the Table Service Client looks like:

private readonly TableServiceClient _tableServiceClient = new
(
    new Uri($"https://{storageAccountName}.table.core.windows.net/"),
    new TableSharedKeyCredential
    (
        storageAccountName,
        storageAccountKey
    )
);

We then call CreateTable() to create our 'Books' table, followed by InsertRecordsIntoTable() to populate it.

public void CreateTable(string tableName)
{
    _tableServiceClient.CreateTableIfNotExists(tableName);
}

The InsertRecordsIntoTable method is where we read the books.json file, convert its contents into Book objects, and then insert them into our table.

Here's the InsertRecordsIntoTable method:

public void InsertRecordsIntoTable(string tableName)
{
    var tableClient = _tableServiceClient.GetTableClient(tableName);

    var jsonFilePath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Assets", "books.json");
    var jsonFileContent = File.ReadAllText(jsonFilePath);

    var books =
        JsonConvert.DeserializeObject<List<Book>>(jsonFileContent)
        ??
        throw new Exception("Unable to deserialize books. Check JSON file and make sure it was included in the build.");

    foreach (var book in books)
    {
        book.PartitionKey = book.Genre;
        book.RowKey = book.Id;

        tableClient.AddEntity(book);
    }
}

The Book class is straightforward. It inherits from ITableEntity, a requirement for any entity you want to store in Azure Table Storage. Here's what our Book class looks like:

public class Book : ITableEntity
{
    [JsonIgnore] // Ignored by Index & Indexer.
    public string PartitionKey { get; set; } = null!;

    [JsonIgnore] // Ignored by Index & Indexer.
    public string RowKey { get; set; } = null!;

    [JsonIgnore] // Ignored by Index & Indexer.
    public DateTimeOffset? Timestamp { get; set; }

    [JsonIgnore] // Ignored by Index & Indexer.
    public ETag ETag { get; set; }

    [SimpleField(IsKey = true)]
    public string Id { get; set; } = null!;

    [SearchableField(AnalyzerName = LexicalAnalyzerName.Values.EnLucene)]
    public string Title { get; set; } = null!;

    [SearchableField(IsFilterable = true)]
    public string Author { get; set; } = null!;

    [SimpleField(IsFilterable = true)]
    public string Genre { get; set; } = null!;

    [SimpleField(IsFilterable = true, IsSortable = true)]
    public int PublishedYear { get; set; }

    [SimpleField(IsSortable = true, IsFilterable = true)]
    public double Price { get; set; }

    public override string ToString()
    {
        return $"Id: {Id}, Title: {Title}, Author: {Author}, Genre: {Genre}, PublishedYear: {PublishedYear}, Price: {Price}";
    }
}

And that's the walkthrough of the setup! By following these steps in the project, you will have created a table in Azure Table Storage and populated it with data, setting the stage for indexing once you run the project.

Once you've executed the code, you can verify the results in your Storage Account. Simply navigate to the 'Books' table by going to your Storage Account > Storage Browser > Table > Books. There, you should find 30 Book records, showcasing the successful data population.

Build the Search Index Infrastructure

We then focus on constructing the Search Index infrastructure. The trickiest part? Defining the Index fields.

Index Field Preparation

Imagine a Search Index as similar to a database table, where fields in the index parallel the columns in a database. Just like defining column types in a database, we define field types in a Search Index. However, there's an additional layer in Search Indexes as we also need to specify the query attributes for each field. This distinction is what sets the Search Index fields apart and tailors them for efficient searching.

Now, let's circle back to our Book class. The fields in this class come with specific annotations, demonstrating how Azure AI Search interprets and utilizes each field. Here’s a quick glance at some of these annotations:

JsonIgnore: This annotation tells Azure to exclude specific fields during index creation. For instance, RowKey is not necessary for our querying purposes and can be overlooked.

[JsonIgnore] // Ignored by Index & Indexer.
public string RowKey { get; set; } = null!;

SimpleField: Used for fields that we want to retrieve but not necessarily search through. The Id field is a perfect example; it's retrievable but not intended for searching.

[SimpleField(IsKey = true)]
public string Id { get; set; } = null!;

SearchableField: Assigned to fields that should be both retrievable and searchable. Consider the Author field, which is also marked as filterable.

[SearchableField(IsFilterable = true)]
public string Author { get; set; } = null!;

These examples illustrate just a few of the annotations and field setups used in Search Indexes. For a comprehensive understanding of how to set field attributes and to explore all the options available, I recommend diving into the documentation on Field definitions and Field attributes.

Index Construction

In Program.cs, we use the CreateAndPopulateBooksSearchIndex method to build our Search Index and its components. The SearchServiceManager class sets up the connection to our Azure Cognitive AI Search service. In this section, we focus on the Index creation, happening in CreateIndex method, called from Program.cs on line 62.

The process:

1. Initialize the Search Index Client.

2. Build the index fields using FieldBuilder.

3. Define the index name and fields.

4. Either create a new index or update an existing one using CreateOrUpdateIndex.

public SearchIndex CreateIndex()
{
    var searchIndexClient = new SearchIndexClient(_searchServiceAuthentication.Uri, _searchServiceAuthentication.Credentials);

    // Define & build the index fields
    var fieldBuilder = new FieldBuilder();
    var searchFields = fieldBuilder.Build(typeof(Book));

    // Set the index name and fields
    var searchIndex = new SearchIndex("my-index", searchFields);

    searchIndexClient.CreateOrUpdateIndex(searchIndex);

    return searchIndex;
}

Populate the Search Index

Next up: transferring data from our Azure Storage Book table to the Search Index.

We start by setting up a Data Source Connection for our Indexer. This connection links our data source (the Azure Storage Book table) to the Indexer. The setup happens in Program.cs, line 65 when we run CreateIndexerDataSource.

The process involves:

1. Creating a Search Indexer Client.

2. Building the Data Source Connection.

3. Either creating a new Data Source Connection or updating the existing one using CreateOrUpdateDataSourceConnection.

public SearchIndexerClient CreateIndexerDataSource
(
    string storageAccountName,
    string storageAccountKey,
    string tableName,
    out SearchIndexerDataSourceConnection searchIndexerDataSourceConnection
)
{
    var searchIndexerClient = new SearchIndexerClient(_searchServiceAuthentication.Uri, _searchServiceAuthentication.Credentials);

    searchIndexerDataSourceConnection = new SearchIndexerDataSourceConnection
    (
        "my-index-data-source",
        SearchIndexerDataSourceType.AzureTable,
        $"DefaultEndpointsProtocol=https;AccountName={storageAccountName};AccountKey={storageAccountKey};EndpointSuffix=core.windows.net",
        new SearchIndexerDataContainer(tableName)
    );

    searchIndexerClient.CreateOrUpdateDataSourceConnection(searchIndexerDataSourceConnection);

    return searchIndexerClient;
}

After establishing the Data Source Connection, we create the Indexer by calling the CreateIndexer method from Program.cs, line 68. This sets up where the data is pulled from and where it's pushed to through the SearchIndexer class instantiation.

public static SearchIndexer CreateIndexer(SearchIndexerDataSourceConnection dataSourceConnection, SearchIndex searchIndex, SearchIndexerClient searchIndexerClient)
{
    var searchIndexer = new SearchIndexer("my-index-indexer", dataSourceConnection.Name, searchIndex.Name);

    searchIndexerClient.CreateOrUpdateIndexer(searchIndexer);

    return searchIndexer;
}

Finally, to start the data transfer, we run the PopulateIndex method in Program.cs, line 71, which resets and then runs the Indexer.

public static void PopulateIndex(SearchIndexerClient searchIndexerClient, SearchIndexer searchIndexer)
{
    searchIndexerClient.ResetIndexer(searchIndexer.Name);

    searchIndexerClient.RunIndexer(searchIndexer.Name);
}

This function does two key things:

1. Resets the change tracking state of indexed documents.

2. Runs the Indexer to begin transferring data into our Search Index.

The Indexer may take around 30 seconds to finish transferring data to the Index. To view results in your Cognitive AI Search service navigate to Indexes > my-index and hit Search.

Querying Data From the Search Index

Before we run the code, let's take a look at some queries that highlight the capabilities of the Search Index. I've set up four distinct queries for us to execute. They're called from Program.cs (lines 85-95), each designed to demonstrate the versatility of our Search Index.

void RunQueriesOnSearchIndexAndPrintResults(string azureSearchServiceEndpoint, string azureSearchServiceAdminApiKey, SearchIndex azureSearchIndex)
{
    var searchIndexDataRetriever = new SearchIndexDataRetriever
    (
        azureSearchServiceEndpoint,
        azureSearchServiceAdminApiKey,
        azureSearchIndex
    );

    var queryOneResults = searchIndexDataRetriever.SearchBooksForGeorgeOrwell();
    queryOneResults.PrintValue();

    var queryTwoResults = searchIndexDataRetriever.FilterBooksCheaperThanTwentyFiveDollarsInDescendingOrder();
    queryTwoResults.PrintValue();

    var queryThreeResults = searchIndexDataRetriever.SortBooksByPublishedYearInAscendingOrderAndGrabTopFive();
    queryThreeResults.PrintValue();

    var queryFourResults = searchIndexDataRetriever.SearchBooksForTitleMatchingMockingBird();
    queryFourResults.PrintValue();
}

Let's break down these queries:

Query 1 - Searching for 'George Orwell': This query rummages through all fields for any mention of 'George Orwell'.

public SearchResults<Book> SearchBooksForGeorgeOrwell()
{
    Console.WriteLine("Query 1: Search for 'George Orwell':");

    var options = new SearchOptions();

    GrabAllBookFieldsFromTheIndex(options);

    var results = _searchClient.Search<Book>("George Orwell", options);

    return results;
}

Query 2 - Filtering Books Under $25: Here, we filter for books priced below $25 and sort them by price in descending order.

public SearchResults<Book> FilterBooksCheaperThanTwentyFiveDollarsInDescendingOrder()
{
    Console.WriteLine("Query 2: Apply a filter to find books cheaper than $25, order by Price in descending order:");

    var options = new SearchOptions
    {
        Filter = "Price lt 25",
        OrderBy = { "Price desc" }
    };

    GrabAllBookFieldsFromTheIndex(options);

    var results = _searchClient.Search<Book>("*", options);

    return results;
}

Query 3 - Sorting by Published Year, Top 5: This query sorts books by their publication year in ascending order, grabbing the first five.

public SearchResults<Book> SortBooksByPublishedYearInAscendingOrderAndGrabTopFive()
{
    Console.WriteLine("Query 3: Search all the books, order by published year in ascending order, take the top 5 results:");

    var options = new SearchOptions
    {
        Size = 5,
        OrderBy = { "PublishedYear asc" }
    };

    GrabAllBookFieldsFromTheIndex(options);

    var results = _searchClient.Search<Book>("*", options);

    return results;
}

Query 4 - Title Search for 'Mockingbird': We focus on the 'Title' field to find books with 'Mockingbird' in their title.

public SearchResults<Book> SearchBooksForTitleMatchingMockingBird()
{
    Console.WriteLine("Query 4: Search the Title field for the term 'Mockingbird':");

    var options = new SearchOptions
    {
        SearchFields = { "Title" }
    };

    GrabAllBookFieldsFromTheIndex(options);

    var results = _searchClient.Search<Book>("Mockingbird", options);

    return results;
}

Running the Project and Viewing Results

Executing the Project

After configuring the variables in your appsettings.json file, simply run the project. That's all there is to it. Should you encounter any issues, refer to the Troubleshooting & Additional Information section for guidance.

Observing the Query Outputs

After running the project, If everything has gone smoothly, we should now see the following results from our queries:

Query 1: Finds '1984' by George Orwell.

Id: 2, Title: 1984, Author: George Orwell, Genre: Dystopian, PublishedYear: 1949, Price: 19.99

Query 2: Lists books under $25, like 'The Great Gatsby' by F. Scott Fitzgerald and 'Moby Dick' by Herman Melville, sorted by price.

Id: 1, Title: The Great Gatsby, Author: F. Scott Fitzgerald, Genre: Fiction, PublishedYear: 1925, Price: 24.99
Id: 9, Title: Book Title 9, Author: Author 9, Genre: Other Genre, PublishedYear: 1959, Price: 24.99           
Id: 8, Title: Moby Dick, Author: Herman Melville, Genre: Adventure, PublishedYear: 1851, Price: 22.99         
Id: 5, Title: Brave New World, Author: Aldous Huxley, Genre: Dystopian, PublishedYear: 1932, Price: 22.99     
Id: 6, Title: The Hobbit, Author: J.R.R. Tolkien, Genre: Fantasy, PublishedYear: 1937, Price: 21.99           
Id: 4, Title: The Catcher in the Rye, Author: J.D. Salinger, Genre: Fiction, PublishedYear: 1951, Price: 20.99
Id: 2, Title: 1984, Author: George Orwell, Genre: Dystopian, PublishedYear: 1949, Price: 19.99                
Id: 3, Title: To Kill a Mockingbird, Author: Harper Lee, Genre: Fiction, PublishedYear: 1960, Price: 18.99    
Id: 7, Title: Pride and Prejudice, Author: Jane Austen, Genre: Romance, PublishedYear: 1813, Price: 18.99

Query 3: Brings up the top 5 books sorted by year, starting with the oldest, like 'Pride and Prejudice' by Jane Austen.

Id: 7, Title: Pride and Prejudice, Author: Jane Austen, Genre: Romance, PublishedYear: 1813, Price: 18.99
Id: 8, Title: Moby Dick, Author: Herman Melville, Genre: Adventure, PublishedYear: 1851, Price: 22.99         
Id: 1, Title: The Great Gatsby, Author: F. Scott Fitzgerald, Genre: Fiction, PublishedYear: 1925, Price: 24.99
Id: 5, Title: Brave New World, Author: Aldous Huxley, Genre: Dystopian, PublishedYear: 1932, Price: 22.99     
Id: 6, Title: The Hobbit, Author: J.R.R. Tolkien, Genre: Fantasy, PublishedYear: 1937, Price: 21.99

Query 4: Successfully locates 'To Kill a Mockingbird' by Harper Lee.

Id: 3, Title: To Kill a Mockingbird, Author: Harper Lee, Genre: Fiction, PublishedYear: 1960, Price: 18.99

Query Outputs Explained

Alright, let's break down what's happening in our query method calls. It's simpler than it looks, I promise!

We're basically doing two things in each query:

1. Setting Search Options: We tailor these options to meet our specific query needs. This is where we decide how we want to search, filter, sort or in other ways, find the data we need.

2. Selecting Fields to Receive: We select which Search Index fields we want to receive in our results.

Each query is crafted to showcase a different aspect of Azure Search's data retrieval prowess. Let me walk you through a couple of examples:

Filtering by Price, Sorted in Descending Order

Consider this query:

var options = new SearchOptions
{
    Filter = "Price lt 25",
    OrderBy = { "Price desc" }
};

var results = _searchClient.Search<Book>("*", options);

Here, we're implementing a filter to identify books priced below (less than, denoted as 'lt') $25, and we're sorting these results in descending (abbreviated as 'desc') order by price. The asterisk '*', used as a wildcard, signifies a broad search without pinpointing a specific term. This approach depends on the 'Price' field being set as both filterable and sortable in our Book class:

[SimpleField(IsSortable = true, IsFilterable = true)]
public double Price { get; set; }

Searching by Title for 'Mockingbird'

Now, let's look at a title-specific search:

var options = new SearchOptions
{
    SearchFields = { "Title" }
};

var results = _searchClient.Search<Book>("Mockingbird", options);

In this query, we're zeroing in on the 'Title' field to find matches for 'Mockingbird'. This approach narrows our search scope to titles, seeking the best match for the given term.

If you're still feeling a bit fuzzy on these concepts, I recommend running the code yourself and tweaking the queries. It's a hands-on way to see how flexible and powerful Azure Search can be. For further reading and more examples, check out the official Azure Search documentation.

By following these steps, you're well-equipped to embark on your own Proof of Concept (POC) projects. The best way to learn is by doing, so dive in and experiment! And hey, once you've mastered it, feel free to flaunt 'Expert Azure Search Index Developer' on your resume – you've earned it!

A Quick Tour of Our Index, Indexer, and Data Source Connection (Optional)

Let's briefly navigate the key components you've set up in Azure AI Search. This guide is focused on where to find these elements and what to expect when you get there.

Take a Closer Look at Your Index

After checking the search results in the Cognitive AI Search service, why not explore a bit more? In your Azure Search Service, navigate to the 'Indexes' section and select 'my-index' once more. Here, you can casually browse through the layout and settings. It's a good spot to see how your data is structured and get familiar with the finer details of your index setup.

Finding the Indexer

For the Indexer, head to the 'Indexers' section within your Search Service. Click on 'my-index-indexer' to see the indexing status and details. This is where the process of feeding data into your index takes place.

Locating the Data Source Connection

To check your Data Source Connection, navigate to 'Data Sources' in the Azure Search Service. Select 'my-index-data-source' to view the configuration that connects your data source to the Indexer.

This is a quick orientation to help you locate and recognize the components you've configured. Each plays a vital role in your Azure AI Search setup, and knowing where to find them is key to managing your search capabilities efficiently.

Navigating Challenges and Opportunities with Azure AI Search Indexes

Balancing Pros and Cons of Azure Indexer

Azure Indexers are a double-edged sword. On the one hand, they offer a quick start, letting you dive into the Search Service without getting bogged down in the complexities of index population. However, as you scale, limitations and errors can become prominent, especially with multiple indexes handling extensive data. While perfect for beginners, for larger-scale operations involving millions of records across numerous indexes, consider investing time in developing a custom indexer.

Caution with Index Infrastructure Changes

Modifying the Search Index Infrastructure requires careful planning. Changes, especially altering field properties, often require a complete index rebuild from scratch. This means losing all data and needing a full re-index, particularly challenging with large datasets. Adding new fields is generally safer, but always plan your index structure with future changes in mind. For detailed guidance, see Azure's index rebuild documentation.

Utilizing Azure AI Search Thoughtfully

Azure AI Search is indeed a powerful tool, yet it's essential to approach its integration thoughtfully. Building your entire application around it can introduce risks. While it's not discouraged, it's crucial to proceed with awareness of these potential challenges.

Preparedness and Backup Strategies

Having backup strategies and contingency plans is sensible, especially for scenarios where the service might face issues. This doesn't diminish the tool's value but highlights the importance of a well-rounded approach to application architecture.

Informed Integration and Exploration

From my own positive experiences with Azure AI Search, I've learned the importance of understanding both its strengths and complexities. It's beneficial to explore Microsoft's documentation in depth and also consider other search service providers. This broader perspective helps in making an informed decision, ensuring Azure AI Search is integrated into your application in a way that maximizes its advantages while being mindful of potential risks.

Essential Tips & Best Practices

Start Slow and Steady

Begin with small-scale Proof of Concept (POC) projects. This approach allows you to test different configurations and understand the tool's capabilities before fully integrating it into your system. As Robert C. Martin aptly puts it, 'The only way to go fast, is to go well.'

Deep Dive into Documentation

Familiarizing yourself with Azure's documentation is key. It helps you understand not just the basics but also the underlying principles of the tool. Don't hesitate to explore offerings from other providers and compare their features and pricing.

Comprehensive Testing is Key

Thorough testing is crucial for long-term success with Search Indexes. This includes validating Search Index data, ensuring smooth index creation during deployments, handling updates, managing indexer timeouts, and more. A well-planned and rigorously tested system can significantly enhance your search capabilities.

Troubleshooting & Additional Information

Dealing with Deployment Issues

If you run into deployment problems, start by checking the deployment logs in your Resource Group. Ensure you're following all the prerequisites as outlined in the documentation for each resource.

Solving Code-Related Problems

The repository provided has been extensively tested. Should you face any issues, please report them on the GitHub issues page, and I'll assist as soon as possible. Remember to configure the appsettings.json file correctly before running the project.

A key point to note: the Books table only supports unique identifiers, so you'll need to delete it if you need to rerun the project.

Looking Ahead: Advanced Search Index Management

In my upcoming posts, I will guide you through the advanced elements of managing Azure AI Search Indexes, catering to those who are incorporating this tool into their systems. We'll navigate together through topics like custom data indexing, complex data modeling, strategies for updating Search Index data, and effective testing methods for Search Indexes and their various components. And there’s more to it. Considering that each business faces unique challenges, we will also explore diverse and complex scenarios. My aim is to provide a path based on my experiences, enabling you to achieve a comprehensive understanding of Search Index management and beyond. And remember, there's always more to learn, so keep exploring on your own!

Wrapping Up Our Journey with Azure AI Search

Our exploration of Azure AI Search has opened a window into efficient, intelligent data management. From creating data sources to building, populating, and querying Search Indexes, we’ve laid the groundwork for you to harness this powerful tool effectively.

The hands-on experience with C# is just the beginning. As we progress, we'll delve into more complex topics such as custom data indexing and dynamic Search Index updates. This exploration will tailor Azure AI Search to suit your specific needs, enhancing your data management strategies.

The road to expertise in Azure AI Search, like any other tool, is a continuous learning process. Embrace the challenges, dive into the documentation, and don't shy away from thorough testing. Each step is an opportunity to grow and innovate.

Thank you for joining this journey. Keep learning and experimenting with Azure AI Search. It's a versatile tool for data management and you're just scratching the surface. Here's to your success in harnessing its full potential!

Additional Resources and Further Reading

General Microsoft Azure AI Search Documentation

• Azure AI Search Service Overview

• Indexes in Azure AI Search

• Indexers in Azure AI Search

• Modeling Complex Data Types in Azure AI Search

• Security overview for Azure AI Search

• Create a search index in the Azure portal

• Query with Search Explorer

• What is Azure Table Storage?

Best Practices & Improving Performance

• Tips for better performance

• Improving your Azure Cognitive Search performance

• Azure Cognitive Search performance: Setting yourself up for success

Visual Resources and Tutorials

• Azure Cognitive Search (more of an overview)

• Cognitive Search - Azure Search with AI (decent demo)

Don’t worry, there’s no heavy lifting required – only some clever coding. With Azure Bicep, your computer and a willingness to learn are all you need. Let’s dive in.

What is a Bicep File?

Bicep files are your gateway to deploying resources in Azure, offering a more straightforward approach than ARM Template files. If you're not familiar with ARM Templates, don't worry. It's helpful to focus solely on Bicep files for now. ARM templates can be challenging to understand and develop. Meanwhile, Microsoft's Bicep files provide equivalent functionality and capabilities but are notably easier to understand and work with, simplifying the experience of managing deployments in Azure.

Bicep Deployment Overview:

1. Writing the Code: Begin by writing the deployment code in a Bicep file.

2. Deployment to Azure: Use tools like the Azure CLI or the Azure portal to deploy the Bicep file to Azure.

3. Conversion to ARM Template Code: Azure automatically converts the Bicep file's code into ARM template code upon deployment.

4. Resource Creation: Finally, the ARM template code is used to create the necessary resources within Azure.

Benefits of Bicep

Wondering about the perks of Bicep files? Let’s look at it this way: Do you prefer automated processes over manual ones? If yes, we’re on the same page.

The primary advantage of using Bicep files is automation in resource deployments. Once you’ve crafted your Bicep file with the desired setup, you can reuse it across various platforms, like the Azure CLI, Azure Releases Pipeline, or even the Azure Portal.

Moreover, Bicep allows for parameter variation in each deployment. This flexibility is invaluable for systems with different environments. You can customize setups for testing and have entirely different configurations for production, ensuring each environment gets precisely what it needs.

What's the plan?

We're going to deploy a "Hello World" Web Application to Azure’s Web App service. It's not just about getting the app up and running; you'll pick up some cool skills along the way, such as:

• Setting up an Azure Resource Group – think of this as your project’s home base in Azure.

• Writing a Bicep file – the blueprint of our deployment plan.

• Discovering Resource Templates for Bicep Deployments – I'll guide you through finding and using resource templates, essential in shaping your Bicep files.

• Crafting a Bicep parameter file – this is where we get clever with our configuration, making our deployment adaptable.

• Deploying your Bicep file using the Azure CLI – you’ll learn not only how to execute the deployment but also how to preview it first to ensure everything goes smoothly.

By the end of this, you’ll not only have your "Hello World" Web Application running on Azure but you'll also gain a comprehensive understanding of resource groups, Bicep files, App Service Plans, and Web App Services. Plus, you’ll become adept at deploying with different parameters. Excited to begin? Let's get started!

I'm Assuming

1. You Have an Azure Subscription: To create resources, an Azure Subscription is required. If you don't have one yet, signing up is straightforward, and you'll immediately receive $200 in credits, which is more than enough for this project.

2. A Prepared Resource Group: This is where we'll host our resources. If you’re not familiar with setting up a resource group in Azure, you can follow the steps outlined here.

3. Basic Programming Knowledge: A fundamental understanding of programming will help you follow along more easily.

4. IDE Installed: Something like VSCode should be on your machine, ready to go.

5. Azure CLI: It's essential for our work. If it's not already set up, you can follow the download and install instructions in this link.

6. Azure CLI - Bicep: Let's make sure you've got the Azure CLI's Bicep extension ready to use. Double-check by following the instructions here.

Let's Get Started!

We're kicking off by setting up our environment. Start by creating a new workspace folder, and inside that, create a file named deploy-app.bicep.

To deploy an App Service, we first need to prepare an App Service plan. This plan acts as a hosting environment for the App Service and defines the region, features, cost, and compute resources. Service plans are referred to as Microsoft.Web/serverfarms in Bicep. You can easily find the template from Microsoft with a quick Google search, and then copy it into our Bicep file. For your convenience, here's the link for easy access.

After trimming down the unnecessary configurations, our code snippet looks like this:

resource symbolicname 'Microsoft.Web/serverfarms@2022-09-01' = {
  name: 'string'
  location: 'string'
  properties: {
    reserved: true
  }
  sku: {
    name: 'string'
  }
  kind: 'string'
}

Next, let's focus on the App Service Bicep template itself. This service is where our site will live. In Bicep, App Services are known as Microsoft.Web/sites. Following the same steps as before, we search, find, and refine the template. Don't worry about the plethora of configuration options; here's a streamlined version for our purpose:

resource appService 'Microsoft.Web/sites@2020-12-01' = {
  name: 'string'
  location: 'string'
  kind: 'string'
  properties: {
    serverFarmId: 'string'
    httpsOnly: bool
    siteConfig: {
      appSettings: [
        {
          name: 'string'
          value: 'string'
        }
      ]
      publicNetworkAccess: bool
    }
  }
}

Now, let's tackle the code deployment to our new App Service. For this, we use a template called Microsoft.Web/sites/sourcecontrols. This template allows us to deploy our code directly from a public GitHub repository. In our case, we'll use it to deploy a simple "Hello World" application. For newcomers, this means that our App Service will automatically pull the code from the specified repository and branch, making the deployment process smooth and automated.

Here's how that part of the Bicep file should look:

resource srcControls 'Microsoft.Web/sites/sourcecontrols@2021-01-01' = {
  parent: 'string'
  name: 'string'
  properties: {
    repoUrl: 'string'
    branch: 'string'
    isManualIntegration: bool
  }
}

By the end of this setup, your deploy-app.bicep file will be a robust blueprint for your Azure deployment. It should look something like this: deploy-app.bicep.

Preparing Your Bicep File: Structuring for Success

A quick note before we continue: If you're eager to get straight to the action, you're welcome to skip ahead to the Creating Parameter Files for Different Environments (Testing vs. Production) section. But if you're here for some handy tips on structuring your Bicep file for success, let's dive in!

Tips on Structuring Your Code

1. Logical Grouping: Organize related resources together. For instance, if you're creating a web app, group all resources related to the web app (like App Service, App Service Plan, and Storage Account) in one section.

2. Use Comments Wisely: Comments are crucial for explaining why something is done a certain way, especially for complex logic. However, avoid over-commenting obvious things as it can clutter your code.

3. Consistent Naming Conventions: Adopt a clear and consistent naming strategy for resources and variables. This not only makes your code more readable but also eases collaboration with others.

4. Modular Approach: Break down your code into modules for reusability and simplicity. Each module should ideally represent a logical unit of deployment, like a network module, a storage module, etc.

Best Practices for Scalability and Maintenance

1. Parameterization: Make use of parameters to avoid hard-coding values, especially those that might change (like environment names, sizes, and SKUs). This practice makes your Bicep files adaptable and easier to scale.

2. Leverage Resource Dependencies: Bicep automatically handles resource dependencies, but understanding and defining these properly ensures that resources are deployed in the correct order.

3. Version Control: Use version control systems like Git to track changes, collaborate with others, and maintain a history of your deployments. This is crucial for any scalable project.

4. Testing: Regularly test your Bicep files in different environments. Testing helps to identify issues early and ensures that your deployments are reliable.

5. Stay Updated: Make sure to keep up with the latest Azure Bicep updates. Regular releases bring new features, improvements, and bug fixes, all of which can greatly enhance your deployment experience.

By following these guidelines, you’ll set a strong foundation for your Azure Bicep files, ensuring they are not only effective and efficient but also scalable and easy to maintain.

Bicep Refinement

To enhance readability, maintainability, and overall deployment efficiency, let's fine-tune the base template by:

1. Extracting Change-Expected Values into Parameters: We're moving values that are likely to change into parameters for easy adjustments.

    param location string = resourceGroup().location // Bicep function returning the resource group location
    param textToReplaceSubtitleWith string
    param repositoryBranch string
   

2. Adding Clear and Concise Comments: Brief yet informative comments are key to understanding our parameters.

    @description('Azure resource deployment location.')
    param location string
   
    @description('The text to replace the default subtitle with.')
    param textToReplaceSubtitleWith string
   
    @description('Branch of the repository for deployment.')
    param repositoryBranch string
   

3. Setting Defaults for Some Parameters: We'll assign default values to parameters that might not get values during deployment.

    @description('Azure resource deployment location.')
    param location string = resourceGroup().location
   
    @description('The text to replace the default subtitle with.')
    param textToReplaceSubtitleWith string = 'This is my default subtitle text. Boring, right?'
   
    @description('Branch of the repository for deployment.')
    param repositoryBranch string = 'main'
   

4. Finalizing the Bicep File: We'll now incorporate our variables and set the remaining values and configurations.

    // App Service Plan Creation
    resource appServicePlan 'Microsoft.Web/serverfarms@2020-12-01' = {
      name: 'myAppServicePlan'
      location: location
      sku: {
        name: 'F1'
      }
      kind: 'app'
      properties: {
        reserved: false
      }
    }
   
    // Web App Creation
    resource appService 'Microsoft.Web/sites@2020-12-01' = {
      name: 'this-is-a-unique-name-for-the-web-app'
      location: location
      properties: {
        serverFarmId: appServicePlan.id
        httpsOnly: true
        siteConfig: {
          appSettings: [
            {
              name: 'TEXT_TO_REPLACE_SUBTITLE_WITH' // This value needs to match the name of the environment variable in the application code
              value: textToReplaceSubtitleWith
            }
            {
              name: 'SCM_DO_BUILD_DURING_DEPLOYMENT' // Build the application during deployment
              value: 'true'
            }
            {
              name: 'WEBSITE_NODE_DEFAULT_VERSION' // Set the default node version
              value: '~20'
            }
          ]
          publicNetworkAccess: 'Enabled'
        }
      }
    }
   
    // Source Control Integration
    resource srcControls 'Microsoft.Web/sites/sourcecontrols@2021-01-01' = {
      parent: appService
      name: 'web'
      properties: {
        repoUrl: 'https://github.com/FarzamMohammadi/hello-world'
        branch: repositoryBranch
        isManualIntegration: true
      }
    }
   

The final result should align with this deploy-webapp.bicep file. Wondering how I know which configurations to use? Well, it's a mix of scouring documentation and good old Google searches. Often, someone else's shared solution is the key to unlocking your deployment puzzles.

Creating Parameter Files for Different Environments (Testing vs. Production)

When deploying applications in Azure, it's common to have separate settings for different environments, such as testing and production. This is where parameter files come into play. They allow you to define environment-specific values without altering the core Bicep file. This approach not only simplifies management but also reduces the risk of errors when moving from testing to production.

For our "Hello World" web app, we've prepared two parameter files: parameters.test.json for testing and parameters.json for production. The primary distinction between these files is the source code branch they target.

parameters.json (Production Environment)

This file, used for the production environment, adheres to the default branch specified in the Bicep file. It doesn't require an explicit repositoryBranch parameter.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "textToReplaceSubtitleWith": {
      "value": "This site was released to Azure using a Bicep file. Biceps are better than ARM templates!"
    }
  }
}

parameters.test.json (Testing Environment)

In this file, we've included the repositoryBranch parameter. This simple yet effective change specifies that Azure should use code from the "test" branch for deployment. It's a practical way to ensure that during the testing phase, we are deploying exactly what's intended for testing, separate from our production environment.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "textToReplaceSubtitleWith": {
      "value": "This site is being tested and was released to Azure using a Bicep file. Let's see if Biceps are better than ARM templates."
    },
    "repositoryBranch": {
      "value": "test"
    }
  }
}

The repositoryBranch parameter in the test file is a key example of how changing one parameter can shift the deployment from production to testing. This helps demonstrate the real-world effects of such variations.

Time to Deploy

Azure CLI Preparation

First things first, we need to authenticate ourselves with the Azure CLI before deploying our Bicep file. It's simple: just run az login. You'll be taken to the Azure portal for a quick sign-in.

Once authenticated, a confirmation message will appear, indicating we're ready to proceed.

Now, let's set the Azure account subscription. Just run az account set --subscription <SUBSCRIPTION_ID>, replacing <SUBSCRIPTION_ID> with your actual subscription ID. And with that, we're set for deployment.

Deploying Bicep With Test Parameters

We'll start by deploying our Bicep file using the parameters.test.json file. Before deploying, it's wise to preview our changes. Using the --confirm-with-what-if option in Azure CLI, we can get a sneak peek of what our deployment will entail.

Run this command in the same directory as your Bicep file:

az deployment group create --resource-group my-rg --template-file deploy-webapp.bicep --parameters parameters.test.json --confirm-with-what-if

Understanding the Command:

• Resource Group: Use the --resource-group tag to specify the Resource Group where your resources will be deployed.

• Bicep File: The --template-file tag is followed by the name of your Bicep file.

• Parameter File: Use the --parameters tag to specify the Parameter JSON file that will override your Bicep parameters.

• Confirm Before Deployment: The --confirm-with-what-if tag gives you a preview of all the changes about to be made. It's your chance to review everything and confirm whether to proceed or not.

This preview helps identify any potential issues. Once confident, proceed with deployment by pressing y and enter.

Note: You can skip the preview and deploy directly using the same command without --confirm-with-what-if.

Monitoring the Deployment Progress

During deployment, check out "Deployments" in your resource group. This area lets you monitor the progress, review the status, and troubleshoot if needed.

After deployment, delve into the details by selecting your deployment, then your uniquely named App Service under "Resources".

Clicking the "Default domain" URL in the overview will take you to your newly deployed site.

If things went smoothly, you should see something like this:

Keep this tab open and let's discuss why we're seeing this "From the Test branch" text.

Executing the Steps: Video Demonstration

For a visual guide on deploying with test parameters, watch the first video. It details each step from "Azure CLI Preparation" to "Understanding the Deployment of the Test Branch."

Understanding the Test Branch Deployment

Our deployment targeted the test branch due to the parameters.test.json file. This file specifies the repositoryBranch as 'test', directing Azure to pull code from this branch.

Deploying Bicep With Production Parameters

Next, let’s deploy using the parameters.json file for a production setup.

Important Step: Disconnect Source Control in App Service

After testing your deployment, there's a crucial step to ensure everything works smoothly. I encountered a bug with the "Source Control" feature that prevented it from updating the branch name to "main" during redeployment. To avoid this issue and proceed successfully, manually disconnect the source control settings:

1. Go to Your App Service: Open the Azure portal and select your App Service created for Bicep deployment.

2. Find Deployment Center: Inside your App Service, locate and click on "Deployment Center."

3. Disconnect: In the Deployment Center, simply click the "Disconnect" button to remove the source control link.

This step is vital for the next part of our deployment process to work correctly.

If you're interested in the exact details of this bug, take a look at the issue I created in the Azure Bicep repository. Visit Azure Bicep Issue #12544 for more information.

Deploying the production code is similar to the test deployment. Use this command:

az deployment group create --resource-group my-rg --template-file deploy-webapp.bicep --parameters parameters.json

With parameters.json missing the repositoryBranch property, deployment defaults to the previously set "main" branch in our Bicep file.

@description('Branch of the repository for deployment.')
param repositoryBranch string = 'main'

After the deployment is complete, the view should now reflect the main branch code, as shown here, without any mentions of the Test branch:

And just like that, you’ve completed not one, but two Bicep deployments, and your website is live for the world to see. Congratulations!!

Additional Video: Production Parameter Deployment Walkthrough

For insights into deploying with the "production" parameters, check out this second video. It illustrates the use of parameters.json file, to simulate various deployment scenarios. This video serves as a practical complement to the first, enhancing your understanding of Bicep deployment flexibility.

Troubleshooting Common Issues

Deployment Troubleshooting

If you encounter non-syntax related issues during deployment, a good starting point is the Deployment logs in your Resource Group. These logs can help pinpoint which resources are causing trouble and clarify error messages.

Azure CLI Issues

For Azure CLI-related problems, the official documentation is a reliable resource. It offers detailed solutions and step-by-step guidance for many common issues.

Additional Tips:

• Check Parameter Mismatches: Ensure your parameter file correctly matches the parameters in your Bicep file. Even small mismatches or typos can lead to unexpected issues.

• Resource Dependencies: The order of resource deployment can be crucial. Review your Bicep file to ensure dependencies are appropriately managed.

• API Version Compatibility: Keep your API versions in Bicep templates up-to-date. Older versions may lead to unexpected behaviors.

• Community Forums and Support: Azure forums and platforms like Stack Overflow are valuable for finding solutions to complex problems.

• Local Validation and Testing: It's important to validate your Bicep files locally before deploying. You can use the Bicep CLI for this. The --confirm-with-what-if tag in Azure CLI is also useful for previewing changes and catching potential issues early.

Expanding Beyond "Hello World"

Next on the agenda is something a bit more advanced: a guide on deploying containerized applications to Azure using Bicep. This is for those looking to take their Azure skills up a notch, so stay tuned!

Got any Bicep or Azure topics you're itching to learn more about? Send me a message or leave a comment below. I'm always keen to collaborate and explore new ideas!

Conclusion: Wrapping Up and Looking Ahead

That’s it for our Azure Bicep guide! We’ve tackled everything from Bicep basics to getting a "Hello World" Web Application up and running on Azure. The goal was to make things straightforward and practical, and I hope you found it useful.

Along the way, we’ve seen how to write, refine, and deploy with Bicep, not to mention a little bit of Azure CLI in action. Hopefully, the video demo made things even clearer.

Up next, I’m planning to cover the deployment of containerized applications using Bicep, so there’s more useful content on the way. If you have any Bicep or Azure topics you’re curious about, feel free to let me know – your input could guide what comes next.

Thanks for joining me on this journey. Here’s to creating killer software solutions, and successful Azure projects! Keep an eye out for what’s next!

Additional Resources and Further Reading

General Microsoft Bicep Documentation:

• Bicep Overview and Basics

• Azure Bicep Official Documentation

• Azure Bicep GitHub Repository

• Azure App Service

• Azure App Serice Plan

• Continuous deployment to Azure App Service

• Bicep CLI commands

• How to deploy resources with Bicep and Azure CLI

Best Practices:

• Azure Bicep Best Practices Guide

Visual Resources and Tutorials:

• Getting Started with Azure Bicep (Video)

• Bicep Advanced Deployments - Part 1 (Video)

• Bicep Advanced Deployments - Part 2 (Video)

Setting up a Resource Group in Azure is easier than you think, but first, what exactly is a Resource Group? In Azure, a Resource Group is a container that holds related resources for an Azure solution. Think of it as a folder where you can group your Azure services (like databases, web apps, and storage accounts) together, making them easier to manage and monitor. This guide will walk you through each step to create your own Resource Group, simplifying the process to make it straightforward and hassle-free.

What You Need

• An Azure Subscription

Steps for Setting Up Your Azure Resource Group

1. Get Started: First, log into your Azure account. This is where everything begins.

2. Find Your Way: Click on "Create a resource" in the Azure dashboard. This is where you can search for all Azure resources.

   

3. Search and Select: Type “Resource Group” in the search bar and hit "Create".

4. 

   Name Your Group: Choose a name for your Resource Group, as this will be its identifier within Azure. Additionally, you can select the Region to specify the geographical location where your data and deployment metadata will be stored. This helps in optimizing the deployment according to your desired location.

5. 

   Check Everything: Go to the bottom of the page and click "Review + create". Just to make sure all looks good.

6. 

   Hit Create: After your settings pass the validation check, click "Create" to start building your Resource Group.

7. 

   Confirmation Time: Once Azure tells you it's done, click "Go to resource group". This takes you to your new group.

8. 

   Take a Look Around: You're now in your Resource Group. Check out its details under "Overview".

Your Resource Group is all set to accommodate new resources, forming an organized hub for managing your Azure solution. To learn more and explore advanced capabilities, refer to the official documentation here: Manage Azure resource groups by using the Azure portal.