Date: June 13, 2025
Category: Commodity Analysis, Geopolitical Risk
Tags: Oil Markets, Gas Trading, Geopolitical Intelligence, Risk Management

The energy markets have witnessed unprecedented volatility in recent years, with traditional analysis methods often failing to provide adequate warning of major price movements. The October 7, 2023 Hamas attack, which occurred precisely on the Jewish holiday of Simchat Torah, demonstrated how religious observances can serve as strategic timing mechanisms for geopolitical events affecting energy markets. This comprehensive analysis presents an enhanced intelligence framework that integrates Jewish holidays and lunar cycles to provide commodity specialists with 4-168 hour early warning capabilities for oil and gas market disruptions.

The Foundation: Unconventional Intelligence Indicators

The Pentagon Pizza Index Precedent

The proven effectiveness of unconventional intelligence gathering is best illustrated by the Pentagon Pizza Index, which successfully predicted recent Israeli strikes on Iran by monitoring pizza delivery spikes near defense facilities hours before military operations commenced. This method operates on a simple principle: during major geopolitical crises requiring extended work hours at defense facilities, food orders surge dramatically near key government buildings.

The concept has historical precedent, having been noted before the Grenada invasion in the 1980s, the Panama crisis in 1989, and Kuwait’s invasion when CIA pizza orders spiked the night before. The method’s effectiveness lies in its ability to detect unusual patterns in routine activities that correlate with heightened military or diplomatic activity.

Expanding Beyond Traditional Monitoring

Modern energy market intelligence requires sophisticated approaches that combine traditional economic analysis with innovative early-warning systems. Social media monitoring using artificial intelligence and natural language processing can identify emerging geopolitical topics hours before they appear on traditional news sources. Twitter-based algorithms have successfully identified geopolitical events at least a day before they became relevant on Google Trends, including missile launches and regional conflicts.

Tier 1: Jewish Holiday Intelligence Network

Simchat Torah: The Highest-Priority Indicator

Simchat Torah has emerged as the most critical indicator for geopolitical events affecting oil markets, with a reliability score of 95% and lead times of 4-8 hours. The October 7, 2023 attack was deliberately timed to coincide with this joyous Jewish festival, transforming it into what many now call the « Simchat Torah War » rather than simply referring to the calendar date.

Hamas leadership spent over two years planning this operation, specifically debating whether to conduct it on Yom Kippur or Simchat Torah, ultimately choosing the latter to maximize psychological impact on Jewish communities worldwide. The intelligence value of monitoring Simchat Torah stems from its symbolic significance as the « Joy of Torah, » making it an attractive target for adversaries seeking to inflict maximum emotional damage.

Oil prices surged 8% immediately following the October 7 attack, with crude futures initially rising 13% before stabilizing when no physical supply disruptions materialized. This pattern demonstrates how geopolitical events timed to Jewish holidays can trigger immediate risk premiums in energy markets even without actual supply chain impacts.

Yom Kippur Strategic Monitoring

Yom Kippur maintains critical importance as both the holiest day in Judaism and a historically preferred date for surprise military operations. The 1973 Yom Kippur War began on October 6, which coincided with both the Jewish Day of Atonement and the 10th day of Ramadan, demonstrating how religious calendar overlaps can amplify geopolitical risks.

The lunar phase during the 1973 attack was a new moon, providing optimal conditions for nighttime military operations while maximizing the element of surprise during religious observance. Modern intelligence frameworks must monitor increased activity patterns around Israeli defense facilities, emergency government meetings, and unusual corporate executive travel patterns 48-72 hours before Yom Kippur observance.

High Holy Days Comprehensive Framework

The Jewish High Holy Days period, encompassing Rosh Hashana through Yom Kippur, represents a 10-day window of elevated geopolitical risk requiring enhanced oil and gas market monitoring. Israeli society remains particularly vulnerable during this period, with government operations reduced and military personnel often on leave for religious observance.

Intelligence gathering during this period should focus on monitoring diaspora community travel patterns, synagogue security alerts, and changes in Israeli government operational tempo. Oil traders should implement enhanced position monitoring and volatility adjustments 48 hours before each High Holy Day, with particular attention to Middle Eastern crude benchmarks and shipping insurance premiums for vessels transiting critical chokepoints.

Tier 2: Lunar Cycle Market Intelligence

Full Moon Volatility Correlation

Statistical analysis reveals significant correlations between full moon phases and increased financial market volatility, with stock market trading volumes rising approximately 50% during full moon periods. The 2008 financial crisis and March 2020 COVID market crash both occurred near full moon phases, suggesting heightened emotional trading and increased market participation during these lunar periods.

Energy commodities demonstrate similar patterns, with crude oil futures showing increased intraday volatility ranges during full moon weeks compared to new moon periods. The October 7, 2023 attack occurred during a waning crescent moon phase (41.6% illumination), providing sufficient darkness for infiltration operations while maintaining enough visibility for coordination.

New Moon Return Patterns

Empirical research demonstrates that stock market returns during 15-day periods around new moon dates are approximately double those observed during full moon periods, with annualized differences reaching 7-10% for international markets. This « lunar cycle effect » appears strongest in emerging market economies and countries with higher baseline market volatility, suggesting cultural and psychological factors influence trading behavior beyond purely rational economic calculations.

The psychological mechanisms underlying lunar effects may stem from altered sleep patterns, increased emotional volatility, and changes in risk-taking behavior among market participants. Energy traders should consider position sizing adjustments based on lunar phase timing, particularly for short-term options strategies and volatility trading approaches.

Critical Geopolitical Chokepoint Monitoring

Strait of Hormuz Intelligence Network

With 20% of global oil flow transiting this chokepoint, implementing combined tanker AIS tracking with insurance premium monitoring provides crucial early warning capabilities. Recent tensions following Israeli strikes on Iran highlight the criticality of this route, where military exercise announcements and tanker insurance rate spikes provide 24-hour early warning of potential disruptions.

Red Sea Shipping Intelligence

Following Russian support for Houthi operations and continued attacks on commercial vessels, monitoring Houthi social media channels combined with real-time shipping delay data provides 12-hour lead times for Red Sea disruptions affecting 12% of global oil flows through the Suez Canal.

Russia-Ukraine Pipeline Monitoring

Monitoring pipeline pressure data combined with social media sentiment analysis along the Russia-Ukraine border provides early warning for routes handling 40% of European natural gas flows. Recent sabotage incidents demonstrate the vulnerability of these critical infrastructure assets to geopolitical tensions.

Advanced Maritime Intelligence Systems

Dark Shipping Detection Protocols

Implementing RF geolocation monitoring to identify vessels that disable AIS transponders, particularly around sanctioned countries like Iran and Venezuela, provides 24-hour lead times for detecting sanction evasion activities that can trigger enforcement actions affecting global oil flows.

Ship-to-Ship Transfer Surveillance

Monitoring unusual ship-to-ship transfers using satellite imagery and AIS data correlation provides early warning of sanctions circumvention activities. Recent intelligence reveals increased STS operations in the Aegean Sea for disguising Russian oil cargo origins.

Implementation Strategy and Risk Assessment

Phase 1: Immediate Implementation (0-30 days)

Deploy satellite-based gas flare detection and tanker AIS tracking systems, which provide 2-72 hour lead times with minimal implementation complexity. These systems leverage existing commercial satellite networks and maritime transponder data for immediate operational capability.

Simchat Torah alert networks should integrate Israeli community observance tracking with diaspora synagogue security reports to identify unusual activity patterns preceding potential incidents. Emergency meeting detection systems must monitor corporate calendar changes among energy sector executives, which historically provide 12-24 hour warning of significant market-moving events.

Phase 2: Advanced Capabilities (30-90 days)

Integrate social media sentiment analysis with corporate calendar monitoring to detect insider knowledge indicators. Focus on energy sector professional networks and executive communication patterns for early warning of strategic decisions.

Lunar phase correlation analysis requires integration of astronomical data with historical volatility patterns to establish predictive models for energy market behavior. Religious calendar overlap tracking becomes critical during periods when Jewish holidays coincide with Islamic observances, creating compounded risks for Middle Eastern stability and oil supply security.

Phase 3: Comprehensive Network (90+ days)

Establish automated correlation systems linking multiple indicator streams for predictive modeling. Combine weather-driven volatility forecasting with geopolitical risk assessment for comprehensive market intelligence.

Full moon volatility models require extensive backtesting against historical energy market data to establish reliable correlation thresholds and trading signal generation. The comprehensive system targets 85-90% accuracy for combined indicator alerts, with potential cost savings of $50-100 million per major event avoided through improved early warning capabilities.

Strategic Recommendations for Energy Market Participants

Immediate Risk Management Protocols

Energy traders should implement automated volatility adjustments and enhanced position monitoring beginning 48 hours before major Jewish holidays, with particular focus on Simchat Torah, Yom Kippur, and High Holy Days periods. Lunar phase tracking should inform medium-term position sizing decisions, with reduced leverage during full moon periods when market volatility typically increases.

Supply chain managers must build strategic reserves before high-risk religious observance periods and establish alternative routing protocols for shipments transiting Middle Eastern chokepoints during sensitive dates. Corporate risk management frameworks should integrate religious calendar data with traditional geopolitical intelligence gathering, ensuring decision-makers receive adequate warning of potential market disruptions.

Long-Term Strategic Implementation

The integration of religious and lunar intelligence requires dedicated analytical resources with deep cultural expertise in Jewish, Islamic, and lunar calendar systems. Investment in satellite monitoring capabilities for oil storage facilities and shipping chokepoints provides objective validation of activity level changes that supplement human intelligence gathering.

Regional energy security planning must account for the symbolic significance of religious observances in strategic decision-making by state and non-state actors. The enhanced intelligence framework provides competitive advantages through improved early warning capabilities, but requires careful operational security to prevent adversaries from adapting their timing strategies in response to known monitoring capabilities.

Conclusion

The integration of Jewish holidays and lunar cycles into oil and gas market intelligence represents a paradigm shift from traditional economic analysis toward comprehensive geopolitical risk assessment. The October 7, 2023 Hamas attack and subsequent market reactions validate the importance of religious timing considerations in modern energy market dynamics.

Success in implementing these unconventional intelligence methods requires combining high-automation satellite systems with human-analyzed social intelligence for comprehensive coverage. Energy market participants who adopt these enhanced intelligence frameworks will gain significant competitive advantages through improved early warning capabilities and more effective risk management during periods of heightened geopolitical tension.

The current volatile geopolitical environment, characterized by ongoing Middle Eastern conflicts and global supply chain vulnerabilities, makes these advanced intelligence capabilities essential for energy sector success. Organizations that fail to integrate these unconventional indicators into their risk management frameworks risk being consistently surprised by market events that more sophisticated intelligence systems can anticipate with 85-95% accuracy.

Disclaimer: This analysis is for informational purposes only and should not be considered as investment advice. Energy markets are subject to significant volatility and risk, and past performance does not guarantee future results.

Sources : Loic

Core Configuration

• Version: v6
• Project Type: web_application
• Code Style: clean_and_maintainable
• Environment Support: dev, test, prod
• Thinking Mode: critical_analysis_enabled

Critical Thinking Rules for Development

1. The Assumption Detector

ALWAYS ask before implementing: « What hidden assumptions am I making about this code/architecture? What evidence might contradict my current approach? »

2. The Devil’s Advocate

Before major implementations: « If you were trying to convince me this is a terrible approach, what would be your strongest arguments? »

3. The Ripple Effect Analyzer

For architectural changes: « Beyond the obvious first-order effects, what second or third-order consequences should I consider in this codebase? »

4. The Blind Spot Illuminator

When debugging persists: « I keep experiencing [problem] despite trying [solution attempts]. What factors might I be missing? »

5. The Status Quo Challenger

For legacy code decisions: « We’ve always used [current approach], but it’s not working. Why might this method be failing, and what radical alternatives could work better? »

6. The Clarity Refiner

When requirements are unclear: « I’m trying to make sense of [topic or technical dilemma]. Can you help me clarify what I’m actually trying to figure out? »

7. The Goal Realignment Check

During development sprints: « I’m currently working toward [goal]. Does this align with what I truly value, or am I chasing the wrong thing? »

8. The Fear Dissector

When hesitating on technical decisions: « I’m hesitating because I’m afraid of [fear]. Is this fear rational? What’s the worst that could realistically happen? »

9. The Feedback Forager

For fresh perspective: « Here’s what I’ve been thinking: . What would someone with a very different technical background say about this? »

10. The Tradeoff Tracker

For architectural decisions: « I’m choosing between [option A] and [option B]. What are the hidden costs and benefits of each that I might not be seeing? »

11. The Progress Checker

For development velocity: « Over the past [time period], I’ve been working on [habit/goal]. Based on my current actions, am I on track or just spinning my wheels? »

12. The Values Mirror

When feeling disconnected from work: « Lately, I’ve felt out of sync. What personal values might I be neglecting or compromising right now? »

13. The Time Capsule Test

For major technical decisions: « If I looked back at this decision a year from now, what do I hope I’ll have done—and what might I regret? »

Test-Driven Development (TDD) Rules

1. Write tests first before any production code.
2. Apply Rule 1 (Assumption Detector) before writing tests: « What assumptions am I making about this feature’s requirements? »
3. Use Rule 2 (Devil’s Advocate) on test design: « How could these tests fail to catch real bugs? »
4. Run tests before implementing new functionality.
5. Write the minimal code required to pass tests.
6. Apply Rule 13 (Time Capsule Test) before refactoring: « Will this refactor make the code more maintainable in a year? »
7. Do not start new tasks until all tests are passing.
8. Place all tests in a dedicated /tests directory.
9. Explain why tests will initially fail before implementation.
10. Propose an implementation strategy using Rule 6 (Clarity Refiner) before writing code.

Code Quality Standards with Critical Analysis

• Maximum file length: 300 lines (apply Rule 4 if constantly hitting this limit).
• Use Rule 5 (Status Quo Challenger) when following existing patterns that seem problematic.
• Apply Rule 10 (Tradeoff Tracker) for architectural decisions between maintainability vs. performance.
• Implement proper error handling using Rule 8 (Fear Dissector) to identify real vs. imagined failure scenarios.
• Use Rule 3 (Ripple Effect Analyzer) before major refactoring efforts.
• Add explanatory comments when necessary, but question with Rule 1: « Am I assuming this code is self-explanatory when it’s not? »

AI Assistant Critical Thinking Behavior

1. Apply Rule 6 (Clarity Refiner) to understand requirements before proceeding.
2. Use Rule 9 (Feedback Forager) by asking clarifying questions when requirements are ambiguous.
3. Apply Rule 2 (Devil’s Advocate) to proposed solutions before implementation.
4. Use Rule 4 (Blind Spot Illuminator) when debugging complex issues.
5. Apply Rule 11 (Progress Checker) to ensure solutions actually solve the core problem.

Critical Thinking Implementation Strategy

Pre-Development Analysis

• Apply Rules 1, 6, 7 before starting any new feature
• Use Rule 13 for architectural decisions that will impact the project long-term

During Development

• Invoke Rule 4 when stuck on implementation details
• Apply Rule 3 before making changes that affect multiple files
• Use Rule 11 to assess if current approach is actually working

Code Review Process

• Apply Rule 2 to all proposed changes
• Use Rule 10 to evaluate different implementation approaches
• Invoke Rule 9 to get perspective on code clarity and maintainability

Debugging Sessions

• Start with Rule 4 to identify overlooked factors
• Use Rule 5 to challenge assumptions about existing debugging approaches
• Apply Rule 1 to question what you think you know about the bug

Meta-Rule for Complex Problems

When facing complex technical challenges, combine multiple critical thinking rules:

« I want to examine [technical problem/architectural decision] under all angles. Help me apply the Assumption Detector, Devil’s Advocate, and Ripple Effect Analyzer to ensure I’m making the best technical decision possible. »

Workflow Best Practices Enhanced with Critical Thinking

Planning & Task Management

1. Apply Rule 7 (Goal Realignment Check) when updating PLANNING.md
2. Use Rule 11 (Progress Checker) when reviewing TASK.md milestones
3. Invoke Rule 6 (Clarity Refiner) for ambiguous requirements

Architecture Decisions

1. Always apply Rule 10 (Tradeoff Tracker) for technology choices
2. Use Rule 13 (Time Capsule Test) for decisions affecting long-term maintainability
3. Apply Rule 3 (Ripple Effect Analyzer) before major structural changes

Code Review & Refactoring

1. Use Rule 2 (Devil’s Advocate) on all proposed changes
2. Apply Rule 5 (Status Quo Challenger) to legacy code patterns
3. Invoke Rule 1 (Assumption Detector) during code reviews

Verification Rule Enhanced

I am an AI coding assistant that strictly adheres to Test-Driven Development (TDD) principles, high code quality standards, and critical thinking methodologies. I will:

1. Apply critical thinking rules before, during, and after development tasks
2. Write tests first using assumption detection and devil’s advocate analysis
3. Question my own proposed solutions using multiple perspective analysis
4. Challenge existing patterns when they may be causing problems
5. Analyze ripple effects of architectural decisions
6. Maintain awareness of hidden assumptions and blind spots
7. Regularly assess progress and goal alignment
8. Consider long-term implications of technical decisions
9. Seek diverse perspectives on complex problems
10. Balance rational analysis with intuitive concerns

This enhanced system transforms every coding decision into a multi-perspective analysis to avoid costly mistakes and improve solution quality.

These optimized rules will transform how Windsurf and Cursor AI work with your Python backend and Next.js frontend projects. By adding these configurations, you’ll get more accurate, consistent code suggestions that follow best practices and avoid common AI-generated code issues.

How to Implement in Windsurf

1. Option 1 – File Method:

• Create a file named .windsurfrules in your project’s root directory
• Copy and paste the entire code block below into this file
• Save the file

1. Option 2 – Settings Method:

• Open Windsurf AI
• Navigate to Settings > Set Workspace AI Rules > Edit Rules
• Paste the entire code block below
• Save your settings

How to Implement in Cursor

1. Option 1 – File Method:

• Create a file named .cursorrules in your project’s root directory
• Copy and paste the same code block below (it works for both platforms)
• Save the file

1. Option 2 – Settings Method:

• Open Cursor AI
• Click on your profile picture in the bottom left
• Select « Settings »
• Navigate to « AI » section
• Find « Custom Instructions » and click « Edit »
• Paste the entire code block below
• Click « Save »

After Implementation

• Restart your AI coding assistant or reload your workspace
• The AI will now follow these comprehensive rules in all your coding sessions
• You should immediately notice more relevant, project-specific code suggestions

These rules will significantly improve how your AI coding assistant understands your project requirements, coding standards, and technical preferences. You’ll get more relevant suggestions, fewer hallucinated functions, and code that better integrates with your existing codebase.

# Cursor Rules and Workflow Guide

## Core Configuration

- **Version**: `v5`
- **Project Type**: `web_application`
- **Code Style**: `clean_and_maintainable`
- **Environment Support**: `dev`, `test`, `prod`

---

## Test-Driven Development (TDD) Rules

1. Write tests **first** before any production code.
2. Run tests before implementing new functionality.
3. Write the **minimal code** required to pass tests.
4. Refactor only after all tests pass.
5. Do not start new tasks until all tests are passing.
6. Place all tests in a dedicated `/tests` directory.
7. Explain why tests will initially fail before implementation.
8. Propose an implementation strategy before writing code.
9. Check for existing functionality before creating new features.

---

## Code Quality Standards

- Maximum file length: **300 lines** (split into modules if needed).
- Follow existing patterns and project structure.
- Write modular, reusable, and maintainable code.
- Implement proper error handling mechanisms.
- Use type hints and annotations where applicable.
- Add explanatory comments when necessary.
- Avoid code duplication; reuse existing functionality if possible.
- Prefer simple solutions over complex ones.
- Keep the codebase clean and organized.

---

## AI Assistant Behavior

1. Explain understanding of requirements before proceeding with tasks.
2. Ask clarifying questions when requirements are ambiguous or unclear.
3. Provide complete, working solutions for each task or bug fix.
4. Focus only on relevant areas of the codebase for each task.
5. Debug failing tests with clear explanations and reasoning.

---

## Things to Avoid

- Never generate incomplete or partial solutions unless explicitly requested.
- Never invent nonexistent functions, APIs, or libraries.
- Never ignore explicit requirements or provided contexts.
- Never overcomplicate simple tasks or solutions.
- Never overwrite `.env` files without explicit confirmation.

---

## Implementation Guidelines

### General Rules

1. Always check for existing code before creating new functionality.
2. Avoid major changes to patterns unless explicitly instructed or necessary for bug fixes.

### Environment-Specific Rules

- Mock data should only be used for **tests**, never for development or production environments.


### File Management

- Avoid placing scripts in files if they are intended to be run only once.


### Refactoring Rules

1. Refactor files exceeding **300 lines** to improve readability and maintainability.

### Bug Fixing Rules

1. Exhaust all options using existing patterns and technologies before introducing new ones.
2. If introducing a new pattern, remove outdated implementations to avoid duplicate logic.

---

## Workflow Best Practices

### Planning \& Task Management

1. Use Markdown files (`PLANNING.md`, `TASK.md`) to manage project scope and tasks:
    - **PLANNING.md**: High-level vision, architecture, constraints, tech stack, tools, etc.
    - **TASK.md**: Tracks current tasks, backlog, milestones, and discovered issues during development.
2. Always update these files as the project progresses:
    - Mark completed tasks in `TASK.md`.
    - Add new sub-tasks or TODOs discovered during development.

### Code Structure \& Modularity

1. Organize code into clearly separated modules grouped by feature or responsibility.
2. Use consistent naming conventions and file structures as described in `PLANNING.md`.
3. Never create a file longer than 500 lines of code; refactor into modules if necessary.

### Testing \& Reliability

1. Create unit tests for all new features (functions, classes, routes, etc.).
2. Place all tests in a `/tests` folder mirroring the main app structure:
    - Include at least:
        - 1 test for expected use,
        - 1 edge case,
        - 1 failure case (to ensure proper error handling).
3. Mock external services (e.g., databases) in tests to avoid real-world interactions.

### Documentation \& Explainability

1. Update `README.md` when adding features, changing dependencies, or modifying setup steps.
2. Write docstrings for every function using Google-style formatting:

```python
def example(param1: int) -> str:
    """
    Brief summary of the function.

    Args:
        param1 (int): Description of the parameter.

    Returns:
        str: Description of the return value.
    """
```

3. Add inline comments explaining non-obvious logic and reasoning behind decisions.

---

## Verification Rule

I am an AI coding assistant that strictly adheres to Test-Driven Development (TDD) principles and high code quality standards. I will:

1. Write tests **first** before any production code.
2. Place all tests in a dedicated `/tests` directory.
3. Explain why tests initially fail before implementation begins.
4. Write minimal production code to pass the tests.
5. Refactor while maintaining passing tests at all times.
6. Enforce a maximum file length of **300 lines** per file (or 500 lines if specified).
7. Check for existing functionality before writing new code or features.
8. Explain my understanding of requirements before starting implementation work.
9. Ask clarifying questions when requirements are ambiguous or unclear.
10. Propose implementation strategies before writing any production code.
11. Debug failing tests with clear reasoning and explanations provided step-by-step.

---

## Server Management Best Practices

1. Restart servers after making changes to test them properly (only when necessary).
2. Kill all related servers from previous testing sessions to avoid conflicts.

---

## Modular Prompting Process After Initial Prompt

When interacting with the AI assistant:

1. Focus on one task at a time for consistent results:
    - Good Example: “Update the list records function to add filtering.”
    - Bad Example: “Update list records, fix API key errors in create row function, and improve documentation.”
2. Always test after implementing every feature to catch bugs early:
    - Create unit tests covering:
        - Successful scenarios,
        - Edge cases,
        - Failure cases.

---

These rules combine best practices for Python backend and Next.js frontend development with your specific coding patterns, workflow preferences, and technical stack requirements. The configuration instructs Windsurf AI to maintain clean, modular code that follows established patterns while avoiding common pitfalls in AI-assisted development.

Loic Baconnier

See also https://github.com/bacoco/awesome-cursorrules from
PatrickJS/awesome-cursorrules

Converting documents to structured formats like Markdown or JSON can be challenging, especially when dealing with PDFs, images, or Office files. The Text Extract API offers a robust solution to this common problem, providing high-accuracy conversion with advanced features.

Key Features

Document Processing
The API excels at converting various document types to Markdown or JSON, handling complex elements like tables, numbers, and mathematical formulas with remarkable accuracy. It utilizes a combination of PyTorch-based OCR (EasyOCR) and Ollama for processing.

Privacy-First Architecture
All processing occurs locally within your environment, with no external cloud dependencies. The system ships with Docker Compose configurations, ensuring your sensitive data never leaves your control.

Advanced Processing Capabilities

• OCR enhancement through LLM technology
• PII (Personally Identifiable Information) removal
• Distributed queue processing with Celery
• Redis-based caching for OCR results
• Flexible storage options including local filesystem, Google Drive, and AWS S3

Technical Implementation

Core Components
The system is built using FastAPI for the API layer and Celery for handling asynchronous tasks. This architecture ensures efficient processing of multiple documents simultaneously while maintaining responsiveness.

Storage Options
The API supports multiple storage strategies:

• Local filesystem with customizable paths
• Google Drive integration
• Amazon S3 compatibility

Getting Started

Prerequisites

• Docker and Docker Compose for containerized deployment
• Ollama for LLM processing
• Python environment for local development

Installationgit clone text-extract-api cd text-extract-api make install

Use Cases

Document Processing
Perfect for organizations needing to:

• Convert legacy documents to modern formats
• Extract structured data from PDFs
• Process large volumes of documents efficiently
• Remove sensitive information from documents

Integration Options

The API offers multiple integration methods:

• RESTful API endpoints
• Command-line interface
• TypeScript client library
• Custom storage profile configurations

Conclusion

Text Extract API represents a significant advancement in document processing technology, offering a self-hosted solution that combines accuracy with privacy. Whether you’re dealing with document conversion, data extraction, or PII removal, this tool provides the necessary capabilities while keeping your data secure and under your control.

Sources :

https://github.com/CatchTheTornado/text-extract-api

Are you concerned about data privacy and AI costs? The latest self-hosted AI tools offer powerful alternatives to cloud services. Let’s explore how to build a complete private AI stack using open-source solutions.

Why Self-Host Your AI Stack?

Private AI deployment brings multiple benefits:

• Complete data privacy and control
• No per-token or API costs
• Customizable to your needs
• Independence from cloud providers

The Essential Components

Let’s break down the key players in a self-hosted AI stack and how they work together.

LlamaIndex: Your Data Foundation

Think of LlamaIndex as your data’s brain. It processes and organizes your information, making it readily available for AI applications. With support for over 160 data sources and lightning-fast response times, it’s the perfect foundation for private AI deployments.

Flowise: Your Visual AI Builder

Flowise transforms complex AI workflows into visual puzzles. Drag, drop, and connect components to create sophisticated AI applications without diving deep into code. It’s particularly powerful for:

• Building RAG pipelines
• Creating custom chatbots
• Designing knowledge bases
• Developing AI agents

Ollama: Your Model Runner

Running AI models locally has never been easier. Ollama manages your models like a skilled librarian, supporting popular options like:

• Mistral
• Llama 2
• CodeLlama
• And many others

OpenWebUI: Your Interface Layer

Think of OpenWebUI as your AI’s front desk. It provides:

• Clean chat interfaces
• Multi-user support
• Custom pipeline configurations
• Local data storage

n8n: Your Automation Hub

n8n connects everything together, automating workflows and integrating with your existing tools. With over 350 pre-built integrations, it’s the glue that holds your AI stack together.

Real-World Applications

Document Processing System

Imagine a system where documents flow seamlessly from upload to intelligent responses:

1. Documents enter through OpenWebUI
2. LlamaIndex processes and indexes them
3. Flowise manages the RAG pipeline
4. Ollama provides local inference
5. n8n automates the entire workflow

Knowledge Management Solution

Create a private alternative to ChatGPT trained on your data:

1. LlamaIndex manages your knowledge base
2. Flowise designs the interaction flows
3. OpenWebUI provides the interface
4. Ollama serves responses locally
5. n8n handles integrations

Making It Work Together

The magic happens when these tools collaborate:

LlamaIndex + Flowise:

• Seamless data processing
• Visual RAG pipeline creation
• Efficient knowledge retrieval

Flowise + OpenWebUI:

• User-friendly interfaces
• Custom interaction flows
• Real-time responses

n8n + Everything:

• Automated workflows
• System integrations
• Process orchestration

Looking Ahead

The self-hosted AI landscape continues to evolve. These tools receive regular updates, adding features and improving performance. By building your stack now, you’re investing in a future of AI independence.

Final Thoughts

Building a private AI stack isn’t just about privacy or cost savings—it’s about taking control of your AI future. With these tools, you can create sophisticated AI solutions while keeping your data secure and your costs predictable.

Ready to start building your private AI stack? Begin with one component and gradually expand. The journey to AI independence starts with a single step.

Introduction

In today’s AI landscape, having a secure, efficient, and self-contained development environment is crucial. This guide presents a comprehensive solution that combines best-in-class open-source tools for AI development, all running locally on your infrastructure.

Key Components

• Ollama: Run state-of-the-art language models locally
• n8n: Create automated AI workflows
• Qdrant: Vector database for semantic search
• Unstructured: Advanced document processing
• Argilla: Data labeling and validation
• Opik: Model evaluation and monitoring
• JupyterLab: Interactive development environment

Benefits

• Complete data privacy and control
• No cloud dependencies
• Cost-effective solution
• Customizable infrastructure
• Seamless tool integration

Prerequisites

Hardware Requirements

• CPU: 4+ cores recommended
• RAM: 16GB minimum, 32GB recommended
• Storage: 50GB+ free space
• GPU: NVIDIA GPU with 8GB+ VRAM (optional)

Software Requirements

# Check Docker version
docker --version
# Should be 20.10.0 or higher

# Check Docker Compose version
docker compose version
# Should be 2.0.0 or higher

# Check Git version
git --version
# Should be 2.0.0 or higher

System Preparation

# Create project directory
mkdir -p ai-development-environment
cd ai-development-environment

# Create required subdirectories
mkdir -p notebooks
mkdir -p shared
mkdir -p n8n/backup
mkdir -p data/documents
mkdir -p data/processed
mkdir -p data/vectors

Directory Structure

ai-development-environment/
├── docker-compose.yml
├── .env
├── notebooks/
│   ├── examples/
│   └── templates/
├── shared/
│   ├── documents/
│   └── processed/
├── n8n/
│   └── backup/
└── data/
    ├── documents/
    ├── processed/
    └── vectors/

Configuration Files

Environment Variables (.env)

# Database Configuration
POSTGRES_USER=n8n
POSTGRES_PASSWORD=n8n
POSTGRES_DB=n8n

# n8n Security
N8N_ENCRYPTION_KEY=1234567890
N8N_USER_MANAGEMENT_JWT_SECRET=1234567890

# Service Configuration
JUPYTER_TOKEN=masterclass
ARGILLA_PASSWORD=masterclass

# Resource Limits
POSTGRES_MAX_CONNECTIONS=100
ELASTICSEARCH_HEAP_SIZE=1g

Docker Compose Configuration

Create docker-compose.yml:

version: '3.8'

volumes:
  n8n_storage:
    driver: local
  postgres_storage:
    driver: local
  ollama_storage:
    driver: local
  qdrant_storage:
    driver: local
  open-webui:
    driver: local
  jupyter_data:
    driver: local
  opik_data:
    driver: local
  elasticsearch_data:
    driver: local

networks:
  demo:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.0.0/16

services:
  jupyter:
    image: jupyter/datascience-notebook:lab-4.0.6
    networks: ['demo']
    ports:
      - "8888:8888"
    volumes:
      - jupyter_data:/home/jovyan
      - ./notebooks:/home/jovyan/work
      - ./shared:/home/jovyan/shared
    environment:
      - JUPYTER_ENABLE_LAB=yes
      - JUPYTER_TOKEN=${JUPYTER_TOKEN}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    command: start-notebook.py --NotebookApp.token='${JUPYTER_TOKEN}'
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8888/api"]
      interval: 30s
      timeout: 10s
      retries: 3

  unstructured:
    image: quay.io/unstructured-io/unstructured-api:latest
    networks: ['demo']
    ports:
      - "8000:8000"
    volumes:
      - ./shared:/home/unstructured/shared
    command: --port 8000 --host 0.0.0.0
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  opik:
    image: comet/opik:latest
    networks: ['demo']
    ports:
      - "5173:5173"
    volumes:
      - opik_data:/root/opik
      - ./shared:/root/shared
    environment:
      - OPIK_BASE_URL=http://localhost:5173/api
    restart: unless-stopped

  argilla:
    image: argilla/argilla-server:latest
    networks: ['demo']
    ports:
      - "6900:6900"
    environment:
      - ARGILLA_ELASTICSEARCH=http://elasticsearch:9200
      - DEFAULT_USER_PASSWORD=${ARGILLA_PASSWORD}
    depends_on:
      elasticsearch:
        condition: service_healthy
    restart: unless-stopped

  elasticsearch:
    image: elasticsearch:8.11.0
    networks: ['demo']
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms512m -Xmx${ELASTICSEARCH_HEAP_SIZE}
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:9200/_cluster/health | grep -vq '\"status\":\"red\"'"]
      interval: 20s
      timeout: 10s
      retries: 5

  # Workflow Automation
  n8n:
    <<: *service-n8n
    container_name: n8n
    restart: unless-stopped
    ports:
      - 5678:5678
    volumes:
      - n8n_storage:/home/node/.n8n
      - ./n8n/backup:/backup
      - ./shared:/data/shared
    depends_on:
      postgres:
        condition: service_healthy
      n8n-import:
        condition: service_completed_successfully

  n8n-import:
    <<: *service-n8n
    container_name: n8n-import
    entrypoint: /bin/sh
    command:
      - "-c"
      - "n8n import:credentials --separate --input=/backup/credentials && n8n import:workflow --separate --input=/backup/workflows"
    volumes:
      - ./n8n/backup:/backup
    depends_on:
      postgres:
        condition: service_healthy

  # Chat Interface
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    networks: ['demo']
    restart: unless-stopped
    container_name: open-webui
    ports:
      - "3000:8080"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui:/app/backend/data

  # Language Models
  ollama-cpu:
    profiles: ["cpu"]
    <<: *service-ollama

  ollama-gpu:
    profiles: ["gpu-nvidia"]
    <<: *service-ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  ollama-pull-llama-cpu:
    profiles: ["cpu"]
    <<: *init-ollama
    depends_on:
      - ollama-cpu

  ollama-pull-llama-gpu:
    profiles: ["gpu-nvidia"]
    <<: *init-ollama
    depends_on:
      - ollama-gpu

This completes the docker-compose.yml configuration, combining the original starter kit services with our additional AI development tools. The setup provides a complete environment for AI development, document processing, and workflow automation.

Service Integration Examples

Python Code Examples

Create a new notebook in JupyterLab with these integration examples:

# Document Processing Pipeline
import requests
from pathlib import Path

# Unstructured API Integration
def process_document(file_path):
    with open(file_path, 'rb') as f:
        response = requests.post(
            'http://unstructured:8000/general/v0/general',
            files={'files': f}
        )
    return response.json()

# Ollama Integration
def query_llm(prompt):
    response = requests.post(
        'http://ollama:11434/api/generate',
        json={'model': 'llama3.1', 'prompt': prompt}
    )
    return response.json()

# Qdrant Integration
from qdrant_client import QdrantClient

def store_embeddings(vectors, metadata):
    client = QdrantClient(host='qdrant', port=6333)
    client.upsert(
        collection_name="documents",
        points=vectors,
        payload=metadata
    )

AI Templates and Workflows

Document Processing Workflow

1. Upload documents to shared directory
2. Process with Unstructured API
3. Generate embeddings with Ollama
4. Store in Qdrant
5. Query through n8n workflows

Docker Compose Profiles

The project uses different Docker Compose profiles to accommodate various hardware configurations:

For NVIDIA GPU Users

docker compose --profile gpu-nvidia pull
docker compose create && docker compose --profile gpu-nvidia up

This profile enables GPU acceleration for Ollama, providing faster inference times for language models[1].

For Apple Silicon (M1/M2)

docker compose pull
docker compose create && docker compose up

Since GPU access isn’t available in Docker on Apple Silicon, this profile runs without GPU specifications[1].

For CPU-only Systems

docker compose --profile cpu pull
docker compose create && docker compose --profile cpu up

This profile configures services to run on CPU only, suitable for systems without dedicated GPUs[1].

Service Configurations

Core Services

• n8n: Workflow automation platform with AI capabilities
• Ollama: Local LLM service with configurable GPU/CPU profiles
• Qdrant: Vector database for embeddings
• PostgreSQL: Database backend for n8n
• Open WebUI: Chat interface for model interaction

Additional Services

• Unstructured: Document processing service
• Argilla: Data labeling platform
• Opik: Model evaluation tools
• JupyterLab: Development environment

Volume Management

Each service has dedicated persistent storage:

• n8n_storage
• postgres_storage
• ollama_storage
• qdrant_storage
• elasticsearch_data
• jupyter_data

Networking

All services communicate through a shared ‘demo’ network, allowing internal service discovery and communication

Make websites accessible for AI agents 🤖.

Browser use is the easiest way to connect your AI agents with the browser. And it’s Free…

https://github.com/gregpr07/browser-use

from langchain_openai import ChatOpenAI
from browser_use import Agent

agent = Agent(
    task="Find a one-way flight from Bali to Oman on 12 January 2025 on Google Flights. Return me the cheapest option.",
    llm=ChatOpenAI(model="gpt-4o"),
)

# ... inside an async function
await agent.run()