Data Storage

SQL has limitations as it is built on relational concepts and relies on binary joins.

The future of databases is shifting towards relational knowledge graphs, allowing the flexibility to work with various data structures beyond tables.

Businesses are moving towards explicitly modeling business semantics and logic, which are often stored in... See more

1. Scalability is crucial - systems need to be designed with the assumption that query volume, document corpus size, indexing complexity etc. could increase by 10x. What works at one scale may completely break at a higher scale.


2. Sharding the index, either by document or by word, is important to distribute the indexing and querying load across machines.

... See more

SQL Studio

Single binary, single command SQL database explorer. SQL studio supports SQLite , libSQL , PostgreSQL , MySQL and DuckDB .

Local SQLite DB File

sql-studio sqlite [sqlite_db]

Remote libSQL Server

sql-studio libsql [url] [auth_token]

PostgreSQL Server

sql-studio postgres [url]

MySQL/MariaDB Server

sql-studio mysql [url]

Local DuckDB File

sq... See more

Datasette is a tool for exploring and publishing data. It helps people take data of any shape, analyze and explore it, and publish it as an interactive website and accompanying API.

Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with... See more

Rottnest : Data Lake Indices

You don't need ElasticSearch or some vector database to do full text search or vector search. Parquet + Rottnest is all you need. Rottnest is like Postgres indices for Parquet. Read more on what it can do for e.g. logs here.

Installation

Local installation: pip install rottnest .

Rottnest supports many different index... See more

Denormalization

Another way Reddit minimizes joins is by using denormalization.

They took all the metadata fields required for displaying an image post and put them together into a single JSONB field. Instead of fetching different fields and combining them, they can just fetch that single JSONB field.

This made it much more efficient to fetch all the... See more

With Quary, engineers can:

• 🔌 Connect to their Database


• 📖 Write SQL queries to transform, organize, and document tables in a database


• 📊 Create charts, dashboards and reports (in development)


• 🧪 Test, collaborate & refactor iteratively through version control


• 🚀 Deploy the organised, documented model back up to the database

View the documentation.

pg_vectorize: a VectorDB for Postgres

A Postgres extension that automates the transformation and orchestration of text to embeddings and provides hooks into the most popular LLMs. This allows you to do vector search and build LLM applications on existing data with as little as two function calls.

This project relies heavily on the work by pgvector... See more

Our Goals

We made it lightweight and kept the efficiency in mind:

• Self-contained
  
  We ship a single dependency-free binary that runs on all Linux distributions


• Fast to deploy, safe to operate
  
  We are sysadmins, we know the value of operator-friendly software


• Deploy everywhere on every machine
  
  We do not have a dedicated backbone, and neither do you,
  
  so

... See more