Data Storage

Expose Delta Tables via REST APIs

Git repo to test 3 architectures to expose delta tables via REST APIs. See also my blogpost here. Architectures can be described as follows:

• Architecture A: Direct, Web App with DuckDB. In this architecture, APIs are directly connecting to the delta table and there is no layer in between. This implies that all data

... See more

Denormalization

Another way Reddit minimizes joins is by using denormalization.

They took all the metadata fields required for displaying an image post and put them together into a single JSONB field. Instead of fetching different fields and combining them, they can just fetch that single JSONB field.

This made it much more efficient to fetch all the... See more

ReadySet is a transparent database cache for Postgres & MySQL that gives you the performance and scalability of an in-memory key-value store without requiring that you rewrite your app or manually handle cache invalidation. ReadySet sits between your application and database and turns even the most complex SQL reads into lightning-fast lookups.... See more

Unlike some other popular algorithms, DiskANN is designed to keep memory usage to a minimum. This makes it a great match for use cases where Turso already excels at.

#Multitenancy

Turso allows for an easy implementation of a database-per-tenant pattern, where databases can be cheaply created on-demand. Keeping memory consumption at bay is critical... See more

Rottnest : Data Lake Indices

You don't need ElasticSearch or some vector database to do full text search or vector search. Parquet + Rottnest is all you need. Rottnest is like Postgres indices for Parquet. Read more on what it can do for e.g. logs here.

Installation

Local installation: pip install rottnest .

Rottnest supports many different index... See more

SQL has limitations as it is built on relational concepts and relies on binary joins.

The future of databases is shifting towards relational knowledge graphs, allowing the flexibility to work with various data structures beyond tables.

Businesses are moving towards explicitly modeling business semantics and logic, which are often stored in... See more

It turns out there's a handy feature in PostgreSQL called row constructor comparisons that allows me to compare tuples of columns. That's exactly what we need. Instead of doing CreateAt > ?1 OR (CreateAt = ?1 AND Id > ?2) , we can do ( CreateAt, Id) > (?1, ?2) . And the row constructor comparisons are lexicographical, meaning that it's... See more

pg_vectorize: a VectorDB for Postgres

A Postgres extension that automates the transformation and orchestration of text to embeddings and provides hooks into the most popular LLMs. This allows you to do vector search and build LLM applications on existing data with as little as two function calls.

This project relies heavily on the work by pgvector... See more

For High Throughput data, Grab uses Apache Avro with a strategy called Merge on Read (MOR) .

Here's the main operations with Merge on Read:

• Write Operations - When data is written, it's appended to the end of a log file. This is much more efficient than merging it in the current data and reduces the latency of writes.


• Read Operations - When you need

... See more