Data Storage
- Always use BUFFERS when running an EXPLAIN . It gives some data that may be crucial for the investigation.
- Always, always try to get an Index Cond (called Index range scan in MySQL) instead of a Filter .
- Always, always, always assume PostgreSQL and MySQL will behave differently. Because they do.
Making a Postgres query 1,000 times faster
pgmock
Demo — Discord
pgmock is an in-memory PostgreSQL mock server for unit and E2E tests. It requires no external dependencies and runs entirely within WebAssembly on both Node.js and the browser.
Installation
npm install pgmock
If you'd like to run pgmock in a browser, see the Browser support section for detailed instructions.
Demo — Discord
pgmock is an in-memory PostgreSQL mock server for unit and E2E tests. It requires no external dependencies and runs entirely within WebAssembly on both Node.js and the browser.
Installation
npm install pgmock
If you'd like to run pgmock in a browser, see the Browser support section for detailed instructions.
stackframe-projects • GitHub - stackframe-projects/pgmock: In-memory Postgres for unit/E2E tests
Datasette is a tool for exploring and publishing data. It helps people take data of any shape, analyze and explore it, and publish it as an interactive website and accompanying API.
Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with... See more
Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with... See more
Datasette
A serverless vector database
built from first principles on object storage: 10-100x cheaper, usage-based pricing, massive scalability
built from first principles on object storage: 10-100x cheaper, usage-based pricing, massive scalability
turbopuffer
We can't share the exact formula for our search ranking, but here are the few parameters we consider:
- Exact match (rank #1)
- Frequency of matching lexemes using ts_rank
- Similarity score using similarity
- Type of record
- Popularity of the search result
- Similarity between the result’s alias and query
- Inverse of the result’s string length
How Levels.fyi Built Scalable Search with PostgreSQL
SQL Studio
Single binary, single command SQL database explorer. SQL studio supports SQLite , libSQL , PostgreSQL , MySQL and DuckDB .
Local SQLite DB File
sql-studio sqlite [sqlite_db]
Remote libSQL Server
sql-studio libsql [url] [auth_token]
PostgreSQL Server
sql-studio postgres [url]
MySQL/MariaDB Server
sql-studio mysql [url]
Local DuckDB File
sq... See more
Single binary, single command SQL database explorer. SQL studio supports SQLite , libSQL , PostgreSQL , MySQL and DuckDB .
Local SQLite DB File
sql-studio sqlite [sqlite_db]
Remote libSQL Server
sql-studio libsql [url] [auth_token]
PostgreSQL Server
sql-studio postgres [url]
MySQL/MariaDB Server
sql-studio mysql [url]
Local DuckDB File
sq... See more
frectonz • GitHub - frectonz/sql-studio: SQL Database Explorer [SQLite, libSQL, PostgreSQL, MySQL/MariaDB, DuckDB, ClickHouse]
For low throughput data, Grab uses Parquet with Copy on Write (CoW) .
Here's the main operations for Copy on Write:
Here's the main operations for Copy on Write:
- Write Operations - Whenever there's a write, you create a new version of the file that includes the latest change. You can also keep the previous version for consistency and rollback purposes. This helps prevent data corruption, incon
The Architecture of Grab's Data Lake
PGlite - Postgres in WASM
PGlite is a WASM Postgres build packaged into a TypeScript client library that enables you to run Postgres in the browser, Node.js and Bun, with no need to install any other dependencies. It is only 3.7mb gzipped.
import { PGlite } from "@electric-sql/pglite"
const db = new PGlite()
await db.query("select 'Hello world' as mes... See more
PGlite is a WASM Postgres build packaged into a TypeScript client library that enables you to run Postgres in the browser, Node.js and Bun, with no need to install any other dependencies. It is only 3.7mb gzipped.
import { PGlite } from "@electric-sql/pglite"
const db = new PGlite()
await db.query("select 'Hello world' as mes... See more
electric-sql • GitHub - electric-sql/pglite: Lightweight Postgres packaged as WASM into a TypeScript library for the browser, Node.js, Bun and Deno
Expose Delta Tables via REST APIs
Git repo to test 3 architectures to expose delta tables via REST APIs. See also my blogpost here. Architectures can be described as follows:
Git repo to test 3 architectures to expose delta tables via REST APIs. See also my blogpost here. Architectures can be described as follows:
- Architecture A: Direct, Web App with DuckDB. In this architecture, APIs are directly connecting to the delta table and there is no layer in between. This implies that all data