Data Storage

7 must-know strategies to scale your database

Indexing:

Check the query patterns of your application and create the right indexes.

Materialized Views:

Pre-compute complex query results and store them for faster access.

Denormalization:

Reduce complex joins to improve query performance.

Vertical Scaling

Boost your database server by adding more CPU, RAM, or... See more

This README is under construction as we work to build a new community driven high performance key-value store.

This project was forked from the open source Redis project right before the transition to their new source available licenses.

This README is just a fast quick start document. We are currently working on a more permanent documentation page.

W... See more

Data bases have gotten so good at this, that the term is almost misleading now. “Base” suggests something rigid, without which the data would slip away. But the data is always there, just bits on a nameless hard disk. The structure and the accessibility that a modern database provides exist completely independently from that hard disk. That’s right... See more

We can't share the exact formula for our search ranking, but here are the few parameters we consider:

• Exact match (rank #1)


• Frequency of matching lexemes using ts_rank


• Similarity score using similarity


• Type of record


• Popularity of the search result


• Similarity between the result’s alias and query


• Inverse of the result’s string length

A serverless vector database

built from first principles on object storage: 10-100x cheaper, usage-based pricing, massive scalability

Rottnest : Data Lake Indices

You don't need ElasticSearch or some vector database to do full text search or vector search. Parquet + Rottnest is all you need. Rottnest is like Postgres indices for Parquet. Read more on what it can do for e.g. logs here.

Installation

Local installation: pip install rottnest .

Rottnest supports many different index... See more

ReadySet is a transparent database cache for Postgres & MySQL that gives you the performance and scalability of an in-memory key-value store without requiring that you rewrite your app or manually handle cache invalidation. ReadySet sits between your application and database and turns even the most complex SQL reads into lightning-fast lookups.... See more

1. Scalability is crucial - systems need to be designed with the assumption that query volume, document corpus size, indexing complexity etc. could increase by 10x. What works at one scale may completely break at a higher scale.


2. Sharding the index, either by document or by word, is important to distribute the indexing and querying load across machines.

... See more

memary: Open-Source Longterm Memory for Autonomous Agents

memary demo

Why use memary?

Agents use LLMs that are currently constrained to finite context windows. memary overcomes this limitation by allowing your agents to store a large corpus of information in knowledge graphs, infer user knowledge through our memory modules, and only retrieve... See more