Nicolay Gerold

@nicolaygerold

All cardsCollections

So what abstractions do we have as of today? For example, let’s take the resource abstraction (Dagster, Prefect, referred to as an operator in Airflow). You abstract complex environments and connections away with a simple construct like that. You have the immediate benefits of defining that once and using it in every task or pipeline with... See more

Data Engineering • Data Orchestration Trends: The Shift From Data Pipelines to Data Products

Data Orchestration and Data Processing

There is no AI Strategy without a Data Strategy

Training AI models requires data. Whether you’re training existing models, developing models from scratch, or simply testing theories, high-quality data is crucial.

Incumbents have the data because they have the customers. They can immediately leverage customers’ data to train models and tune... See more

Jason Cohen • AI startups require new strategies: This time it's actually different

I would differentiate between data for existing behaviors vs new behaviors. The world and our tools are constantly changing. I think AI is one of those drivers. I think the startups who can either facilitate new behaviors the best (e.g. Cursor for AI-assisted coding) or who can create entirely new behaviors have a great way to generate unique training data.

Taking existing data and making it workable with AI is a lot of work and takes skilled labor (AI + domain expertise). I wouldn’t say incumbents have the complete edge. They have the edge in certain applications and behaviors under the assumptions these behaviors are not changing to render their applications useless.

𝘱𝘦𝘳𝘧𝘰𝘳𝘮𝘢𝘯𝘤𝘦: it will improve your LLM performance on given use cases (e.g., coding, extracting text, etc.). Mainly, the LLM will specialize in a given task (a specialist will always beat a generalist in its domain)

𝘤𝘰𝘯𝘵𝘳𝘰𝘭: you can refine how your model should behave on specific inputs and outputs, resulting in a more robust product

𝘮𝘰𝘥𝘶𝘭𝘢𝘳𝘪𝘻𝘢𝘵𝘪𝘰𝘯:... See more

Shortwave — rajhesh.panchanadhan@gmail.com [Gmail alternative]

LLMs

Motivation for finetuning

When it comes to adding drag and drop to your application, you almost certainly need a third-party library. The most popular choice in the past has been React Beautiful DnD. As of 2024, it is no longer receiving regular updates.

Going forward, a solid replacement for drag and drop is to use DnD Kit . It’s lightweight, very flexible, and the... See more

📚 React Libraries You Should Use In 2024

Frontend-Tools

Frontend Fuzzy Search

@m31coding/fuzzy-search is a frontend library for searching objects with ids (entities) by their names and features (terms). It is

Fast: A query takes usually well below 10 ms.

Accurate: Powered by n-grams with a novel approach of character sorting.

Multilingual: The language-agnostic design of the algorithm enables operation

GitHub - m31coding/fuzzy-search: A fast, accurate and multilingual fuzzy search library for the frontend.

Frontend-Tools

Real-time transcription powered by enterprise-grade Whisper ASR

Published on

Oct 2023

We’re happy to announce the general availability of Gladia’s live transcription, powered by a proprietary version of Whisper large-v2.

The feature utilizes WebSocket technology to transcribe audio and video in real-time at latency as low as 400 milliseconds and... See more

Gladia - Real-time transcription API powered by enterprise-grade Whisper ASR

Tools

Weclome to Ark UI

The headless component library for a wide range of JavaScript frameworks

Motivation

Most popular component libraries are designed to work with a specific JavaScript framework. While this setup works fine for individuals and small businesses, it can pose challenges for larger organizations or agencies dealing with various clients or... See more

Introduction | Ark UI

Frontend-Tools

Surya

Surya is a document OCR toolkit that does:

OCR in 90+ languages that benchmarks favorably vs cloud services

Line-level text detection in any language

Layout analysis (table, image, header, etc detection)

Reading order detection

GitHub - VikParuchuri/surya: OCR, layout analysis, reading order, line detection in 90+ languages

Data Loading

FastStream brokers provide convenient function decorators @broker.subscriber and @broker.publisher to allow you to delegate the actual process of:

consuming and producing data to Event queues, and

decoding and encoding JSON encoded messages

These decorators make it easy to specify the processing logic for your consumers and producers, allowing you to... See more

airtai • GitHub - airtai/faststream: FastStream is a powerful and easy-to-use Python framework for building asynchronous services that interact with event streams such as Apache Kafka and RabbitMQ.

Producing: One part of your system (a producer) wants to send a message, which could be anything like an order from a customer, a sensor reading, a log message, etc. Consuming: Another part of your system (a consumer) wants to receive and process that message. FastStream acts as a postman, ensuring the message is delivered accurately and efficiently from producer to consumer. Applications: E-commerce Platform: When a customer places an order, a message regarding that order needs to be communicated to various services like Order Management, Inventory Management, Payment Gateway, and more. FastStream will handle these messages, validate the order data, and ensure that all relevant microservices are informed in real-time, ensuring a smooth ordering process. IoT Sensor Network: Consider a network of IoT devices sending sensor readings (temperature, humidity, etc.) to a central server. FastStream ensures that all these readings are valid, documented, and are efficiently communicated to the central server for further analysis or action. Financial Transactions: In a financial application, transactions and trading data can be passed through FastStream to validate transactions, communicate them to various analytic and logging microservices, and even trigger alerts or actions based on the data.