Show HN: Efficient intermediate data sharing for Kedro pipelines https://ift.tt/1IDo2Zg

By Mustansir Malik
Show HN: Efficient intermediate data sharing for Kedro pipelines Data processing pipelines are becoming increasingly complex, and intermediate data sharing is becoming the bottleneck, especially for data-intensive analytics and data preprocessing in machine learning and AI. This blog shows the possibility of efficient data sharing in data science pipelines, which naturally fits the settings of Kubernetes. It demonstrates how existing codebases can benefit from it without requiring an overhaul of the engineering effort. https://ift.tt/u4LpRKV July 19, 2023 at 03:23PM

Comments

Post a Comment

Popular posts from this blog

Show HN: Samsar One Version One is now live https://ift.tt/THDkKSc

By Mustansir Malik
Show HN: Samsar One Version One is now live Hi all, 6 months ago I started building out an app which combines all the SoTA gen-ai models out there into one platform which anyone could use to create 2-4 min long videos starting from scratch, starting from an idea. I received a ton of feedback from HN community & others along the way. Today excited to announce version one of app.samsar.one, a platform which allows anyone to easily and consistently create long, coherent and consistent generative-ai videos while picking and choosing their model starting from merely an idea. https://app.samsar.one/ October 31, 2024 at 03:36PM
Read more

Show HN: NewsCatcher's Hyperlocal News API – Granular City-Level News Feeds https://ift.tt/9UNMDR7

By Mustansir Malik
Show HN: NewsCatcher's Hyperlocal News API – Granular City-Level News Feeds We’ve launched NewsCatcher’s Hyperlocal News API to provide city-level news feeds for market analysis, localized apps, and data analytics. Why We Built It: Aggregating hyperlocal news is complex. Local news comes from diverse sources, many covering national topics, making it hard to isolate local events like transit updates, crime, extreme weather alerts, or infrastructure projects. Searching by location alone can lead to irrelevant results, especially with city names that overlap or match common words (e.g., Orange, Lake). What It Does: Our API provides precise, location-based news by processing 185K articles daily and tagging each with its associated_town down to the town level with 92% accuracy. Covering 31,000 U.S. locations, it uses advanced NLP to accurately associate location names, even when they’re common words. It enables you to: - Access near-real-time, location-focused news feeds - Filter by top...
Read more

Show HN: Krixik – Easily sequence small/specialized AI models (pip-installable) https://ift.tt/9nDHzUF

By Mustansir Malik
Show HN: Krixik – Easily sequence small/specialized AI models (pip-installable) Our initial intention was to experiment with a smartbook idea, but we couldn't find a good prototyping tool for small/specialized AI models. That rabbit hole took us through vector databases, through infra for small AI, and finally here. We’re particularly proud of the accessibility/simplicity of our code syntax. Krixik’s model library is limited to eleven model types (modules) and a few dozen models, but we will significantly expand it. Other enhancements, like a local client and several 3rd-party integrations, are also planned. You require API credentials to try Krixik. You can quickly get them through this form: https://ift.tt/013RxOa And here’s a demo video (our YouTube channel also has a couple dozen example videos in it): https://youtu.be/WpSSYLfvfdM https://ift.tt/hCwuGmP November 4, 2024 at 08:33PM
Read more