Show HN: Continuous-eval – Granular evaluation of GenAI pipelines https://ift.tt/UCT8HXd

By Mustansir Malik
Show HN: Continuous-eval – Granular evaluation of GenAI pipelines Hi HN - we are the creators of “continuous-eval”, an open-source tool to test and evaluate generative AI apps. "Continuous-eval" came from our efforts to measure, validate and improve the reliability of a finance AI copilot we were developing for banks. End-to-end evaluation was not enough for us. We wanted to have granular evaluations that help pinpoint the bottlenecks and identify what / how to improve. We’ve since developed more metrics and made the framework more flexible so it can evaluate components like agent tool use, code change, retrieval steps, etc. Let us know what you think of our approach to GenAI App evaluation. https://ift.tt/VDiCjfu February 26, 2024 at 12:11AM

Comments

Post a Comment

Popular posts from this blog

Show HN: Samsar One Version One is now live https://ift.tt/THDkKSc

By Mustansir Malik
Show HN: Samsar One Version One is now live Hi all, 6 months ago I started building out an app which combines all the SoTA gen-ai models out there into one platform which anyone could use to create 2-4 min long videos starting from scratch, starting from an idea. I received a ton of feedback from HN community & others along the way. Today excited to announce version one of app.samsar.one, a platform which allows anyone to easily and consistently create long, coherent and consistent generative-ai videos while picking and choosing their model starting from merely an idea. https://app.samsar.one/ October 31, 2024 at 03:36PM
Read more

"Love Story Is Written In..." Ex-India Opener On MS Dhoni's Knock vs LSG

By Mustansir Malik
Chennai Super Kings treated their fans with a remarkable return to Chepauk after they registered a 12-run victory over Lucknow Super Giants in their IPL 2023 match on Monday. Asked to bat first, CSK posted a huge total of 217/7 in 20 overs, with Ruturaj Gaikwad smashing 57 off 31 balls. Later, LSG were restricted at 205/7 despite a quick knock of 53 runs off 22 balls by Kyle Mayers . Apart from such a thrilling encounter, one thing which will definitely be remembered as the major highlight of the season, will be MS Dhoni 's spectacular cameo off 12 runs off 3 balls. During the match, Dhoni marked his return to his fortress to Chepauk with two massive sixes off Mark Wood the first two deliveries he faced.  Seeing the vintage form of the CSK skipper, former India batter Aakash Chopra stated that Dhoni's epic cameo and CSK's win by 12 runs is a "love story" written in heaven. "MS Dhoni scored 12 runs and the victory margin in Chennai's win against Luckn...
Read more

Show HN: Infinite SCP Generator https://ift.tt/U1G9ONQ

By Mustansir Malik
Show HN: Infinite SCP Generator I work at .txt, the Outlines maintainers, and did this as a fun side project. I used Outlines to structure language model output in the format of an SCP entry. SCP entries have a specific structure. It's easy to get a language model to generate these structures via JSON without fail -- no parsing errors, no trailing commas, no missing brackets. Outlines ensures that all text generated will follow the SCP structure. We're also testing an internal structured text API, and figured this might be a good way to test out repeated, regular calls. Entries are generated at midnight and uploaded to a simple website. Check out the resulting entries here: https://ift.tt/BbXd1KH https://ift.tt/aehVMKr October 24, 2024 at 11:46PM
Read more