Fredrik talks to Matt Topol about Arrow and how the Arrow ecosystem is evolving. Arrow is an open source, columnar in-memory data format designed for efficient data processing and analytics - which means passing data between things without needing to transform it, and ideally even without needing to copy it.
What makes the ecosystem grow, and why is it very cool to have Arrow on the GPU? What is the connection between Arrow, machine learning, and Hugging face? Matt emphasizes the value of open standards, even as they work with or within more closed systems they can help open things up, and help bring about more modular solutions so that developers can focus on doing their core area really well.
This episode can be seen as a follow-up to episode 567, where Matt first joined to discuss everything Arrow.
Recorded during Øredev 2024.
Thank you Cloudnet for sponsoring our VPS!
Comments, questions or tips? We a re @kodsnack, @tobiashieta, @oferlundand @bjoreman on Twitter, have a page on Facebook and can be emailed at [email protected] if you want to write longer. We read everything we receive.
If you enjoy Kodsnack we would love a review in iTunes! You can also support the podcast by buying us a coffee (or two!) through Ko-fi.
Links
- Matt
- Matt's Øredev 2023 talks: State of the Apache Arrow ecosystem: How your project can leverage Arrow! and Leveraging Apache Arrow for ML workflows
- Previous episodes with Matt
- Øredev 2024
- Matt's Øredev 2024 talks - on Arrow ADBC and Composable and modular data systems
- ADBC - Arrow database connectivity
- Arrow
- Snowflake
- Snowflake drivers for ADBC
- Bigquery
- The Bigquery driver
- Microsoft Fabric
- Duckdb
- Postgres
- SQLite
- Arrow flight - RPC framework for services based on Arrow data
- Arrow flight SQL
- Micros