There is definitely a gap, and I don't think that Arrow tries to fill that. But I don't think that its wrong to have multiple implementations doing the same thing right? We have PostgresQL vs MySQL, both seem valid choices to me.
A SQL like query engine has its place. An in memory DataFrame also has its place. I think the wide-spread use of pandas proves that. I only think we can do that more efficient.
With regard to C++ vs Rust arrow. The memory underneath is the same, so having an implementation in both languages only helps more widespread adoption IMO.
Thank you for your work! I've decided to kick the tires after reading your Python book, I think you understimate the clarity of the API you have exposed which, honestly, looks a fair bit more sane than the tangled web that pandas is.
A SQL like query engine has its place. An in memory DataFrame also has its place. I think the wide-spread use of pandas proves that. I only think we can do that more efficient.
With regard to C++ vs Rust arrow. The memory underneath is the same, so having an implementation in both languages only helps more widespread adoption IMO.