We have built Walden, a small data lake for (mostly) solitary use, consisting of a set of configurations and images for deployment into a Kubernetes cluster. We are releasing the code as free and open source software, hoping to lower some of the barriers to entry to the world of big data and AI. Check it out on our github, or read below for more info!
Rust looks like an increasingly important part of the future of scientific programming. It's as fast as C++, as flexible as Python, and does packaging better than even R. It is also very difficult to learn, and demands a lot of cognitive investment. On balance, the ability to write clean, fast, and safe code is worth it, but be prepared to re-learn programming.
The way we do science is evolving. We are building to cope with the change.