Seamlessly run
machine learning on Kubernetes
Airflow
Spark
Databricks
Lambda
Kubernetes
Kubernetes
Airflow
Spark
Databricks
Lambda
Kubernetes
Aqueduct is an MLOps framework that allows you to define and deploy machine learning and LLM workloads on any cloud infrastructure.
# Use an existing LLM.
vicuna = aq.llm_op('vicuna_7b', engine='eks-us-east-2')
features = vicuna(
raw_logs,
{
prompt:
"Turn this log entry into a CSV: {text}"
}
)
# Or write a custom op on your favorite infrastructure!
@op(
engine='kubernetes',
# Get a GPU.
resources={'gpu_resource_name': 'nvidia.com/gpu'}
)
def train(featurized_logs):
return model.train(features) # Train your model.
train(features)
Integrated with your cloud
modern ML teams
Python-native
Define your ML tasks in vanilla Python — no more YAML configs, Dockerfiles, or DSLs to worry about.
Integrated with your cloud
Aqueduct workflows can run on any cloud infrastructure you already use — choose from Kubernetes, Spark, Airflow, Lambda, or Databricks.
Native LLM support
Integrate powerful LLMs into your pipelines with a single API call — all without any operational overhead.
Deep visibility into data & code
Regardless of where your code is running, Aqueduct captures and validates the code and data at every stage, so you know what ran and when it ran.
Easy to debug
Every function run has error logs and stack traces, so you can pinpoint errors quickly.
Runs securely in your cloud
Aqueduct is fully open-source, so you can be sure your code and data is always where it's supposed to be.
Trusted by top machine learning teams
What our users are saying
Jack Reynolds
Machine Learning Engineer
Aqueduct gives me a comprehensive view of the data flow in my ML pipelines. Today, this context is scattered across a notebook and a couple Miro boards, but these pipelines change so fast that it's hard to keep track of them. To see all of my pipelines end-to-end and to see everything light up green is going to give me the confidence that I need to know everything's working and how well it's working.
Pablo Vega-Behar
Director of Data Science, Sparks & Honey
Aqueduct makes it easy to add a couple decorators to your codebase and automatically capture metrics, track them over time, and enforce constraints on those measurements over time. I don't have to think about where or how I track these things because Aqueduct does it for me.
Anchit Desai
Lead Engineer, Replate
Our previous infrastructure was built by data scientists and engineers with little knowledge of each others' best practices. It worked but wasn't ideal for us. Aqueduct streamlines production data science by providing a simple Pythonic API that makes it easy to get models into production. We can focus on delivering better models rather than maintaining cloud infrastructure.
Why Aqueduct?
Get more value out of machine learning, faster. With Aqueduct, you can run experiments more quickly, deploy models faster, and debug failures effectively.
Centralize your machine learning code, data, and metadata in a single place. Always know what's running, whether it worked, and who's responsible.
Have confidence that your models and predictions are behaving like they're supposed to, and proactively detect failures before stakeholders and customers complain.
Avoid vendor and cloud lock-in by using general-purpose, system-agnostic APIs. Experiment more nimbly with new tools and infrastructure.
Get started with Aqueduct
Fully open-source and easy to setup on your laptop or in your cloud
Check out the code
See how Aqueduct works, make a suggestion, and share your feedback — we'd love to hear from you!
Stay up to date with Aqueduct
© 2023 Aqueduct, Inc. All rights reserved.