Infrastructure and performance for AI/ML/DL workloads.
This section curates infrastructure, cloud, and DevOps resources for AI/ML/DL workloads. It covers compute/storage options, accelerators (CPU/GPU/TPU/IPU), platforms, tooling, and performance references with practical links.
Home · 📚 Data · 📓 Notebooks · 🧰 Tools · ☁️ Infrastructure
- System / Infra
- Compute & Storage
- Grid computing / Super computing
- Cloud services
- Tools
- CPU
- FPGA
- GPU
- TPU
- IPU
- Performance
- Related
- Misc
- Contributing
- Data
- NLP
- Infrastructure & Cloud
- Tools & Frameworks
- Notebooks
- Generative AI
- Computer Vision
- serveo.net - Serveo is an SSH server just for remote port forwarding. When a user connects to Serveo, they get a public URL that anybody can use to connect to their localhost server. See link for other SSH and related alternatives, useful to be able to serve resources across devices i.e. access GPU or other hardware accelerators from another device remotely. | How to forward my local port to public using Serveo? | Serveo on GitHub
- Inlets by Alex Ellis | Get started | Video
- KnockKnock by @huggingface | tweet
- Cray Computers | Artificial Intelligence | Accel AI | Cryp-em | Autonomous Vehicles | Geospatial AI
- #ipu
- Lambda Labs
- NGD Systems: Technology [deadlink] | Solutions - High Compute Storage, Scalable Computational Storage [deadlink] | NGD Systems: Ensuring AI Advancement with Intelligent Storage
- Grid Engine: wikipedia | Univa website | Datasheet
- BOINC - High-Throughput Computing with BOINC | Tech Docs | Download BOINC | GitHub
- Cray Computers - Supercomputing as a Service
- vast.ai - GPU Sharing Economy. One simple interface to find the best cloud GPU rentals. Reduce cloud compute costs by 3X to 5X
- paperspace - The first cloud built for the future. Powering next-generation applications and cloud ML/AI pipelines. Paperspace is built to scale with your team - pay as you go option for individuals.
- NextJournal - The notebook for reproducible research
- valohai | docs | blogs | GitHub | Videos | Showcase | Slack | @valohaiai - Valohai is a machine learning platform. It runs your experiments in the cloud, tracks your experiment history and streamlines data science workflows. DEEP LEARNING MANAGEMENT PLATFORM. Machine Orchestration, Version Control and Pipeline Management for Deep Learning.
- Lambda Cloud GPU Instances - GPU Instances for Deep Learning & Machine Learning
- NavOps - Cloud Migration for HPC | Datasheet
- Verne Global: HPC Cloud | NVIDIA DGX Ready
- Weights and Biases | Learn more about WandB
- Marvin AI: About Marvin AI | Apache Marvin AI: MLOps platform | GitHub | Video
- RealityEngine.ai | Research | Blogs
- Videos
- Notebooks
- Workshop: Unsupervised Learning and Deep Learning Based Forecasting: Anomaly Workbook | Forecasting Workbook
- AutoML Core Concepts and Hands-On Workshop: Regression Notebook | Classification Notebook
- Workshop: Large Scale Deep Learning Recommender
- Reality Engines Demo
- Accelerating AI Training with MLPerf Containers and Models from NVIDIA NGC
- Running AI Models in the Cloud: site | video | Docs | Getting started
- snakemake - The Snakemake workflow management system is a tool to create reproducible and scalable data analyses. Slides | PyPi
- plz - Plz (pronounced "please") runs your jobs storing code, input, outputs and results so that they can be queried programmatically.
- valohai | docs | blogs | GitHub | Videos | Showcase | Slack - Valohai is a machine learning platform. It runs your experiments in the cloud, tracks your experiment history and streamlines data science workflows. DEEP LEARNING MANAGEMENT PLATFORM. Machine Orchestration, Version Control and Pipeline Management for Deep Learning.
- Seldon - Model deployment platform, on kubernetes clusters. | docs | github | use-cases | blogs | videos | Seldon's opensource library for MachineLearning model inspection and interpretation
- Arize AI | docs | certification | resources | Slack - Model monitoring and observability platform. Community edition offers model performance tracing, data quality checks, explainability, and drift detection -- including embedding drift detection for CV and NLP models.
- kedro | other kedro projects | docs | Kedro-Viz | kedro-examples | Blogs | Video | gitter.im/py-sprints/kedro | pypi - Kedro is a workflow development tool that helps you build data pipelines that are robust, scalable, deployable, reproducible and versioned.
- Lambda Stack - One-line installation of TensorFlow, Keras, Caffe, Caffe, CUDA, cuDNN, and NVIDIA Drivers for Ubuntu 16.04 and 18.04.
- Apache Airflow - Airflow is a platform to programmatically author, schedule and monitor workflows. Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies.
- Nextflow - Data-driven computational pipelines. Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages.
- StackHPC suites of repositories: AI, ML, DL, Cloud, HPC | StackHPC
- cortex - Machine learning deployment platform: Deploy machine learning models to production
- #Uber introduces #Fiber, an #AI development and distributed training platform for methods including reinforcement learning and population-based learning.| Uber Open-Sources Fiber - A New Library For Distributed Machine Learning
- A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin
- H2O Framework for Machine Learning
- ML Framework: Introducing Ludwig, a Code-Free Deep Learning Toolbox | Ludwig is a toolbox built on top of TensorFlow that allows to train and test deep learning models without the need to write code
- ML Pipelines
- Large SVDs Dask + CuPy + Zarr + Genomics
- Determined AI | About Niel Conway | Determined: Open-source Deep Learning Training Platform
- ML Framework by Abhishek Thakur
- See also: Data > Programs and Tools
- Probing the CPU (Linux/MacOS)
- Zero overhead performance capturing: use
/proc/interruptsand/proc/softirqs - Non-zero overhead, less accurate: use the PMU (capture on- and off-core events)
- Zero overhead performance capturing: use
- Probing the CPU (Windows)
- perfview - general profiling on Windows
- perfview for .net - excellent overview by Sasha Goldshtein
- Neural Magic: GPU-class performance on CPU
- Intel
- Intel® Developer Zone
- Intel® AI Developer Home Page
- Intel® AI Developer Webinar Series | All webinars listing
- nGraph - Unlocking next-generation performance with deep learning compilers: webinar | slides | homepage | github
- Intel Debug memory & threading bugs: Webinar slides | Intel inspector | | Inspector Docs | Intel® Parallel Studio XE | Intel® System Studio
- Intel Analysers/Profilers:
- Tuning applications for multiple architectures
- Also see Intel in Courses
Thanks to the great minds on the mechanical sympathy mailing list for their responses to my queries on CPU probing.
- Using FPGAs for Datacenter Acceleration | Windows AI | Intel® Distribution of OpenVINO™ Toolkit: Develop Multiplatform Computer Vision Solutions
- Also see FPGA in Courses
- Know your GPU
- GPU Server 1 of 2 | GPU Server 2 of 2 | Applications of GPU servers - checkout the manufacturers
- Embedded Vision Solutions for NVIDIA Jetson Series | Embedded Vision Family Brochure
- Avermedia Box PC & Carrier (works with NVIDIA Jetson): 1 | 2
- Accelerating Wide & Deep Recommender Inference on GPUs
- Create GPU Arrays and Move to DL Frameworks with DLPack
- GPU Accelerated data viz tools
- This tool is nice to monitor not only RAPIDS but also deep learning workloads
- InstaDeep™ powers AI as a Service with shared NVMe: Excelero NVMesh™ feeds unlimited streams of data to GPU-based systems with local performance for AI and ML end-users
- See NVIDIA's RAPIDS
- A NumPy-compatible array library accelerated by CUDA
- For ML Practitioners, from @NVIDIAGTC, a catalog of resources for @ApacheSpark on GPUs, using @RAPIDSai, and other #NVIDIA's Libraries (Deployment on @GCPcloud | Architectural e-Book | Use cases for Adobe & Verizon | Pipelines & Hyperparameter Tuning)
- Explore GPU Acceleration in the Intel® DevCloud (slides)
- Offload Your Code from CPU to GPU … and Optimize It
- Profile DPC++ and GPU Workload Performance
- Command-line interface
- Examples
- GraphCore | Videos: Simon Knowles - More complex models and more powerful machines | Graphcore tech Concept | A new kind of hardware designed for machine intelligence - GraphCore Presentations | VIDEO: SCALING THROUGHPUT PROCESSORS FOR MACHINE INTELLIGENCE
- What makes the IPU's architecture so efficient
- Watch the Graphcore quick guide to the #IPU LinkedIn Post
- Dissecting the Graphcore IPU Architecture via Microbenchmarking
- Learn how to develop and train models for the Graphcore #IPU using TensorFlow
- Graphcore Whitepaper: DELL DSS8440 GRAPHCORE IPU SERVER
- Product brief: IPU-MACHINE: M2000
- MOOR INSIGHTS - GRAPHCORE SOFTWARE STACK: BUILT TO SCALE
- INTELLIGENT MEMORY FOR INTELLIGENT COMPUTING
- Graphcore Benchmarks
- Graphcore GitHub
- MLPerf - Fair and useful benchmarks for measuring training and inference performance of ML hardware, software, and services.
- MLPerf introduces machine learning inference benchmark suite...
- ONE DEEP LEARNING BENCHMARK TO RULE THEM ALL
- mlbench: Distributed Machine Learning Benchmark - A public and reproducible collection of reference implementations and benchmark suite for distributed machine learning algorithms, frameworks and systems.
- EEMBC MLMark Benchmark - The EEMBC MLMark benchmark is a machine-learning (ML) benchmark designed to measure the performance and accuracy of embedded inference.
- DeepOBS: A Deep Learning Optimizer Benchmark Suite
- Deep Learning Benchmarking Suite | HPE Deep Learning Cookbook
- Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors
- Performance profiling in TF 2 (TF Dev Summit '20)
Contributions are very welcome, please share back with the wider community (and get credited for it)!
Please have a look at the CONTRIBUTING guidelines, also have a read about our licensing policy.
#cloud-devops-infra · ↑ Back to top · ← Back home
- [AutoML Core Concepts and Hands-On Workshop](https://www.youtube.com/watch?v=QbqsOcX7KZo&feature=em-lbcastemail)
- [Episode 3 Handling Categorical Features in Machine Learning Problems](https://lnkd.in/e9Qc5fe)
- How to harness the Powers of the Cloud TPU
- PMLB - a large benchmark suite for machine learning evaluation and comparison
- The PlaidML Tensor Compiler - [webinar](https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?&eventid=2026509&sessionid=1&username=&partnerref=&format=fhaudio&mobile=false&flashsupportedmobiledevice=false&helpcenter=false&key=B27628973F7FA8B9758983E373E36ED1&text_language_id=en&playerwidth=1000&playerheight=700&overwritelobby=y&eventuserid=246511746&contenttype=A&mediametricsessionid=207230377&mediametricid=2857349&usercd=246511746&mode=launch)
- [Webinar slides: offload your code to GPU (part 1)](https://event.on24.com/event/23/51/32/1/rt/1/documents/resourceList1590781996922/s_webinarslides1590781995277.pdf)
- [Intel® DevCloud for oneAPI](https://devcloud.intel.com/oneapi/)
- [TVM is an open deep learning compiler stack for CPUs, GPUs, and specialized accelerators. It aims to close the gap between the productivity-focused deep learning frameworks, and the performance- or efficiency-oriented hardware backends](https://tvm.apache.org/docs/index.html)
- How-tos
- All tutorials
- Cloud TPU tools
- Performance Guide
- TPU Estimator API
- Using BFloat
- Advanced Guide to Inception V3 on Cloud TPU
- 👉 Docker CLI & Dockerfile Cheat Sheet 👈
- [oneAPI Toolkits](https://software.intel.com/content/www/us/en/develop/tools/oneapi.html#oneapi-toolkits)
- [Intel® Advisor](https://software.intel.com/content/www/us/en/develop/tools/advisor.html)
- [Hello, TPU in Colab notebook](https://colab.research.google.com/drive/1MefSa2P6UP-gO2S0-dCjIjbvRxOnewZK#scrollTo=llcFb_P_BNPM)
- [Useful TPU and Model example](https://colab.research.google.com/drive/1F8txK1JLXKtAkcvSRQz2o7NSTNoksuU2#scrollTo=mQnZM5JYlRvs)
- [Measure Performance on TPU, in a notebook](https://colab.research.google.com/drive/11VnRHgG_067fwPGhMwBz0SmplLsf9X5h)
- [Web traffic prediction](https://adaptpartners.com/technical-seo/website-traffic-prediction-with-google-colaboratory-and-facebook-prophet/)
- [GAN example, TPU version](https://colab.research.google.com/drive/1EkZPH6UE_I1a2TQfDDpjjqA7Na0_qd6v)