Please refer to the official docs at kubeflow.org.
Quick Links
Get Involved
Please refer to the Community page.
Machine Learning Toolkit for Kubernetes
Kubeflow is a Cloud Native platform for machine learning based on Google’s internal machine learning pipelines. Please refer to the official docs at kubeflow.org. Quick Links Prow test dashboard Prow jobs dashboard PR
257 Open
More issues
Closed
Implements case 1 and case 2 in the new kfctl semantics Design doc reference
- [x] make platform args optional
- [x] implement new semantics for build and apply
Current workflow for kfctl -
New sematics single command install:
# move into empty directory
kfctl apply -f <path to config file / URL>
New semantics build, edit, apply:
# move into empty directory
kfctl build -f <path to config file / URL>
kfctl apply
Old semantics:
kfctl init kf-app --config <path to config file / URL>
cd kf-app
kfctl generate <platform>
kfctl apply <platform>
/cc @yanniszark @jlewi @kkasravi
Related to: #3518
This change is 
size/L lgtm approved cla: yes
opened Aug 30, 2019 by swiftdiaries 101
Closed
Trying to install Kubeflow v.70 on on-prem cluster (kubernetes v15.7/Ubuntu 18.04) but getting below error:
[email protected]:/opt/kubeflow/kf-test# kfctl apply -V -f ${CONFIG_FILE} INFO[0000]
Notice anonymous usage reporting enabled using spartakus To disable it If you have already deployed it run the following commands: cd $(pwd) kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus
For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/
filename="coordinator/coordinator.go:120" INFO[0000] Deleting cachedir /opt/kubeflow/kf-test/.cache/manifests because Status.ReposCache is out of date filename="kfconfig/types.go:464" INFO[0000] Fetching https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz to /opt/kubeflow/kf-test/.cache/manifests filename="kfconfig/types.go:485" Error: failed to build kfApp from URI /opt/kubeflow/kf-test/kfctl_existing_arrikto.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter093015857/archive: EOF Usage: kfctl apply -f ${CONFIG} [flags]
Flags: -f, --file string Static config file to use. Can be either a local path: export CONFIG=./kfctl_gcp_iap.yaml or a URL: export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.0.yaml export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_existing_arrikto.0.7.0.yaml export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.0.yaml export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml kfctl apply -V --file=${CONFIG} -h, --help help for apply -V, --verbose verbose output default is false
failed to build kfApp from URI /opt/kubeflow/kf-test/kfctl_existing_arrikto.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter093015857/archive: EOF
The Kubeflow installation was working fin when tried it a few days ago. Trying to install it in a new cluster and getting this error. Can you please help?
Thanks, Job Varkey
priority/p0 platform/onprem kind/bug area/enterprise_readiness
opened Jan 9, 2020 by jobvarkey 84
Closed
/kind process
We need to identify who will be driving the 1.1 release. These folks should then
- Identify timelines for the release
- e.g. cutoff dates for branch cuts
- Target dates for RCs
This would probably be a good opportunity to update some of the processes and policies around releases. https://github.com/kubeflow/kubeflow/blob/master/docs_dev/releasing.md
Area | release czar | Tracking Issue --- | --- | --- aws | @Jeffwan | ~~#5057~~ | centraldashboard | | ~~#5068~~ docs | | kubeflow/website#1984 fairing | @jinchihe | ~~kubeflow/fairing#503~~ | feast | @woop | | gcp | @jlewi | kubeflow/gcp-blueprints#46 | katib | @andreyvelich | kubeflow/katib#1211 | kfctl | @krishnadurai, @crobby | kubeflow/kfctl#352 | kfserving | @yuzisun @animeshsingh | kubeflow/kfserving#648 | manifests | @krishnadurai | kubeflow/manifests#1252 metadata | | minikf | @vkoukis | | multiuser | @yanniszark @bmorphism | #5067, #5068 notebooks | @kimwnasptd , @jtfogarty | #5060, #5068 | pipelines | @Bobgy | kubeflow/pipelines#3961 | training | @johnugeorge @andreyvelich @Jeffwan | kubeflow/common#97
Target Dates
- Branch cut June 19
kind/feature priority/p0 kind/process area/engprod
opened May 18, 2020 by jlewi 73
Closed
We'd like to make it super easy to go from writing code in a notebook to training that model distributed.
Experience might be something like
- User writes code in notebook and executes in Jupyter lab
- User clicks a button which allows user to fill in various settings e.g. number of GPUs
- User clicks train
Under the hood this would cause
- A docker image to be built
- A TFJob/PyTorch/K8s Job to be created and fired off.
I think the biggest challenge is that we probably don't want to execute all code in the notebook. Typically, there's some amount of refactoring that needs to be done to convert a notebook into a python module suitable for execution in a bash job.
As a concrete example
Here's the notebook for our GitHub Issue summarization example
Here's the corresponding python module used when training in a K8s job.
The python module only executes a subset of cells in particular those to
- Define model architecture
- train the model
Rather than try to auto-convert a notebook like the github issue example, I think we should require users structure their code to facilitate the conversion.
My suggestion would be to allow any functions defined in the notebook to be used as entry points. So for the GitHub issues summarization the user would have a cell like the following
from keras.callbacks import CSVLogger, ModelCheckpoint
def train_model(output)
script_name_base = 'tutorial_seq2seq'
csv_logger = CSVLogger('{:}.log'.format(script_name_base))
model_checkpoint = ModelCheckpoint('{:}.epoch{{epoch:02d}}-
val{{val_loss:.5f}}.hdf5'.format(script_name_base),
save_best_only=True)
batch_size = 1200
epochs = 7
history = seq2seq_Model.fit([encoder_input_data, decoder_input_data],
np.expand_dims(decoder_target_data, -1),
batch_size=batch_size,
epochs=epochs,
validation_split=0.12, callbacks=[csv_logger, model_checkpoint])
seq2seq_Model.save(output)
train('seq2seq_model_tutorial.h5')
If user structures their code this way, we should be able to manually create and invoke a suitable container entry point. Something like the following
- Use nbconvert to convert from ipynb to python code
- Post process the python code
- Strip out any statements not inside a function (except imports)
- Create a CLI for the functions using a library like PyFire
- Build a Docker image that is Notebook image + code
A variant of this idea would be to use metaml (by @wbuchwalter ). metaml uses metaparticle to allow people to annotate their python code with information needed to then run it on K8s (e.g. distributed using TFJob). If we went this approach I think the flow would be
- Run nbconvert to go from ipynb -> py
- Use metaparticle/metaml tool chain to build the docker image and submit the job.
@willingc @yuvipanda Is there existing tooling in the Jupyter community other than nbconvert to convert notebooks to code suitable for asynchronous batch execution?
/cc @wbuchwalter @gaocegege @yuvipanda @willingc
priority/p1 area/jupyter area/0.4.0
opened Jul 19, 2018 by jlewi 67
Closed
We should have a Kubeflow logo.
priority/p2 priority/p0
opened Jan 31, 2018 by jlewi 62
Closed
I want to add tolerations to the pods for my Jupyter server. I am able to inject other things like Labels and Annotations, this would allow me to also use PodDefault to inject tolerations.
This change is
size/L lgtm approved cla: yes ok-to-test
opened Mar 11, 2020 by wdhorton 60
Closed
Easy way to track your experiments in kubeflow
kind/discussion lifecycle/stale
opened Feb 19, 2018 by chrisheecho 59
Open
The workspace volume path of "/home/jovyan" was previously hardcoded. This is problematic for configurations which customize this value in the spanner_ui_config and created a disconnect in which the presented workspace path in the UI did not match what is actually used on the backend.
With this fix, the workspace path presented to the user in the UI is the same that is used when creating the statefulset for the notebook.
A previous fix was to use the templatedPath parameter passed back from the UI, but it seems semantically incorrect to treat a template as a materialized value.
size/S needs-ok-to-test cla: yes
opened Oct 15, 2021 by timcharper 3
Open
/kind question
I'm trying to add a Resource Quota to the default profile that is generated when new users log in. Can someone point me to docs on this or explain how it can be done? Thanks!
kind/question
opened Oct 15, 2021 by cupdike 0
Open
Fix #6187
size/S needs-ok-to-test cla: yes
opened Oct 14, 2021 by sunzhaochang 2
Open
/kind bug
What steps did you take and what happened:
- Add resourceQuotaSpec for kubeflow-user-example-com profile as follows:
apiVersion: kubeflow.org/v1
kind: Profile
metadata:
name: kubeflow-user-example-com
spec:
owner:
kind: User
name: [email protected]
resourceQuotaSpec:
hard:
cpu: "4"
memory: 4Gi
We can get the kf-resource-quota resource and it works as expected.
$ kubectl get resourcequotas -n kubeflow-user-example-com
NAME AGE REQUEST LIMIT
kf-resource-quota 74s cpu: 2090m/4, memory: 4486Mi/4Gi
- Remove the resourceQuotaSpec for kubeflow-user-example-com profile as follows:
apiVersion: kubeflow.org/v1
kind: Profile
metadata:
name: kubeflow-user-example-com
spec:
owner:
kind: User
name: [email protected]
It can be found that kf-resource-quota has not been deleted.
$ kubectl get resourcequotas -n kubeflow-user-example-com
NAME AGE REQUEST LIMIT
kf-resource-quota 6m24s cpu: 2090m/4, memory: 4486Mi/4Gi
What did you expect to happen:
Delete the kf-resource-quota resource after remove resourceQuotaSpec from the profile.
Environment:
- Kubeflow version: : 1.4
- Kubernetes platform: kind
- Kubernetes version: v1.21.1
kind/bug
opened Oct 14, 2021 by sunzhaochang 0
Open
Currently, the central-dashboard sends EVERY client header passed to its /add-contributor/xxx and /remove-contributor/xxx APIs on to kfam's /kfam/v1/bindings API.
(NOTE: these APIs are used in the "manage contributor" page to add/remove contributors)
This means central-dashboard sends requests to /kfam/v1/bindings with invalid :authority headers (with the hostname of the istio gateway, rather than the internal kfam Service).
This causes significant issues when you enable istio-sidecars on the kfam Pods, as istio will reject the "invalid" HTTP request, preventing you from adding/removing contributors.
This PR is very simple, it defines a list of "auth headers" which are safe to pass on, and deletes all others before forwarding to /kfam/v1/bindings.
(NOTE: The only "auth header" that kfam actually reads is USERID_HEADER, however, I have included cookie and authorization in case users are validating these with their istio configs.)
size/S cla: yes
opened Oct 13, 2021 by thesuperzapper 2
Open
/kind question
Question: How to quicikly Private Deployment Kubeflow? We need to do private deployment in many time, so how to quickly deployment kubeflow in private cloud or on-bare env？
There will be a lot of network problems in regional or private deployment in China. Does our community have a solution in the future?
Thanks.
opened Oct 13, 2021 by Zhiyuancheng 1
Open
/kind bug
What steps did you take and what happened: Attempting to install KF on a 1.20.2 cluster using 1.3.0 release and the instructions here: https://v0-6.kubeflow.org/docs/started/k8s/kfctl-existing-arrikto/#log-in-with-ldap-active-directory
All steps succeed, all pods are up and running and active directory is properly seeded, but when I attempt to login with ldap, I receive the following error in the UI (screenshot below)
What did you expect to happen: Im able to login successfully with the available default credentials.
Anything else you would like to add:
Environment:
- Kubeflow version: 1.3.0
- kfctl version: (use
kfctl version): na - Kubernetes platform: minikube
- Kubernetes version: 1.20.2
- OS (e.g. from
/etc/os-release): ubuntu 18.04
kind/bug
opened Oct 11, 2021 by akchinSTC 0
v1.4.0(Oct 11, 2021)
Web apps:
- Migrate to Angular 12 from 8: https://github.com/kubeflow/kubeflow/pull/6004 @DavidSpek
Internationalization progress:
- Internationalization to web apps https://github.com/kubeflow/kubeflow/pull/5880 @wg102, @Jose-Matsuda
- Use Angular's i18n solution https://github.com/kubeflow/kubeflow/pull/6065 @kimwnasptd
Central Dashboard:
- Namespaced menu items https://github.com/kubeflow/kubeflow/pull/5995 @toshi-k
- Make it possible to add Namespaced menu items #5871 @toshi-k
- Scrollable side-bar https://github.com/kubeflow/kubeflow/pull/5964 @toshi-k
- Add support for Models web app https://github.com/kubeflow/kubeflow/pull/6085 @kimwnasptd
Jupyter web app
- Jupyter web app fix for autoscaling GPU nodegroups https://github.com/kubeflow/kubeflow/pull/6171 @kimwnasptd
- Fix limits calculation when limitFactor is none in Jupyter web app #6058 @kimwnasptd
- Make jupyter-web-app parse workspace volume MountPath #5952 @anencore94
- JWA: Don't override assets with logos ConfigMap #5942 @kimwnasptd
- Add fonts as assets to service #5691 @saffaalvi @wg102
- The number of gpu must be set as string #5891 @juliusvonkohout
Notebooks
- Remove virtualservice timeout to prevent websocket disconnect #6126 @abe-hpe
- Correct missing predicates in controller watches#5873 @filintod
- Improve the README of the example Notebook servers https://github.com/kubeflow/kubeflow/pull/6165 @thesuperzapper
TensorBoards
- tensorboard-controller: fix binding issue #5925 @DavidSpek
PodDefaults
- Bump Golang version for PodDefaults, TensorBoard Controller and KFAM https://github.com/kubeflow/kubeflow/pull/6180 @kimwnasptd
- Use namespace from Admission Review in PodDefaults https://github.com/kubeflow/kubeflow/pull/6052 @henrysecond1
- Add support for ServiceAccountName and AutomountServiceAccountName to PodDefaults #5939 @hopper-signifyd
Other improvements
- Add kustomize tests in CI/CD https://github.com/kubeflow/kubeflow/pull/5919 @DavidSpek
- Format checks for the Volumes web app https://github.com/kubeflow/kubeflow/pull/5820 @kimwnasptd
- Make i18n work with prefixes https://github.com/kubeflow/kubeflow/pull/6034 @kimwnasptd
- Allow user to add/delete labels to user namespace using ConfigMap #5761 @zijianjoy
v1.4-rc.0(Sep 14, 2021)
The first RC for the 1.4 release
v1.3.1-rc.0(Jun 25, 2021)
v1.3.1-rc.0 release for components owned by WG-Notebooks.
v1.3.0(May 27, 2021)
v1.3.0 stable release for components owned by WG-Notebooks.
v1.2.0(Nov 20, 2020)
To deploy Kubeflow, please follow the instruction
- kfctl https://github.com/kubeflow/kfctl/releases/tag/v1.2.0
- manifest https://github.com/kubeflow/manifests/releases/tag/v1.2.0
Changelog
1.2.0 (2020-11-20)
Please check full change log here
v1.2-rc.0(Nov 9, 2020)
Please help test v1.2 kubeflow release.
The binaries are the same as published in https://github.com/kubeflow/kfctl/releases/tag/v1.2-rc.0
Information
| Category: Linux / Miscellaneous |
| Watchers: 384 |
| Star: 10.8k |
| Fork: 1.8k |
| Last update: Oct 15, 2021 |
Contents
Resource links
Share this repo
Latest Repos
Related Repos
1.3k
This is a pyTorch implementation of Tabnet (Arik, S. O., & Pfister, T. (2019). TabNet: Attentive Interpretable Tabular Learning
Raifhack-DS-2021 https://raifhack.ru/ - Команда Звёздочка Siamese TabNet Сиамская TabNet предсказывает стоимость объекта недвижимости с price_type=1,
360
Pytorch Tutorial, Pytorch with Google Colab, Pytorch Implementations: CNN, RNN, DCGAN, Transfer Learning, Chatbot, Pytorch Sample Codes
115
Mixup: Beyond Empirical Risk Minimization in PyTorch
This is an unofficial PyTorch implementation of mixup: Beyond Empirical Risk Minimization. The code is adapted from PyTorch CIFAR.
The results:
I only tested using
deep-dream-in-pytorch
Pytorch (https://github.com/pytorch/pytorch) implementation of the deep dream (https://en.wikipedia.org/wiki/DeepDream) computer vision algorithm
Installation
Install Jupyter notebook with Anacon
101
LIS-YNP (Life Is Short-You Need Pytorch)
Based on Official Pytorch Tutorial and yunjey's pytorch-tutorial, this repository provides tutorial code for deep learning researchers to learn PyTorch .
There is also a PyTorch tutor
132
PyTorch On Angel, arming PyTorch with a powerful Parameter Server, which enable PyTorch to train very big models.
122
A Universal PyTorch Source-Build Docker Template Template repository to build PyTorch from source on any version of PyTorch/CUDA/cuDNN. PyTorch built
848
fpn.pytorch Pytorch implementation of Feature Pyramid Network (FPN) for Object Detection
Introduction
This project inherits the property of our pytorch implementation of faster r-cnn. Hence, it also has the following unique feat
198
Siamese-RPN (PyTorch Implementation)
This is my implementation for Siamese Region Proposal Network with PyTorch.
Python version: 3.6 (3.5 or 3.4 may be also okay even though I did not try)
PyTorch version: 0.4.0 or higher
NOTE
408
PyTorch-LBFGS: A PyTorch Implementation of L-BFGS
Authors: Hao-Jun Michael Shi (Northwestern University) and Dheevatsa Mudigere (Facebook)
What is it?
PyTorch-LBFGS is a modular implementation of L-BFGS, a popular qua
1.2k
High quality, fast, modular reference implementation of SSD in PyTorch 1.0
This repository implements SSD (Single Shot MultiBox Detector). The implementation is heavily influenced by the projects ssd.pytorch, pytorch-ssd and mask
MixNet-Pytorch
A PyTorch implementation of MixNet
Overview
A PyTorch implementation of MixNet architecture: MixNet: Mixed Depthwise Convolutional Kernels. Based on MobileNetV2, found by Neural Architecture Search, rep
671
FlowNetPytorch
Pytorch implementation of FlowNet by Dosovitskiy et al.
This repository is a torch implementation of FlowNet, by Alexey Dosovitskiy et al. in PyTorch. See Torch implementation here
This code is mainly inspired fr
322
PyTorch implementation of TRPO
Try my implementation of PPO (aka newer better variant of TRPO), unless you need to you TRPO for some specific reasons.
This is a PyTorch implementation of "Trust Region Policy Optimization (TRPO
771
StackGAN-v2
StackGAN-v1: Tensorflow implementation
StackGAN-v1: Pytorch implementation
Inception score evaluation
Pytorch implementation for reproducing StackGAN_v2 results in the paper StackGAN++: Realistic Image
PyTorch implementation of HMAX
PyTorch implementation of the HMAX model that closely follows that of the MATLAB implementation of The Laboratory for Computational Cognitive Neuroscience:
http://maxlab.neuro.georgetown.edu/hmax.h
222
PyTorch REINFORCE
PyTorch implementation of REINFORCE. This repo supports both continuous and discrete environments in OpenAI gym.
Requirement
python 2.7
PyTorch
OpenAI gym
Mujoco (optional)
Ru
356
PyTorch-ADDA
A PyTorch implementation for Adversarial Discriminative Domain Adaptation.
Environment
Python 3.6
PyTorch 0.2.0
Usage
I only test on MNIST -> USPS, you can just run the following com
160
PyTorch CapsNet: Capsule Network for PyTorch
A CUDA-enabled PyTorch implementation of CapsNet (Capsule Network) based on this paper: Sara Sabour, Nicholas Frosst, Geoffrey E Hinton. Dynamic Routing Between Capsules. NIPS 2017