Operators Scope
Overview
A namespace-scoped operator watches and manages resources in a single Namespace, whereas a cluster-scoped operator watches and manages resources cluster-wide.
An operator should be cluster-scoped if it watches resources that can be created in any Namespace. An operator should be namespace-scoped if it is intended to be flexibly deployed. This scope permits decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions.
By default, operator-sdk init
scaffolds a cluster-scoped operator. This document details conversion of default
operator projects to namespaced-scoped operators. Before proceeding, be aware that your operator may be better suited
as cluster-scoped. For example, the cert-manager operator is often deployed with cluster-scoped
permissions and watches so that it can manage and issue certificates for an entire cluster.
IMPORTANT: When a Manager instance is created in the main.go
file, the
Namespaces are set via Manager Options as described below. These Namespaces should be watched and
cached for the Client which is provided by the Manager.Only clients provided by cluster-scoped Managers are able
to manage cluster-scoped CRD’s. For further information see: CRD scope doc.
Manager watching options
Watching resources in all Namespaces (default)
A Manager is initialized with no Namespace option specified, or Namespace: ""
will
watch all Namespaces:
...
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
LeaderElection: enableLeaderElection,
LeaderElectionID: "f1c5ece8.example.com",
})
...
Watching resources in a single Namespace
To restrict the scope of the Manager’s cache to a specific Namespace set the Namespace
field
in Options:
...
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
LeaderElection: enableLeaderElection,
LeaderElectionID: "f1c5ece8.example.com",
Namespace: "operator-namespace",
})
...
Watching resources in a set of Namespaces
It is possible to use MultiNamespacedCacheBuilder
from
Options to watch and manage resources in a set of Namespaces:
...
namespaces := []string{"foo", "bar"} // List of Namespaces
...
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
LeaderElection: enableLeaderElection,
LeaderElectionID: "f1c5ece8.example.com",
NewCache: cache.MultiNamespacedCacheBuilder(namespaces),
})
...
In the above example, a CR created in a Namespace not in the set passed to Options
will not be reconciled by
its controller because the Manager does not manage that Namespace.
IMPORTANT: Note that this is not intended to be used for excluding Namespaces, this is better done via a Predicate.
Restricting Roles and permissions
An operator’s scope defines its Manager’s cache’s scope but not the permissions to access the resources. After updating the Manager’s scope to be Namespaced, the cluster’s Role-Based Access Control (RBAC) permissions should be restricted accordingly.
These permissions are found in the directory config/rbac/
. The ClusterRole
in role.yaml
and ClusterRoleBinding
in role_binding.yaml
are used to grant the operator permissions to access and manage its resources.
NOTE For changing the operator’s scope only the role.yaml
and role_binding.yaml
manifests need to be updated.
For the purposes of this doc, the other RBAC manifests <kind>_editor_role.yaml
, <kind>_viewer_role.yaml
,
and auth_proxy_*.yaml
are not relevant to changing the operator’s resource permissions.
Changing the permissions to Namespaced
To change the scope of the RBAC permissions from cluster-wide to a specific namespace, you will need to use Role
s
- Inform the Namespace to the Manager
By default, the Manager does not have any namespace specified in main.go
, and hence it will watch all the namespaces. In order to restrict the controllers to watch a specific namespace, specify it while creating the manager. Update the NewManager
to inform the Namespace, in our Memcahced
example it would like:
...
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
LeaderElection: enableLeaderElection,
LeaderElectionID: "f1c5ece8.example.com",
Namespace: "memcached-operator-system", // operator namespace
})
...
- Change the scope of the RBAC permissions from cluster-wide to a specific namespace
You will need to use Role
s and RoleBinding
s instead of ClusterRole
s and ClusterRoleBinding
s, respectively.
RBAC markers
defined in the controller (e.g controllers/memcached_controller.go
)
are used to generate the operator’s RBAC ClusterRole (e.g config/rbac/role.yaml
). The default
markers don’t specify a namespace
property and will result in a ClusterRole
.
Update the RBAC markers
in <kind>_controller.go
with namespace=<namespace>
where the Role
is to be applied, such as:
// +kubebuilder:rbac:groups=cache.example.com,namespace=memcached-operator-system,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=cache.example.com,namespace=memcached-operator-system,resources=memcacheds/status,verbs=get;update;patch
- Run
make manifests
to updateconfig/rbac/role.yaml
. In our example it would like:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
creationTimestamp: null
name: manager-role
namespace: memcached-operator-system
- Replace
ClusterRoleBinding
withRoleBinding
andClusterRole
withRole
in theconfig/rbac/role_binding.yaml
file such as:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: manager-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: manager-role
subjects:
- kind: ServiceAccount
name: default
namespace: system
Using environment variables for Namespace
Instead of having any Namespaces hard-coded in the main.go
file a good practice is to use environment
variables to allow the restrictive configurations
Configuring Namespace scoped operators
- Add a helper function in the
main.go
file:
// getWatchNamespace returns the Namespace the operator should be watching for changes
func getWatchNamespace() (string, error) {
// WatchNamespaceEnvVar is the constant for env variable WATCH_NAMESPACE
// which specifies the Namespace to watch.
// An empty value means the operator is running with cluster scope.
var watchNamespaceEnvVar = "WATCH_NAMESPACE"
ns, found := os.LookupEnv(watchNamespaceEnvVar)
if !found {
return "", fmt.Errorf("%s must be set", watchNamespaceEnvVar)
}
return ns, nil
}
- Use the environment variable value:
...
watchNamespace, err := getWatchNamespace()
if err != nil {
setupLog.Error(err, "unable to get WatchNamespace, " +
"the manager will watch and manage resources in all namespaces")
}
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
LeaderElection: enableLeaderElection,
LeaderElectionID: "f1c5ece8.example.com",
Namespace: watchNamespace, // namespaced-scope when the value is not an empty string
})
...
- Define the environment variable in the
config/manager/manager.yaml
:
spec:
containers:
- command:
- /manager
args:
- --enable-leader-election
image: controller:latest
name: manager
resources:
limits:
cpu: 100m
memory: 30Mi
requests:
cpu: 100m
memory: 20Mi
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
terminationGracePeriodSeconds: 10
NOTE WATCH_NAMESPACE
here will always be set as the namespace where the operator is deployed.
Configuring cluster-scoped operators with MultiNamespacedCacheBuilder
- Add a helper function to get the environment variable value in the
main.go
file as done in the previous example (e.ggetWatchNamespace()
) - Use the environment variable value and check if it is a multi-namespace scenario:
...
watchNamespace, err := getWatchNamespace()
if err != nil {
setupLog.Error(err, "unable to get WatchNamespace, " +
"the manager will watch and manage resources in all Namespaces")
}
options := ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
LeaderElection: enableLeaderElection,
LeaderElectionID: "f1c5ece8.example.com",
Namespace: watchNamespace, // namespaced-scope when the value is not an empty string
}
// Add support for MultiNamespace set in WATCH_NAMESPACE (e.g ns1,ns2)
if strings.Contains(watchNamespace, ",") {
setupLog.Info("manager set up with multiple namespaces", "namespaces", watchNamespace)
// configure cluster-scoped with MultiNamespacedCacheBuilder
options.Namespace = ""
options.NewCache = cache.MultiNamespacedCacheBuilder(strings.Split(watchNamespace, ","))
}
...
- Define the environment variable in the
config/manager/manager.yaml
:
...
env:
- name: WATCH_NAMESPACE
value: "ns1,ns2"
terminationGracePeriodSeconds: 10
...
Updating your CSV’s installModes
If your operator is integrated with OLM, you will want to update your CSV base’s
spec.installModes
list to support the desired namespacing requirements. Support for multiple types of namespacing
is allowed, so supporting multiple install modes in a CSV is permitted. After doing so, update your
[bundle][bundle-quickstart] or package manifests by following the linked guides.
Watching resources in all Namespaces (default)
Only the AllNamespaces
install mode is supported: true
by default, so no changes are required.
Watching resources in a single Namespace
If the operator can watch its own namespace, set the following in your spec.installModes
list:
- type: OwnNamespace
supported: true
If the operator can watch a single namespace that is not its own, set the following in your spec.installModes
list:
- type: SingleNamespace
supported: true
Watching resources in multiple Namespaces
If the operator can watch multiple namespaces, set the following in your spec.installModes
list:
- type: MultiNamespace
supported: true