MongoDB Enterprise Kubernetes Operator Production Notes¶
On this page
- Control Your Deployments with Policies Set in OPA Gatekeeper
- Deploy the Recommended Number of MongoDB Replica Sets
- Ensure Proper Persistence Configuration
- Name Your MongoDB Service with its Purpose
- Specify CPU and Memory Resource Requirements
- Use Multiple Availability Zones
- Co-locate
mongos
Pods with Your Applications - Use Labels to Differentiate Between Deployments
- Verify Permissions
- Enable HTTPS
- Enable TLS
- Enable Authentication
- Configure Readiness Probe Overrides for Externally Accessed Kubernetes Operator Deployments
This page details system configuration recommendations for the MongoDB Enterprise Kubernetes Operator when running in production.
- All sizing and performance recommendations for common MongoDB deployments through the Kubernetes Operator in this section are subject to change. Do not treat these recommendations as guarantees or limitations of any kind.
- These recommendations reflect performance testing findings and represent
our suggestions for production deployments. We ran the tests on a cluster
comprised of seven AWS EC2 instances of type
t2.2xlarge
and a master node of typet2.medium
. - The recommendations in this section don’t discuss characteristics of any specific deployment. Your deployment’s characteristics may differ from the assumptions made to create these recommendations. Contact MongoDB Support for further help with sizings.
Control Your Deployments with Policies Set in OPA Gatekeeper¶
To control, audit, and debug your production deployments, you can use policies for the Gatekeeper Open Policy Agent (OPA). Gatekeeper contains CustomResourceDefinitions for creating and extending deployment constraints through the constraint templates.
The Kubernetes Operator offers a list of Gatekeeper policies that you can customize and apply to your deployments.
Each Gatekeeper policy consists of:
<policy_name>.yaml
fileconstraints.yaml
file that is based on the constraint template
You can use binary and configurable Gatekeeper policies:
- Binary policies allow or prevent specific configurations, such as preventing deployments that don’t use TLS, or deploying only specific MongoDB or Ops Manager versions.
- Configurable policies allow you to specify configurations, such as the total number of replica sets that will be deployed for a specific MongoDB or Ops Manager custom resource.
To use and apply Gatekeeper sample policies with the Kubernetes Operator:
Install the OPA Gatekeeper on your Kubernetes cluster.
Review the list of available constraint templates and constraints:
Navigate to the policy directory, select a policy from the list and apply it and its constraints file:
Review the Gatekeeper policies that are currently applied:
List of Sample OPA Gatekeeper Policies¶
The Kubernetes Operator offers the following sample policies in this OPA examples GitHub directory:
Location | Policy Description |
---|---|
Debugging | Blocks all MongoDB and Ops Manager resources. This allows you to use the log output to craft your own policies. To learn more, see Gatekeeper Debugging. |
mongodb_allow_replicaset | Allows deploying only replica sets for MongoDB resources and prevents deploying sharded clusters. |
mongodb_allowed_versions | Allows deploying only specific MongoDB versions. |
ops_manager_allowed_versions | Allows deploying only specific Ops Manager versions. |
mongodb_strict_tls | Allows using strict TLS mode for MongoDB deployments. |
ops_manager_replica_members | Allows deploying a specified number of Ops Manager replica set and Application Database members. |
ops_manager_wizardless | Allows installing Ops Manager in a non-interactive mode. |
Deploy the Recommended Number of MongoDB Replica Sets¶
We recommend that you use a single instance of the Kubernetes Operator to deploy up to 20 replica sets in parallel.
You may increase this number to 50 and expect a reasonable increase in the time that the Kubernetes Operator takes to download, install, deploy, and reconcile its resources.
For 50 replica sets, the time to deploy varies and might take up to 40 minutes. This time depends on the network bandwidth of the Kubernetes cluster and the time it takes each MongoDB Agent to download MongoDB installation binaries from the Internet for each MongoDB cluster member.
To deploy more than 50 MongoDB replica sets in parallel, use multiple instances of the Kubernetes Operator.
Ensure Proper Persistence Configuration¶
The Kubernetes deployments orchestrated by the Kubernetes Operator are stateful. The Kubernetes container uses Persistent Volumes to maintain the cluster state between restarts.
To satisfy the statefulness requirement, the Kubernetes Operator performs the following actions:
- Creates Persistent Volumes for your MongoDB deployment.
- Mounts storage devices to one or more directories called mount points.
- Creates one persistent volume for each MongoDB mount point.
- Sets the default path in each Kubernetes container to
/data
.
To meet your MongoDB cluster’s storage needs, make the following changes in your configuration for each replica set deployed with the Kubernetes Operator:
- Verify that persistent volumes are enabled in
spec.persistent
. This setting defaults totrue
. - Specify a sufficient amount of storage for the Kubernetes Operator
to allocate for each of the volumes. The volumes store the data
and the logs.
- To set multiple volumes, each for data, logs, and the
oplog
, usespec.podSpec.persistence.multiple.data
. - To set a single volume to store data, logs, and the
oplog
, usespec.podSpec.persistence.single
.
- To set multiple volumes, each for data, logs, and the
The following abbreviated example shows recommended persistent storage sizes.
For a full example of persistent volumes configuration, see replica-set-persistent-volumes.yaml in the Persistent Volumes Samples directory. This directory also contains sample persistent volumes configurations for sharded clusters and standalone deployments.
Name Your MongoDB Service with its Purpose¶
Set the spec.service
parameter to a value that identifies
this deployment’s purpose, as illustrated in the following example.
See also
Specify CPU and Memory Resource Requirements¶
In Kubernetes, each Pod includes parameters that allow you to specify CPU resources and memory resources for each container in the Pod.
To indicate resource bounds, Kubernetes uses the requests and limits parameters, where:
- request indicates a lower bound of a resource.
- limit indicates an upper bound of a resource.
The following sections illustrate how to:
For the Pods hosting Ops Manager, use the default resource limits configurations.
Set CPU and Memory Utilization Bounds for the Kubernetes Operator Pod¶
When you deploy replica sets with the Kubernetes Operator, CPU usage for Pod used to host the Kubernetes Operator is initially high during the reconciliation process, however, by the time the deployment completes, it lowers.
For production deployments, to satisfy deploying up to 50 MongoDB replica sets or sharded clusters in parallel with the Kubernetes Operator, set the CPU and memory resources and limits for the Kubernetes Operator Pod as follows:
spec.template.spec.containers.resources.requests.cpu
to 500mspec.template.spec.containers.resources.limits.cpu
to 1100mspec.template.spec.containers.resources.requests.memory
to 200Mispec.template.spec.containers.resources.limits.memory
to 1Gi
If you don’t include the unit of measurement for CPUs, Kubernetes interprets
it as the number of cores. If you specify m
, such as 500m, Kubernetes
interprets it as millis
. To learn more, see
Meaning of CPU.
The following abbreviated example shows the configuration with recommended CPU and memory bounds for the Kubernetes Operator Pod in your deployment of 50 replica sets or sharded clusters. If you are deploying fewer than 50 MongoDB clusters, you may use lower numbers in the configuration file for the Kubernetes Operator Pod.
Note
Monitoring tools report the size of the node rather than the actual size of the container.
Example
For a full example of CPU and memory utilization resources and limits for the Kubernetes Operator Pod that satisfy parallel deployment of up to 50 MongoDB replica sets, see the mongodb-enterprise.yaml file.
Set CPU and Memory Utilization Bounds for MongoDB Pods¶
The values for Pods hosting replica sets or sharded clusters map to the requests field for CPU and memory for the created Pod. These values are consistent with considerations stated for MongoDB hosts.
The Kubernetes Operator uses its allocated memory for processing, for the WiredTiger cache, and for storing packages during the deployments.
For production deployments, set the CPU and memory resources and limits for the MongoDB Pod as follows:
spec.podSpec.podTemplate.spec.containers.resources.requests.cpu
to 0.25spec.podSpec.podTemplate.spec.containers.resources.limits.cpu
to 0.25spec.podSpec.podTemplate.spec.containers.resources.requests.memory
to 512Mspec.podSpec.podTemplate.spec.containers.resources.limits.memory
to 512M
If you don’t include the unit of measurement for CPUs, Kubernetes interprets
it as the number of cores. If you specify m
, such as 500m, Kubernetes
interprets it as millis
. To learn more, see
Meaning of CPU.
The following abbreviated example shows the configuration with recommended CPU and memory bounds for each Pod hosting a MongoDB replica set member in your deployment.
Example
For a full example of CPU and memory utilization resources and limits for Pods hosting MongoDB replica set members, see the replica-set-podspec.yaml file in the MongoDB Podspec Samples directory.
This directory also contains sample CPU and memory limits configurations for Pods used for:
- A sharded cluster, in the sharded-cluster-podspec.yaml.
- Standalone MongoDB deployments, in the standalone-podspec.yaml.
Use Multiple Availability Zones¶
Set the Kubernetes Operator and StatefulSets to distribute all members of one replica set to different nodes to ensure high availability.
The following abbreviated example shows affinity and multiple availability zones configuration.
Example
In this example, the Kubernetes Operator schedules the Pods deployment to
the nodes which have the label kubernetes.io/e2e-az-name
in e2e-az1
or
e2e-az2
availability zones. Change nodeAffinity
to
schedule the deployment of Pods to the desired availability zones.
See the full example of multiple availability zones configuration in replica-set-affinity.yaml in the Affinity Samples directory.
This directory also contains sample affinity and multiple zones configurations for sharded clusters and standalone MongoDB deployments.
See also
Co-locate mongos
Pods with Your Applications¶
You can run the lightweight mongos
instance on the same node
as your apps using MongoDB. The Kubernetes Operator supports standard Kubernetes
node affinity and anti-affinity
features. Using these features, you can force install the mongos
on the same node as your application.
The following abbreviated example shows affinity and multiple availability zones configuration.
The podAffinity
key determines whether to install an application
on the same Pod, node, or data center as another application.
To specify Pod affinity:
- Add a label and value in the
spec.podSpec.podTemplate.metadata.labels
YAML collection to tag the deployment. Seespec.podSpec.podTemplate.metadata
, and the Kubernetes PodSpec v1 core API. - Specify which label the
mongos
uses in thespec.mongosPodSpec.podAffinity
.requiredDuringSchedulingIgnoredDuringExecution.labelSelector
YAML collection. ThematchExpressions
collection defines thelabel
that the Kubernetes Operator uses to identify the Pod for hosting themongos
.
Example
See the full example of multiple availability zones and node affinity configuration in replica-set-affinity.yaml in the Affinity Samples directory.
This directory also contains sample affinity and multiple zones configurations for sharded clusters and standalone MongoDB deployments.
Use Labels to Differentiate Between Deployments¶
Use the Pod affinity Kubernetes feature to:
- Separate different MongoDB resources, such as
test
,staging
, andproduction
environments. - Place Pods on some specific nodes to take advantage of features such as SSD support.
See also
Verify Permissions¶
Objects in the Kubernetes Operator configuration use the following default permissions.
Kubernetes Resources | Verbs |
---|---|
Configmaps | Require the following permissions:
|
Secrets | Require the following permissions:
|
Services | Require the following permissions:
|
StatefulSets | Require the following permissions:
|
Pods | Require the following permissions:
|
Namespaces | Require the following permissions:
|
Enable HTTPS¶
The Kubernetes Operator supports configuring Ops Manager to run over HTTPS.
Enable HTTPS before deploying your Ops Manager resources to avoid a situation
where the Kubernetes Operator reports your resources’ status as Failed
.
See also
Enable TLS¶
The Kubernetes Operator supports TLS encryption. Use TLS with your MongoDB deployment to encrypt your data over the network.
The configuration in the following example enables TLS for the replica set. When TLS is enabled, all traffic between members of the replica set and clients is encrypted using TLS certificates.
To learn more about securing your MongoDB deployments using TLS, see Deploy a Replica Set.
The default TLS mode is requireTLS
. You can customize it using the
spec.additionalMongodConfig.net.ssl.mode
configuration
parameter, as shown in the following abbreviated example.
See the full TLS configuration example in replica-set.yaml in the TLS samples directory. This directory also contains sample TLS configurations for sharded clusters and standalone deployments.
Enable Authentication¶
The Kubernetes Operator supports X.509, LDAP, and SCRAM user authentication.
Note
For LDAP configuration, see the
spec.security.authentication.ldap.automationLdapGroupDN
setting.
You must create an additional CustomResourceDefinition for your MongoDB users and the MongoDB Agent instances. The Kubernetes Operator generates and distributes the certificate.
See the full X.509 certificates configuration examples in the x509 Authentication directory in the Authentication samples directory. This directory also contains sample LDAP and SCRAM configurations.
Example Deployment CRD¶
Configure Readiness Probe Overrides for Externally Accessed Kubernetes Operator Deployments¶
If you create custom services that require external access to MongoDB custom
resources deployed by the Kubernetes Operator, and use readiness probes
in Kubernetes, set the publishNotReadyAddresses
setting in Kubernetes to true
.
The publishNotReadyAddresses
setting indicates that an agent that
interacts with endpoints for this service should disregard the service’s
ready
state. Setting publishNotReadyAddresses
to true
overrides the
behavior of the readiness probe configured for the Pod hosting your service.
By default, the publishNotReadyAddresses
setting is set to false
.
In this case, when the Pods that host the MongoDB custom resources in the
Kubernetes Operator lose connectivity to Cloud Manager or Ops Manager, the
readiness probes configured for these Pods fail.
However, when you set the publishNotReadyAddresses
setting to true
:
- Kubernetes does not shut down the service whose readiness probe fails.
- Kubernetes considers all endpoints as ready even if the probes for the Pods hosting the services for these endpoints indicate that they aren’t ready.
- MongoDB custom resources are still available for read and write operations.
See also
- Kubernetes API Reference and search for
publishNotReadyAddresses
- DNS for Services in Pods
- Configure Readiness Probes