The translation of the article is specially prepared for the students of the course , which starts today!
Have you ever launched a new service in production? Or perhaps you've been involved in supporting such services? If so, what guided your decisions? What is beneficial for production, and what is detrimental? How do you train new team members for releases or the support of existing services?
Most companies, regarding industrial operation practices, ultimately adopt a 'Wild West' approach. Each team, through trial and error, independently determines the tools and best practices. However, this often affects not only project success but also the engineers.
The trial-and-error method creates an environment where blame is frequently sought, and responsibility is shifted around. In such an environment, learning from mistakes and not repeating them becomes increasingly difficult.
Successful organizations:
- recognize the need for production guidelines,
- study best practices,
- begin discussions on production readiness when developing new systems or components,
- ensure compliance with production readiness rules.
Preparing for production includes the 'review' process. Reviews can take the form of a checklist or a set of questions. They can be performed manually, automatically, or by both means. Instead of static requirement lists, templates for checklists can be created that are tailored to specific needs. This way, engineers can inherit knowledge and have sufficient flexibility when required.
When should you check a service for production readiness?
It is beneficial to conduct readiness checks not only right before the release but also when passing it to another operations team or a new employee.
Perform a check when:
- You are releasing a new service into production.
- You are transferring the operation of a production service to another team, such as SRE.
- You are handing over the operation of a production service to new employees.
- You are organizing technical support.
Production readiness checklist
Some time ago, as an example, I created a production readiness checklist. Although this list originated while working with Google Cloud clients, it will be useful and applicable beyond Google Cloud.
Design and development
- Develop a reproducible build process that does not require access to external services and is not dependent on the failure of external systems.
- During the design and development phase, define and set SLOs for your services.
- Document the availability expectations of the external services you depend on.
- Avoid single points of failure by removing dependencies on a single global resource. Replicate the resource or use a fallback option when the resource is not available (for example, a hard-coded value).
Configuration Management
- Static, small, and non-sensitive configurations can be passed through command-line parameters. For everything else, use configuration storage services.
- Dynamic configurations should have fallback settings in case the configuration service becomes unavailable.
- Development environment configuration should not be linked to production configuration. Otherwise, this could lead to development access to production services, causing privacy issues and data leaks.
- Document what can be configured dynamically, and describe the fallback behavior if the configuration delivery system becomes unavailable.
Release Management
- Thoroughly document the release process. Describe how releases impact SLOs (for example, temporary latency increases due to cache misses).
- Document canary releases.
- Develop a plan for analyzing canary releases and, where possible, implement automatic rollback mechanisms.
- Ensure that rollbacks can use the same processes as deployments.
Observability
- Ensure that a set of metrics necessary for SLOs is being collected.
- Ensure that you can distinguish between client-side and server-side data. This is important for troubleshooting issues.
- Set up alerts to reduce operational overhead. For example, eliminate alerts triggered by routine operations.
- If you are using Stackdriver, enable GCP platform metrics in your dashboards. Set up alerts for GCP dependencies.
- Always propagate incoming traces. Even if you are not participating in tracing, this will allow lower-level services to debug production issues.
Protection and security
- Ensure that all external connections are encrypted.
- Ensure that your production projects have the correct IAM settings.
- Use networks to isolate groups of virtual machine instances.
- Use a VPN for secure connections to remote networks.
- Document and monitor user access to data. Ensure that all user access to data is audited and logged.
- Ensure that debugging endpoints are restricted by ACL.
- Sanitize user input. Set payload size limits for user input.
- Ensure that your service can selectively block incoming traffic for individual users. This allows you to block violations without affecting other users.
- Avoid external endpoints that initiate a large number of internal operations.
Capacity planning
- Document how your service scales. For example: number of users, size of incoming payloads, number of incoming messages.
- Document resource requirements for your service. For example: number of allocated virtual machine instances, number of Spanner instances, specialized hardware like GPUs or TPUs.
- Document resource constraints: resource type, region, etc.
- Document quota limits for creating new resources. For example, the limit on the number of GCE API requests if you are using the API to create new instances.
- Consider conducting load tests to analyze performance degradation.
That's it. See you in class!
Source: habr.com
