Blog

Analyzing Technical Processes for GCP

Nov 17, 2022

Christopher Shaun Godwin

Author

Architects are involved in many different types of technical processes:

Continuos Deployment
Continuous Delivery
Post-mortem Analysis
Development Lifecycle Planning
Testing
Validation
Business continuity
Disaster Recovery(DR)

Here we will discuss these processes in relation to business needs and goals. We will learn to focus on and define these processes rather than simply follow them.

Software Development Lifecycle(SDLC)

The Software development lifecycle are the steps that software, and those who engineer it, goes through from beginning to end to create and host a service. This includes 12 phases. In some cases these are collapsed or combined to fewer, or some are regarded as pre-SDLC steps.

Proposal
Scope Analysis
Planning
Requirements Analysis
Design
Development
Integration & Testing
Implementing
Documentation
Operations
Maintenance
Disposition


Software Development Lifecycle - Wikipedia

Every phase does work that is required to produce quality software. It is a cycle because you reiterate over these steps until the software is no longer used. The process could start over after the Maintenance step and begin at any one of these beginning steps. After a software is deployed, the next iteration of that software could be as complex as having another Proposal created by the captains of those duties. Or 2nd time the process iterates, it loops back to the Development phase depending if the next iteration requirements are already known. Proposal, scope analysis, planning and requirements analysis can even be done by non-developers or teams of analysts.

For this reason we’re going to jump right into Planning.

Planning

Planning is a step performed by the Project Manager. They’ll create all the spaces which track work, all the spaces where the documentation, solution architect design document, specifications and roadmaps will live. They’ll create the roadmap for the different project phases. They’ll create the templates for spring planning, sprint retros, creation of overarching tasks often called ‘epics’.

Requirements Analysis

This may be done by developers and architects together. The goal is to fully understand the needs and wants of the proposal and find potential ways to meet them. The problem is discussed and ideas are put together to meet those problems. Here the solutions are not designed but considered. Any spikes that are needed to sus out requirements are performed by developers or other engineers. A spike is a short development period where a developer tries a feature to gain knowledge needed for planning a full fledged effort. That effort achieves those requirements in the context of existing systems. Spikes are often isolated to proof of concepts. Proof of concept projects might exist here and iterate back to requirements for an actual project.

In this phase you’re trying to:

Grasp the scope of the needs and wants of the proposal
Track and assess of all possible solutions
Evaluate the cost benefits of the different paths toward a solution

Understanding the scope is a matter of both knowledge of the domain in question: if a mail problem, familiarity with mail operations and development; it is also a matter of systems and software knowledge of the existing infrastructure. Domain knowledge, for example, is knowing that kubernetes secrets are not secure. Systems and software knowledge, is knowing where you’ll inject and use the google libraries to fetch secrets from GSM. This is precisely why developers, architects, and reliability engineers all engage together in this phase.

When finding solutions for your problem, you need to be able to filter them out without trying them. The solutions you’re filtering in your search are those that aren’t feasible, do not fit your use case, or don’t fit within your limitations. Once you know the limits of the project, you can search for possible solutions. If your Google Secret Manager project has a limit placed on it that it must work for in-house apps and third party apps, the direction you’ll go into will be different. The direction differs from the one you’d take without filtering on this rubric. You’ll also consider if commercial software meets your needs at a better cost than you can.

Purchased or Free and Open Source Software(FOSS) can meets a wide range of use cases faster than developing something new. They also have the benefit of the ability to focus on other easier to solve problems. Purchased software or purchased FOSS support can help offset the costs of provisioning new services. This disadvantages are potential licensing models and costs and being locked into a feature set that doesn’t evolve with your needs.

You can decide to build from scratch, from a framework, or from an opensource project. There are different considerations with each of these. How much modification does ready made software require, what languages and formats does it exist in, do you have to acquire talent to work with it. Consider the lifecycles of the software you use. For instance, if you build docker images from other images, knowing the release cycles of those will help you be able to create new releases at the time new operating systems are released. Paying attention to the popularity and maintainers of the application can help you know if a project has become deprecated. You can avoid deprecated software if you do not want to deal with becoming the new maintainer or updater to the software within your use of it. Or you could choose actively maintained software to fork and modify so that you can roll in security backports from the upstream project into your project.

Scratch allows for full control but involves the most work, most maintenance, most planning, having the team with the talent and skillsets needed, most resolution of issues.

Once you have several viable solutions to consider, spike the one first with the greatest cost benefit. You’ll know this because you can do a Cost Benefit Analysis on all these options we’ve discussed.

Cost Benefit Analysis

Part of Analysis is the cost benefit analysis of meeting the requirements with your various solution options. When asked to justify the decisions in your project you’ll be asked for this and be able to contrast the different values of each solution. As part of this you’ll calculate the ROI for the different options to arrive at the solutions value. At the end of this phase you’ll decide which solutions you’ll pursue in the Design.

Design

As part of the design phase, you’ll plan out how the software will work, the structure of the schemas and endpoints, and the functionality that these will achieve. This phase starts with a high level design and ends in a detailed one.

The High Level design is an inventory of all the top levels of parts of the application. Here you’ll identify how components will interact as well their overarching functions. You might work up UML or mermaid diagrams describing parts and interactions.

The Detailed design is a plan of implementation of each of these parts. These parts will be modularized in though and broken down into the most sensible and efficient anatomies in which for them to exist. Some of the things planned include, error codes or pages, data structures, algorithms, security controls, logging, exit codes and wire-frames for user interfaces.

During the design phase, its best to work directly with the users of the system as you would work with other disciplines during other phases. The Users of a system will have a closer relationship to the requirements. In this phase developers will choose which frameworks, libraries, and dependencies.

Develop, Test, Implement

Under development, software is created by engineers and built as artifacts which are pushed to a repository. These artifacts are deployed into an operating system either with a package manager, ssh, direct copying, a build process or via Dockerfile commands. Artifacts can have within them code, binaries, documentation, configuration, or raw mime/type files.

In this phase developers might use tools like ‘VSCode’, analysis applications, administration tools; while changes are committed with source control tools that have gitOps attached to them. All these processes are in the domain of an Architect to conceive and track when designing a project.

Developers will also test as part of the commands they give the continuous integration(CI) system. Well before the CI steps are created, the developer has created unit and integration tests and knows the commands to run them so that the automation team can include them in the creation of the CI portion of the development operations. There are language specific unit tests but generally the integration tests the API endpoints and you have a choice of software for that.

Documentation

Documentation is crucial to the SDLC because it lets others using the software know how to operate the software. This is often your DevOps team handling automation in deployments. Developer documentation can be in the form of inline comments within the code, but also developers should release a manual as a README.md file in the source control repository root. A README.md file should exist in every folder where a different component has different usage instructions.

You entire solution architecture design should be documented. For a lot of companies this is a page in a intranet wiki like Confluence.

Maintenance

This is the practice of keeping the software running and updated. In Agile software practices, developers maintain code and run deployment pipelines to development environments which graduate to higher environments. In a fully agile environment, automation engineers create the pipelines but an automation release team approves the barriers so that developer initiated deployments can be released to production under supervision during a release window.

Keeping a service running includes logging, monitoring, alerting and mitigation. Some of this work includes log rotation and performance scaling. Developers control log messages but infrastructure developers like cloud engineering teams might create the terraform modules that automation engineers use to automatically create alerts and logging policies.

Continuous Integration / Continuous Delivery(CICD)

Continuous integration is the practice of building code every time there is a change to a code base. This usually starts with a commit to a version control system. If the branch or tag of the commit is part of the rules for the continuous part, then the integration part will take place automatically. Integration pipelines often have built, test, and push steps.

Continuous deployment is often the practice of deploying new artifacts as soon as they are available. If a repository’s continuous integration settings builds a package and places it in the repo, continuous deployment systems polling for new artifacts may trigger a deployment pipeline when it finds one. So once a new version is added to nexus or a deb-repository, CD systems often send that artifact down the line.

Business Use

The cornerstone of CI/CD is that individual features can be added quickly, unlike the past’s methods which had to weave several new features together into a major release. Instead, new features are built on different feature branches, those feature branches have builds, those builds can be deployed quickly and then once tested the feature branch can be merged into one of the trunks. The version control system acts as an integration engine which takes all these features and incorporates them together, if you’re using trunk based development. In the context of hosted services, users get a risk free but up-to-date experience.

CI/CD is testing heavy. In real life production pipelines, tests are over 50% or more of the pipeline steps and is used throughout the workflows as steps. Automated tests allow the test cases to pass or fail without human intervention. This means that services can be tested with scripted steps, and then deployed if those steps succeed. This prevents deployments and the building of artifacts that do not pass tests.

In certain critical cases, continuous deliver isn’t possible as the safety risk is too high to deploy the latest code. Sometimes code needs to be hand certified and hand installed.

VCS

The foundation of Continuous Deployment / Continuous Integration is Version control of software source code. When developers checkout code to work on it and improve it, they get it from a git repository. They make their changes, and push them to the git repository. Git makes a revision and keeps both copies. Points in time in the revisions are called references. Branches and tags are references. You can merge two disparate code bodies by merging two references. A request to merge two references is caled a pull request. So to merge one branch like feature/my-latest-change into develop you’d create a pull request from my ‘feature branch’ into the ‘trunk’ which in this case is develop.

This is how basic version control works with source code. When you commit, often the repository server will notify listening services that code is updated. Those services will look at the repo and if they find a build instruction file they will do the steps listed in the file. This way when we want to build our software, we put all the means to do it in that file. When new commits are made to the repo, listeners will build the application based on our instructions.

If there are no code updates, listeners, or build instructions, there is no continuous part and no integration is happening. In the ancient software world, a developer would commit code and send an integration engineer release notes in an email and the integration engineer would run and babysit a build script while the developer went and got coffee. Now the developer makes a commit and then watches a job console with output logs from the build without communicating with other engineers… they still get coffee while the build runs.

Secrets Management

Pipelines

Fixing Incident Culture

Enterprise IT Processes

Business Continuity & Disaster Recovery

Summary

blah

Exam Essentials

blah

Official Resources

Architecting for Reliability

Nov 17, 2022

Christopher Shaun Godwin

Author

A reliable system is one people can get to now. Reliability is the probability a system can be reached and used without failure and Availability is a measure of how available a system is to be used within a given period of time.

Cloud Operations Suite

In an environment of constant change, hyper scaling, frequent deployments, and business demand, you cannot maintain reliable systems without metics and insights.

There are some problems you’ll come up against such as needing additional compute power, having to handle seasonal ups and downs, errors or crashing under load, storage filling up, a memory ceiling causes cache to cycle too often and therefore cause latency. The ways things can go wrong a several and in a distributed hyper-scaled environment you’ll run into 1 in a million problems as well. That is why we need detailed information about the operation of the resources in our project.

Cloud Operations Suite which used to be known as Stackdriver has several operations products:

Cloud Logging
- Log Router
Cloud Monitoring
- Alerts
- Managed Prometheus
Service monitoring
Latency management
Performance and cost management
Security management

Cloud Logging has the Log Router which is a built in part of Cloud Logging. The Cloud Logging API receives each log message and then send it to the Log Router which stores log based metrics, an then sends those messages to log sinks which store those entires in logs in a Cloud Storage bucket. Cloud Monitoring receives these log metrics and user defined sinks can send entries to BigQuery for longer storage retention. The default retention for Cloud Logging is 30 das.

Cloud Monitoring is Google’s managed product in which you can setup alerting policies to alert you or your team when things go wrong. Things go wrong in the form of failed health or status checks, metrics over defined thresholds, and failed uptime checks. Policies can be defined so that uptime checks can meet certain requirements. Cloud Monitoring has several integrations for notifications which include Slack and custom webhooks. Alerting Policies are Google’s way of user defined criteria for notifications about problems.

These are the three major services which when combined increase observability into your operations in GCP.

Cloud Monitoring

Monitoring is collecting measurements about what hardware, infrastructure and performance. For example, CPU minimum and maximum, CPU averages, disk usage as well as capacity, network throughput and latency, application response times, memory utilization, 1/5/15 minute load averages. These metrics are generally time series. Metrics usually have a timestamp, a name and a value. Sometimes they can have other attributes like labels as is the case in GCP. GCP auto defines metrics but you can define your own metrics using BigQuery queries while having the Log Router send the custom logs to BigQuery. The timestamp is usually epoch time while the value is some value like percent of disk capacity used, web1_disk_usage might be the name of the metric.

Cloud Monitoring has an API which you can query for time series data based on name or resource, offers grouping resource groups based on attribute, list members of resource groups, list metrics and descriptors, listing descriptors of monitored resources and objects.

Dashboards

Some out of the box dashboards are created when you create certain resources such as a Cloud Run instance or a firewall rule, or a Cloud SQL instance. Otherwise you can create the dashboards that you need for your project’s golden signals and other operational metrics that are important to your workload. Users can fully customize them to their needs and to their specific data. Like development, creating dashboards in GCP can often be a cyclical process because you have to create displays which help you quickly diagnose problems at scale. You may start out with planned Key Performance Indicators(KPIs) but then you might drop some and tune into others.

Cloud Monitoring Alerts

When you monitor for problems and use your metrics data in dashboards, you may move on to automatic alerts so you don’t have to monitor the dashboards. This allows you to notify the correct parties when incidents occur. Normally your cloud infrastructure is structured in a way that auto-healing remediates problems, but in the case where auto healing can’t fix an issue. Crashed pods for instance, are restarted when their liveness probes meet the failure criteria and the RestartPolicy allows for it.

Alerts trigger when time series data goes above or below a certain threshold and can be integrated with third party notifications systems such as MS Teams and Slack. Policies specify these conditions, who to notify, and also specify a way to specify the data you’re selecting the resources to alert on. Conditions are used to determine unhealthy states so they can be fixed. It is up to the architect of the policy to figure out how to define what is unhealthy. It could be a port not responding, an http code, or how long ago a file was written, as long as it can be exposed as a metric.

It is easy to create false or flapping alerts and therefore you’ll have to adjust the timing and thresholds for your conditions. You can also increase reliability by setting automatic remediation responses. When a CPU utilization alert is set, for instance you can add new VMs to a group, you can run a job that runs kubectl patch a kubernetes deployment’s Horizontal Pod Autoscaler(HPA) to raise the replicaCount ceiling and then lower it after load is decreased.

All of Google’s managed products like BigTable and Cloud Spanner do not need to be monitored because Google manages the incident response. Switching to these services can help you reduce the amount of monitoring, notifying and alerting you have overall. Of course the recommended approach on migrating to managed services with regard to alerting is to monitor throughput and latency on managed services though resource monitoring like cpu and memory are not needed on them. This is especially true if you are connecting to in-cloud managed DBs from on-prem workloads through VPN or interconnects. Hybrid and Multi-cloud latencies are metric points of monitoring that should be shown on a dashboard and included in notifications.

Cloud Logging

Cloud Logging is a log collection service that either has log collection agents or collects logs from managed services like GKE naturally. Log entries are not time series and occur when system events happen. The /var/log/syslog or /var/log/messages in a Linux VM collects messages about several services together, but there are other logs like /var/log/auth.log or /var/log/lastlog. These logs record data about who is currently logged in and the most recent shell sessions respectively. So these logs are only filled when users trigger login events, either on the consol or remotely. Processes may run garbage collection or some kind of file de-fragmentation and print log messages.

Cloud Logging can store logs from any GCP resource or on-premises resource. In Cloud Logging logs can be searched and queried, exported to BigQuery. When you use Cloud Log Analytics log data is automatically exported to BigQuery. You may also choose to send logs to Pub/Sub and have them consumable by third party log software such as Splunk.

FOSS Monitoring

The popular free and open source(FOSS) tools such as Prometheus and Grafana can be used with Cloud Monitoring. Prometheus is controlled by the CNFN who controlls Kubernetes. Prometheus scrapes HTTP(S) endpoints and collects data and stores it in a multi-dimensional way based on attributes. This is well suited to querying with PromQL, the query language used with the project.

Google Managed Prometheus provides a monitoring agent which uses Google’s in-memory time-series database called Monarch. Grafana used in conjunction with Prometheus can display metrics in graphs from several data sources. Grafana has the ability to directly query data sources and monitoring services.

Continuous Delivery

Managing releases are an important part of the software lifecycle development process. Some releases are more involved and more complex than others. Releases are often interdependent and therefore need high levels of planning and coordination on behalf of development teams and release engineering teams. The better release management and deployment strategies you have the more reliable your services will become. In an agile and continuously deployed environment, there are pipelines that deploy new artifacts to dev, test, staging, and production often called intg, qa, uat, and prod. intg and qa are what are considered the lower environments and those experience lower load but a higher and more frequent rate of iterations. So intg and qa get the most deployments. These frequent deployments to development and testing environments allows developers to go back to the planning stage before a change gets to production. This happens in the case it doesn’t pass 100% of the tests or function 100% of the time.

So problems are worked out early on and once they get to Unattended Automated Testing(UAT), the programmatic list of tests in a production similar environment under production similar load validates the release to go to production. Some pipelines have fully automated and unimpeded ascents to higher environments, however the more critical workloads in Fortune 500 enterprises all have barriers to production on services that will have customer impact in the case of a release failure.

Even with that, errors get into production and need to be fixed quickly, which is where this release management using DevOps principles of Continuous Deployment allows for a pull request to be merged and tagged, automatically built and polled by the CD triggers, and in minutes be sent out to all the environments ready for the approval barrier to succeed so that the hot fix makes it to production quickly.

This is an effective way to produce fixes quickly while reducing risk in releases. In this model, all the access to make release imacting changes are given to the developer who runs these pipelines when needed or when triggered automatically, while some release and integration engineers approve and perform the production runs and service swaps.

Testing in continuous deployment pipelines involves acceptance and regression tests, while unit and integration tests are usually part of Continuous Integration. The exception is that a lot of deployment code might have validations and unit testing as part of their runs. This is the case in terraform Infrastructure as Code(IaC) and with configuration management pipelines like salt and puppet. Tests usually define expected states and the resource being tested say an endpoint such as /health which prints the artifact version. The endpoint is what is checked and the state is the key and value expected. The test for that state passes when the endpoint is fetched and the real value and key are compared to the expected state. If the value was lower than expected the service has regressed and the regression testing will fail. In the case of a unit test, a yaml file might contain input, and the unit test in puppet processes the function. It contrasts the output to the expected output that the developer defined in the test. Several of these definitions when related constitute a test that would run before the deployment code executives the active part of the workflow.

Integration tests can exist at all different layers that code exists: in a repository, running as a service, testing dependent APIs. Integration tests can tests for things such as a name longer than the amount of allow characters that the backend will receive per the database schema. Integration tests are different than Unit tests as they span all the units of code together in a running artifact.

Acceptance testing are generally testing if the release being deployed meets the business needs the software was designed to meet such as a customer being able to open a new account, change their account data, review it and delete their account. This is an example of an acceptance test for a root business goal of onboarding new customers.

Some times an automation department will order a whole environment tier just for performance testing and load testing. With this you can understand how your application wil fail or perform under load. You can use load testing to simulate so many transactions per minute. While load testing you can do chaos engineering and make things go wrong to see how customers will be impacted. This teases out bugs, latency tuning problems, memory tuning problems, database connection ceilings and subsequent timeouts.

Deployment Methodologies

Service swaps are done typically in a blue green or canary deployment style. There are a few different popular deployment strategies:

Big Bang
Rolling
Canary
Blue Green
Swap Only

	Big Bang	Rolling	Canary	Blue Green
Expense	$	$	$	$$
Risk	very high	high	low	very low
Complexity	low	low	mid	high

Big Bang deployments, often called “complete” deployments simply update all instances of the software wherever they occur according to the recommended approach in the release notes. On a linux server that uses rpm packages as the method of deployment delivery, a service is stopped, the RPMs are installed with yum, dnf, or rpm directly, database deltas are applied if they are included in the release, and the service is started again. This may happen in series or parallel on all the systems to which it will be applied. This process can be run by script, package configuration and package manager, or a configuration management tool like Ansible, Salt or Puppet. Before continuous deployment was popular, this was the most used deployment style and it was performed manually at first, then with automation.

This is the cheapest as it only ever involves one copy of your infrastructure to be alive at one time.

Rolling deployments are the second cheapest because in some contexts you only need two copies of your infrastructure running while the latest one boots and becomes available. Once its healthy the previous versions are terminated. This is the case with Cloud Run and Kubernetes Pods with regard to rolling deployments. Otherwise with VMs, rolling deployments upgrades one server, tests or problems and then after a time moves onto another until the deployment is rolled out.

This kind of deployment is database delta risky because changes to the database which are not additions might cause 9 out of 10 servers fail running the older version. In this example scenario, 90% of your customers are impacted until the rollout progresses to 2 servers, then 80% suffer until the 3rd server has its deployment updated. If you don’t have db deltas or only ever append to you schemas, the risk is considerably less and only impacting a subset of customers at a time becomes an advantage.

Canary deployments are a type which releases new artifacts to infrastructure that receives a test amount of live traffic. When no errors are detected in the deployment, the rest of the traffic is routed to the new infrastructure. In the case of VMs this can be in the form of creating a new Managed Instance Group(MIG) with a new image that has been built with the new code. It can use its existing disk image but run some configuration management code to perform the upgrade, or it can have a new version label applied to it to be selected for an ssh script which does the deployment. In the case of containers, this comes in the form of a new docker tag, a new docker deployment and some routing magic which is automatically built into services like GKE and Cloud Run. There are several ways to choose users whose traffic is routed to the canary deployment.

Blue Green strategies are those which use two environments, one active while the other is inactive. When deployment pipelines run, they keep track of which service, either blue or green is active. When the deployment workflow performs the release, it releases to the inactive set of infrastructure which is receiving not traffic. Verifications, regressions and production tests validate the inactive deployment and then the workflow switches all the traffic to the new deployment at once.

While the most expensive route because it requires constantly maintaining two copies of identical infrastructure, it mitigates the most risk. Firstly, failed deployments result in a failed iteration and no change in routing therefore customers continue using the older version of the service. Then, if a live active deployment fails, the traffic can be swapped to the inactive service to reinstate an older version of the software without any new releases. The iteration can fail and the developer can take the feedback and begin again fixing the issues and cutting a new release version for a new deployment. In a blue green strategy you have to decide if both versions will connect to the same database. If you ony append to your schema this is fine, otherwise database deltas which edit, rename, remove or change tabes and felids you may consider running a blue and green database. Configure each service with either of these and when swapping the traffic, you change an environment variable selecting the database and restart the service. In kubernetes this is as simple as running kubectl set env on the deployment. You can run this command in swap deployment workflows for pod, replicationcontroller, deployment, daemonset statefulset, cronjob, replicaset.

With blue green deployments you’ll have to also script workflows which swap any urls of the services from active to inactive. This ensures that all the active services point to active urls while all the inactive deployments point to inactive endpoints. You can accomplish this in manually in the application config prior to deployment, or you can script this as part of your deployment swap workflows. Inside a GKE namespace, the nginx service is actively routing services to the nginx-blue pod while the nginx-stage service routes to the nginx-green pod. The nginx pods all proxy content for application pods called app. So nginx_blue needs to point its configuration at app_blue, which then connect to database_blue. Both the nginx and the app pods will need their urls swapped via kubectl set env or kubectl patch.

Continuous Integration

This is the practice of building code with triggers that listen to each commit to a repository or set of repositories. The CI jobs are configured to run syntax validation, vulnerability scanning, unit tests, code quality test uploads, and pushing artifacts to artifact repositories. CI jobs might be single stage, multistage, create java artifacts, create deb and rpm packages and then repackage them in docker files. There are several CI suites which drive integration from Jenkins to Bamboo. Google Cloud Build is googles managed and serverless Continuous Integration product. With it you can host source code in Cloud Source Repositories, or sync them there. Cloud Build Triggers can then listen to the repository and trigger the jobs configured in the cloudbuild.yaml file stored in the triggering repo.

You can also manually build Continuous Integration pipelines. Configuring build pipelines is a much more consistent way to ensure artifact quality than manual integrations and manually running the build steps by hand.

Reliability Engineering

Reliability engineering is mostly about building resiliency in pipelines, in the software, in the performance of services under load. One such example is a vanilla linux postfix mail server which uses linux users and groups as the main source of mail accounts. If users send mail using SMTP auth and check it with Imap, the shadow and passwd files are being queried ever time mail sends and receives. Additionally, when users change their password at the same time, there’s always a chance of collision in that the files in question become corrupted. It suffers form simultaneous writes from two different processes. Not collecting password changes an queuing them one at a time with a success confirmation between each change, that corruption will never happen. The acts of Setting up a message queue to collect password jobs, writing an agent which reads the queue, and does the work while tracking what work as been done successfully or with errors, and what work is undone are all efforts of Reliability Engineering(RE).

RE takes place on any layer of the technology surrounding a service from the code that runs services to the code that deploys services. Ensuring quality on every distinct layer is an SREs job.

Load is something you cannot plan for exactly how much you’ll have. Errors happen at rates because there are certain chances of an error occurring. When you increase load, you not only increase frequency of known errors, you pull out of the chaotic universe higher magnitude errors and lower frequencies. These are you one in a million errors, that say Ticket Master might face every day since they’re doing 100k transactions per second in some cases.

Overload

You can guarantee that at some point you’ll experience increased load and need to scale. If you aren’t using a service like Cloud Run’s autoscale then you’ll have to manually configure and reconfigure each service to handle the load per that service’s resource usage. Even in that case if you’re running a Cloud SQL instance you’ll have to vertically or horizontally scale it at some point.

Its best to design for this possibility at the beginning. The more user-facing a service is, the more reliability engineering will surround that service. Internal services and things like batch services which can fail cyclically and then at some point reach eventual processing, we don’t necessarily have to worry about unless we have inter-team SLAs which we have to honor.

You can simply shed load, meaning you can respond to requests greater than a system can handle with error codes instead of the application. This isn’t a clean approach though it is an approach. Based on revenue, business needs, you can shed load from priority services last and tertiary services first.

You can handle overload by degrading the accuracy or quality of the services. Switch ‘contains’ filters to ‘beginswith’ filters to reduce load on the database. Reduce latency everywhere you can but Instead of delivering full images deliver thumbnails to reduce load and restore higher resolution delivery later.

Upstream throttling is another way to deal with overload, you limit the calls or requests that you make on crippled systems. You can cache requests and process them later, enter requests into a message queue and process them later. You can switch from instant operations to queued operations modes, reducing load you can later offload to batch processing like profile edits or other non critical parts of your application. Spotify used a combination of CDNs and peer to peer client network to handle overload. The first 10 seconds of a song are loaded from a server and the rest of the file is loaded from other spotify users who have recently listened to a track.

If you build in a trip switch into your app, and then use monitoring to flip the trip switch, you application can decide to cache requests and process them when batch processes are performed like a wordpress cron job, for instance. You can flip the trip switch back when load returns to normal and the logic in your app will return its mode of behavior to the default behavior. When applications have built-in internal responses to overload, they become more reliable and they can log these occurrences for increase observability.

Cascading Failures

Cascading failures are those whose effect becomes the cause of another failure. If a database has a disk error, application instances fail, and then proxy instances fail. This is the simplest and easiest form but consider when the application is generally mostly functioning but particular operations are inefficiently written in code and so they create unnecessary cycles. Certain days when certain jobs run and there are intermittent failures, and everything retries three times before completing. This is like cars backing up on the highway because they have to try three times to change into the lane that goes their way. The traffic gets backed up and it affects not only the cars here, but also the cars in queue to arrive here. This can compound and remain a problem long after the initial prime cause is removed from the situation.

In a cascading failure, you may have a resource consumption problem that is the root cause and have issues determining on which system the root cause is happening. You can Upstream throttle in this case, and apply any overload strategy. If you have increased observability, say a dashboard for every impact causing signal, you can quickly see all the failed services in the cascade. You can have them organized and ordered by dependency so that you eye can go right to the problem. You can order your tests the same way in reverse so that things at the bottom of the stack like database and db disk size are the first tests so that you and run a test to identify the last responding service in a stack to quickly locate the root. So deal with these as you would overload, including using degraded levels of service. Windows introduced safe mode as a way to reliably boot your computer amid problems enabling users to make changes, and fix the issues, rebooting into disabled mode. Windows safe mode boots into a degraded level of service and sheds some load by not enabling it.

When mitigating overload with autoscaling, consider that you need to set the thresholds as low as they need to be so that the load does eat up the resource gap by the time the new resources become available. If you set your Horizontal Pod Austoscalers to add a new replica when one container reaches 90% of CPU resources, but it takes 156 seconds to start a new pod, but only 100 seconds to eat up the remaining 10% of the resources, there will be a period of 56 seconds of unavailability. You’ll need to set your thresholds lower or work on a speedier boot time on our containers.

Scaling down too quickly is also a concern and if your scaling down thresholds are too low, you might create a flapping situation where pods or instances are created and then destroyed repeatedly.

Reliability Testing

The reason why you want to test is increased observability, but also: have you ever put a bed-sheet on alone? One change to one area tugs an unexpected change in another area. You have to iterate too many times to arrive at your goals. With testing, you peg values as non-moving targets and the more you add the more of the field you successes and errors you can orient your self against. It’s like pinning the sheet into place on one side while you tighten it on the other side getting out every wrinkle. This ensures that all processes stop right when you make a change that disrupts expected states and values.

Unit tests do this for software, are written and performed by developers and then incudes into the continuous integration process. Integration tests ensure that the units of a feature perform as a whole represented as a function. System tests are those which tests all components under a simple set of conditions that represent sanity checks. System tests that are called performance tests do this same process while placing several repeated requests simulating load. Regression tests are system tests which check to make sure past issues continue to be resolved in future releases. Reliability stress tests are those which do not limit the load but increase it continually until something breaks. The configuration and memory management of a java application, or instance is adjusted and the tests are rerun. You approach this until you exceed at minimum 20% growth over your highest load.

Stress tests are often used to simulate and understand cascading failures. This will inform your monitoring goals and strategy. Chaos engineering puts load on a system an then just randomly causes probably problems to see how the system will respond in order to tease out mitigation responses before they occur.

Incident Management and Post-mortem sessions

Incidents are major problems. Severe incidences are those that impact services which have Service Level Agreements. Severe incidents can be defined as those which impact multiple teams and multiple different type of customer experiences on the service-level. Incident management is the set of duties surrounding incidents and include remediation and fixing the incident, recording details about the state of the incident as it initially occurred and a history of all the decision surrounding the incident. Incident management duties often include making calls to involved parties in an escalation tree.

Notify a captain who coordinates the incident response.
Call a working session with available response teams from operations, automation, and development teams.
Analyze the problem, make corrections
Audit all actions taken into a log for the post-mortem analysis

Incident management focuses on correcting the service-level disruption as soon as possible. There should be less concern with why it failed but how it will be fixed. Incident management focuses on correcting the service-level disruption as soon as possible. There should be less concern with why it failed but how it will be fixed.

The post mortem should focus on a blameless cause of the incident. Blameless postmortems create less of an environment of fear which reduces cognition. Cognition is key to production solutions which fix future versions of this problem. In the spectrum of problems one can have there are patterns, unique to your app, that will form in incidents. If you catch and fix each one, 20% of all fixes will negate 80% of the errors. This zipfy statistic is what allows startups to launch on a startup amount of effort. As an application matures, engineers take on the remaining 80% of fixes which are one offs which apply to fringe cases that only affect 20% of the customers.

Incidents/Bugs	Fixes	Customer Affect
Wide field	20%	80%
Narrow field	80%	20%

This zipfy pareto principle is basically a law of nature and governs everything.

Summary

Reliability is a measure of how available the system is over a period of time. Creating reliable systems is a discipline involving application design and development, deployment methodologies, Incident management, Continuous Testing and more. Continuous Integration and Delivery managed code releases and bring sanity and mitigate risk in what was traditionally quickly changing process. Systems Reliability Engineering involves software development that includes operations goals, things like safe modes with degraded services or upstream throttling. Architects must understand that systems will fail, and that the best way to live with failures are to have defined service level objectives service level indicators, monitor services to detect incidences, and learn from failures by to risk assessment and mitigation techniques.

Exam Essentials

Understand monitoring, logging, and alerting in gcp and in relation to reliability
Be able to design for continuous deployments and integration
Be versed in kinds of tests use in reliability engineering
Understand that Reliability Engineering(RE) is a collaboration of operations and development goals combined on all levels of the system to reduce the risk of conflicting interests between development and operations.
Understand that RE includes planning for unplanned load, cascading failures, and responding to incidents
Understand that testing is a cornerstone of reliability engineering

Official Resources

Architecting GCP Solutions for Security and Legal Compliance

Nov 3, 2022

Christopher Shaun Godwin

Author

Identity and Access Management

Identity and Access Management or IAM is a service which lets you specify which users can perform which actions in the cloud. IAM includes the following objects:

Identities and Groups
Resources
Permissions
Roles
Policies

Identities users and service accounts, groups are collections of those. The Identity entity itself is the thing which is granted access. When you perform any actions in GCP, you must first authenticate against an identity, either on the Console or with the gcloud command. Identities are also called ‘members’. There are three kinds of core identities: Google account, Service Accounts, and Cloud Identity Domains.

Google accounts are members that represent users who access resources in GCP. Active directory users often are synced as Google accounts. Service accounts are accounts systems and programs use. Your terraform instances in an Enterprise environment might be created by a service account with the appropriate IAM roles or permissions to do so. Service accounts are denoted by a service account id in projects/{{project}}/serviceAccounts/{{email}} notation, or email notation sa-name@iam.gserviceaccounts.com. GKE service accounts are the same as compute service accounts. All compute operations run as the default compute service account.

Cloud Identity is an Identity as a service managed product which creates identities that are not tied to Google accounts. You can interface this with Active Directory OIDC and SAML.

Federating Google Cloud with Active Directory

Groups are collections of Identities belonging together. A group is the object that binds the members or the entity they’re associated with. The kind of members of a Google Group in IAM are service accounts and Google accounts. G Suite users and domains are group identities in GCP.

All of these: identities, groups and service accounts can be granted permissions or roles on Resources. A Resource is any GCP object.

Resources:

Compute Instances
Storage Buckets
GSM Secrets
Projects
etc…

Every resource has both granular permissions that correspond to any action that can be done on that resource and predefined roles which represent workloads a person may be assigned with regard to the resource(i.e. developer, viewer, administrator).

Permissions correspond to specific actions like getting, listing, or deleting a resource.

Cloud Run IAM permissions examples:

Permission	Description
run.services.get	View services, excluding IAM policies.
run.services.list	List services.
run.services.create	Create new services.
run.services.update	Update existing services.
run.services.delete	Delete services.
run.services.getIamPolicy	Get an IAM policy.
run.services.setIamPolicy	Set an IAM policy.

In Enterprise level companies, these fine grained permissions are more often used. Small companies may use the roles or even basic roles. If you’re going for a least privilege principal of access, then steering clear of the roles and only granting permissions will provide this. You’ll collect job roles from the team, and consider the privileges needed to do that work. Secrets accessor can be granted on the project level or the secret level. Enterprise companies will want to place it on the secret level. They’ll want to group the secrets to a service which accesses it and create a specific service account it will impersonate so that only that service can access it secrets and not the secrets of other services. The exam will not require you to know the permissions, however, knowing how granular they can be is what the exam creators will expect GCP Certified Architect’s to know.

Roles are groups of these permissions bound together in a role which you assign to an identity or group in order to grant access. Identities can have multiple roles. Roles apply across the project.

Cloud Run IAM predefined roles examples:

Role	Permission	Description
(roles/run.developer)	…	…
(roles/run.developer)	run.jobs.create	Create Cloud Run jobs
(roles/run.developer)	run.jobs.delete	Delete Cloud Run jobs
(roles/run.developer)	run.jobs.get	Get Cloud Run jobs
(roles/run.developer)	run.jobs.list	List cloud Run jobs
(roles/run.developer)	run.jobs.run	Run a job in Cloud Run
(roles/run.developer)	run.jobs.update	Update a Cloud Run job
(roles/run.developer)	…	so on and so forth

Applying these to an identity can be done at the Org, Folder or Project level and would apply to all sub resources in one of those three. Predefined roles are those like the above example. They pre-exist and are pre-defined collections of permissions. Other kinds of roles exist named basic roles which were the roles that existed before IAM. Basic roles apply to every resource and are Viewer, Editor, and Owner. The Viewer role gives read only access to resources, the editor grants change and delete access to resources which the Owner role inherits. Additionally the Owner role can assign roles and manage permissions to resources.

You can grant basic roles per resource so you can make on identity or group owner over certain Compute Managed Instance Groups while giving Owner to other MIGs. Owner role over resources allows users to set up a billing account for those resources. Its best to consider basic roles legacy and avoid them when possible.

Custom roles are those which are created by you where you group a permissions set into a custom role which you grant to identities or groups. This can help you adhere closer to the lease privilege access principle. Some developer roles allow you to set anything in a space where sometimes a thing should be restricted, like production. You would use a custom role to include all the things that a developer role has without the ability to write code and instead limit writes only to pull requests to the master branch.

Policies are Json definitions or directives called a binding which specifies which identities are bound to which roles and permissions. The IAM API allows you to set or get policies and test permissions. Policies can be set on the Organization, individual projects or folders of projects and are inherited infinitely deep.

IAM also has Conditions that are written in a logic language called CEL that is a versatile way to define access granting logic so that things like resource tagging may trigger granting access to certain groups over that resource based simply upon its attributes. Conditions can apply to the following services:

Cloud Storage
Compute Engine
Cloud KMS
GSM
Resource Manager
Cloud SQL
Bigtable
IAP

Google recommends these best practices for using IAM in a secure way.

Lease Privilege Principles

Do not ever use Basic roles in production
- Role recommendations will recommend a role to replace basic roles.
- Policy Simulator will allow you to ensure this replacement will not change access
Consider each layer of workload of your app is untrusted, give each one its own serviceaccount and grant only the permissions the app needs.
Consider that all child Resources inherit the permissions of their parent Resources. Don’t grant project level roles when Resource level roles will suffice.
Grant permissions or roles on the smallest scope needed.
Specify who can impersonate which service accounts
Limit who can create and access service accounts.
Take care who you grant Project IAM Admin and Folder IAM Admin
Conditional bindings can allow access to expire
Consider granting privileged access only on a just-in-time basis.

Service Accounts

Rotate your service account keys using the IAM service account API.
Label the service account with a deploy name that tells you about what it is for and what it has access to.
Don’t leave the service account keys in email, check them into code, or leave them in the Downloads directory.
Audit changes to your policies with Cloud Audit Logs
Export logs to Cloud Storage for preservation
Audit who has the ability to change your allow policies on your projects.
Limit access to logs per least privilege principles
Use the Cloud Audit Logs to audit who has service account key access

Policy management

If identities need to access all projects in an organization, grant access at the organization level.
Use groups instead of users when possible.

Bad Actors will look for Service Account Keys in these locations:

Source code repositories of open-source projects
Public Cloud Storage buckets
Public data dumps of breached services
Compromised Email inboxes
File shares
Backup storage
Temporary file system directories

Identity Aware Proxies(IAP)

IAP are Layer 7 proxies which are capable of allowing or denying HTTP(S) requests based on IAM policy and identity membership. If a user making the request doesn’t have an identity associated with it, the user will be redirected to a Google Oath page to sign into to a Google account or single signon account. Once an identity is associated with the request, and if the identity is allowed to access the resource, then the IAP forwards the connection to its destination.

Using IAP Proxies in front of app are ways you can limit access to parts or all of your application based on Google account. IAP for On-Premises Apps is Googles way of protecting Apps in Hybrid-Cloud Networking environments with IAM.

Workload Identity

Workload Identity is a way to grant IAM roles and permissions to external identities. If you want a Kubernetes service account to have certain permissions in GCP, the secretAccessor role for instance, workload identity federation is the IAM feature which will allow you to do that. Workload Identity providers do the magic of connecting the external entity to the workload defined. These providers either use SAML or OAuth 2.0 token exchange.

Providers supported:

AWS
Azure Active Directory
On-premises Active Directory
Okta
Kubernetes clusters

Organization Policy Constraints

Organizations can have limits placed on them for any number of attributes of the org’s resources. You can prevent certain actions from being taken by identities or service accounts. For instance, if you want all CloudFunctions to work through the VPC in a given project, you can create and then apply a constraint against constraints/cloudfunctions.requireVPCConnector. Depending on the constraint, it may apply to a set o Google services, or to specific services. You can find a full list here.

Data Security and Encryption

Encryption is the process of masquerading data that is in one form into another form using encoding algorithms which produce results that are impractical to convert back without having the cypher keys. Encryption at rest is usually denoting filesystem encryption. Encryption in transit usually refers to things like SSL over TCP or HTTPS encryption.

Within the ecosystem of Google Cloud, Encryption at rest occurs at the hardware level, at the data infrastructure level, and using file encryption. On the infrastructure level the data is grouped into chunks and each one is encrypted. Using AES 256 and 128 encryption, Google can either use encryption keys Google creates and manages or customer managed keys in Cloud KMS.

Cloud SQL encrypts all data together with one key in the same instance. Cloud Spanner, Cloud Bigtable, and Cloud Firestore using an infrastructure encryption mechanism. In storage systems, the data is grouped into chunks which can be several gigabytes in size, and each chunk is encrypted with a data encryption key(DEK) which Google encrypts with key encryption keys(KEKs). DEKs are stored near the chunks they encrypt and sent to a centralized store where they are encrypted by the KEKs which are also stored in a centralized location. If data is changed or added to a chunk, a new key is created and the chunk re-encrypted. Keys are never reused with regard to chunks. Access control lists refer to some of the chunks’ unique identifiers. All these chunks are stored on drives which have hardware encryption built into their chips.

Encryption in transit or encryption-in-motion protects against network interceptors and middle men. Data in transit in GCP on the Google network may not be encrypted but is authenticated at every transfer. Data in GCP that is outside the borders of the Google network is always encrypted. All incoming traffic to Google Cloud goes through the Google Frontend which runs on distributed global loadbalancers and protects against DDoS attacks. All communication to Google Cloud uses either TLS or QUIC. Within the Google network Application Layer Transport Security(ALTS) to authenticate and encrypt most intra-network connections.

Users do not have to create resources or set anything up to enable this encryption but they cannot control or manipulate the default Google Managed keys. Rather they can use their own keys with Cloud KMS. By default, DEKs and KEKs are rotated by Google. When a system tries to access a chunk, it requests the DEK from the key management system which authenticates the calling service, and then it sends the DEK to the storage system that decrypts the data.

Cloud KMS is a managed service for customer controlled encryption keys. It handles generating, importing and storing the keys within Google for application layer encryption on services such as Cloud Storage, BigQuery and Bigtable.

Cloud HSM is Google’s support for FIPS 140-2 keys using them only in Level 3 hardware modules which are tamper-evident and respond to tamper attempts.

Customer Supplied Keys is the option for using your own key management entirely. Keys are generated and kept onpremises and passed along with API calls which only use them in memory never storing them to disk. This way, Google can encrypt or decrypt the data with the customer supplied keys. This customer provided key is used to create a new customer derived key in combination with a per-persistent disk cryptographic nonce. In many cases, the customer supplied key is used to seed other keys that only stay in memory except for the nonce. Cloud External Key Manager(EKM) is the service which allows one to use third party management of keys and sets up Cloud KMS to consume them.

Cloud Storage supports ACLs in finegrained access mode to mirror support for them in Amazon S3 buckets to aid migrations, but this support is considered legacy. Otherwise buckets support IAM access at the bucket and project levels in uniform access mode. You can also use url signatures to grant temporary access to objects. Storage Buckets can also be made available publicly.

With Cloud Storage, signed policy documents can be created and set to restrict uploads based on sizes, type and file attribute based restrictions. It is a best practice to write checksums for all uploads and verify them. Google recommends creating and using CRC32C vs MD5 checksums due to its support of composite objects that are created with parallel uploads.

You can Secure your GKE or Anthos clusters with binary authorization, istio and mesh networking(ASM), cert manager, OPA policies and create all your elevated access service accounts with ACM.

Security Observability

Evaluation of security practices starts with increased observability into the different layers and components of the application you’re working with. This starts with understanding if your access controls and IAM policies work correctly. Otherwise you’re unaware the security measures put in place to run the application are working.

Auditing your policies begins with reviewing them and what has happened in your projects audit logs. The Cloud Logging agent will collect most common logs needed and can be configured to collect specific logins and accesses. Cloud Audit Logs is a logging service which records administrative operations taken in your project. Audit Logs are saved for a limited amount of time so they need to be exported to Cloud Storage or BigQuery if regulations require retaining for a longer amount of time. Logging can export messages to Pub/Sub as JSON messages, to ‘Logging’ datasets in BigQuery, or as JSON files to Cloud Storage. When everything is sufficiently logged, you can create access monitoring and run audit queries which that scan for anomalies which can be reported. Turning on a Google Artifact Registry’s automatic scan for vulnerabilities is an example of increasing security observability.

Penetration testing simulates an attack, particularly on a network interface of a host or a firewall. These tests connect with services and detect security vulnerabilities in running services. The solution is to often upgrade or patch an application so that it is no longer vulnerable.

The first phase of Penetration Testing is Reconnaissance where testers scope out the target much like a burglar looking for ways in. All information that can be gathered is gathered like Apache’s ServerToken string. Recon phase testing might include social aspects where the tester learns everything they can about the operators who do have access to the target system. This might come in the form of phishing or leaving a USB key near someone’s car in the parking lot. This phase can be short or long.

The second step is Scanning. Once information is gathered, points of access on the network like IPs and ports are scanned, http endpoints have their root and header capabilities fetched and tested, commonly vulnerable urls are checked to see if they exist to determine if an access vector is present.

Gaining Access is a phase where the information gathered and a scanned access vector is exploited to obtain access to the breeched system. Maintaining access is what happens when parts of the exploit or other exploits are stored or hidden in the filesystem, obfuscated, set to sleep or listen for commands from some remote uri. They may even scrub logs hiding their tracks.

It is recommended for highly secure environments to create automatic pentesting tools that run automatically and log to Cloud Logging, from which you can draw monitoring alerts or reports.

Security Principles

Three main principles apply when we discuss Cloud Security: Separation of Duties, Least Privilege, Defense in Depth.

Separation of Duties

Separation of Duties, especially combined with these two other principles, creates a strong accountability and oversight in the work. Separation of Duties means code committers aren’t the same as pull-request approvers. When multiple people have a scope of duties that are closely related, the impact and risk of internal bad actors is reduced.

Developers use pipelines created by reliability engineers through DevOps principles, but often they are not allowed to approve pipeline steps in the higher environments such as production. Small teams may have a harder time accomplishing this.

Least Privilege

Least privilege is the principle of giving only the access that is needed. Working in least privilege focused companies is often a headache as nothing is easy to setup, it often takes coming up against an access denial to know what requests you need to make of the access teams. It may take you weeks to set up something it’d take you days to do if you had full access. This is because when access is denied, despite planning, and requests are made for grants, documentation has to be updated, the Solution Architecture Document may need to be updated, several security teams may need to reapprove your project after discovery of new facets of the work, you might need to wait on a Cloud Solutions team to produce a terraform module which provides your needed resources for a part of the project.

If you have microservices that use serviceaccounts to access resources, separate the serviceaccounts into ones that represent the workload, so that resources are grouped and only the services which need to access their resources will be able to do so.

IAM roles and permissions can be granted to satisfy whatever schema you can conceive. Once roles are granted, or custom roles created, you can use the Recommendation Engine to help prune unnecessary principle grants in IAM.

Defense in Depth

This is the practice of controlling security at multiple levels of your application using the tools of those layers. For instance, If you treat a kubernetes pod as if it has a bad actor built into its image, we can distrust the filesystem as a safe place to store sensitive data. We can exclude secrets from env vars and use Google’s SDK to request the directly from the secret manager api upon startup of our application. This assures the secrets are only in memory and our bad actor now can be inside our pod and not be able to know the sensitive information.

So like a stairway of distrust we design while considering:

The Network is compromised
The Cluster or VM is compromised
The disk is compromised
Root is compromised
The Application is compromised

We are trying to reduce the above list to just the last item:

The Application is compromised

As an SRE, SRE Manager or an Architect, it is important to know that last item is the responsibility of the application development team to secure their code and app. The other items on the list, we can as SREs design around. We can introduce securityContexts on pods or containers which mount the root FS readonly. We can ask the app team to modify those applications so they only write to volumes. We can design around this stairway of distrust. If every connection is suspect, then when we secure them with istio and certificates then we can be fulfilling the principles of Defense in Depth.

Regulations

Regulations are a big part of organizations and business. Every industry and company is regulated. Understanding where those regulations intersect with your design decisions is the same thing as knowing the impact they’ll have on your project. Cloud Architects should know how these regulations apply to them and how to stay compliant with regulations like US medical industry’s HIPPA/HITECH, Europe’s GDPR, and COPPA.

The exam will cover these as well as Sarbanes-Oxley.

HIPPA/HITECH

This is the law which applies to medical records in the United States. It is designed to protect personal information and privacy.

The Health Insurance Portability and Accountability Act (HIPAA) was enacted in 1996 to improve the portability and continuity of health insurance coverage. The HITECH Act, enacted as part of the American Recovery and Reinvestment Act of 2009, promotes the adoption and meaningful use of health information technology. Both HIPAA and HITECH place privacy, security, and breach notification requirements on covered entities and their business associates.

As a cloud architect, it is important to be aware of HIPAA and HITECH and how they impact the handling of health information in the cloud. HIPAA and HITECH impose requirements on covered entities and business associates with respect to the security, privacy, and confidentiality of health information. These requirements must be met when storing or transmitting

The HIPAA Security Rule is a federal law that establishes national standards for the security of electronic protected health information. The Rule requires covered entities to implement security measures to protect the confidentiality, integrity, and availability of PHI.

What are the Security Rule Safeguards? The HIPAA Security Rule is a set of standards that must be met in order to ensure the confidentiality, integrity, and availability of electronic protected health information (PHI).

There are four main types of safeguards that must be in place in order to meet the requirements of the Security Rule: administrative, physical, technical, and organizational. Administrative safeguards are policies and procedures that must be put in place in order to protect PHI, while physical safeguards are measures taken to secure the physical environment in which PHI is stored. Technical safeguards are security measures used to protect electronic personal health information. Organizational safeguards are measures taken by an organization to protect the personal information of its clients, employees, and other individuals it deals with. Organizational safeguards are specified under Section 164.308 of the HIPAA Security Rule. Organizations must be able to design and implement appropriate administrative, technical, and physical safeguards to protect the privacy and security of individuals’ health information.

The most common technical safeguards are authentication, authorization, integrity, confidentiality, and availability.

The Privacy Rule requires entities covered by HIPAA to identify the personal health information (PHI) of individuals in certain transactions and maintain that information in an identifiable form only for legitimate business purposes.

The European Union’s (EU) General Data Protection Regulation (GDPR) came into effect on 25 May 2018, replacing the previous EU data protection legislation from 1995.

Under the new rules, organizations handling personal data of EU citizens must comply with a variety of requirements covering privacy by design, consent for data use, and access to personal information.

The GDPR treats Controllers and Processors differently. A controller is any person, organization or company that controls the collection and use of personal data. A person or company that collects data on people for their own use is called a processor. Any processor that uses personal data to create a valuable asset is required to identify the asset as a data subject’s must be informed and give consent to.

In the event of a data breach (e.g. leaked passwords), data processors must notify the data controllers who have to notify the government and the people whose data was breached.

SOX

The Sarbanes-Oxley (SOX) Act is a set of rules and regulations that help ensure the accuracy and transparency accounting information in publicly traded companies.

The act was introduced by Senator Paul Sarbanes of Maryland in 2002. The primary purpose of the act is to ensure that public information served by companies is accurate and complete.

In addition, the act requires companies to disclose any material weaknesses in their internal control over financial reporting.

What rules do they put in place? As far as IT Architects are concerned, the act requires the prevention of falsification and deletion of records, retention of certain records for defined periods.

This includes measures to increase transparency, and may include: periodic auditing compliance with SOX, developing a plan to disclose material information on a regular basis, ensuring that employees understand the company’s reporting process and comply with it, developing training programs to help employees recognize potential conflicts of interest, and creating a culture in which employees feel confident to raise issues without fear of being sued.

requirement to implement tamper-prevention controls
requirement for an annual audits
requirement to keep data confidential

Childrens’ Online Privacy Protection Act(COPPA)

COPPA is a United States law passed in 1998 which requires websites and online service to restrict what they do reguarding the personal information of children under the age of 13. Websites which serve this audience must:

Notify Parents before collecting data about their child
Allow Parents to block such collection
Give Parents access to the data collected
Give Parents the choice of how such data is used
Have clear and understandable privacy policies
Retain the data only for the length of time for which it is needed
Maintain confidentiality, integrity and availability of the collected data.

All the data covered by the law aren’t limited to but specifically mention the identifying information in the data, such as name, dwelling, photographs.

ITIL Framework

ITIL is a standard of IT management practices that dovetails business goals with common IT activities. ITIL has 34 practices that are grouped into General, Service, and Technical practices. General are strategy, risk management, disaster recovery, architecture, project and security management. Service management items are analytics and analysis, service design, capacity and performance, incident management, and asset management. Technical practices are those which include management of deployments, infrastructure, and software development practices. Businesses adopt something like the ITIL because its a magic box of best practices that fits many different scenarios. It creates a repeatable standard which can simplify a lot of trouble and guesswork in IT management.

Summary

Designing secure systems that will live in GCP starts with access and ends with compliance and touches everywhere in between. IAM is used to give access to identities which are users, groups or serviceaccounts, Permissions, custom roles, predefined roles, and basic roles provide for just about any concievable combination of access and limits. Policies ensure that company wide standards are enforced.

Encryption is everywhere and its power can be placed within the customer’s hands. Least privilege, defense-in depth, and proper auditing fill in the gaps.

Exam Essentials

Understand all the different parts of IAM and how they interact
Understand that roles are simply groups of permissions which go together
Basic roles are legacy and should be avoided when possible
Understand that access can be granted at the resource, project and folder levels
Understand that Policies use bindings to associate roles with resources
Understand the hierarchy of Organizations, Folders, Projects and inheritance
Understand Google’s Encryption at Rest and in Transit, know the AES bit level for each
Understand DEKs, KEKs, and how they’re used and interact
Understand all the types of managing keys
Understand pentesting and auditing
Understand the best practices for security
Understand how to use access and storage classes to achieve compliance

Official Resources

Architecting GCP Network Solutions

Oct 31, 2022

Christopher Shaun Godwin

Author

OSI Model Layers

Physical, the actual metal, wires, electrons and plastic ethernet plugs. You’ll find wifi’s radio frequency here because radio is physical phenomenon. In Quantum networking this layer are the entangled particles and the equipment uses to read and write to them plus the equipment used to connect to that. Voltage is sometimes the physical layer in Ethernet Over Power. With tin cans on a string, this layer is the cans, string and the vocal vibrations traveling through them.
Data Link, ARP, Mac Addresses, Collision avoidance. This is broken into two mini-layers, the first is media access control(MAC) and the second is Logical Link Control(LLC). The second acts as a negotiator between the MAC layer and the third ‘Network’ layer.
Network, this is where IP Addresses live. Keep in mind these network layers are the layers of a packet sent over the network. This is the base layer for packets. A packet is data encapsulated in a route with source and destination addresses.
Transport. The protocol that makes this process work known by all networking devices speak is Transmission Control Protocol(TCP) or User Datagram Protocol(UDP). The Protocol identifier stored in a packet lives in this layer.
Session, this layer manages handshakes. An SMTP connection timeout would exist on this layer. TLS handshakes happen here. An https packet is fully encrypted, so a request to a server asking for a url cannot be understood unless it is decrypted, then it can be seen. Inside layer 4 lives an encrypted layer 5 envelope in the case oF HTTPS connections. Layer 5 is the encrypted data, while layer 6 is the decrypted data.
Presentation, A GET / request is in this layer. Mappings of network resources to application resources in the OS kernel happen at layer 6.
Application, This is the later applications connect to in order to do networking. A webbrowser fetches web pages from this layer. This layer one might consider a data format. A TXT file vs a Json file. Mime types exist at this Layer. Layer 7 in the packet is the raw data unenveloped by network dressing that tells the network about it.

OSI Model as an Out of Town Drive

Gravel, Concrete, Rebar, Paint, Reflectors, Lights, Engine, Fuel, Speed Limit Sign
The Lane
Connected Roads
Vehicle Tags, Driving Skills, Driving Laws
The Trip Session
The Itinerary of the Trip
The People on the Trip

Architects only need to worry about layers 3, 4, and 7 with regard to load balancers, gateways, proxies, firewall rules, subnets, and traffic flow.

IP Addresses, FW Rules, and Routing

CIDR means classless inter-domain routing notation. It’s a way of simplifying the subnet mask by only specifying the bits. Understanding CIDR notation and IPV4 should be sufficient for the exam.

Networking in the cloud and in general works with IP Networking. IP networks are groups of devices. Subnets are spaces that identifiers live. A subnet is a street in a neighborhood. If all the addresses are single digits, then only 10 houses on that street are addressable. This is how IP networking works. You have to add more digits, or break the street into east street and west street to fit more addresses on that street.

In this way, networks are partitioned by their octets and subnet masks. Additionally they are partitioned with firewalls, NATing and public vs private IP spaces. Computers on the same physical network have an ARP table which maps IPs to MAC addresses as well as routing tables which map certain networks to specific network interfaces. IPV4 uses a four octet notation. Each octet represents numbers from 0-255. 0.0.0.0 is the internet. 255.255.255.255 is a subnet mask. Routers usually sit on the first or last ip in a network: 1 or 254. In binary to count to 255 you need 8 bits: 11111111, while 255 can be represented in hexadecimal as FF. but both have the same number of bits. So the highest number in an IPv6 block(FFFF) is 65535. That means that the IPv6 block has an entire IPv4 class B network within just one of its 8 groups: F0d8:0000:0000:0000:0000:0000:0000:0000. No IPv6 knowledge is required.

You’ll use CIDR ranges to specify subnets in GCP. You can learn subnetting in IPv4 or use tools online or in the shell like ipcalc to find the right amount of addresses for your private networks. Remember to consider growth. No overlapping subnets can be created in a VPC and each subnet must be uniquely defined.

In IP Networks, there are public and private networks. Certain online committees like the Internet Engineering Tas Force(IETF) process documents lie those called RFCs which define internet open standards. RFC 1918 designates these subnets for internal private use:

10.0.0.0/8

$ ipcalc 10.0.0.0/8
Address:   10.0.0.0                 00001010. 00000000.00000000.00000000
Netmask:   255.0.0.0 = 8            11111111. 00000000.00000000.00000000
Wildcard:  0.255.255.255            00000000. 11111111.11111111.11111111
=>
Network:   10.0.0.0/8               00001010. 00000000.00000000.00000000
HostMin:   10.0.0.1                 00001010. 00000000.00000000.00000001
HostMax:   10.255.255.254           00001010. 11111111.11111111.11111110
Broadcast: 10.255.255.255           00001010. 11111111.11111111.11111111
Hosts/Net: 16777214                  Class A, Private Internet

172.16.0.0/12

$ ipcalc 172.16.0.0/12
Address:   172.16.0.0           10101100.0001 0000.00000000.00000000
Netmask:   255.240.0.0 = 12     11111111.1111 0000.00000000.00000000
Wildcard:  0.15.255.255         00000000.0000 1111.11111111.11111111
=>
Network:   172.16.0.0/12        10101100.0001 0000.00000000.00000000
HostMin:   172.16.0.1           10101100.0001 0000.00000000.00000001
HostMax:   172.31.255.254       10101100.0001 1111.11111111.11111110
Broadcast: 172.31.255.255       10101100.0001 1111.11111111.11111111
Hosts/Net: 1048574               Class B, Private Internet

192.168.0.0/16

$ ipcalc 192.168.0.0/16
Address:   192.168.0.0              11000000.10101000. 00000000.00000000
Netmask:   255.255.0.0 = 16         11111111.11111111. 00000000.00000000
Wildcard:  0.0.255.255              00000000.00000000. 11111111.11111111
=>
Network:   192.168.0.0/16           11000000.10101000. 00000000.00000000
HostMin:   192.168.0.1              11000000.10101000. 00000000.00000001
HostMax:   192.168.255.254          11000000.10101000. 11111111.11111110
Broadcast: 192.168.255.255          11000000.10101000. 11111111.11111111
Hosts/Net: 65534                     Class C, Private Internet

Above, Hosts/Net shows the total number of ip addresses on the network.

Firewall rules control the flow of traffic over any network. In a VPC in GCP, you’ll find firewall rules are part of the network. Traffic flowing into a network is called ingress, and traffic which exits the network is called egress.

Respectively, firewall rules fall into the categories of controlling either ingressive or egressing traffic. Implied firewall rules exist by default. The first one blocks all ingressive traffic and the second one allows all egressing traffic. These rules cannot be deleted and they aren’t listed, they’re implied. To override them you make other rules with a higher priority. If traffic enters or exits the network, its properties are matched to all the rules in order of priority. When a match occurs the rules are no longer processed. Therefore a higher priority rule allowing all HTTPS traffic into the network that matches an incoming packet will allow the packet and not move on to the lower priority implied rule that blocks all traffic.

Rule priority is processed from low to high, low being 65535 and the highest being 0. The two implied rules have a priority of 65535.

There are four default rules designated on each default VPC network.

default-allow-internal: allows all VPC traffic to and from the VPC
default-allow-ssh: allows ssh from outside the network to any instance within the network
default-allow-rdp: allows Remote Desktop Protocol(RDP) connections from any source to any VPC destination
default-allow-icmp: allows ping to ingress into the VPC

These four rules have a priority of 65534 and are therefore the second lowest.

Ingress rules can specify the source ip while egress rules can specify the destination. To get more granular than that you can use network tagging in your firewall rules, and then tag compute resources. Otherwise all rules can specify an allow or deny action, the targets to which the rule applies, the protocol, the port, and enforcement status(enabled or disabled). Firewall rules exist in Google’s network at the global scale so all a Projects rules apply to every location within which the project has resources.

Cloud Router is a Border Gateway Protocol(BGP) software router in the cloud which advertises its IPs to networks outside of the cloud. When it interacts with those networks, it learns IP information about them. These public routers then speak to each other to map and remap the internet to physical connections. In this way, a ip range can be moved from one internet provider to another when they both allow BGP to communicate over them. This allows physical internet connectivity redundancy.

Cloud Router handles routing for the following services:

Dedicated & Partner Interconnects
High Availability VPNs
Router appliances

Cloud armor is an application layer(OSI Layer 7) web applications firewall(WAF) what protects against DDoS. attacks, cross-site scripting, and Database injections. The preconfiguration for Cloud Armor uses rules mitigating OWASPs top ten threat list. Cloud Armor has security policies that filter connections that use attack methodologies allowing the ones free of them to pass. Policies are available as preconfiguration while allowing for manually configured policies. Rules are defined with a rules language, but policies can also simply specify whitelists of trusted parties.

Virtual Private Clouds(VPCs) are networks which exist in the cloud at the global scale, so VPCs in Google span all regions. VPCs have subnets and all resources that use internal ips, which are Compute Engine based services for the most part. Cloud Run and App Engine can connect to VPC resources through a Serverless VPC connector, though the connectors for each.

Though VPCs are global, subnet resources are regional resources, since there is no overlap between subnets, each region’s subnet resources must be unique from other subnet resources in any region including unique from those within the same region. When VPCs are created you can specify automatic creation of subnets for different regions, or you can choose a custom provisioning of subnets for the regions involved. /29 subnets are the smallest allowed networks within a VPC.

VPSs can be set to one of three modes:

default: the mode selected when creating a new project
auto-mode: an automatic mode that creates subnets in every region
custom: allows full control of subnetting for production and high security environments

Auto-mode uses this range to create a subnet in every region automatically:

$ ipcalc 10.128.0.0/9
Address:   10.128.0.0           00001010.1 0000000.00000000.00000000
Netmask:   255.128.0.0 = 9      11111111.1 0000000.00000000.00000000
Wildcard:  0.127.255.255        00000000.0 1111111.11111111.11111111
=>
Network:   10.128.0.0/9         00001010.1 0000000.00000000.00000000
HostMin:   10.128.0.1           00001010.1 0000000.00000000.00000001
HostMax:   10.255.255.254       00001010.1 1111111.11111111.11111110
Broadcast: 10.255.255.255       00001010.1 1111111.11111111.11111111
Hosts/Net: 8388606               Class A, Private Internet

The VPC reserves four ip addresses from every subnet. Shared VPCs are shared from one project to another. This may be part of an organizational structure, or collaboration between parts of a company. Google recommends using one VPC because its easier to manage. However large enterprises will ignore this.

Shared VPCs are how the resources across several projects can be on the same network. This works because the host project defines service projects. The firewall rules for the resources can exist on the project but apply the shared VPC. You can specify that all future subnets are shared in a host project or just specific subnets.

You can take this further and delineate network and project duties partitioning them among teams and therefore separating their privileges. As long as the host project and service projects are in the same organization, shared VPCs can be used. Migrations are the exception.

When projects are in different organizations and need to communicate over a network, they can use network peering. VPC Network peering allows two VPCs to communicate with one another via RFC 1918 private ranges. Organizations usually communicate over the internet with public ips. If a lot of private communication exists between companies, they’ll use a VPN to communicate over private networks. VPC Network peering is an alternative to these approaches.

VPC Network peering might be used by an organization wanting to make their services available to its customers who are different organizations in GCP. A Concert company might make a private cloud network available to the ticketing vendor and the marketing vendor so that the concert organization can coordinate ticketing and sales from booths within the venue.

Companies might use organizations as part of a higher segmentation of their projects and may have a need for organizations to communicate over its peered VPC.

VPC Network Peering:

has lower latency, doesn’t travel over the internet
as an alternative to public ips, a peered VPC is a reduced attack surface
egress between peered VPC is free

Peered VPCs have their own firewall rule definitions from the VPC that is within an organization. A single VPC can have up to 25 connections peered at maximum. VPC peering works with Compute base services which receive a private IP. With peering, both peers must set up the configuration and the configurations must match. If a peer deletes their side’s configuration, the peering will cease and go into inactive mode. Peering doesn’t add latency.

Hybrid-Cloud Networking

Hybrid-Cloud Networking is networking which spans clouds or to onprem datacenters. When only separate public clouds are involved, Multi-cloud Networking is involved. But when an onprem datacenters is involved with one or more public clouds, Hybrid-Cloud Networking is the term applied. Services which connect to onprem databases thorough a dedicated or partner interconnect is considered Hybrid-cloud networking as is something like Anthos Service Mesh in a hybrid context.

Top 5 workloads staying onprem according to Dell:

Unstructured Data Analytics is staying onprem 31% of the time due to a more secure environment for which the data to live.
Structured Data Management & Analytics for the same reasons.
Business Applications like ERM, ERP, CRM
Engineering/Technical

Top 5 workloads moving to the cloud:

Databases
Batch processing, File lifecycle
Backups, Disaster Recovery
Petabyte scale data warehouses
Scaled workloads, Compute Workloads, Stateless kubernetes applications

Data warehouses in the cloud like BigQuery can use onprem sources, and the interconnect between cloud and onpremises datacenters needs to have the capacity for that connectivity. You must know the projected bandwidth usage and adequately plan for not only growth but redundancy for critical operations. This keeps the network reliable under load.

Latency is also a consideration. Running stateless GKE applications that connect to an onpremises database can expect a 2000 millisecond latency accessing a moderate payload event when they run on the fastest and most compute specialized nodes. The bottleneck is entirely the connectivity between datacenters and cloud regions. This is less of an issue with customer non-facing applications, however with things like JAM stack APIs running in cloud, this affects page load and the quickness of your app.

One way to handle latency is to use caching in the cloud so that the calls back to onprem databases or APIs will only take long once in a while. One might take a local database and sync it with mongo mirror or add a replica to a local MySQL database in the cloud to reduce latency and continue to meet SLAs.

Network Topologies:

Mirrored topology: General onprem resources are exactly mirrored in the cloud
Meshed topology: All resources can connect with all resources
Gated egress topology: Onprem APIs are made available to the cloud
Gated ingress topology: Cloud APIs are made available to onprem services
Gated egress and ingress topology: both the prior two
Handover topology: Onprem data is uploaded to the cloud to be used by cloud services

Your choice of these depends on workload distribution, latency, throughput, and existing topology.

The ways to implement Hybrid-Cloud Networking are by three different means:

Cloud VPN
Cloud Interconnect (either direct or partner)
Direct Peering

Cloud VPNs are services that create a virtual private connection between your VPC in Google and your other networks. Cloud VPNs are IPSec tunnels and so they require public static IPs on both ends. Google offers an HA VPN and VPN Classic. The HA VPN uses two connections to one HA VPN gateway, each connection comes with their own external IP addresses. The HA option has 99.99% availability. VPN Classic provides 99.9% availability with one connection and endpoint. Both option support 3Gbps. When data egresses to the VPN is it encrypted and when it ingresses into the destination network it is decrypted. Cloud VPNs operate with Internet Key Exchange(IKE) protocol.

Cloud Interconnects provide direct connections between GCP and onpremises networks. Highly available interconnects use two connections. Interconnects are available in 10Gbps and 100Gbps bandwidths. Partner interconnects are available between 50Mbps to 50Gbps bandwidths. Google’s interconnects terminate at one of Google’s Points of Port(PoP). If you are not near enough to a PoP, you interconnect to a third party near you who has connections near one.

Interconnects are:

Private
VPC addresses are available to onpremises networks without NAT or encryption
You can scale up interconnects

Interconnect scaling chart:

	Dedicated	Partner
Unscaled	10/100Gbps	50Mbps-50Gbps
Scaled	80/200Gbps	80Gbps

80Gbps connections use eight 10Gbps combined and 200Gbps interconnects use two 10Gbps combined.

Direct Network Peering is used when you need to affect the BGP routing from GCP and Google Workspace services. Peering doesn’t utilize any part of GCP, rather it affects the internets routing matrix so that your public resources route directly to you. Google recommends simply using interconnects when not needing to connect to Workspace services.

Private Service Connect for Google APIs connects Google’s public API to private locations without the need for egressing over the public side of the network. Private Service connect can be configured to point to private.Googleapis.com(all-apis) or restricted.Googleapis.com(vpc-sc).

Private Service Connect for Google APIs with Consumer HTTP(S) offers the same service but connects to internal loadbalancers inside your VPC which forwards the correct requests to the correct API.

Private Google Access connects custom domains to Google’s APIs through a VPC’s internet gateway. With this option you have to create the DNS records you’re using and the dns records that point to the all-apis or vpc-sc api domains.

Private Google Access for Onpremises Hosts is access that allows onpremises hosts to connect to private Google resources over Cloud VPN or Cloud Interconnect.

Private Service Connect for Published Services allows you to privately connect to services in a different VPC that has published their service using the Private Service Connect for Service Producers.

Private Service Access is network access used by Serverless GCP resources to connect to VPC resources over IP when VPC peering is used.

Serverless VPC Access is used by serverless resources to connect to VPC resources using an internal IP address. This option uses VPC Connectors to connect from Cloud Run, Cloud Functions, and App Engine Standard.

Load Balancing

GCP has five different loadbalancers(LBs) for different use cases. Is your workload balanced between addresses in a region or across several regions? Does the LB receive internal, external, or both internal and external traffic? What are the protocols of the connections being balanced?

GCP Loadbalancers:

Network TCP/UDP
Internal TCP/UDP
HTTP(S) Proxy
SSL Proxy
TCP Proxy

multiregional=>condition: Multi-Regional Balancing?
https=>condition: HTTP(S)?
ssl=>condition: SSL?
tcp=>condition: TCP?
intorext=>condition: Internal traffic?
internallb=>operation: Internal TCP/UDP
externallb=>operation: Network TCP/UDP
httpstraffic=>operation: HTTP(S) Proxy
ssllb=>operation: SSL Proxy
tcplb=>operation: TCP Proxy
e=>end: End

multiregional(yes)->https->e
multiregional(no)->intorext
intorext(yes)->internallb
intorext(no)->externallb
https(yes)->httpstraffic
https(no)->ssl
ssl(yes)->ssllb
ssl(no)->tcp
tcp(yes)->tcplb

HTTP(S) Load balancers are Layer 7 LBs and specifically handle http traffic. For other SSL purposes, like loadbalancing SMTP TLS you’d use the SSL LB as it is also a Layer 7 LB which operates on other protocols. For everything else, there’s TCP. You would use any of these three if you are balancing across two or more regions.

Additional Networking Services in GCP

Service Directory is a managed service discovery meta- database. Service directory can be accessed by a number of means, clouds, and GCP Services.

Cloud CDN is a managed content delivery network enabling global latency reduction for data access of files such as images or documents. Cloud CDN can pull content from Compute Engine Managed Instance Groups, App Engine, Cloud Run, Cloud Functions, and Cloud Storage.

Cloud DNS is a managed and globally distributed hosting service for the Domain Name System. Cloud DNS supports public and private DNS zones. Private zones are visible within the VPC and public zones are published to the internet.

Summary

Virtual Private Clouds are global resources which contain your addressed services. VPCs have various ways of having serverless environments connect to them, or private connections out to Google APIs from VPCs with no egress to the internet. Connecting VPCs to onpremises networks is done through hard connection and network management of the flow of traffic over interconnects which can be Highly Available as can Cloud VPNs.

Hybrid Cloud Networking, either with Interconnects, VPNs or Direct Peering allow workloads to span between local datacenters and cloud resources. Architects include latency, network topology, transfer time, maximum throughput, and room for growth.

Load Balancing handles different use cases with 5 types of loadbalancers, 2 regional, and 3 global.

Exam Essentials

Grasp VPCs
Understand VPC Sharing
Understand Firewall Rules, priorities, and direction
Know CIDR notation, lean how to subnet in your head or with ipcalc
Understand Hybrid-cloud Networking(HCN)
Understand when to use HCN
Know the advantages and disadvantages of each HCN option
Understand Private Access Services
Understand GCP Load Balancing

Official Resources

Architecting Storage Solutions in Google Cloud Storage

Oct 31, 2022

Christopher Shaun Godwin

Author

Object Storage

Object Storage is common to all cloud systems and has its roots way back in 2006 with Amazon S3 and Rackspace Files/OpenStack Swift and Google Cloud Storage in 2010. These systems are for storing files or documents as objects as opposed to a directory filesystem. Instead of hierarchical the particulate nature of object storage treats everything atomically. You can’t seek and read parts of the file, you can’t tail off of object storage. You can get, put, delete objects. Their organization depends on the system.

Organization

Buckets in GCP are containers filled with these particular objects. Objects when updated create new versions, you cannot update an old version with a new file. Once a version is there they’re immutable or unchangeable. The bucket is the logical definition with the IAM permissions that the objects inherit. Therefor you’ll give write access to all the objects in the bucket to any accounts with write access. You can place individual IAM permissions upon individual objects. There is an illusion of a directory structure because the file /pictures/2022-10-20/picture.jpg on a file system would be named picture.jpg and live in the folder /2022-10-20/ which in turn lives in the folder /pictures/. However, with object storage, /pictures/2022-10-20/picture.jpg is the entire filename.

Buckets must be uniquely named from all other buckets in the cloud owned by all other users. Buckets cannot be renamed or automatically copied to a new bucket. Objects don’t have to be uniquely named.

Bucket name best practices:

Bucket names shouldn’t have personal information.
Use DNS naming standards.
Use UUIDs or GUIDs if you have buckets in any real quantity.
Don’t upload objects with time series based filenames in parallel
Don’t name objects in sequence if uploading them in parallel
It’s best to use the fully qualified subdomain

One way to access cloud storage is through a FUSE mount. FUSE (Filesystem in Userspace) is a software interface that allows users to create and access virtual filesystems. This can be useful for mounting cloud storage buckets so that they can be accessed like any other local filesystem. To do this, first install the FUSE package for your operating system. Then, create a directory that will serve as the mount point for the bucket. For example, if you want to mount a bucket named “mybucket” on your local machine, you would create a directory named “mybucket” in your home directory. Next, use the fuse-bucket tool to mount the bucket. To use FUSE with Cloud Storage, you first need to install the FUSE library and the gcsfuse tool. Once these are installed, you can use the gcsfuse command to mount a bucket. For example, the following command will mount a bucket named mybucket.

Storage classes

GCP has different classes of storage:

Standard
Nearline
Coldline
Archive

Different storage classes in Google Cloud Storage offer different benefits for different workloads. The most basic storage class, Standard, is great for storing data that is accessed frequently. The next class, Nearline, is ideal for data that is accessed less frequently, but still needs to be accessed quickly. The last class, Coldline, is perfect for data that is infrequently accessed and can tolerate higher retrieval costs. By understanding the different workloads and access patterns, users can select the most appropriate storage class for their needs and optimize their Google Cloud Storage experience.

The Standard storage class is designed for frequently accessed data. Data stored in the Standard storage class is charged based on how much you store.

Nearline storage is a type of cloud storage that is similar to online storage but with lower availability and higher latency. Nearline storage is typically used for data that is not accessed more often than once every 30 days but needs to be stored for long-term retention. Costs are calculated based on how often you access the data and how much you store.

Coldline is a class of storage that was announced by Google in October 2016. It is designed for data that doesn’t need to be frequently accessed, such as historical logs or data archival. The storage itself is designed for files accessed less than once per year. It has a higher retrieval cost than nearline.

Archive storage is the lowest cost storage option in Google Cloud with the highest retrieval costs. It is specifically for data that you don’t need to access more than once a year, such as historical data, backup files, or log files. This is great for compliance storage of files that never need to be accessed.

Feature Summary

Feature	Standard	Nearline	Coldline	Archive
Multiregion SLA	99.95%	99.9%	99.9%	99.9%
Region SLA	99.9%	99.0%	99.0%	99.0%
Latency	millisecond access	millisecond access	millisecond access	millisecond access
Frequency	Often	1x30 days	1x90 days	1x1 year
Capabilities	Video, Multimedia, Business Continuity, Transcoding, Data analytics, General Compute	Backup, Long-tail content, Rarely accessed docs	Archive, Source File Escrow, Disaster Recovery Testing	Compliance Retention, Disaster Recovery

Costs Summary

Cost	Standard	Nearline	Coldline	Archive
Size	$0.020/GB	$0.010/GB	$0.004/GB	$0.0012/GB
Retrieval	$0.00/GB	$0.01/GB	$0.02/GB	$0.05/GB

Example use-cases for Google Cloud Storage:

Hosting website static assets (images, JS, CSS)
Distributed backup and disaster recovery
Storing data for analytics and Big Data processing
Storing data for internet of things devices
Storing data for mobile apps
Storing data for gaming applications
Storing data for video and audio streaming
Collaboration and sharing of files non-persistent attached storage
Security and compliance data
Geospacial data storage
In combination with Cloud Functions

These examples leverage both the storage classes and the atomic treatment of the objects themselves. Architects must understand the differences between these storage classes.

Network Attached Storage

Network Attached Storage (NAS) is a type of storage that allows files to be accessed over a network. NAS devices typically connect to a network using Ethernet and can be used by any computer on the network.

Google Cloud Filestore is a NAS service that provides high performance, scalable file storage for applications running on Google Cloud Platform. Cloud Filestore is built on top of Google Cloud Storage and offers the same benefits as other Cloud Storage products, such as high availability, durability, and security.

Cloud Filestore is a good choice for applications that require low latency access to files, such as video editing, media streaming, and scientific computing. Cloud Filestore is also a good choice for applications that require high throughput.

Google Cloud Filestore is a high-performance, managed file storage service for applications that require a file system interface and a shared filesystem. It supports industry-standard file system protocols such as NFSv3 and SMB. Google Cloud Filestore is available in three storage tiers: Basic, High Scale, and Enterprise.

Basic HDD, Good
Basic SSD, Great
High Scale SSD, Better
Enterprise, Best

Basic

The basic Filestore option strikes a good match for file sharing, software development, and use as a backend service with GKE workloads. You can opt for either hard disk drives (HDD) or solid state disks (SSD) when choosing storage, but SSDs provide higher performance at higher cost. For HDD, the I/O performance is reliant on the provisioned capacity, with peak performance increasing when the storage capacity exceeds 10 TiB. For SSD, the performance is fixed no matter the storage capacity.

High Scale

High-scale SSD storage tiers instances are ideal for performing large-scale computing tasks such as DNA sequencing and data analysis for financial services. It gives fast throughput with the ability to scale up and down with demand.

Enterprise

Enterprise tier is designed for enterprise-grade NFS workloads, critical applications (for example, SAP), and GKE workloads. It supports regional high availability and data replication over multiple zones for resilience within a region.

Service Tier	Provisionable capacity	Scalability	Performance	Availability	Data recovery	Monthly Pricing
Basic HDD	1–63.9 TiB	Up only in 1 GiB units	Standard fixed	Zonal	Backups	$204.80($0.20/GiB)
Basic SSD	2.5–63.9 TiB	Up only in 1 GiB units	Premium fixed	Zonal	Backups	$768.00($0.30/GiB)
High Scale SSD	10–100 TiB	Up or down in 2.5 TiB units	Scales with capacity	Zonal	None	$3,072.00($0.30/GiB)
Enterprise	1–10 TiB	Up or down in 256 GiB units	Scales with capacity	Regional	Snapshots	$614.40(0.60/GiB)

Filestore Networking

Cloud file store can connect to a Virtual Private Cloud (VPC) network either by using VPC Network Peering or Private Services Access. When connecting to a VPC network with standalone VPC, when creating an Instance within a Host Project of a Shared VPC, or when accessing the Filesystem from an On-Premises network, you can use VPC Network Peering. When connecting from a Service Project to a Shared VPC, or when using Centralized IP Range Management for Multiple Google Services, you need to use the Private Services Access.

Filestore Access Controls

Iam roles only grant you management access on the GCP resource but file access is managed with unix permissions in an octet format 0777, chown and chgrp.

Databases

Google cloud has several different database options. Relational, NoSQL, Analytical.

Relational Databases

Relational databases have tables with fields which can to refer to fields in other tables. An Example:

User Table:

ID	Name	Age
0	Jeff	35
8	John	35

Jobs Table:

ID	Job Title
25	Software Engineer
8	CEO
0	Director of Engineering

From the example above we can see that these two tables relate on the ID colum, they are relational. So Jeff is Director of Engineering.

Relational databases are built to support a query language and minimize problems with the data often called anomalies. In the above two tables, ID 25 doesn’t exist in the user table so the first row in the Jobs table above is a data anomaly. When fields are properly related, deleting a record in one should cascade to the others. These constraints are part of table schemas. Relational databases conform to ACID(atomicity, consistency, isolation, and durability) transaction models.

ACID Transactions

Atomicity means the whole transaction is done or none at all. A transaction is indivisible for relational databases to work.
Consistency Means when a transaction is complete, the database is constrained to a consistent state so that all foreign key reference a primary key, all unique keys are unique and the database is in an integral state.
Isolation Isolation means that parts of transactions cannot be mixed. Meaning strict grouping and ordering of transaction data in buffers.
Durability Durability means that when a transaction is complete, its change will be immediately reflected in requests for the data that was changed even if the database crashes after the completed transaction.

Cloud SQL

Cloud SQL offers MySQL server, Microsoft SQL server, or PostgresSQL via managed VMs. Google will perform upgrades and backups and let you specify maintenance times. Failovers are automatically managed and healing is an automatic process. Regional Databases are perfect for Cloud SQL. Cloud SQL supports databases up to 30 Terabytes.

All data is encrypted at rest and in transit
Data is replicated across the region to other zones
Failover to replicas is automatic
Standard tools and libraries can connect to Cloud SQL as if they’re connecting to MySQL, SQL Server, or Postgres
Logging is integrated as well as monitoring

Cloud SQL Machine Type Examples

Legacy Type	vCPUs	Memory(MB)	Machine Type
`db-f1-micro`	1	614	n/a
`db-g1-small`	1	1700	n/a
`db-n1-standard-1`	1	3840	`db-custom-1-3840`
`db-n1-standard-2`	2	7680	`db-custom-2-7680`
`db-n1-standard-4`	4	15360	`db-custom-4-15360`
`db-n1-standard-8`	8	30720	`db-custom-8-30720`
`db-n1-standard-16`	16	61440	`db-custom-16-61440`
`db-n1-standard-32`	32	122880	`db-custom-32-122880`
`db-n1-standard-64`	64	245760	`db-custom-64-245760`
`db-n1-standard-96`	96	368640	`db-custom-96-368640`
`db-n1-highmem-2`	2	13312	`db-custom-2-13312`
`db-n1-highmem-4`	4	26624	`db-custom-4-26624`
`db-n1-highmem-8`	8	53248	`db-custom-8-53248`
`db-n1-highmem-16`	16	106496	`db-custom-16-106496`
`db-n1-highmem-32`	32	212992	`db-custom-32-212992`
`db-n1-highmem-64`	64	425984	`db-custom-64-425984`
`db-n1-highmem-96`	96	638976	`db-custom-96-638976`

Shared core types db-f1-micro and db-g1-small are not covered by Google’s Cloud SQL SLA.

By default a Cloud SQL instance is a single machine in a single zone, but high availability options for provisioning additional failover and read replicas in additional zones exist. Additionally you can add read replicas in different regions. This is one way to migrate data between regions and to do disaster recovery testing. Failover replica’s are automatically promoted from read replicas to master in the case of failure.

GCP’s Database Migration Service is designed for MySQL and PostgresSQL workloads and will continuously replicate data from on-premises or other clouds. It performs and initial snapshot of the database and then leverages the native replication features of your database to continually migrate the data. You can also perform lift and shift migrations with this tool in addition to continuous. Cloud SQL scales well only vertically and not well horizontally. More memory and CPU power is needed for bigger workloads, they aren’t sharded across several devices in a workload agnostic manner.

Cloud Spanner

Cloud Spanner is a globally consistent and distributed database that provides the highest level of horizontal scalability as any relational database on the biggest network of its kind. It is fully managed and scales to multiple regions. Spanner supports relational schemas and 2011 ANSI SQL as well as Postgres dialects. Supporting instance consistency rather than “eventual consistency” as is the case with Cloud SQL read replicas, so the risk of data anomalies that eventual models produce are reduced.

Example Use Cases:

Stock trading systems that want to enable global purchasing of a security at a current price at a known time of day.
Shipping companies who need a consistent view of their global distribution network, the status of packages and the sending of global notifications.
Global inventory for a company like Sony Playstation.

Spanner provides a 5 9s availability which means less than 5 minutes of downtime per year. Its fully managed and like other managed database services in GCP its upgraded, backed up, and failover is managed. Data is also encrypted and at rest and in transit.

Analytical Databases

Analytical databases are usually data warehouses. We’ve described some data lake and data warehouse options in Google Cloud’s Hadoop and Spark offering. Though they’re used or ETL, Hadoop data lakes can be used as the data from which analytical systems can draw their data.

BigQuery(BQ)

Hadoop cannot do analytics but BigQuery is able to provide insights and is an analytics solution. Its queries scan large amounts of data and can perform data aggregation. BigQuery uses SQL and is serverless, managed and scales automatically.

BQ Analytics

Big Query is built upon Dremel, Colossus, Borg and Jupiter. Dremel maps queries to extraction trees with leaves called slots. Slots read information from storage and do a bit of processing on the data. Branches on the tree aggregate the data. Colossus is distributed filesystem by Google that offers encryption and replication. Borg is a request router that can handle rerouting during node failure. Jupiter is a petabyte per second network built by Google with rack aware placement which improves fault tolerance and throughput and requires less replication.

While other databases group rows together, in BigQuery, the data in the same column are stored together in an columnar structure called Capacitor. Capacitor supports nested fields and is used because the analytics and business intelligence filtering only happens on a small number of columns compared to a traditional application’s filtering of a number of columns in a row.

BigQuery has batch and streaming jobs to load the data, jobs can export the data, run queries or copy data. Projects contain objects called a dataset that are regional or multi-regional. Regional is straight forward, is what it sounds like. But with multi-regional you either choose the United States or Europe and Google copies the dataset into multiple regions within the continent you’ve chosen.

BigQuery bills on size stored as well as the query size and the data scanned when running the query. For this reason it is advisable to partition your query to specifically the time when the data occurred. Use less broad queries for smaller ones and less data scanned while running the query. You can read more about BigQuery Pricing. For this reason, don’t use queries to view the structure of the tables use bq head or use the Preview Option on the console. You can also use --dry-run to test command line queries which will tell you the number of bytes the query would have returned. You’re not billed for errors or queries whose results are returned from cache.

BQ IAM Roles

Access permissions in all of GCP’s products are granted by IAM, which generally has predefined roles for its products. The roles in IAM for BigQuery are:

roles/bigquery.dataViewer can list projects, tables, and access table data.
roles/bigquery.dataEditor has the permissions of dataViewer and can create and change tables and datasets.
roles/bigquery.dataOwner has dataEditor and can delete tables and datasets.
roles/bigquery.metadataViewer can list tables, datasets and projects.
roles/bigquery.user can list projects, tables has metadataViewer, and can create jobs and datasets.
roles/bigquery.jobUser Can list projects, create queries and jobs.
roles/bigquery.admin Can do any BigQuery operation.

In addition to these overarching roles, granular access can be given to Google service accounts, Google groups, etc over organizations, projects, datasets, tables and table views.

Loading Data into BQ

You can batch load or stream load data into BigQuery.

Batch Loading

Through ETL and ELT processes, data is typically batch loaded into a data warehouse through combining some sort of extraction, loading and transformation. Jobs which load the data into BigQuery can use files as objects in Cloud Storage, files on your local filesystem. Files can be Avro, CSV, ORC, and Parquet formats.

The Data Transfer Service in BigQuery loads the data from other services such as Youtube, Google Ads and Google Ad Manager, Google’s SaaS products and third-party sources. The Storage Write API is used to load data in a batch and process the records in one shot atomically, meaning the whole thing goes in or none of it does. Big Query can load data from Cloud Datastore and Cloud Firestore.

Stream Loading

To stream data into BigQuery you can use the Storage Write API or Cloud Dataflow which uses a runner in Apache Beam to write the data directly to BigQuery tables from a job in Cloud Dataflow. The Storage Write API will ingest the data with high throughput and ingest each record only once.

NoSQL Databases

GCP has four NoSQL databases: BigTable, Datastore, Cloud Firestore and Redis via Cloud MemoryStore(especially with RDB snapshotting).

Cloud Bigtable

Bigtable is a wide column multidimensional database that supports petabyte size databases for analytics, operational use, and time series data for Internet of Things(IoT) sensors. It’s ability to handle time series data well means it is good for marketing, advertisement, financial data and graphs.

Bigtable supports latencies lower than 10ms, Stores at the Petabyte scale, replicates into multiple regions, supports Hadoop HBase interfacing, data is stored in the Colossus filesystem, and metadata is stored in the cluster directly.

Data is stored in tables with key to value maps and each row stores information about the entry which is indexed by a row-key. Columns are grouped into column families like collections and a table can container multiple column families.

Tables are sectioned into blocks of contiguous rows called tablets. These tablets are stored in Colossus. Hotspots occur when you make the row key associated with a workload. For instance, if you make the row key the user ID, the heavier use users will all write to one tablet server. Design the workloads so that they’re as distributed as possible, and if hotspots still do occur you can limit or throttle the keys that cause the problem. Find out more about Bigtable hotspots.

Bigtable has support for the HBase API, so one can migrate from Hadoop HBase to Bigtable. Bigtable is the best option for migrating Cassandra databases to Google Cloud. One can create Bigtable as a multi-cluster and multi-regional and Google will take care of replicating the data. Multi-cluster systems can have their workloads separated, one being the read cluster and the other being assigned a write workload. The cluster replication procedures will assure that both cluster reach “eventual consistency”.

Cloud Datastore

Datastore is a fully managed, autoscaled, flexible structure NoSQL database for storing json objects called documents. It is superseded by Cloud Firestore. Datastore doesn’t have tables, it has what is known as a ‘kind’. Kinds contain entities. Datastore does have relational column called a property and it has a key instead of a primary key.

Cloud Firestore

This is the next product iteration of Cloud Datastore. Firestore is consistent, has two data models(collections and documents). Firestore operates under Datastore mode or Firestore mode, for supporting the latest document database features. Firestore is strongly consistent in either mode. Firestore in Datastore mode is strongly consistent where as Cloud Datastore is eventually consistent. Firestore offers millions of writes a second and the fully featured mode can handle millions of connections.

Memorystore

Managed as other products are, Memory store comes in two forms, Redis and Memcached. You can use memory caches for message processing, database caching, session sharing, etc. Memory caches are generally nonpersistent, but Redis can be configured to snapshot to dir and start again with that same data.

Memorystore for Redis

Redis is a memory datastore designed to return information with sub-millisecond latency. You can store many data types in Redis. Instance memory ceilings top out at 300GB with 12 Gigabit networking. Caches can be replicated across zones for 3 nines availability. As a managed service, Google handles updates, upgrades, syncing and failing over to other instances.

Memorystore for Redis comes in two tiers:

Basic
Standard

Basic is a single server with no replication, Standard is a multi-zonal replication and failover model.

Memorystore for Memcached

Memcached is an opensource cache which was first written for LiveJournal to perform query results caching, session caching, and data caching. Memcached nodes within a cluster called an ‘instance’ must all have the same cpu and memory geometry. So the same amount of resources on each node. Instances can have 20 nodes max, nodes can utilize a max of 32vCPUs and 256GB of memory, with a total cluster memory size of 5TB. This integrated service can be accessed from other services.

Data retention & Lifecycle Management

Data has lifecycles, is fresh, becomes inactive over time, must be archived or pruned. Different types of data have different stages, not only that they can be differently required. As an architect, you must track and handle these data lifecycles for a project or migration.

Storage requirements often impact how policies can be implemented. That is why intimate knowledge of various storage attributes is required of Cloud Architects.

Considering these things is a matter of knowing all your data and the types of data. From there you can record how quickly data must available to be accessed. Then knowing the frequency of access for each type helps define your retention planning and your planning the management of its lifecycles.

Frequency	Solution
Sub-millisecond	Cloud Memorystore, Bigtable, Firestore
frequently	Cloud Storage, Database, NoSQl, Document database
infrequently	Cloud Storage Coldline
not accessed archived	Cloud Storage Archive
not accessed	Prune

In Cloud Storage, one can create triggers that run based on the age of an object or file, the versions of that file, the object’s storage class, actions can include deleting, manipulating or changing the storage class of the object. So when objects are old and not accessed, they can be migrated to different classes. Retention policies can be created and when their specifications are not yet satisfied, they are locked into place guaranteeing their retention under the conditions specified in the policy.

Network and Latency

Latency is a big consideration in overall cloud design. There are decisions you can make that impact latency without knowing their consequences if you are unfamiliar with the particulars of different storage cloud products. Reducing latency is as simple as:

Replicating data into regions across customer locations
Distributing data over a CDN
Using the Premium Network Tier
Using services like Firestore or Spanner which are already global

Summary

GCP has Relational, Analytical, and Unstructured Databases. There are four kinds of cloud storage systems:

Cloud Storage for objects
NAS via Cloud Filestore
Databases
Memory Caches

GCP Relational Databases:

Cloud SQL: Eventual Consistency
Cloud Spanner: High Consistency

GCP Analytical Databases:

BigQuery: Columnar

NoSQL Databases:

Bigtable
Datastore
Firestore

Exam Essentials

Understand all the Storage Systems in GCP
Understand: Standard, Nearline, Coldline, Archive classes in Cloud Storage
Understand: Cloud Filestore NAS features, accessing from Compute
Know how to deploy Cloud SQL as a single server or with replication
Understand horizontal scalability in GCP Storage options
Be familiar with BigQuery as a data warehouse
Be familiar with BigTables Petabyte Scale Options and Operations
Be familiar with migrating data to GCP
Understand GCP’s JSON Document stores
Understand Caching services
Understand data retention and lifecycle management
Understand how to consider latency when designing storage for GCP

Blog

Software Development Lifecycle(SDLC)

Planning

Requirements Analysis

Cost Benefit Analysis

Design

Develop, Test, Implement

Documentation

Maintenance

Continuous Integration / Continuous Delivery(CICD)

Business Use

VCS

Secrets Management

Pipelines

Fixing Incident Culture

Enterprise IT Processes

Business Continuity & Disaster Recovery

Summary

Exam Essentials

Official Resources

Cloud Operations Suite

Cloud Monitoring

Dashboards

Cloud Monitoring Alerts

Cloud Logging

FOSS Monitoring

Continuous Delivery

Deployment Methodologies

Continuous Integration

Reliability Engineering

Overload

Cascading Failures

Reliability Testing

Incident Management and Post-mortem sessions

Summary

Exam Essentials

Official Resources

Identity and Access Management

Lease Privilege Principles

Service Accounts

Policy management

Identity Aware Proxies(IAP)

Workload Identity

Organization Policy Constraints

Data Security and Encryption

Security Observability

Security Principles

Separation of Duties

Least Privilege

Defense in Depth

Regulations

HIPPA/HITECH

GDPR

SOX

Childrens’ Online Privacy Protection Act(COPPA)

ITIL Framework

Summary

Exam Essentials

Official Resources

OSI Model Layers

OSI Model as an Out of Town Drive

IP Addresses, FW Rules, and Routing

VPCs, Sharing, Subnets, and Peering

Hybrid-Cloud Networking

Load Balancing

Additional Networking Services in GCP

Summary

Exam Essentials

Official Resources

Object Storage

Organization

Storage classes

Feature Summary

Costs Summary

Network Attached Storage

Basic

High Scale

Enterprise

Filestore Networking

Filestore Access Controls