GCP Digest Issue 7

31 August 2020
Back to homepage

Google Cloud has launched a new product called Workflows, which allows you to orchestrate and automate Google Cloud and HTTP-based API services with serverless workflows via YAML config, without the need to write glue code. Will it take off?

Other than that, Game Servers, which is a fully-managed version Agones, an open source game server management project that runs on Kubernetes, is now in GA. Furthermore, it is also highlighted that Cloud Logging is becoming more and more enterprise-ready with enhanced logs routing via log sinks. 

Also, the much-awaited MySQL 8.0 is now available in Cloud SQL.

Happy reading!
- Jonathan Lin

What's new across Google Cloud databases

  • Spanner:
    • The Spanner Emulator, now generally available, lets application developers do correctness testing when developing an application.
    • Spanner now offers new multi-region configurations for Asia and Europe with 99.999% availability
  • Bigtable:
    • Managed backups
    • Table-level identity access management (IAM) and audit logs for admin activity
    • Expanded support and SLA for single-node production instances
  • Firestore:
    • Richer query language
    • C++ client library
    • Firestore Unity SDK
    • Firestore Key Visualizer gives you better visibility into usage patterns and performance (coming soon)
  • Cloud SQL:
    • More maintenance controls and cross-region replication features (enterprise capabilities)
    • Point-in-time-recovery for Postgres
  • Bare Metal Solution (e.g. Oracle):
    • Now available in even more regions
Check out the blog post.

Workflows (New GCP product)

This is genuinely exiting. I almost missed this because there wasn't a blog post for it. With Workflows, you can orchestrate and automate Google Cloud and HTTP-based API services with serverless workflows. Via a YAML file configuration, you can configure complex workflows in Google Cloud without writing glue code. See product page and Next OnAir video

Enhanced natural features in Google Maps

Google Maps is rolling out new visual improvements that bring even more detail and granularity to the map, making it easier to understand what an area looks like whether you’re exploring virtually or planning a visit. Essentially a Google Map map will look more like what you see if you were floating in the air, looking down at a geographic area. More in the blog post

Google Cloud Game Servers now GA

Earlier this year, Google Cloud announced the beta release of Game Servers, a managed service built on top of Agones, an open source game server scaling project. Game Servers uses Kubernetes for container orchestration and Agones for game server fleet orchestration and lifecycle management, providing developers with a modern, simpler paradigm for managing and scaling games. Game Servers is now in GA.

By simplifying infrastructure management, Game Servers empowers developers to focus their resources on building better games for their players. More in the blog post.

New GKE Dataplane V2 with eBPF

As more and more enterprises adopt Kubernetes, the gamut of use cases is widening with new requirements around multi-cloud, security, visibility, scalability, and customization. These new requirements all have something in common: they need a more programmable dataplane that can perform Kubernetes-aware packet manipulations without sacrificing performance.

The Extended Berkeley Packet Filter (eBPF) is a new Linux networking paradigm that exposes programmable hooks to the network stack inside the Linux kernel. The ability to enrich the kernel with user-space information—without jumping back and forth between user and kernel spaces—enables context-aware operations on network packets at high speeds. GKE Dataplane V2, an opinionated dataplane that harnesses the power of eBPF and Cilium, an open source project that makes the Linux kernel Kubernetes-aware using eBPF. Now in beta, GCP is also using Dataplane V2 to bring Kubernetes Network Policy logging to Google Kubernetes Engine (GKE).

Use case: Enterprises want to be able to quickly identify abnormal traffic patterns such as pods that are unexpectedly talking to the internet and denial-of-service attacks. With Kubernetes Network Policy logging, you can now see all allowed and denied network connections directly in the Cloud Logging console to troubleshoot policies and spot irregular network activity. More in the blog post.

CPU overcommit for sole-tenant nodes now GA

Sole-tenant nodes in GCP are physical Compute Engine servers dedicated solely to your workloads. With CPU overcommit for sole-tenant nodes, you can over-provision your dedicated host virtual CPU resources by up to 2X. CPU overcommit automatically reallocates virtual CPUs across your sole-tenant nodes from idle VM instances to VM instances that need additional resources. More in the blog post.

Simulate high-volume data streams with Dataflow

GCP has now made it easier for developers to generate large amounts of synthetic data with very high queries per second (QPS). To help with this, GCP has announced the launch of a new Dataflow Flex template called Streaming Data Generator that is capable of publishing unlimited high-volume JSON messages to a Google Cloud Pub/Sub topic. More in the blog post.

Better log storage and management for Cloud Logging

A selection of what’s new in Cloud Logging:
  • Logs buckets (beta) 
  • Logs views (alpha) 
  • Regionalized log storage (alpha) 
  • Customizable retention (generally available)
  • Cloud Logging Router (generally available - new functionality in beta)
  • Exploring and analyzing logs (generally available)
  • New logs viewer
  • Histograms
  • Field explorer 
  • Regular expression support 
  • Logging Dashboard
Cloud Logging Router allows you to manage all log destinations consistently using log sinks, and all log sinks can also support exclusions, making it easier to configure the logs you want to the right destination. You can also now route logs from one project to another or even use aggregated log sinks from across folders or organization level for security and ease of maintenance.

There's really a lot going on in the logging area that feels quite overwhelming to wrap my head around. Can anyone really accuse GCP of not being enterprise-friendly anymore? More in the blog post.

Multi-language Dataflow pipelines

Google Cloud has added a new, more services-based architecture called Runner v2 (available to anyone building a pipeline) to Dataflow that includes multi-language support for all of its language SDKs. This addition of what the Apache Beam community calls “multi-language pipelines” lets development teams within your organization share components written in their prefered language and weave them into a single, high-performance, distributed processing pipeline. More in the blog post.

High-resolution and extended user-defined metrics in Cloud Monitoring

Higher resolution custom, Prometheus, and agent metrics are now generally available, and can be written at a granularity of 10 seconds (compared to 60 seconds before). More in the blog post.

Also, extended metric retention for custom and Prometheus metrics in Cloud Monitoring is now GA, increasing retention from 6 weeks to 24 months. Extended retention for custom and Prometheus metrics is enabled by default. More in the blog post

MySQL 8 available in Cloud SQL

Cloud SQL now support MySQL 8. This means you get access to a variety of powerful new features—such as instant DDL statements (e.g. ADD COLUMN), atomic DDL, privilege collection using roles, window functions, and extended JSON syntax - to help you be more productive. More in the blog post

Orchestrating Data Fusion pipelines with Cloud Composer

Cloud Composer is a workflow orchestration tool based on Apache Airflow. Data Fusion is Google Cloud’s fully managed, cloud-native data integration service that is built on the open source CDAP platform. Data Fusion helps users build and manage ETL and ELT data pipelines through an intuitive graphical user interface.

Google Cloud has now announced that you can now orchestrate and manage your Data Fusion pipelines in Cloud Composer using a rich set of Cloud Data Fusion operators. More in the blog post.

Beta? GA?

The list below is best-effort and not meant to be exhaustive.

Entered GA (or new features)
  • AI Platform Deep Learning Containers: TensorFlow Enterprise 2.3 environments
  • AI Platform Training: Set a maximum time that you are willing to wait between the moment when you create a training job and the moment when AI Platform Training starts running the job
  • Artifact Registry: Use Pub/Sub to configure notifications for changes in Docker repositories
  • BigQuery ML: Matrix Factorization model support
  • Cloud Build: Cloud Build GitHub App now allows users to view triggers by name on GitHub
  • Cloud Load Balancing: Setting up Internal HTTP(S) Load Balancing in a Shared VPC service project
  • Cloud Logging: Suggested queries based on the context of your Google Cloud project
  • Cloud Trace: Supports search by the trace ID
  • Dataproc: New Personal Cluster Authentication feature
  • AI Platform Prediction: Compute Engine (N1) machine types for online prediction 
  • AI Platform Prediction: GPUs for online prediction
  • BigQuery: Explicitly assign projects to use on-demand pricing with Reservations
  • Cloud Run: Allocate up to 4GiB of memory to a service
  • Resource Manager: Organization policy for restricting protocol forwarding creation, Organization policy for restricting Cloud Load Balancing creation
  • Game Servers
  • Cloud Load Balancing: Organization policy constraint for restricting Cloud Load Balancing creation
  • Compute Engine: Organization Policy for restricting protocol forwarding creation
  • Anthos 1.4.2
  • Text-to-Speech: Four new English (US) voices, four new Chinese (Hong Kong) voices
  • Security Command Center: Audit logs
  • Managed Service for Microsoft Active Directory: Terraform support
  • Compute Engine: Compute Engine committed use discounts for SUSE Linux Enterprise Server for SAP images
  • BigQuery: Using service account credentials with scheduled queries
  • Speech-to-Text: New On-Prem API
  • Cloud SQL for MySQL: MySQL 8.0
  • Traffic Director: Advanced traffic management features with proxyless gRPC applications
  • BigQuery: INFORMATION_SCHEMA views for BigQuery reservations
  • Anthos Service Mesh: Support for joining multiple clusters from different projects into a single Anthos Service Mesh on Google Kubernetes Engine, Citadel CA support for gcp profiles
  • Pub/Sub: Subscription detachment
Entered Beta For more product updates, visit Google Cloud release notes


  • Next OnAir (Weekly for 9 weeks, starting July 14)


See you next time!

Send me GCP Digest

One newsletter, every two weeks.

We care about the protection of your data. Read our Privacy Policy.