Google Cloud VPC のログの収集

最後の更新
PDFとして保存

Google Cloud VPC からのログ収集の手順。

このページでは、Google Cloud Platform (GCP) サービスからログを取り込むための Sumo パイプラインについて説明し、Google Cloud VPC からログを収集する手順を紹介します。

Collection process for GCP services

The key components in the collection process for GCP services are: Google Stackdriver, Google Cloud Pub/Sub, and Sumo’s Google Cloud Platform (GCP) source running on a hosted collector.

The integration works like this: Google Stackdriver collects logs from GCP services. Once you’ve configured the pipeline shown below, the logs collected by Stackdriver will be published to a Google Pub/Sub topic. A Sumo GCP source on a hosted collector subscribed to that topic ingests the logs into Sumo.

The configuration process is as follows.

Configure a GCP source on a hosted collector. You'll obtain the HTTP URL for the source, and then use Google Cloud Console to register the URL as a validated domain.
Create a topic in Google Pub/Sub and subscribe the GCP source URL to that topic.
Create an export of GCP logs from Stackdriver. Exporting involves writing a filter that selects the log entries you want to export, and choosing a Pub/Sub as the destination. The filter and destination are held in an object called a sink.

See the sections below for instructions.

Stackdriver からの Google Cloud VPC ログのエクスポートの作成

GCP で [Logging (ログ)] に移動します。
[Exports (エクスポート)] に移動します。[Create Export (エクスポートの作成)] をクリックします。
GCP サービスとして [GCE Subnetwork (GCE サブネットワーク)] を選択してログを絞り込みます。
右側の [Edit Export (エクスポートの編集)] ウィンドウで次の操作を実行します。
1. [Sink Name (Sink 名)] を設定します。たとえば、"gcp-subnetwork" です。
2. [Sink Service (Sink サービス)] を "Cloud Pub/Sub" に設定します。
3. [Sink Destination (Sink ターゲット)] を作成したばかりの Pub/Sub トピックに設定します。たとえば、pub-sub-logs です。
4. [Create Sink (Sink の作成)] をクリックします。

ログメッセージのサンプル

{
  "message": {
    "data": {
      "insertId": "h7cue3dc1fr",
      "jsonPayload": {
        "bytes_sent": "1836",
        "connection": {
          "dest_ip": "34.192.224.100",
          "dest_port": 443,
          "protocol": 6,
          "src_ip": "10.128.0.3",
          "src_port": 56552
        },
        "dest_location": {
          "city": "Ashburn",
          "continent": "America",
          "country": "usa",
          "region": "Virginia"
        },
        "end_time": "2018-01-26T12:35:10.115UTC",
        "packets_sent": "20",
        "reporter": "SRC",
        "rtt_msec": "49",
        "src_instance": {
          "project_id": "bmlabs-loggen",
          "region": "us-central1",
          "vm_name": "vm-selectstar-collector-again",
          "zone": "us-central1-c"
        },
        "src_vpc": {
          "project_id": "bmlabs-loggen",
          "subnetwork_name": "default",
          "vpc_name": "default"
        },
        "start_time": "2018-01-26T12:35:10.115UTC"
      },
      "logName": "projects/bmlabs-loggen/logs/compute.googleapis.com%2Fvpc_flows",
      "receiveTimestamp": "2018-01-26T12:35:10.115UTC",
      "resource": {
        "labels": {
          "location": "us-central1-c",
          "project_id": "bmlabs-loggen",
          "subnetwork_id": "3656133720937113003",
          "subnetwork_name": "default"
        },
        "type": "gce_subnetwork"
      },
      "timestamp": "2018-01-26T12:35:10.115UTC"
    },
    "attributes": {
      "logging.googleapis.com/timestamp": "2018-01-26T12:35:10.115UTC"
    },
    "message_id": "172581793992900",
    "messageId": "172581793992900",
    "publish_time": "2018-01-26T12:35:10.115UTC",
    "publishTime": "2018-01-26T12:35:10.115UTC"
  },
  "subscription": "projects/bmlabs-loggen/subscriptions/push-to-sumo"
}

クエリの例

サブネット ID 別の平均レイテンシ (ms)

_collector="HTTP Source for GCP Pub/Sub" logName resource timestamp
| json "message.data.resource.type" as type 
| parse regex "\"logName\":\"(?<log_name>[^\"]+)\"" 
| where type = "gce_subnetwork" | where log_name matches "projects/*/logs/compute.googleapis.com%2Fvpc_flows"
| json "message.data.resource" as resource | json field=resource "labels.location","labels.project_id","labels.subnetwork_id","labels.subnetwork_name" as zone,project,subnetwork_id,subnetwork_name nodrop
| json "message.data.resource.labels", "message.data.jsonPayload" as labels, payload
| json field=payload "src_instance","dest_instance" as src_instance,dest_instance nodrop 
| json field=payload "src_vpc.vpc_name","dest_vpc.vpc_name" as src_vpc,dest_vpc nodrop
| json field=payload "connection.src_ip","connection.dest_ip","connection.dest_port","connection.src_port" as src_ip,dest_ip,dest_port,src_port 
| json field=src_instance "project_id", "zone", "region", "vm_name" as src_project, src_zone, src_region, src_vm nodrop 
| json field=dest_instance "project_id", "zone", "region", "vm_name" as dest_project, dest_zone, dest_region, dest_vm nodrop
| json field=payload "bytes_sent","rtt_msec","packets_sent"  as bytes, rtt,packets  
| timeslice 1m
| avg(rtt) as latency by _timeslice, subnetwork_id, subnetwork_name 
| transpose row _timeslice column subnetwork_id,subnetwork_name

Collection process for GCP services

Stackdriver からの Google Cloud VPC ログのエクスポートの作成

ログ メッセージのサンプル

クエリの例

ログメッセージのサンプル