メインコンテンツまでスキップ
Sumo Logic Japanese

Collect Logs for Google App Engine (Google App Engine のログの収集)

Google App Engine アプリケーションをインストールする前に、ログ収集を設定しておく必要があります。

このページでは、Google Cloud Platform (GCP) サービスからログを取り込むための Sumo パイプラインについて説明し、Google App Engine からログを収集する手順を紹介します。

Collection process for GCP services

The key components in the collection process for GCP services are:  Google Stackdriver, Google Cloud Pub/Sub, and Sumo’s Google Cloud Platform (GCP) source running on a hosted collector. 

The integration works like this: Google Stackdriver collects logs from GCP services. Once you’ve configured the pipeline shown below, the logs collected by Stackdriver will be published to a Google Pub/Sub topic. A Sumo GCP source on a hosted collector subscribed to that topic ingests the logs into Sumo.

monitor-gcp-services.png 

The configuration process is as follows. 

  1. Configure a GCP source on a hosted collector. You'll obtain the HTTP URL for the source, and then use Google Cloud Console to register the URL as a validated domain.  
  2. Create a topic in Google Pub/Sub and subscribe the GCP source URL to that topic.
  3. Create an export of GCP logs from Stackdriver. Exporting involves writing a filter that selects the log entries you want to export, and choosing a Pub/Sub as the destination. The filter and destination are held in an object called a sink. 

See the sections below for instructions.

Stackdriver からの App Engine ログのエクスポートの作成

このステップでは、App Engine ログを Stackdriver から前のステップで作成した Pub/Sub トピックにエクスポートします。

  1. GCP コンソールの左側のペインの [STACKDRIVER] セクションで [Logging (ログ)] をクリックします。
    gcp6.png
  2. [Exports (エクスポート)] に移動します。[Create Export (エクスポートの作成)] をクリックします。
    gcp7.png
  3. 中央のペインで、サービスとして"GAE Application" を選択してログを絞り込みます。 
    gcp8.png

  4. 右側の [Edit Export (エクスポートの編集)] ペインで次の操作を実行します。

    1. [Sink Name (Sink 名)] を設定します。たとえば、"gce-applications" です。
    2. [Sink Service (Sink サービス)] を「Cloud Pub/Sub」に設定します。
    3. [Sink Destination (Sink ターゲット)] を Pub/Sub トピックに設定します。たとえば、"pub-sub-logs" です。
    4. [Create Sink (Sink の作成)] をクリックします。

ログ メッセージのサンプル

{ "message": { "data": { "httpRequest": { "status": 200 }, "insertId": "5a0e593e000d3494692375e0", "labels": { "clone_id": "00c61b117c27a4ec1f60970fdd9db838ca7ce717bfea65f950122a708e22eff25699065ba561" }, "logName": "projects/bmlabs-loggen/logs/appengine.googleapis.com%2Frequest_log", "operation": { "first": true, "id": "5a0e593e00ff07071f75c6d2240001737e776b2d6465760001626967736b7966343a363932382d30333566343662000100", "last": true, "producer": "appengine.googleapis.com/request_id" }, "protoPayload": { "@type": "type.googleapis.com/google.appengine.logging.v1.RequestLog", "appEngineRelease": "1.9.54", "appId": "s~bmlabs-loggen", "cost": 9.957699999999999e-8, "endTime": "2018-01-26T11:35:11.374UTC", "finished": true, "first": true, "host": "bigskyf4.bmlabs-loggen.appspot.com", "httpVersion": "HTTP/1.1", "instanceId": "00c61b117c27a4ec1f60970fdd9db838ca7ce717bfea65f950122a708e22eff25699065ba561", "instanceIndex": -1, "ip": "0.1.0.2", "latency": "0.404415s", "line": [ { "logMessage": "Setting in-memory RuntimeSettings:6928-035f46b.", "severity": "DEBUG", "sourceLocation": { "file": "/base/data/home/apps/s~bmlabs-loggen/bigskyf4:6928-035f46b.405569666796645761/base/runtime_settings.py", "functionName": "_get_data", "line": "180" }, "time": "2018-01-26T11:35:11.374UTC" }, { "logMessage": "TASK-INFO: {\"gae_latency_seconds\": 0.16422700881958008, \"task_eta\": 1510889790.3144629, \"execution_count\": 0, \"ran\": 1510889790.4786899, \"expected_eta\": 1510889790.3136301, \"retry_count\": 0, \"latency_seconds\": 0.16505980491638184}", "severity": "DEBUG", "sourceLocation": { "file": "/base/data/home/apps/s~bmlabs-loggen/bigskyf4:6928-035f46b.405569666796645761/base/deferred/deferred.py", "functionName": "_log_task_info", "line": "752" }, "time": "2018-01-26T11:35:11.374UTC" }, { "logMessage": "Deferred 8a3a365c-4ed3-40ae-8a1c-faff06a9464c running function audit.actions._write with args = () and kwargs = {'account': <authmodel.auth.Account object at 0xf46cae50>, 'entity_keys': [datastore_types.Key.from_path(u'Membership', 4674727938359296L, _app=u's~bmlabs-loggen')], 'timestamp': datetime.datetime(2017, 11, 17, 3, 36, 30, 307040), 'audit_type': 'account_deactivated', 'user': <authmodel.auth.WFUser object at 0xf41fd730>, 'message': u'Service Docs exited the service_docs account.', 'ip_address': '52.4.205.181'}.", "severity": "INFO", "sourceLocation": { "file": "/base/data/home/apps/s~bmlabs-loggen/bigskyf4:6928-035f46b.405569666796645761/base/deferred/deferred.py", "functionName": "_run_function", "line": "574" }, "time": "2018-01-26T11:35:11.374UTC" }, { "logMessage": "TASK-DONE: {\"queue_name\": \"audit-log\", \"worker_id\": \"8a3a365c-4ed3-40ae-8a1c-faff06a9464c\", \"task_name\": \"27563511594509108661\"}", "severity": "INFO", "sourceLocation": { "file": "/base/data/home/apps/s~bmlabs-loggen/bigskyf4:6928-035f46b.405569666796645761/base/deferred/deferred.py", "functionName": "_run_deferred", "line": "921" }, "time": "2018-01-26T11:35:11.374UTC" }, { "logMessage": "gstats: ['event'] messages are disabled by default.", "severity": "DEBUG", "sourceLocation": { "file": "/base/data/home/apps/s~bmlabs-loggen/bigskyf4:6928-035f46b.405569666796645761/external_libs/gstats/client.py", "functionName": "__init__", "line": "424" }, "time": "2018-01-26T11:35:11.374UTC" }, { "logMessage": "gstats: statsd sent 16 metric keys with 18 metric values in 2 udp packets in [17, 15] ms", "severity": "DEBUG", "sourceLocation": { "file": "/base/data/home/apps/s~bmlabs-loggen/bigskyf4:6928-035f46b.405569666796645761/external_libs/gstats/listeners/statsd.py", "functionName": "send", "line": "362" }, "time": "2018-01-26T11:35:11.374UTC" }, { "logMessage": "GSTATSDATA: eJytlU1v2zAMhv9KofPqWHJkOQF6G7Dr2u60oRAEm0uM+iOTqLRBkP8+Sh3W2d3QOPPNskTx4cvX9JFh3YJ2YGtwbH11ZHZXauzRNLrsfYfx3d40Pm5/O8bzDk27oyUTKVfXnF9z9SXN1lm+ztKk4ELl6Vf24eoljs6p08OJlmjc4w8PHhILaA+XJ5CKDxOkpwdatYCmMmjilTuDW3rofNOcQnJ4BiqM7o67vqsx5GOt+33PhAKFGubPlvItQSz1T4Sw47C3oPeZ/uxRV94arPtuBqSlSIdIXL2DRHtjjV4bFNTyAe3yJuViepNaaEtTbkF/glnVyfhyyCIo23mRMhvpem6kHOec3o+BYe58d+uBPps5XSNHrhF/adLZptkY0I1B6MoDTZSy7yo3ZLwEMc/FyEYJQReF4lLKTBRZvuITjHU/s7EKNTbWfzX5IzSAMGuLi5F+fDUNMfb43TxSqF9KlE0NcWKwDYWgu7lJk1WiIgJY91IVC+9S9i8OZnxVo276DXvFYQtttot4ZFHBd7AWqkU8mJgyqOUS/WRrhBCz7V1kcPrpUVewZ7GMwy7ejqRyG35BkTcMOAKLAIHYOxp6FSQ0US+Ze3z89yN7koo/AQ4fOc0=", "severity": "DEBUG", "sourceLocation": { "file": "/base/data/home/apps/s~bmlabs-loggen/bigskyf4:6928-035f46b.405569666796645761/external_libs/gstats/listeners/log.py", "functionName": "send", "line": "190" }, "time": "2018-01-26T11:35:11.374UTC" }, { "logMessage": "gstats: .send() took 36 ms", "severity": "DEBUG", "sourceLocation": { "file": "/base/data/home/apps/s~bmlabs-loggen/bigskyf4:6928-035f46b.405569666796645761/external_libs/gstats/client.py", "functionName": "send", "line": "678" }, "time": "2018-01-26T11:35:11.374UTC" }, { "logMessage": "PROFILE: eJyt0U9rwjAYBvCvUnL2YFpj586C180eZZSQPmKxaSB5WxTpd1+6DdY2sFXx9v6BXx7e3BguUDmVGuw1Sl7Wi4iZFvYEWfjB0rdKVpXz9eHGisZKKk3tu/hn40u2A2VXR9AZSXLMbxxsW6qeZO5r0w/N8ehAvdotoqdhyT/YRNDQSqoTRkY6Nfj9Bv8zSDYTEUGSVAyYt2bKFJKkI2ORt8mI2sRBnuVA2jf1ewN7ncvFafBrfDPwtqhAmK2F6cT9J0/Cm3MxOnpdTBwLbQi5M+r8/cYvJlYBtnoYS3n30X0CIfoHPA==", "severity": "INFO", "sourceLocation": { "file": "/base/data/home/apps/s~bmlabs-loggen/bigskyf4:6928-035f46b.405569666796645761/external_libs/appstats_logger/middleware.py", "functionName": "_stop_recording", "line": "119" }, "time": "2018-01-26T11:35:11.374UTC" }, { "logMessage": "appstats: http://bigskyf4.bmlabs-loggen.appspot.com/stats/details?time=1510889790466", "severity": "INFO", "sourceLocation": { "file": "/base/data/home/apps/s~bmlabs-loggen/bigskyf4:6928-035f46b.405569666796645761/lib/appstats/recording.py", "functionName": "save", "line": "739" }, "time": "2018-01-26T11:35:11.374UTC" } ], "megaCycles": "196", "method": "POST", "moduleId": "bigskyf4", "referrer": "https://bmlabs-loggen.wdesk.org/auth/logout/", "requestId": "5a0e593e00ff07071f75c6d2240001737e776b2d6465760001626967736b7966343a363932382d30333566343662000100", "resource": "/_ah/queue/deferred/audit.actions._write", "responseSize": "117", "startTime": "2018-01-26T11:35:11.374UTC", "status": 200, "taskName": "27563511594509108661", "taskQueueName": "audit-log", "traceId": "a736a2573bd0d51cce823884cc2b3460", "urlMapEntry": "main.app", "userAgent": "AppEngine-Google; (+http://code.google.com/appengine)", "versionId": "6928-035f46b" }, "receiveTimestamp": "2018-01-26T11:35:11.374UTC", "resource": { "labels": { "module_id": "bigskyf4", "project_id": "bmlabs-loggen", "version_id": "6928-035f46b", "zone": "us14" }, "type": "gae_app" }, "severity": "INFO", "timestamp": "2018-01-26T11:35:11.374UTC" }, "attributes": { "logging.googleapis.com/timestamp": "2018-01-26T11:35:11.374UTC" }, "message_id": "172694189887110", "messageId": "172694189887110", "publish_time": "2018-01-26T11:35:11.374UTC", "publishTime": "2018-01-26T11:35:11.374UTC" }, "subscription": "projects/bmlabs-loggen/subscriptions/sumo-test" }

クエリのサンプル

一定期間にわたるステータス コード

_collector="HTTP Source for GCP Pub/Sub" logName resource timestamp | json "message.data.resource.type" as type | parse regex "\"logName\":\"(?<log_name>[^\"]+)\"" | where type = "gae_app" | where log_name matches "projects/*/logs/appengine.googleapis.com%2Frequest_log" | json "message.data.resource.labels" as labels | json field=labels "module_id", "version_id", "zone", "project_id" as service, version, zone, project | json "message.data.protoPayload.appId", "message.data.protoPayload.status" as app_id, status_code | timeslice 30m | if(status_code matches "20*", 1, 0) as resp_200 | if(status_code matches "30*", 1, 0) as resp_300 | if(status_code matches "40*", 1, 0) as resp_400 | if(status_code matches "50*", 1, 0) as resp_500 | if(!(status_code matches "20*" or status_code matches "30*" or status_code matches "40*" or status_code matches "50*"), 1, 0) as resp_others | sum(resp_200) as tot_200, sum(resp_300) as tot_300, sum(resp_400) as tot_400, sum(resp_500) as tot_500, sum(resp_others) as tot_others by _timeslice
  • この記事は役に立ちましたか?