SIEM on Amazon OpenSearch Service Workshop「4. ログ分析応用編」の環境をローカルで構築する

2023年2月3日 · 約9分

Maintainer of this blog

SIEM on Amazon OpenSearch Serviceを使う機会があったのですが、セキュリティ調査は奥が深い！！もっと勉強がしたい！！！ と思っていたところ、ワークショップを発見しました。

どうせならじっくりゆっくり学習したいので、AWS上ではなくローカル環境で構築してみました。

OpenSearch Serviceを起動

Docker Composeで起動します。こちらを参考にイメージのバージョンを2.3.0にしました。

docker-compose.yaml
version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # disables security plugin entirely in OpenSearch by setting plugins.security.disabled: true in opensearch.yml
      - "discovery.type=single-node" # disables bootstrap checks that are enabled when network.host is set to a non-loopback address
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
    networks:
      - opensearch-net

volumes:
  opensearch-data1:

networks:
  opensearch-net:

注記

シングルノードで起動しました

サンプルデータを登録

2.3. ハンズオンデータのリストアをみると、CloudFormationでデータのリストアができるようです。

Launch Stackをクリックすると、CloudFormationのスタック作成の画面に遷移します。

ここで選択されているテンプレートを見てみましょう。

siem-on-amazon-opensearch-service-workshop.template
Description: Sample log for SIEM Workshop
Resources:
  LambdaRestoreSampleLog068F67FB:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: 'aes-siem-ap-northeast-1'
        S3Key: 'siem-on-amazon-opensearch-service/v2.9.0/assets/restore_samplelog.zip'
      Role: !ImportValue 'role-deploy'
      Environment:
        Variables:
          accountid: !Ref 'AWS::AccountId'
          dashboards_url: !ImportValue 'dashboards-url'
          region: !Ref 'AWS::Region'
      FunctionName: aes-siem-restore-samplelog
      Handler: index.lambda_handler
      MemorySize: 128
      Runtime: python3.8
      Timeout: 300
  LambdaRestoreSampleLogCurrentVersion0E787C04807e3ffecc10a3f8140a46b16caf5f3e:
    Type: AWS::Lambda::Version
    Properties:
      FunctionName: !Ref 'LambdaRestoreSampleLog068F67FB'
      Description: 2.9.0
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain
  ExecAesSiemRestoreSampleLog:
    Type: AWS::CloudFormation::CustomResource
    Properties:
      ServiceToken: !GetAtt 'LambdaRestoreSampleLog068F67FB.Arn'
Parameters: {}
Rules: {}

Lambda関数ですね。関数のコードはS3に配置されていますので、取得します。

aws s3 cp s3://aes-siem-ap-northeast-1/siem-on-amazon-opensearch-service/v2.9.0/assets/restore_samplelog.zip ./

取得できました。Zipを展開してみます。

tree -L 1
.
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── bin
├── certifi
├── certifi-2022.12.7.dist-info
├── charset_normalizer
├── charset_normalizer-2.1.1.dist-info
├── idna
├── idna-3.4.dist-info
├── index.py
├── opensearch_py-2.0.1.dist-info
├── opensearchpy
├── requests
├── requests-2.28.1.dist-info
├── requirements.txt
└── siemlog.zip

リストアするデータがsiemlog.zipで用意されています。これをLambdaで投入する仕組みとなっています。

index.pyが関数ハンドラーです。localhostのOpenSearchへデータをリストアするので、少し修正します。

boto3のインポートを削除
不要な環境変数を削除
ホスト名をlocalhostに、ポートを9200に変更
SSL通信を行わないように変更

index.py
 #!/usr/bin/env python3
 # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
 # SPDX-License-Identifier: MIT-0
 __copyright__ = ('Copyright Amazon.com, Inc. or its affiliates. '
                  'All Rights Reserved.')
 __version__ = '2.9.0'
 __license__ = 'MIT-0'
 __author__ = 'Akihiro Nakajima'
 __url__ = 'https://github.com/aws-samples/siem-on-amazon-opensearch-service'
 
 import json
 import os
 import urllib.error
 import urllib.parse
 import urllib.request
 import zipfile
 
-import boto3
 from opensearchpy import AWSV4SignerAuth, OpenSearch, RequestsHttpConnection
 
-accountid = os.environ['accountid']
-region = os.environ['region']
-dashboards_url = os.environ['dashboards_url']
 
 
 def initial_event_check_and_exit(event, context, physicalResourceId):
     if event:
         print(json.dumps(event))
     if event and 'RequestType' in event and "Delete" in event['RequestType']:
         # Response For CloudFormation Custome Resource
         response = {}
         send(event, context, "SUCCESS", response, physicalResourceId)
         return(json.dumps(response))
 
 
 def send(event, context, responseStatus, responseData, physicalResourceId=None,
          noEcho=False):
     # https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/cfn-lambda-function-code-cfnresponsemodule.html
     responseUrl = event['ResponseURL']
     print(responseUrl)
 
     response_body = {}
     response_body['Status'] = responseStatus
     response_body['Reason'] = ('See the details in CloudWatch Log Stream: '
                                '' + context.log_stream_name)
     response_body['PhysicalResourceId'] = (
         physicalResourceId or context.log_stream_name)
     response_body['StackId'] = event['StackId']
     response_body['RequestId'] = event['RequestId']
     response_body['LogicalResourceId'] = event['LogicalResourceId']
     response_body['NoEcho'] = noEcho
     response_body['Data'] = responseData
 
     print('DEBUG: ' + str(response_body))
     json_response_body = json.dumps(response_body)
 
     print("Response body:\n" + json_response_body)
 
     headers = {'content-type': 'application/json', }
     req = urllib.request.Request(
         event['ResponseURL'], json_response_body.encode(),
         headers=headers, method='PUT')
     try:
         res = urllib.request.urlopen(req)
         print("Status code: " + str(res.status))
     except Exception as e:
         print("send(..) failed executing requests.put(..): " + str(e))
 
 
 def extract_backup_data():
     local_unzip_dir = '/tmp/x/'
     file_name = 'siemlog.zip'
     with zipfile.ZipFile(file_name) as existing_zip:
         existing_zip.extractall(local_unzip_dir)
 
 
 def print_results(results):
     print("Number of loaded documents: " + str(len(results['items'])))
 
 
 def restore_backup_data():
-    eshost = dashboards_url.split('/')[2]
+    eshost = 'localhost'
-    credentials = boto3.Session().get_credentials()
-    awsauth = AWSV4SignerAuth(credentials, region)
     es = OpenSearch(
-        hosts=[{'host': eshost, 'port': 443}], http_auth=awsauth, use_ssl=True,
-        http_compress=True, verify_certs=True, retry_on_timeout=True,
+        hosts=[{'host': eshost, 'port': 9200}], use_ssl=False,
+        http_compress=True, verify_certs=False, retry_on_timeout=True,
         connection_class=RequestsHttpConnection, timeout=60)
 
     samplelog_list = [
         "log-aws-guardduty-workshop", "log-aws-cloudtrail-workshop",
         "log-aws-securityhub-workshop", "log-aws-vpcflowlogs-workshop",
         "log-linux-secure-workshop"]
     size = 0
     putdata_list = []
     for samplelog in samplelog_list:
         print("index: " + samplelog)
         with open("/tmp/x/" + samplelog) as json_file:
             for line in json_file:
                 data = json.loads(line)
                 putdata_list.append(
                     {"index": {"_index": data['_index'], "_id": data['_id']}})
                 putdata_list.append(data['_source'])
                 size += len(str(data))
                 if size > 6000000:
                     results = es.bulk(putdata_list)
                     putdata_list = []
                     size = 0
                     print_results(results)
             results = es.bulk(putdata_list)
             print_results(results)
             putdata_list = []
             size = 0
 
 
 def lambda_handler(event, context):
     physicalResourceId = 'restore-workshop-samplelog'
     try:
         extract_backup_data()
         restore_backup_data()
     except Exception as e:
         print('Exception occured: ' + str(e))
         response = {"failed_reason": e}
         if event and 'RequestType' in event:
             send(event, context, "FAILED", response, physicalResourceId)
 
     response = {"result": "ok"}
     if event and 'RequestType' in event:
         # Response For CloudFormation Custome Resource
         send(event, context, "SUCCESS", response, physicalResourceId)
     return(json.dumps(response))
 
 
 if __name__ == '__main__':
     lambda_handler(None, None)

実行します。

python3 index.py

index: log-aws-guardduty-workshop
Number of loaded documents: 10
index: log-aws-cloudtrail-workshop
Number of loaded documents: 755
index: log-aws-securityhub-workshop
Number of loaded documents: 228
index: log-aws-vpcflowlogs-workshop
Number of loaded documents: 3977
Number of loaded documents: 291
index: log-linux-secure-workshop
Number of loaded documents: 4792