すずけんメモ

技術メモです

re:Invent 2018でみてきたセッション / あとでみるセッションのメモ

今年もre:Inventにいってきました。ということで参加してきたセッションやらワークショップについてのメモです。今回は1000セッションを超える規模ということもありすべては見れていないので、あとからみるためのセッションのメモも書いています。

新サービス類やイベント全般に関してはajitofmで話したのでそちらもどうぞ! ajitofm 36: Can't Wrap Up re:Invent 2018 within 1 hour

見たもの

見た順番に、記憶に残っているものを。

ARC334 - Scaling Push Messaging for Millions of Netflix Devices

Netflix built Zuul Push, a massively scalable push messaging service that handles millions of always-on, persistent connections to proactively push time-sensitive data, like personalized movie recommendations, from the AWS Cloud to devices. This helped reduce Netflix’s notification latency and the Amazon EC2 footprint by eliminating wasteful polling requests. It also powers Netflix’s integration with Amazon Alexa. Zuul push is a high-performance async WebSocket/SSE server. In this session, we cover its design and how it delivers push notifications globally across AWS Regions.  Key takeaways include how to scale to large numbers of persistent connections, differences between operating this type of service versus traditional request/response-style stateless services, and how push messaging can be used to add new, exciting features to your application.

@raksoras

Zuulの話。どのようにGatewayとしてコネクションを管理するか(古いコネクションをrecycleする)、優先度キューの扱い、GatewayをAutoscalingさせるための指標についてなど。

github.com

DVC306 - Serverless: It All Started in Vegas

This talk dives into Trustpilot's journey to serverless compute. The journey starts at re:Invent 2016 and follows how the company fast-tracked its adoption within its engineering organization using a "serverless first" engineering principle. A representative from Trustpilot shares lessons learned and insights gained from running over 200 AWS Lambda functions with 12M invocations/day in production. Also covered are fun stories of what helped the company adopt serverless, how to make those stories actionable, a review of architectural patterns, and a discussion of why they choose serverless over traditional compute every day.

@martinbuberl

Lambdaをごりごり適用してEC2を減らしていった話。2年でLambda Function数が6倍になったらしい。 GitHub HookをLambdaでさばくのはいいアイデアかもしれない。

会社の開発原則: GitHub - trustpilot/principles: Trustpilot's Engineering Principles

実際にはECSでもLambdaでもよく、EC2からワークロードを逃してきたっていうのがメインの話。

NET404-R - [REPEAT] Elastic Load Balancing: Deep Dive and Best Practices

Elastic Load Balancing (ALB & NLB) automatically distributes incoming application traffic across multiple Amazon EC2 instances for fault tolerance and load distribution. In this session, we go into detail on ELB configuration and day-to-day management. We also discuss its use with Auto Scaling, and we explain how to make decisions about the service and share best practices and useful tips for success. Finally, Netflix joins this session to share how it leveraged the authentication functionality on Application Load Balancer to help solve its workforce identity management at scale.

もろもろのupdatesを交えつつ使い方について解説するセッション。

スロースタートつかっていこうという話。 Application Load Balancer のターゲットグループ - Elastic Load Balancing

あとNetflixの Sr.Security Engineerの人がALBでのCognitoとの連携の話をしてた。 Application Load Balancer を使用してユーザーを認証する - Elastic Load Balancing

AVR 202 Sumerian Workshop

Sumerian、高度なマインクラフトといった感じで面白かった。

Amazon Sumerian – VR アプリケーションや AR アプリケーションの構築 で3DモデルつくってWebVRに書き出す、というのもやった。Sumerian自体初めて触った。

こういうのが動く。壁のスイッチを押すと、部屋の電気がついて話し始めます。: http://tinyurl.com/arv202demo

Polly, Lex, Rekogtion, Comprehend, Translate, Transcribeあたりと連携できる。なので音声認識して適当に話す、というのを各サービスと連携して動かせる。

Workshopでは小さい部屋をつくって人間を配置した。壁のスイッチで部屋の上につるしたライトのon/offをするというのも書いた。これはState Machineをスイッチの挙動につけることができ、他のコンポーネントの状態を変更する、といったことが簡単に書ける。こんな感じ Amazon Sumerian ステートマシン - Amazon Sumerian

Workshopではやらなかったけど、Scriptingもできるらしい。 AMAZON SUMERIAN SCRIPTING API

ADT201-L - Leadership Session: Digital Advertising - Customer Learning & the Road Ahead

In this session, learn how experienced leaders in digital advertising respond to the rapid evolution and sophistication of the advertising market driven by innovation and groundbreaking technology. Our customers share real-world applications they've leveraged in the cloud and how they see the media landscape changing as adoption of AI in the space becomes more widespread. Learn about existing and upcoming advancements and how they affect digital transformation in the years to come. Come away with ideas on how you can apply these learnings to your technology stack.

  • Karl Bunch - Worldwide Technical Leader, Digital Advertising
  • Dave Pickles - Chief Technology Officer & Founder, The Trade Desk
  • Alex Collmer - Founder & CEO, VidMob
  • Patrick Wentling - Communications/PR Manager, The Trade Desk


re:Invent中で数少ないAD関連のセッション。VidMobがビデオのクリエイティブをいかに分析できるようにしてるかって話が面白かった。VidMobは動画広告クリエイティブの解析ツール。どうやっているかというと、

  • RekognitionでビデオのObjectを認識。それぞれtimestampごとに属性の移り変わりを記録。
  • ユーザのインタラクションをtimestampで結合し、クリエイティブの中のどの要素がエンゲージメントにつながったのかを取得。

ということをしている。あと音もこれに加わる。

Karlさんのいう「AI as new data source」がこれを端的に表している。つまりRekognitionがつけたタグによって分析可能なデータになるということ。

CON360-R1 - [REPEAT 1] From Monolith to Microservices (And All the Bumps along the Way)

Applications built on a microservices-based architecture and packaged as containers bring several benefits to your organization. In this session, Duolingo, a popular language-learning platform and an Amazon ECS customer, describes its journey from a monolith to a microservices architecture. We highlight the hurdles you may encounter, discuss how to plan your migration to microservices, and explain how you can use Amazon ECS to manage this journey.

まあどんどんコンテナに突っ込めばいい感じに移行できるよねっていうセッション。

BAP401-R - [REPEAT] Build a Voice-Based Chatbot for Your Amazon Connect Contact Center

Learn how easy it is to incorporate a voice-based Amazon Lex chatbot into your Amazon Connect contact center. In this workshop, we walk you through configuring your own Amazon Connect contact center, implementing a chatbot, and using it in your workflows to deliver a personalized voice-based caller experience. You also have the opportunity to further personalize caller experiences by using AWS Lambda to access caller information from your customer data system. Leave this workshop with a functioning Amazon Connect contact center and a voice-enabled chatbot that you can continually modify to your business needs. Come prepared to build by bringing your laptop and a phone to make test calls.

  • Yasser El-Haggan - Principal Solutions Architect, AWS
  • Kevan Mah - Principal Product Manager, AWS

これは面白かった。コールセンターをつくるWorkshop。 Amazon Connect(簡単に使えるクラウド型コンタクトセンター)|AWS をつかう。

次のものをつくった。

  • コールセンターが立ち上がる。ネタはITサポートデスク的なもの。
  • 「パスワードリセットしたいんですけど」というとパスワードリセットできる。
  • 「パソコンつかないんですけど」というと「電源きれてるんじゃない?がんばってね」といって電話を切られる。(冷たい!)
  • 電話してきた顧客の名前をVoiceChat経由でDynamoDBに保存。(Connect -> Lambda -> DynamoDB)次に電話をかけたとき、もし名前があればその名前で返してくれる。「こんにちは鈴木さん」的な。

じゃあどうやってるかというとAmazon ConnectでもState Machine、というかここではContact Flowとよばれるものをつかう。こんな感じのGUIで応答を制御できる。

ちなみにPollyで日本語を話せるようにはしたので電話からは日本語で話してくれる。が、Lexが日本語対応してないので返答は英語でしなければいけないという特殊なコールセンターになったのでした。

AIM395 - [NEW LAUNCH!] Easily add real-time recommendations to your applications with Amazon Personalize

新しく出たAmazon PersonalizeのWorkshop。

Amazon Personalize – Real-Time Personalization and Recommendation for Everyone | AWS News Blog

プログラムを書かずにPersonalizeができる!ってことでどんなものかと見に行ったけど、Workshop自体は微妙だったので途中で抜けてドキュメントを読んだ。

HLC303 - Data Patterns and Analysis with Amazon Neptune: A Case Study in Healthcare Billing

In this session, learn how to better analyze your data for patterns and inform decisions by pairing relational databases with a number of AWS services, including the graph database service, Amazon Neptune. Additionally, hear about the use of AWS Glue and Apache Ranger for data cataloging and as a baseline for query and dataset resolution. Learn about the use of AWS Fargate and AWS Lambda for serverless provisioning of complex data and how to do data rights management at scale on an enterprise data lake. As a case study, hear how Change Healthcare is building an Intelligent Health Platform (IHP) using these services to help standardize and simplify a number of healthcare workflows, including payment processing, which have traditionally been both complex and disconnected from healthcare event data.

  • Mark Weiler - Sr. Solutions Architect, AWS
  • John DeMastri - SVP, R&D - Platforms and Analytics, Change Healthcare

Neptune活用事例。医療系のサービスのBillingをいい感じにするぞ、っていう狙い。複数のサービスをまたいで計算しなければならないタスクがあり、それをGraphでやったら便利っていう話。ただモデリングとかはでてこなかったので具体的イメージがわきづらかった。

AWS DeepRacer - The MGM SpeedWay

今回のおもしろデバイス枠(と勝手に呼んでいる)、DeepRacerのワークショップ。DeepRacerは強化学習でつくったモデルを組み込める自動運転カー(物理) + そのマネジメントサービス。

AWS DeepRacer - the fastest way to get rolling with machine learning

次のことをやった。

  • シミュレーションでコースを走らせる。これはその後物理コースで走らせるものと同じつくりになっている。
  • 徐々にチューニングしていく。報酬関数を調整する。
  • レーニングをひたすら回す。

レーニングはAWS Console上からできる。HLSで動画も転送してくれて、報酬スコアがどう変動しているかがみえるようになっている。Twitterに動画貼ったのでそちらをどうぞ。

報酬の関数はこんな感じ

def reward_function(on_track, x, y, distance_from_center, car_orientation, progress, steps, throttle, steering, track_width, waypoints, closest_waypoint):

    '''
    @on_track (boolean) :: The vehicle is off-track if the front of the vehicle is outside of the white
    lines

    @x (float range: [0, 1]) :: Fraction of where the car is along the x-axis. 1 indicates
    max 'x' value in the coordinate system.

    @y (float range: [0, 1]) :: Fraction of where the car is along the y-axis. 1 indicates
    max 'y' value in the coordinate system.

    @distance_from_center (float [0, track_width/2]) :: Displacement from the center line of the track
    as defined by way points

    @car_orientation (float: [-3.14, 3.14]) :: yaw of the car with respect to the car's x-axis in
    radians

    @progress (float: [0,1]) :: % of track complete

    @steps (int) :: numbers of steps completed

    @throttle :: (float) 0 to 1 (0 indicates stop, 1 max throttle)

    @steering :: (float) -1 to 1 (-1 is right, 1 is left)

    @track_width (float) :: width of the track (> 0)

    @waypoints (ordered list) :: list of waypoint in order; each waypoint is a set of coordinates
    (x,y,yaw) that define a turning point

    @closest_waypoint (int) :: index of the closest waypoint (0-indexed) given the car's x,y
    position as measured by the eucliedean distance

    @@output: @reward (float [-1e5, 1e5])
    '''

    import math
    # Example Centerline following reward function

    marker_1 = 0.1 * track_width
    marker_2 = 0.25 * track_width
    marker_3 = 0.5 * track_width

    reward = 1e-3
    if distance_from_center >= 0.0 and distance_from_center <= marker_1:
        reward = 1
    elif distance_from_center <= marker_2:
        reward = 0.5
    elif distance_from_center <= marker_3:
        reward = 0.1
    else:
        reward = 1e-3  # likely crashed/ close to off track

    return float(reward)

ちなみにその後物理コースで走らせたら全然うまく走らず、1周2分かかりました・・。

SRV409-R1 - [REPEAT 1] A Serverless Journey: AWS Lambda Under the Hood

Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers. However, under the hood, there is a sophisticated architecture that takes care of all the undifferentiated heavy lifting for the developer. Join Holly Mesrobian, Director of Engineering, and Marc Brooker, Senior Principal of Engineering, to learn how AWS architected one of the fastest-growing AWS services. In this session, we show you how Lambda takes care of everything required to run and scale your code with high availability

  • Marc Brooker - Senior Principal Engineer, AWS
  • Holly Mesrobian - Director of Engineering, AWS

Lambda Functionの裏側をどうつくってるかという話。Firecrackerも発表され、Lambdaの基盤も変遷が聞けてよかった。 特にMarcさんのIsolation、KVM上でどうLambdaのスケジューリングの問題に対処したか、そしてFirecrackerでその問題にどう対処したかという話は面白かった。

もうVideoとスライドがあがっていたのでどうぞ。

AWS re:Invent 2018: [REPEAT 1] A Serverless Journey: AWS Lambda Under the Hood (SRV409-R1) - YouTube A Serverless Journey: AWS Lambda Under the Hood (SRV409-R1) - AWS re:…

あとでチェックするセッション

DynamoDB関連セッションはみておく。今回モデリング系のワークショップとかもあったけど行けなかった・・。

aws.amazon.com

というのとお気に入りにいれておいてみてなかったやつのリストをどうぞ。

// https://www.portal.reinvent.awsevents.com/connect/interests.ww でconsoleを開いてスケジュールされていなかったやつだけ出す
$x('//div[contains(@class, "resultRow") and not(contains(@class, "scheduled"))]/div[1]/a').map(e=>e.innerText).sort()
ADT301 - Create a Serverless Web Event Pipeline
ADT302 - Democratize Data Preparation for Analytics & Machine Learning: A Hands-On Lab
ADT303 - ML for Real-Time Self-Service Trend Detection & Root Cause Analysis: A Hands-On Lab
AIM303-R - [REPEAT] Create Smart and Interactive Apps with Intelligent Language Services on AWS
AIM348 - Translating Web Content Easily with Language Services from AWS
AIM362 - Crowdsourcing Data Collection with Amazon Mechanical Turk
AIM363 - [NEW LAUNCH!] Introducing Amazon Textract: Now in Preview
AIM364 - [NEW LAUNCH!] Extract Insights from Millions of Documents with Amazon Textract
AIM366 - [NEW LAUNCH!] Introducing Amazon Elastic Inference: Reduce Deep Learning Inference Cost up to 75%
AIM368 - [NEW LAUNCH!] Introducing Amazon SageMaker RL - Build and Train Reinforcement Learning models on Amazon SageMaker
AIM369 - [NEW LAUNCH!] Introducing Amazon SageMaker Ground Truth: Build High-Quality and Accurate ML Training Datasets
AIM396-S - ML Best Practices: Prepare Data, Build Models, and Manage Lifecycle
AIM401-R1 - [REPEAT 1] Deep Learning Applications Using TensorFlow, ft. Siemens Financial Services
AIM409 - Build a "Who's Who" App for Your Media Content
AIM414 - Sequence-to-Sequence Modeling with Apache MXNet, Sockeye, and Amazon SageMaker
AIM417 - Build a Searchable Media Library & Moderate Content at Scale Using Machine Learning
AIM422 - Fraud Detection and Prevention Using Amazon SageMaker and Amazon Neptune
ALX403-R2 - [REPEAT 2] Supercharge Any Alexa Skill by Understanding What Games Do
ALX405-R2 - [REPEAT 2] Build a Game for Echo Buttons - an Alexa Gadget!
ANT310 - Architecting for Real-Time Insights with Amazon Kinesis
ANT322-R1 - [REPEAT 1] High Performance Data Streaming with Amazon Kinesis: Best Practices
ANT377 - Migrate from Netezza to Amazon Redshift: Best Practices with Financial Engines
ANT379 - [NEW LAUNCH!] Introducing ML Insights with Amazon QuickSight
ANT397 - [NEW LAUNCH!] Amazon Kinesis Data Analytics for Java Applications
ANT398 - [NEW LAUNCH!] Introducing Amazon Managed Streaming for Kafka (Amazon MSK)
ANT401-R - [REPEAT] Deep Dive and Best Practices for Amazon Redshift
API330 - Predictive Scaling for More Responsive Applications
ARC205-R1 - [REPEAT 1] Scaling Up to Your First 10 Million Users
ARC308 - Chaos Engineering and Scalability at Audible.com
ARC340 - [NEW LAUNCH!] Introducing AWS Ground Station – Fully managed Ground Station as a Service
ARC341 - [NEW LAUNCH!] AWS Global Accelerator Deep Dive for Building Highly Available Services
ARC407 - A Chronicle of Airbnb Architecture Evolution
ARC408-R1 - [REPEAT 1] Under the Hood of Amazon Route 53
ARV301 - Meet Luke, Cristine, Grace and Preston - Sumerian Hosts
ARV401-R - [REPEAT] Customizing the Environment in Your AR/VR Scene
ARV403-R - [REPEAT] Machine Learning in the Edge - Enhanced by Amazon Sumerian
CMP203-R - [REPEAT] Amazon Linux 2: A Stable, Secure, High-Performance Linux Environment
CON301-R1 - [REPEAT 1] Mastering Kubernetes on AWS
CON315-R1 - [REPEAT 1] Deploying Microservices using AWS Fargate
CON318-R - [REPEAT] Running a High-Performance Kubernetes Cluster with Amazon EKS
CON365 - [NEW LAUNCH!] Advancing Software Procurement in a Containerized World with the New AWS Marketplace for Containers
CON367 - [NEW LAUNCH!] Introducing AWS App Mesh – service mesh on AWS
CTD203-R1 - [REPEAT 1] Amazon Prime Video: Delivering the Amazing Video Experience
CTD306 - Any Given Thursday, Friday, Saturday: How Pac-12 Streams Hundreds of Live Events
CTD415-R - [REPEAT] Customizing Content Delivery with Lambda@Edge
CTD416 - A Few Milliseconds in the Life of an HTTP Request
DAT310-R - [REPEAT] Building Your First Graph Application with Amazon Neptune - Workshop
DAT314 - Migrating Your NoSQL Database to Amazon DynamoDB
DAT320 - Becoming a Nimble Giant: How Amazon DynamoDB Serves Nike at Scale
DAT365 - Separating Fact from Fiction: Amazon DynamoDB Scaling and Adaptive Capacity
DAT401 - Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB
DAT404-R - [REPEAT] Advanced Design Patterns for Amazon DynamoDB - Workshop
DAT404-R1 - [REPEAT 1] Advanced Design Patterns for Amazon DynamoDB - Workshop
DAT406 - Netflix: Iterating on Stateful Services in the Cloud
DEM120 - [NEW LAUNCH!] How we made aibo smart: A journey through Serverless and IoT on AWS, and a view of AWS RoboMaker integration
DEV303-R - [REPEAT] Instrumenting Kubernetes for Observability Using AWS X-Ray and Amazon CloudWatch
DEV313-S - Shift-Left SRE: Self-Healing with AWS Lambda Functions
DEV322-R - [REPEAT] What's New with the AWS CLI
DEV379-R2 - [REPEAT 2] Introduction to Version 3 of the AWS SDK for JavaScript (TypeScript)
DEV411-R1 - [REPEAT 1] Infrastructure as Code: AWS Best Practices
ENT360 - [NEW LAUNCH!] Scaling HPC Applications on EC2 with Elastic Fabric Adapter
GAM301 - Supercell – Scaling Mobile Games
GPSCT302 - Building a Serverless Space Invaders Game on AWS
GPSCT303 - Blockchain Architecture Design Patterns
GPSCT311 - Tailor-Made SaaS: Multi-Tenant Customizations with AWS Lambda
GPSCT404 - Architecting a Real-World Microservices Architecture and DevOps Strategy on AWS
GPSTEC317 - Building Blockchain Platforms Beyond a Proof of Concept
IOT367 - [NEW LAUNCH!] Introducing AWS IoT Events
IOT368 - [NEW LAUNCH!] Introducing AWS IoT SiteWise
MAE203 - Hollywood's Cloud-Based Content Lakes: Modernized Media Archives
MAE310 - UnicornTrivia - Building a Live Trivia Mobile App Using AWS AppSync, AWS Amplify, and AWS Media Services
MFG301 - Optimize Smart Factories Using Data Lakes and Machine Learning on AWS
MOB306 - Build a Photo-Sharing App with AI-Powered Face and Object Detection
MOB311 - Monetize Your Mobile App with Amazon Mobile Ads
NET312 - Another Day in the Life of a Cloud Network Engineer at Netflix
NET330 - [NEW LAUNCH!] Introduction to AWS Global Accelerator 
NET331 - [NEW LAUNCH!] Introducing AWS Transit Gateway
NET402 - [NEW LAUNCH!] AWS Transit Gateway and Transit VPCs, Reference Architectures for Many VPCs
NET416 - [NEW LAUNCH!] How to Architect for Multi-Region Redundancy Using Anycast Ips
ROB201-R1 - [NEW LAUNCH!] [REPEAT 1] Announcing AWS RoboMaker: A New Cloud Robotics Service
ROB301-R1 - [NEW LAUNCH!] [REPEAT 1] ROS and AWS RoboMaker
ROB302-R1 - [NEW LAUNCH!] [REPEAT 1] Build and Deploy Robot Applications Easily with AWS
SEC329 - AWS Encryption SDK: The Busy Engineer's Guide to Client-Side Encryption
SEC403 - Five New Security Automations Using AWS Security Services & Open Source
SRV305-R1OFV - [OVERFLOW] [REPEAT 1] Inside AWS: Technology Choices for Modern Applications (Red-Venetian)
SRV346-R - [REPEAT] Build and Test a Serverless Error-Tracking Tool in One Weekend
SRV375 - [NEW LAUNCH!] Lambda Layers
STG212 - [NEW LAUNCH!] Introducing S3 Batch Operations: Managing Billions of Objects in Amazon S3 at Scale
STG303-R - [REPEAT] Deep Dive on Amazon S3 Security and Management
STG324 - [NEW LAUNCH!] Introducing AWS DataSync - Simplify, automate, and accelerate online data transfer
STG378-R2 - [REPEAT 2] Querying Data in Place with AWS Object Storage Features and Analytics Tools
STG398-R - [NEW LAUNCH!][REPEAT] Optimizing Costs in Amazon S3 Creating Cost Efficiencies w/ Amazon S3 Storage Classes & Introducing S3 Intelligent-Tiering
STG401-R1 - [REPEAT 1] Building a Data Lake in Amazon S3 & Amazon Glacier