cloudwatch-winston-logger

Example

const { Logger } = require('./utils/winstonLogger');

let logger = new Logger(context, { metadata0: 'xyz' });

let metadata1 = "abc";
let metadata2 = "def";
let metadata3 = "ghi";
logger.info(event);
logger.addMetadata({ metadata1 });
logger.info(event);
logger.addMetadata({ metadata2 },  { metadata3 });
logger.warn(event);
logger.removeMetadata('metadata1');
logger.http(event);
logger.removeMetadata('metadata2',  'metadata3');
logger.report(event);

Output

{"level":"info", "message":{"key":"value"}, "metadata0":"xyz"}
{"level":"info", "message":{"key":"value"}, "metadata0":"xyz", "metadata1":"abc"}
{"level":"warn", "message":{"key":"value"}, "metadata0":"xyz", "metadata1":"abc", "metadata2":"def", "metadata3":"ghi"}
{"level":"http", "message":{"key":"value"}, "metadata0":"xyz", "metadata2":"def", "metadata3":"ghi"}
{"level":"report", "message":{"key":"value"}, "metadata0":"xyz"}

Example

  try {
    await Promise.all(jobs);
  } catch (err) {
    logger.error(new Error(err));
  }

Cloudwatch Output

{
    "awsRequestId": "b04c613a-64e5-4fb6-b2c0-5085f971ded6",
    "level": "error",
    "message": "ReferenceError: jobs is not defined",
    "stack": [
        "Error: ReferenceError: jobs is not defined",
        "    at Runtime.handler (/var/task/src/index.js:34:18)",
        "    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1089:29)"
    ]
}

AVP-auth-api

This API is backed by a Lambda Authorizer that uses Amazon Verified Permissions (AVP) as a cedar policy store and authentication evaluation engine. This can faliciate RBAC & ABAC logic.

Setup

1. Deploy the Application with a given stack name & AVP namespace (used in policy logic)
2. Create a user in the deployed Cognito user pool, Add that user to the READ and or WRITE group & log in to the appClient to retrieve a JWT token.
3. Make a request to the API url from the stack outputs with the Authorization header as the JWT token.
4. If the Cognito user is in the Cognito read group then they'll have access to the /read endpoint. If they've in the write group then they'll have access to the /write endpoint.

Unity Snake

Simple Snake game made in Unity

AWS-EC2-minecraft-server-mesh

This setup launches x EC2 instances, each running a Minecraft server. A custom Bungee plugin connects all the servers, forming a mesh network that allows players to move between them seamlessly using the /server <EC2 instance name> command in Minecraft chat.

Setup

See .gitlab-ci.yml for automated deployment.

1. Deploy the plugin-template.yaml CloudFormation stack.

2. Build the BungeeServerList Bungee plugin and upload it to the stack's S3 bucket:

   • Package the plugin using mvn, e.g. mvn clean package
   • Upload the resulting JAR file (in target/) to the S3 bucket created by the stack.

3. Deploy the template.yaml stack:

   • Provide the plugin's S3 path via the pluginDirectory CloudFormation parameter.
   • Configure the number of EC2 instances via the scalingGroupDesiredSize parameter.
   • Optionally, configure a domain on the Network Load Balancer in front of the Minecraft server mesh.

   a. Setup without a domain:

   • Leave hostedZoneId, domainName, and subDomainName empty. The stack will not create a Route53 record.

   b. Setup with a domain:

   • Provide values for hostedZoneId, domainName, and subDomainName. This will create a Route53 record in front of the Network Load Balancer.

Minecraft Server Proxy

This setup leverages BungeeCord with a custom Java plugin that enables dynamic server discovery, live reloading of proxy target lists, and smooth player session transfers between servers in the mesh.

This is how the plugin displays the list of proxy targets to the players

The Lost Dungeon 3

Made using Java 16 - Version 16.0.1
Included Libraries:openjfx-16 & com.google.code.gson:gson:1.4

Example room:

Example of the boss room:

Hilbert-Curve

infinite space-filling fractal

Order 7: Order 8: Order 9:

API-query-RDS

---

Setup:

1. Deploy project to aws
2. Copy the RDSDBSecretArn output by the stack
3. Go to RDS & select Query on the database
4. enter these credentials with the Secrets manager ARN as the RDSDBSecretArn
5. create the Customers Table

CREATE TABLE Customers (
  CustomerId int,
  FirstName varchar(255)
)

1. create a few example customers

INSERT INTO Customers (CustomerId, FirstName) VALUES (100, 'Name');
INSERT INTO Customers (CustomerId, FirstName) VALUES (101, 'Name1');
INSERT INTO Customers (CustomerId, FirstName) VALUES (102, 'Name2');

1. make a GET request to the HttpApiUrl output by the stack. Making a GET request to https://ID.execute-api.region.amazonaws.com/CustomerId/100. You will recieve:

[
    {
        "CustomerId": 100,
        "FirstName": "Name"
    }
]

Arduino-AWS-IoT-Core

Arduino

This uses the Arduino MKR WiFi 1010 to publish to AWS IoT over MQTT

Setup

1. Find your AWS Account & Region specific AWS IoT Broker Endpoint - This can be found here: https://console.aws.amazon.com/iot/home#/settings

2. create the AWS IoT certificate

aws iot create-keys-and-certificate --set-as-active

The Response will be:

{
  "certificateArn": "arn:aws:iot:{Region}:{AccountId}:cert/2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificateId": "2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificatePem": "-----BEGIN CERTIFICATE-----\n{Certificate}\n-----END CERTIFICATE-----\n",
  "keyPair": {
    "PublicKey": "-----BEGIN PUBLIC KEY-----\n{Public Key Material}\n-----END PUBLIC KEY-----\n",
    "PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\n{Private Key Material}\n-----END RSA PRIVATE KEY-----\n"
  }
}

1. Prepare arduino_secrets.h file

   1. Enter your Wifi name & password
   2. Enter your Account & Region specific AWS IoT Broker Endpoint from Step 1
   3. Enter the complete Device Certificate & Device Private Key from Step 2

2. deploy the template.yaml including the certificateArn parameter from step 2

3. upload the .ino file to Arduino using the Arduino IDE

4. Published messages will now invoke a Lambda

   1. Arduino code
   2. Cloudwatch Logs

Arduino Logs

iot-hub-website

Amplify Setup

1. In order to use Amplify you will need to generate and provide an access token for it to be able to build the frontend. The required scopes are api, read_api & read_repository
2. When deploying the stack to AWS, it can optionally be configured with a domain in Route53 via the domain params.
3. When the stack deploys you will then need to trigger an Amplify job to build the web frontend similar to what's done in the update-frontend job in the .gitlab-ci.yml file

Arduino Setup

1. Find your AWS Account & Region specific AWS IoT Broker Endpoint - This can be found here: https://console.aws.amazon.com/iot/home#/settings

2. create the AWS IoT certificate

aws iot create-keys-and-certificate --set-as-active

The Response will be:

{
  "certificateArn": "arn:aws:iot:{Region}:{AccountId}:cert/2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificateId": "2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificatePem": "-----BEGIN CERTIFICATE-----\n{Certificate}\n-----END CERTIFICATE-----\n",
  "keyPair": {
    "PublicKey": "-----BEGIN PUBLIC KEY-----\n{Public Key Material}\n-----END PUBLIC KEY-----\n",
    "PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\n{Private Key Material}\n-----END RSA PRIVATE KEY-----\n"
  }
}

1. Prepare arduino_secrets.h file in arduino/iot-lightbulb/

   1. Enter your Wifi name & password
   2. Enter your Account & Region specific AWS IoT Broker Endpoint from Step 1
   3. Enter a unique identifier for the device.
   4. Enter the complete Device Certificate & Device Private Key from Step 2

2. deploy the template.yaml including the certificateArn parameter from step 2

3. upload the .ino file to Arduino using the Arduino IDE

4. The board will now listen on the ${DEVICE_ID}/${INCOMING_TOPIC} topic for events targeted for this device

Usage

1. Create a cognito user in the userpool
2. Associate a device to the user in the devicesTable

{
 "userId": "${Cognito user sub}",
 "deviceId": "light-bulb",
 "name": "demo light bulb thing",
 "type": "LIGHT_BULB"
}

This associated only allows logged in users to interact with their own devices

Devices before logging in

Login page

Devices after logging in

The On/Off switch for the device will send a message to the arduino over MQTT to turn the LED On or Off.

Arduino setup

When the Arduino receives a message over MQTT it will either be told to turn the LED ON or OFF. The arduino will then set the voltage of the relevant data pin to turn the LED On or Off

wave-function-collapse

Initial testing:

The small boxes are representing a blank tile

tests with more complicated tile sheets after adding dynamic rules

addition of asymmetric tiles

proof of concept dungeon layout

could be used for The-Lost-Dungeon-4?

ECS-MC-network

This setup launches x ECS containers, each running a Minecraft server classified as a lobby or a server. All of these servers are accessiable behind an ECS proxy layer allowing players to move between the servers they're connected to.

Setup

See .gitlab-ci.yml for automated deployment.

1. Deploy the container-registry.yaml CloudFormation stack.

2. Build the Proxy & Server docker images to the ECR repository

3. Deploy the template.yaml stack:

   • Provide the ECR Repositories' ARN via the ImageUri CloudFormation parameter.
   • Optionally, configure a domain on the Network Load Balancer in front of the ECS Proxies.

   a. Setup without a domain:

   • Leave hostedZoneId, domainName, and subDomainName empty. The stack will not create a Route53 record.

   b. Setup with a domain:

   • Provide values for hostedZoneId, domainName, and subDomainName. This will create a Route53 record in front of the Network Load Balancer.

Minecraft Server Proxy

This setup leverages BungeeCord with a custom Java plugin that enables dynamic server discovery, live reloading of proxy target lists, and smooth player session transfers between ECS hosted servers.

This is how the plugin displays the list of proxy targets to the players

EC2-gitlab-runner-fleet

---

Instance:

• ImageId: ami-084e8c05825742534 (eu-west-2)
• InstanceType: t2.micro

Runners:

• executor: docker
• image: gitlab-runner:latest
• privileged: true (docker-in-docker)

---

Setup:

1. Get Gitlab runner token:
   • create a group in your gitlab
   • acquire a token for that group: https://gitlab.com/groups/YOUR_GROUP/-/runners
2. Download the template.
3. Go to AWS Cloudformation, create stack, upload template.
4. enter a stack name, your token & a scalingGroupDesiredSize (1 will work).
5. finish creating the stack.
6. the ec2 instances & runners will be created.
7. Cloudwatch logs /aws/ec2/STACK_NAME-terminal & /aws/ec2/STACK_NAME-gitlab will show the instances internal logs
8. after less than 5 mins the runners will be visible in https://gitlab.com/groups/YOUR_GROUP/-/runners names after the instance they're running on.

Terminating an instance will result in the runner being unregistered before the instance is terminated

AWS-Transfer-Server

---

Setup:

1. deploy the template to cloudformation
2. enter the parameters
   
3. configuration with a domain:
   1. including the hostedZoneId, domainName & domainCertArn parameters will:
      1. Setup the Certificate on the server & enable FTPS as a protocol
      2. Create a Route 53 Record for the domainName
   2. including the hostedZoneId, subDomainName, domainName & domainCertArn parameters will:
      1. Setup the Certificate on the server & enable FTPS as a protocol
      2. Create a Route 53 Record for the subDomainName.domainName

---

Usage

The server's url will be output by the stack

The server can now be connected to over FTP, SFTP & FTPS (if domain is configured) FileZilla:

The Identity provider function will be invoked where the payload can be evaludated & integrated into your environment.

{
  "username": "user",
  "sourceIp": "10.23.0.6",
  "protocol": "FTP",
  "serverId": "s-0123456789ABCDEF",
  "password": "password"
}

Terraform-Aurora-Global

This demo utilises an AWS RDS Aurora Serverless acting as a global cluster manged by Terraform. It first deploys the primary cluster (the writer) to the specified primaryRegion in variables.tf and it then deploys a cluster (the reader) in each region specified in the secondaryRegions variable in variables.tf. It also generates and configured RDS to authorise with an AWS Secrets Manager Secret which is also replicated to all necessary regions along with the regional KMS Key. As a simple way to view the data in the RDS database, the application launches a website in a public subnet which queries the app database in RDS to list all users.

Setup

See .gitlab-ci.yml for automated deployment.

1. To initialise Terraform and deploy to AWS it requires an S3 bucket be created to store state. Ensure a bucket is created and initialise Terraform like

   terraform init \
     -backend-config="bucket=${BUCKET_NAME}" \
     -backend-config="key=tf-rds-aurora-global" \
     -backend-config="region=${BUCKET_REGION}"
   

2. Configure the desired variables in variables.tf.

   • primaryRegion This specifies the primary AWS Region which the writer cluster will be created in. A KMS Key and Secrets Manager Secret will also be created.
   • secondaryRegions This is an array of AWS Regions for which a replica reader cluster will be created in. A replica KMS Key and Secrets Manager Secret will also be created.
   • domain Optional Route53 domain in the target account. This will cause Route53 Latency records to be created in all specified regions which route to the public EC2 running a website allowing viewing of the users in the app Database.
   • subDomain Optional subdomain to prefix the domain with when creating the Route53 records.

3. To see what resources Terraform will deploy you can run terraform plan and if that looks ok you can run terraform apply --auto-approve which will deploy all resources.

4. Once the resources are deployed you'll need to populate the database with some test data before you can view it on the EC2 website.

   • Go to AWS Secrets Manager, find the Secret that Terraform created and copy its Secret ARN
   • Now go to the AWS console for Aurora and RDS then Query editor.
   • To connect to the cluster, select it from the dropdown, select Connect with a Secrets Manager ARN and enter the previously copied Secret ARN and enter these commands

   CREATE DATABASE app;
   

   use app;
   
   CREATE TABLE users (
     id INT PRIMARY KEY,
     name VARCHAR(100)
   );
   

   use app;
   
   INSERT INTO users (id, name) VALUES (1, 'Alice');
   INSERT INTO users (id, name) VALUES (2, 'Bob');
   

5. Now that's some test data in the DB you can go to the AWS console for EC2, find the websiteInstance that Terraform created and go to the website http://{Public IPv4 address}

6. There is an equivalent EC2 website created in the regions specified for primaryRegion and secondaryRegions which query all data created in the users database as the global cluster replicates it across all replica clusters.

7. Once done, you can teardown the resources Terraform created with terraform destroy --auto-approve

Flocking Simulation

Flocking simulation of boids with configurable alignment, cohesion & separation

Setup

1. run the code
2. a window with n boids will appear - the boids will move together as a flock

Game Of Life

Conway's Game of Life

ECS-Github-Runners

setup

1. Generating a Github Org token. An Admin will need to generate an access token allowing access to Webhooks & Self-Hosted Runners

1. Deploy the stack using your cli of choice. You will need to specify:
   1. Your Organisation Name - githubOrgNameParam
   2. Your Access Token - githubOrgTokenParam
2. When the stack finishes creating it will:
   1. Create a webhook on your organisation - The webhook id will be saved as an SSM Parameter
   2. Create the initial runner token to be used by the ecs runners. This token only lasts 60 minutes & will be rotated every 55 minutes
3. The stack is now ready to receive webhook events. It will scale out an ECS container when it receives a webhook event for a queued workflow and scale it in once the job finishes

ECS Container

The container used by the Github runners is amazonlinux:2023. It then installs a few libraries and registers the github runner to your org. It can take up-tp a minute to install the necessary libraries. Using a custom docker image can minimize start-up times and provide all the libraries commonmly needed for your workflows.

AWS-Asynchronous-api

Implementation of an asynchronous AWS API using SQS to buffer & batch requests with dynamo as a central store for the events, their status and their result. The processFileFunction artificially pauses for 2 - 8 seconds to simulate processing with message.

setup

Deploy the Stack - The API url will be Output from the stack.
Make a POST request to: the /StartQuery endpoint with any body (it doens't validate the body) - You will recieve a response body like:

{
  "MessageId": "4b39d05e-1add-40fe-8c77-6255e5700522",
  "Status": "QUEUED"
}

Make a GET request to the /QueryResults endpoint with the MessageId query parameter - If the message is still in the QUEUED status then you will recieve:

{
  "MessageId": "4b39d05e-1add-40fe-8c77-6255e5700522",
  "Status": "QUEUED"
}

If The message is in the FINISHED status you will recieve

{
  "ttl": "1689212154",
  "Status": "FINISHED",
  "MessageId": "4b39d05e-1add-40fe-8c77-6255e5700522"
}

A-Star Path Finding Algorithm

Implementation of A-Star Path Finding Algorithm

setup

1. Run the code
2. Left click = start square
3. right click = finishing square

result

Green - Start point
Red - End point
Dark-Blue - options that have been evaluation
Light-Blue - options that are next to be evaluated
Purple - The shortest path to the end

ecs-cluster & ecs-cluster-image-repo are individual repositories / AWS stacks

ecs-cluster

ecs-cluster can be deployed on its own given the imageRepoName parameter is updated to point to an already existing image / ECR repo

Changing the image might need more resources to be assigned to the Task Defination

---

Setup

first, deploy ecs-cluster-image-repo to AWS then run the deployDockerImage Gitlab Job to publish the react-app as a docker image to an ECR repository

This job will also run the scripts/envs.sh bash script that appends NEXT_PUBLIC_ to all AWS stack outputs, saves them all to a .env file and compiles it into the docker image for dynamic use of variables inside the image instead of hardcoding values.

The docker image also exposes port 8081 which the ecs-cluster directs web traffic to.

secondly, deploy ecs-cluster to AWS. The deployment will automatically create an ECS service with a running task. This task can be accessed via the load balancer DNS name that is exported as an output of the ecs-cluster stack. e.g. http://ecs-cluster-lb-12345678.eu-west-2.elb.amazonaws.com/

The ecs-cluster gitlab jobs can also be used to administrate ECS

restartAllTasks - gets the number of tasks running e.g. 3 then launches 3 more while shutting down the previous 3. This process will refresh the image in the task if the image has been updated.

stopAllTasks - stops all tasks to save costs

startOneTask - starts 1 task

EC2 instances backed by EFS

The template in this repository defines an Applicaiton Load Balancer, backed by an Auto Scaling Group that launches public instances across 3 AZ's that mount a shared Elastic File System

---

Instance:

• ImageId: ami-084e8c05825742534 (eu-west-2)
• InstanceType: t2.micro
• UserData:
  • install & configure amazon-cloudwatch-agent
  • install & start httpd
  • mount instance to template defined EFS

---

Setup:

1. deploy the template - speficy stack parameters
   
2. By default this will create a VPC with the default CIDR parameter. The Auto Scaling Group will launch 2 instances in that VPC behind the Load Balancer. Each instance will mount the EFS to the efsMountPoint directory & touch a file into that directory.
3. After resource creation, each instance will be streaming its terminal logs to cloudwatch. Checking the efsMountPoint direcotry in any instance will show as many files as there are running instances.

Slime Simulation

AWS-Serverless-Multi-Region-API

This is a proof of concept global AWS API using the Serverless framwork. It includes a base api stack made up of a global dynamo table & it's replicas, A KMS key and many SSM parameters to pass relevant Ids & Arns to the node stacks.

The node stacks create a GraphQL API with a HTTP Proxy fronted by a Latency Route53 record with Cloudwatch metric or Lambda health checks for regional failover. The node stacks are designed to be region agnostic and can be deployed into any region that already has a global table replica in. The GraphQL API communicates with the replica in its own region to provide the lowest possible latency to access data across the globe.

The base api stack also includes a KMS key that's replicated in each node stack. The intention of this was to be used by the Lambda Authorizer for the region specific GraphQL API to authorize encrypted credentials stored in the global table as a 'global' authentication mechanism. This was preferable compared to AWS managed regional API keys (would fail if requests were routed to the region that the key wasn't created in) or cognito authorization which are inherently non-global

Arduino-AWS-IoT-Core

Arduino

This uses the Arduino MKR WiFi 1010 to publish Temperature & Humidity sensor data to AWS Timestream through AWS IoT Core using MQTT

Setup

1. Find your AWS Account & Region specific AWS IoT Broker Endpoint - This can be found here: https://console.aws.amazon.com/iot/home#/settings

2. create the AWS IoT certificate

aws iot create-keys-and-certificate --set-as-active

The Response will be:

{
  "certificateArn": "arn:aws:iot:{Region}:{AccountId}:cert/2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificateId": "2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificatePem": "-----BEGIN CERTIFICATE-----\n{Certificate}\n-----END CERTIFICATE-----\n",
  "keyPair": {
    "PublicKey": "-----BEGIN PUBLIC KEY-----\n{Public Key Material}\n-----END PUBLIC KEY-----\n",
    "PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\n{Private Key Material}\n-----END RSA PRIVATE KEY-----\n"
  }
}

1. Prepare arduino_secrets.h file

   1. Enter your Wifi name & password
   2. Enter your Account & Region specific AWS IoT Broker Endpoint from Step 1
   3. Enter a unique identifier for the device.
   4. Enter the complete Device Certificate & Device Private Key from Step 2

2. deploy the template.yaml including the certificateArn parameter from step 2. The template will listen on a topic with the same name as the stack.

3. upload the .ino file to Arduino using the Arduino IDE

4. The board will now publish the Temperature & Humidity data from the DHT22 sensor and publish it to Timestream through AWS IoT Core

Arduino Logs

Timestream Example

Setup

deploy glue-bucket-crawler, wait for it to be finished then deploy test-api-v2

execute lambda test-api-v2-create-user with event:

{ "Username": "customer0", "Password": "Password123!", "Email": "a@b.com" }

go to the user in cognito, add them to the group test-api-v2-UserPoolClient-FileGroup

add file to S3 bucket athena-bucket-agb43j/customer0/file.json

Make a GET request to the /File endpoint with the Authorization: Basic Y3VzdG9tZXIwOlBhc3N3b3JkMTIzIQ==

All elements of the json file will be returned.

supporrted query parameters: token, updated_at, limit & page

File must contain updated_at timestamp e.g. 2022-11-12T19:48:02.404Z to use the updated_at query parameter.

plop-project-scaffolding

create a project from a cli terminal:

---

It will create a project like so:

Quadtree

Wiki Page
tree structure to store data

Green rectangle is the search area & red dots are the found objects in the search area

twitch-vod-downloader

Twitch is a livestreaming platform that allows content creators to livestream video to their Twitch channel for people to view and interact with. When a livestream ends, it is then made available as a VOD (Video On Demand) to be watched by people who missed the livestream.

Twitch only allows previous broadcasts to be available for 7 days, or 60 days for their Twitch Partners. After this period, the broadcast is no longer available and, unless downloaded by the streamer, it effectively becomes lost media.

This system is intended to support the automatic archival of content creators' livestreams by downloading the VOD asset and enabling rewatching it from a user interface.

Setup

See .gitlab-ci.yml for automated deployment.

1. This application supports logging in by deferring to Twitch via OAuth 2.0. Deploying this application will require registering a Twitch Developer App and then creating a Confidential Twitch application. You will need to create an AWS Secrets Manager secret with the format:

{
  "clientId": "{your client id}",
  "clientSecret": "{your client secret}",
  "url": "https://id.twitch.tv/oauth2/token"
}

1. Deploy the container-registry.yaml CloudFormation stack to host the ECR images. E.g. sam deploy --no-fail-on-empty-changeset --template-file container-registry.yaml --stack-name twitch-vod-downloader-registry --s3-bucket $S3_DEPLOY --capabilities CAPABILITY_NAMED_IAM --region eu-west-1
2. Build the Manifest and DownloadVideo docker images and push them to the ECR repository from step 2. The image tags should be Manifest and DownloadVideo to match what the ECS task definitions in template.yaml expect.
3. Deploy the template.yaml with a Domain, HostedZoneId, DomainCert & optionally DomainPrefix for Amplify to be configured with a Route53 domain. The AmplifyAppRepoURL, AmplifyAppRepoBranch & AmplifyAppRepoToken Parameters need to be provided for Amplify to build the frontend. The TwitchCredentialsSecretPath parameter needs to be specified, which is the ARN of the Secret created in step 1 and the ImageUri parameter will need to be provided as the ECR repository URI for ECS to fetch the docker images.

Frontend

The frontend UI integrates with Twitch using an OIDC Cognito Identity Provider. Visiting the frontend redirects you to the Twitch login. After signing in, Twitch redirects you back to the application.

You are then presented with a list of Twitch Streamers that the application is archiving the VODs for.

Clicking on a Twitch Streamer will load a list of their videos which have been downloaded and are ready to view.

Registering a Streamer

Registering a new Streamer can be done by making this Appsync mutation

mutation MyMutation {
  createStreamer(username: "pokimane") {
    pk
    sk
    userDisplayName
    userProfileImageUrl
    userName
    userId
  }
}

Dockerized puppeteer webbot

A key piece of this system is taking the url of the Streamer's video and retrieving the HLS manifest url of that video. Twitch's API does not allow this, resulting in a dockerized ECS puppeteer bot which loads the page & intercepts traffic for the HLS manifest url.

Data structures

Streamer

{
 "pk": "STREAMER",
 "sk": "STREAMER#44445592",
 "userDisplayName": "pokimane",
 "userId": "44445592",
 "userName": "pokimane",
 "userProfileImageUrl": "https://static-cdn.jtvnw.net/jtv_user_pictures/912232e8-9e53-4fb7-aac4-14aed07869ca-profile_image-300x300.png"
}

Video

{
 "pk": "VIDEO#44445592",
 "sk": "VIDEO#2458896948",
 "processedManifestUri": "s3://twitch-vod-downloader-video-bucket/44445592/2458896948/index.m3u8",
 "processedManifestUrl": "https://djpcyzz1kzchy.cloudfront.net/44445592/2458896948/index.m3u8",
 "twitchManifestUrl": "https://d2nvs31859zcd8.cloudfront.net/d3c2fadcaee0a608c2d6_pokimane_321069980156_1747261099/chunked/index-dvr.m3u8",
 "twitchVideoUrl": "https://www.twitch.tv/videos/2458896948",
 "userId": "44445592",
 "videoDuration": "11m47s",
 "videoId": "2458896948",
 "videoProcessingStatus": "COMPLETE",
 "videoPublishedTime": "2025-05-14T22:18:25Z"
}

Appsync-Messaging-App

Trying it out

Do you want to see the power of Appsync real-time data? https://chat.matthands.co.uk

Setup

1. In order to use Amplify you will need to generate and provide an access token for it to be able to build the frontend. The required scopes are api, read_api & read_repository
2. When deploying the stack to AWS, it can optionally be configured with a domain in Route53 via the domain params. WAF can also optionally be setup on the GraphQL API.
3. When the stack you will then need to trigger an Amplify job to build the web frontend similar to what's done in the update-frontend job in the .gitlab-ci.yml file

Usage

From the console

1. Subscribe to receive messages from a channel. E.g. AWS

2. Publish a message to that channel

3. Receive that message via the subscription

From the Website

1. Enter a channel. The website will display a list of example channels but anything following https://{domain}/channels/{channelName} will work. Entering a channel will subscribe you to receive messages pushed to that channel
2. When any client pushes messages you that channel. all connected clients will receive it over the subscription.

EC2-Gitlab-Instance

---

Instance:

• ImageId: ami-084e8c05825742534 (eu-west-2)
• InstanceType: t2.medium

---

Setup:

1. deploy the template to cloudformation
2. enter the parameters
3. configuration with a domain:
   1. including the hostedZoneId, domainName, subDomainName & domainCertArn parameters will:
      1. Create a HTTPS loadbalancer targetGroup & listener with the domainCertArn on port 443
      2. Create a dns record in the hostedZoneId for subDomainName.domainName e.g. gitlab.example.co.uk
      3. Configure the gitlab instance for the domain subDomainName.domainName
   2. not including the hostedZoneId, domainName, subDomainName & domainCertArn parameters will:
      1. Omit the HTTPS loadbalancer targetGroup & listener
      2. Not create any dns records
      3. Configure the gitlab instance to be accessible from the loadbalancer e.g. {loadBalancerName}-1234567890.AWS::Region.elb.amazonaws.com
4. Give time for the instance to create, it will be accessible from the dns record or the public ELB domain

Notes

Q: Why is a load balancer needed? A: The Gitlab CE installation creates & signs its own HTTPS certificate which some browsers warn about when trying to access the site. The load balancer allows port 443 to be listened on & inject your domain certificate when using HTTPS to resolve this issue.

Q: How do backups work? A: The gitlab.rb file is configured to send the Gitlab backup, the gitlab.rb file & gitlab-secrets.json. A backup will occur everyday at 00:00. A backup can also be preform by running the preform-backup SSM document.

The default username is root & the userData script sets the password to gitlabRootPassword stack parameter, The default being Password123!

---

Self-Managed Gitlab CE

Once you finish setting up Gitlab CE you can login, create groups & repos without issue. You can even clone them locally (setup ssh), add files then push them back to your Gitlab. Additionally, You can also register your own runners on a global or group level check this out. These runners can then create resources in aws using a template.yaml & gitlab-ci.yaml.