API-query-RDS

---

Setup:

1. Deploy project to aws
2. Copy the RDSDBSecretArn output by the stack
3. Go to RDS & select Query on the database
4. enter these credentials with the Secrets manager ARN as the RDSDBSecretArn
5. create the Customers Table

CREATE TABLE Customers (
  CustomerId int,
  FirstName varchar(255)
)

1. create a few example customers

INSERT INTO Customers (CustomerId, FirstName) VALUES (100, 'Name');
INSERT INTO Customers (CustomerId, FirstName) VALUES (101, 'Name1');
INSERT INTO Customers (CustomerId, FirstName) VALUES (102, 'Name2');

1. make a GET request to the HttpApiUrl output by the stack. Making a GET request to https://ID.execute-api.region.amazonaws.com/CustomerId/100. You will recieve:

[
    {
        "CustomerId": 100,
        "FirstName": "Name"
    }
]

EC2-gitlab-runner-fleet

---

Instance:

• ImageId: ami-084e8c05825742534 (eu-west-2)
• InstanceType: t2.micro

Runners:

• executor: docker
• image: gitlab-runner:latest
• privileged: true (docker-in-docker)

---

Setup:

1. Get Gitlab runner token:
   • create a group in your gitlab
   • acquire a token for that group: https://gitlab.com/groups/YOUR_GROUP/-/runners
2. Download the template.
3. Go to AWS Cloudformation, create stack, upload template.
4. enter a stack name, your token & a scalingGroupDesiredSize (1 will work).
5. finish creating the stack.
6. the ec2 instances & runners will be created.
7. Cloudwatch logs /aws/ec2/STACK_NAME-terminal & /aws/ec2/STACK_NAME-gitlab will show the instances internal logs
8. after less than 5 mins the runners will be visible in https://gitlab.com/groups/YOUR_GROUP/-/runners names after the instance they're running on.

Terminating an instance will result in the runner being unregistered before the instance is terminated

EC2-Gitlab-Instance

---

Instance:

• ImageId: ami-084e8c05825742534 (eu-west-2)
• InstanceType: t2.medium

---

Setup:

1. deploy the template to cloudformation
2. enter the parameters
3. configuration with a domain:
   1. including the hostedZoneId, domainName, subDomainName & domainCertArn parameters will:
      1. Create a HTTPS loadbalancer targetGroup & listener with the domainCertArn on port 443
      2. Create a dns record in the hostedZoneId for subDomainName.domainName e.g. gitlab.example.co.uk
      3. Configure the gitlab instance for the domain subDomainName.domainName
   2. not including the hostedZoneId, domainName, subDomainName & domainCertArn parameters will:
      1. Omit the HTTPS loadbalancer targetGroup & listener
      2. Not create any dns records
      3. Configure the gitlab instance to be accessible from the loadbalancer e.g. {loadBalancerName}-1234567890.AWS::Region.elb.amazonaws.com
4. Give time for the instance to create, it will be accessible from the dns record or the public ELB domain

Notes

Q: Why is a load balancer needed? A: The Gitlab CE installation creates & signs its own HTTPS certificate which some browsers warn about when trying to access the site. The load balancer allows port 443 to be listened on & inject your domain certificate when using HTTPS to resolve this issue.

Q: How do backups work? A: The gitlab.rb file is configured to send the Gitlab backup, the gitlab.rb file & gitlab-secrets.json. A backup will occur everyday at 00:00. A backup can also be preform by running the preform-backup SSM document.

The default username is root & the userData script sets the password to gitlabRootPassword stack parameter, The default being Password123!

---

Self-Managed Gitlab CE

Once you finish setting up Gitlab CE you can login, create groups & repos without issue. You can even clone them locally (setup ssh), add files then push them back to your Gitlab. Additionally, You can also register your own runners on a global or group level check this out. These runners can then create resources in aws using a template.yaml & gitlab-ci.yaml.

AWS-Transfer-Server

---

Setup:

1. deploy the template to cloudformation
2. enter the parameters
   
3. configuration with a domain:
   1. including the hostedZoneId, domainName & domainCertArn parameters will:
      1. Setup the Certificate on the server & enable FTPS as a protocol
      2. Create a Route 53 Record for the domainName
   2. including the hostedZoneId, subDomainName, domainName & domainCertArn parameters will:
      1. Setup the Certificate on the server & enable FTPS as a protocol
      2. Create a Route 53 Record for the subDomainName.domainName

---

Usage

The server's url will be output by the stack

The server can now be connected to over FTP, SFTP & FTPS (if domain is configured) FileZilla:

The Identity provider function will be invoked where the payload can be evaludated & integrated into your environment.

{
  "username": "user",
  "sourceIp": "10.23.0.6",
  "protocol": "FTP",
  "serverId": "s-0123456789ABCDEF",
  "password": "password"
}

AWS-Serverless-Multi-Region-API

This is a proof of concept global AWS API using the Serverless framwork. It includes a base api stack made up of a global dynamo table & it's replicas, A KMS key and many SSM parameters to pass relevant Ids & Arns to the node stacks.

The node stacks create a GraphQL API with a HTTP Proxy fronted by a Latency Route53 record with Cloudwatch metric or Lambda health checks for regional failover. The node stacks are designed to be region agnostic and can be deployed into any region that already has a global table replica in. The GraphQL API communicates with the replica in its own region to provide the lowest possible latency to access data across the globe.

The base api stack also includes a KMS key that's replicated in each node stack. The intention of this was to be used by the Lambda Authorizer for the region specific GraphQL API to authorize encrypted credentials stored in the global table as a 'global' authentication mechanism. This was preferable compared to AWS managed regional API keys (would fail if requests were routed to the region that the key wasn't created in) or cognito authorization which are inherently non-global

iot-hub-website

Amplify Setup

1. In order to use Amplify you will need to generate and provide an access token for it to be able to build the frontend. The required scopes are api, read_api & read_repository
2. When deploying the stack to AWS, it can optionally be configured with a domain in Route53 via the domain params.
3. When the stack deploys you will then need to trigger an Amplify job to build the web frontend similar to what's done in the update-frontend job in the .gitlab-ci.yml file

Arduino Setup

1. Find your AWS Account & Region specific AWS IoT Broker Endpoint - This can be found here: https://console.aws.amazon.com/iot/home#/settings

2. create the AWS IoT certificate

aws iot create-keys-and-certificate --set-as-active

The Response will be:

{
  "certificateArn": "arn:aws:iot:{Region}:{AccountId}:cert/2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificateId": "2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificatePem": "-----BEGIN CERTIFICATE-----\n{Certificate}\n-----END CERTIFICATE-----\n",
  "keyPair": {
    "PublicKey": "-----BEGIN PUBLIC KEY-----\n{Public Key Material}\n-----END PUBLIC KEY-----\n",
    "PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\n{Private Key Material}\n-----END RSA PRIVATE KEY-----\n"
  }
}

1. Prepare arduino_secrets.h file in arduino/iot-lightbulb/

   1. Enter your Wifi name & password
   2. Enter your Account & Region specific AWS IoT Broker Endpoint from Step 1
   3. Enter a unique identifier for the device.
   4. Enter the complete Device Certificate & Device Private Key from Step 2

2. deploy the template.yaml including the certificateArn parameter from step 2

3. upload the .ino file to Arduino using the Arduino IDE

4. The board will now listen on the ${DEVICE_ID}/${INCOMING_TOPIC} topic for events targeted for this device

Usage

1. Create a cognito user in the userpool
2. Associate a device to the user in the devicesTable

{
 "userId": "${Cognito user sub}",
 "deviceId": "light-bulb",
 "name": "demo light bulb thing",
 "type": "LIGHT_BULB"
}

This associated only allows logged in users to interact with their own devices

Devices before logging in

Login page

Devices after logging in

The On/Off switch for the device will send a message to the arduino over MQTT to turn the LED On or Off.

Arduino setup

When the Arduino receives a message over MQTT it will either be told to turn the LED ON or OFF. The arduino will then set the voltage of the relevant data pin to turn the LED On or Off

Unity Snake

Simple Snake game made in Unity

A-Star Path Finding Algorithm

Implementation of A-Star Path Finding Algorithm

setup

1. Run the code
2. Left click = start square
3. right click = finishing square

result

Green - Start point
Red - End point
Dark-Blue - options that have been evaluation
Light-Blue - options that are next to be evaluated
Purple - The shortest path to the end

ECS-Github-Runners

setup

1. Generating a Github Org token. An Admin will need to generate an access token allowing access to Webhooks & Self-Hosted Runners

1. Deploy the stack using your cli of choice. You will need to specify:
   1. Your Organisation Name - githubOrgNameParam
   2. Your Access Token - githubOrgTokenParam
2. When the stack finishes creating it will:
   1. Create a webhook on your organisation - The webhook id will be saved as an SSM Parameter
   2. Create the initial runner token to be used by the ecs runners. This token only lasts 60 minutes & will be rotated every 55 minutes
3. The stack is now ready to receive webhook events. It will scale out an ECS container when it receives a webhook event for a queued workflow and scale it in once the job finishes

ECS Container

The container used by the Github runners is amazonlinux:2023. It then installs a few libraries and registers the github runner to your org. It can take up-tp a minute to install the necessary libraries. Using a custom docker image can minimize start-up times and provide all the libraries commonmly needed for your workflows.

The Lost Dungeon 3

Made using Java 16 - Version 16.0.1
Included Libraries:openjfx-16 & com.google.code.gson:gson:1.4

Example room:

Example of the boss room:

AWS-Asynchronous-api

Implementation of an asynchronous AWS API using SQS to buffer & batch requests with dynamo as a central store for the events, their status and their result. The processFileFunction artificially pauses for 2 - 8 seconds to simulate processing with message.

setup

Deploy the Stack - The API url will be Output from the stack.
Make a POST request to: the /StartQuery endpoint with any body (it doens't validate the body) - You will recieve a response body like:

{
  "MessageId": "4b39d05e-1add-40fe-8c77-6255e5700522",
  "Status": "QUEUED"
}

Make a GET request to the /QueryResults endpoint with the MessageId query parameter - If the message is still in the QUEUED status then you will recieve:

{
  "MessageId": "4b39d05e-1add-40fe-8c77-6255e5700522",
  "Status": "QUEUED"
}

If The message is in the FINISHED status you will recieve

{
  "ttl": "1689212154",
  "Status": "FINISHED",
  "MessageId": "4b39d05e-1add-40fe-8c77-6255e5700522"
}

wave-function-collapse

Initial testing:

The small boxes are representing a blank tile

tests with more complicated tile sheets after adding dynamic rules

addition of asymmetric tiles

proof of concept dungeon layout

could be used for The-Lost-Dungeon-4?

ECS-MC-network

This setup launches x ECS containers, each running a Minecraft server classified as a lobby or a server. All of these servers are accessiable behind an ECS proxy layer allowing players to move between the servers they're connected to.

Setup

See .gitlab-ci.yml for automated deployment.

1. Deploy the container-registry.yaml CloudFormation stack.

2. Build the Proxy & Server docker images to the ECR repository

3. Deploy the template.yaml stack:

   • Provide the ECR Repositories' ARN via the ImageUri CloudFormation parameter.
   • Optionally, configure a domain on the Network Load Balancer in front of the ECS Proxies.

   a. Setup without a domain:

   • Leave hostedZoneId, domainName, and subDomainName empty. The stack will not create a Route53 record.

   b. Setup with a domain:

   • Provide values for hostedZoneId, domainName, and subDomainName. This will create a Route53 record in front of the Network Load Balancer.

Minecraft Server Proxy

This setup leverages BungeeCord with a custom Java plugin that enables dynamic server discovery, live reloading of proxy target lists, and smooth player session transfers between ECS hosted servers.

This is how the plugin displays the list of proxy targets to the players

AWS-EC2-minecraft-server-mesh

This setup launches x EC2 instances, each running a Minecraft server. A custom Bungee plugin connects all the servers, forming a mesh network that allows players to move between them seamlessly using the /server <EC2 instance name> command in Minecraft chat.

Setup

See .gitlab-ci.yml for automated deployment.

1. Deploy the plugin-template.yaml CloudFormation stack.

2. Build the BungeeServerList Bungee plugin and upload it to the stack's S3 bucket:

   • Package the plugin using mvn, e.g. mvn clean package
   • Upload the resulting JAR file (in target/) to the S3 bucket created by the stack.

3. Deploy the template.yaml stack:

   • Provide the plugin's S3 path via the pluginDirectory CloudFormation parameter.
   • Configure the number of EC2 instances via the scalingGroupDesiredSize parameter.
   • Optionally, configure a domain on the Network Load Balancer in front of the Minecraft server mesh.

   a. Setup without a domain:

   • Leave hostedZoneId, domainName, and subDomainName empty. The stack will not create a Route53 record.

   b. Setup with a domain:

   • Provide values for hostedZoneId, domainName, and subDomainName. This will create a Route53 record in front of the Network Load Balancer.

Minecraft Server Proxy

This setup leverages BungeeCord with a custom Java plugin that enables dynamic server discovery, live reloading of proxy target lists, and smooth player session transfers between servers in the mesh.

This is how the plugin displays the list of proxy targets to the players

Hilbert-Curve

infinite space-filling fractal

Order 7: Order 8: Order 9:

AVP-auth-api

This API is backed by a Lambda Authorizer that uses Amazon Verified Permissions (AVP) as a cedar policy store and authentication evaluation engine. This can faliciate RBAC & ABAC logic.

Setup

1. Deploy the Application with a given stack name & AVP namespace (used in policy logic)
2. Create a user in the deployed Cognito user pool, Add that user to the READ and or WRITE group & log in to the appClient to retrieve a JWT token.
3. Make a request to the API url from the stack outputs with the Authorization header as the JWT token.
4. If the Cognito user is in the Cognito read group then they'll have access to the /read endpoint. If they've in the write group then they'll have access to the /write endpoint.

plop-project-scaffolding

create a project from a cli terminal:

---

It will create a project like so:

ecs-cluster & ecs-cluster-image-repo are individual repositories / AWS stacks

ecs-cluster

ecs-cluster can be deployed on its own given the imageRepoName parameter is updated to point to an already existing image / ECR repo

Changing the image might need more resources to be assigned to the Task Defination

---

Setup

first, deploy ecs-cluster-image-repo to AWS then run the deployDockerImage Gitlab Job to publish the react-app as a docker image to an ECR repository

This job will also run the scripts/envs.sh bash script that appends NEXT_PUBLIC_ to all AWS stack outputs, saves them all to a .env file and compiles it into the docker image for dynamic use of variables inside the image instead of hardcoding values.

The docker image also exposes port 8081 which the ecs-cluster directs web traffic to.

secondly, deploy ecs-cluster to AWS. The deployment will automatically create an ECS service with a running task. This task can be accessed via the load balancer DNS name that is exported as an output of the ecs-cluster stack. e.g. http://ecs-cluster-lb-12345678.eu-west-2.elb.amazonaws.com/

The ecs-cluster gitlab jobs can also be used to administrate ECS

restartAllTasks - gets the number of tasks running e.g. 3 then launches 3 more while shutting down the previous 3. This process will refresh the image in the task if the image has been updated.

stopAllTasks - stops all tasks to save costs

startOneTask - starts 1 task

EC2 instances backed by EFS

The template in this repository defines an Applicaiton Load Balancer, backed by an Auto Scaling Group that launches public instances across 3 AZ's that mount a shared Elastic File System

---

Instance:

• ImageId: ami-084e8c05825742534 (eu-west-2)
• InstanceType: t2.micro
• UserData:
  • install & configure amazon-cloudwatch-agent
  • install & start httpd
  • mount instance to template defined EFS

---

Setup:

1. deploy the template - speficy stack parameters
   
2. By default this will create a VPC with the default CIDR parameter. The Auto Scaling Group will launch 2 instances in that VPC behind the Load Balancer. Each instance will mount the EFS to the efsMountPoint directory & touch a file into that directory.
3. After resource creation, each instance will be streaming its terminal logs to cloudwatch. Checking the efsMountPoint direcotry in any instance will show as many files as there are running instances.

Arduino-AWS-IoT-Core

Arduino

This uses the Arduino MKR WiFi 1010 to publish Temperature & Humidity sensor data to AWS Timestream through AWS IoT Core using MQTT

Setup

1. Find your AWS Account & Region specific AWS IoT Broker Endpoint - This can be found here: https://console.aws.amazon.com/iot/home#/settings

2. create the AWS IoT certificate

aws iot create-keys-and-certificate --set-as-active

The Response will be:

{
  "certificateArn": "arn:aws:iot:{Region}:{AccountId}:cert/2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificateId": "2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificatePem": "-----BEGIN CERTIFICATE-----\n{Certificate}\n-----END CERTIFICATE-----\n",
  "keyPair": {
    "PublicKey": "-----BEGIN PUBLIC KEY-----\n{Public Key Material}\n-----END PUBLIC KEY-----\n",
    "PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\n{Private Key Material}\n-----END RSA PRIVATE KEY-----\n"
  }
}

1. Prepare arduino_secrets.h file

   1. Enter your Wifi name & password
   2. Enter your Account & Region specific AWS IoT Broker Endpoint from Step 1
   3. Enter a unique identifier for the device.
   4. Enter the complete Device Certificate & Device Private Key from Step 2

2. deploy the template.yaml including the certificateArn parameter from step 2. The template will listen on a topic with the same name as the stack.

3. upload the .ino file to Arduino using the Arduino IDE

4. The board will now publish the Temperature & Humidity data from the DHT22 sensor and publish it to Timestream through AWS IoT Core

Arduino Logs

Timestream Example

Flocking Simulation

Flocking simulation of boids with configurable alignment, cohesion & separation

Setup

1. run the code
2. a window with n boids will appear - the boids will move together as a flock

cloudwatch-winston-logger

Example

const { Logger } = require('./utils/winstonLogger');

let logger = new Logger(context, { metadata0: 'xyz' });

let metadata1 = "abc";
let metadata2 = "def";
let metadata3 = "ghi";
logger.info(event);
logger.addMetadata({ metadata1 });
logger.info(event);
logger.addMetadata({ metadata2 },  { metadata3 });
logger.warn(event);
logger.removeMetadata('metadata1');
logger.http(event);
logger.removeMetadata('metadata2',  'metadata3');
logger.report(event);

Output

{"level":"info", "message":{"key":"value"}, "metadata0":"xyz"}
{"level":"info", "message":{"key":"value"}, "metadata0":"xyz", "metadata1":"abc"}
{"level":"warn", "message":{"key":"value"}, "metadata0":"xyz", "metadata1":"abc", "metadata2":"def", "metadata3":"ghi"}
{"level":"http", "message":{"key":"value"}, "metadata0":"xyz", "metadata2":"def", "metadata3":"ghi"}
{"level":"report", "message":{"key":"value"}, "metadata0":"xyz"}

Example

  try {
    await Promise.all(jobs);
  } catch (err) {
    logger.error(new Error(err));
  }

Cloudwatch Output

{
    "awsRequestId": "b04c613a-64e5-4fb6-b2c0-5085f971ded6",
    "level": "error",
    "message": "ReferenceError: jobs is not defined",
    "stack": [
        "Error: ReferenceError: jobs is not defined",
        "    at Runtime.handler (/var/task/src/index.js:34:18)",
        "    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1089:29)"
    ]
}

Quadtree

Wiki Page
tree structure to store data

Green rectangle is the search area & red dots are the found objects in the search area

Game Of Life

Conway's Game of Life

AWS-Amplify-Weather-App

uses the openweathermap api to retrieve data

Setup

1. deploy weather-app-2-frontend to gitlab, make note of its project_id
2. create an access token that allows READ REPO access to the frontend - this will be used by Amplify
3. get an api access token from openweathermap
4. deploy backend & setup with CICD variables FRONTEND_REPO_PROJECT_ID, AMPLIFY_TOKEN & WEATHER_API_TOKEN
5. run the update-frontend CICD job on either repo to have Amplify get the most recent frontend commit on the specified branch & deploy it to the backend
6. check the Amplify website

Slime Simulation

Arduino-AWS-IoT-Core

Arduino

This uses the Arduino MKR WiFi 1010 to publish to AWS IoT over MQTT

Setup

1. Find your AWS Account & Region specific AWS IoT Broker Endpoint - This can be found here: https://console.aws.amazon.com/iot/home#/settings

2. create the AWS IoT certificate

aws iot create-keys-and-certificate --set-as-active

The Response will be:

{
  "certificateArn": "arn:aws:iot:{Region}:{AccountId}:cert/2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificateId": "2c0f72bf-b230-4c49-9e0e-cbf177926d96",
  "certificatePem": "-----BEGIN CERTIFICATE-----\n{Certificate}\n-----END CERTIFICATE-----\n",
  "keyPair": {
    "PublicKey": "-----BEGIN PUBLIC KEY-----\n{Public Key Material}\n-----END PUBLIC KEY-----\n",
    "PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\n{Private Key Material}\n-----END RSA PRIVATE KEY-----\n"
  }
}

1. Prepare arduino_secrets.h file

   1. Enter your Wifi name & password
   2. Enter your Account & Region specific AWS IoT Broker Endpoint from Step 1
   3. Enter the complete Device Certificate & Device Private Key from Step 2

2. deploy the template.yaml including the certificateArn parameter from step 2

3. upload the .ino file to Arduino using the Arduino IDE

4. Published messages will now invoke a Lambda

   1. Arduino code
   2. Cloudwatch Logs

Arduino Logs

Setup

deploy glue-bucket-crawler, wait for it to be finished then deploy test-api-v2

execute lambda test-api-v2-create-user with event:

{ "Username": "customer0", "Password": "Password123!", "Email": "a@b.com" }

go to the user in cognito, add them to the group test-api-v2-UserPoolClient-FileGroup

add file to S3 bucket athena-bucket-agb43j/customer0/file.json

Make a GET request to the /File endpoint with the Authorization: Basic Y3VzdG9tZXIwOlBhc3N3b3JkMTIzIQ==

All elements of the json file will be returned.

supporrted query parameters: token, updated_at, limit & page

File must contain updated_at timestamp e.g. 2022-11-12T19:48:02.404Z to use the updated_at query parameter.

AWS Webcoket API

This template.yaml defines a Websocket API that uses dynamo to store active connectionIds and push updates downstream to connections

---

Setup:

1. Deploy repository to AWS
2. Get WebSocketAPIendpoint from the stack outputs - E.g. wss://ue22gqann8.execute-api.eu-west-2.amazonaws.com/$default
   

---

Usage

1. Connect to WebSocketAPIendpoint
   1. Every 5 mins the connectionUpdaterFunction will run and push a message to all active connections.
   2. Send a configured action to the API such as

{
    "action": "createuser"
}

Appsync-Messaging-App

Trying it out

Do you want to see the power of Appsync real-time data? https://chat.matthands.co.uk

Setup

1. In order to use Amplify you will need to generate and provide an access token for it to be able to build the frontend. The required scopes are api, read_api & read_repository
2. When deploying the stack to AWS, it can optionally be configured with a domain in Route53 via the domain params. WAF can also optionally be setup on the GraphQL API.
3. When the stack you will then need to trigger an Amplify job to build the web frontend similar to what's done in the update-frontend job in the .gitlab-ci.yml file

Usage

From the console

1. Subscribe to receive messages from a channel. E.g. AWS

2. Publish a message to that channel

3. Receive that message via the subscription

From the Website

1. Enter a channel. The website will display a list of example channels but anything following https://{domain}/channels/{channelName} will work. Entering a channel will subscribe you to receive messages pushed to that channel
2. When any client pushes messages you that channel. all connected clients will receive it over the subscription.