Configuring the Cloud Intelligence Dashboards

The Cloud Intelligence Dashboards are an open-source framework that provides customers with actionable insights and optimisation opportunities at scale within an organisation. These dashboards help organisations drive financial accountability, whilst helping them to optimise cost, track usage goals, and implement best-practices for governance.

The dashboards provided by this Solution have come from the GitHub Repository and provide a variety of Amazon QuickSight Dashboards to provide visibility over the spend in there AWS Accounts.

Architecture Overview

Pre-Requisites for the Solution

1. Determine which AWS Account that you want to have the Amazon QuickSight Dashboards available within. Typically, I recommend using an account such as Shared-Services but this should not be the Management Account.
2. In addition, when the Cloud Intelligence Dashboards are implemented into the Environment; Amazon QuickSight will be configured so that it is integrated directly with AWS IAM Identity Center and thereby enabling federation with the 3rd Party Identity Provider. As part of this integration, there is a dependency on 3 groups being established that exist in AWS IAM Identity Center. It’s recommended that these are provisioned in you Centralised Identity Provider and provisioned via SCIM into IAM Identity Center as follows:

• QuickSight-Admin: This will be used to provide group members with administrative access permissions to Amazon QuickSight and will provide the ability to manage all QuickSight resources like users, groups, data sources, datasets, analyses, dashboards etc.
• QuickSight-Author: This will be used to provide group members with the permissions to create, edit and view analyses and dashboards in Amazon QuickSight but will not allow administrative actions.
• QuickSight-Reader: This will be used to provide group members with limited read-only access to analyses and dashboards in Quicksight. This will allow the members to view and run existing analyses and dashboards but not create or modify any content.

Installation for the Solution

There are several steps that need to be carried out in order to get the Solution to deploy.

Create Cost and Usage Reports

1. Deploy the following CloudFormation Template into the AWS Account that you have chosen in the Pre-Requisites Step 1. Update the Parameters below and leave all other values as they are:

2. Deploy the following CloudFormation Template into the Management Account. Update the Parameters below and leave all other values as they are:

Once these 2 Templates have been deployed this will have deployed the an S3 Bucket in both of the Accounts. In the Management Account the S3 Bucket will be used for storing the AWS Cost and Usage Reports in and will also set up S3 Bucket Replication between the 2 AWS Accounts.

Note: Neither of the S3 Buckets will have an S3 Lifecycle Policy configured, but could be added easily enough based on the Clients needs.

We now need to wait at least 24 hours for the AWS Cost and Usage Reports to generate Data.

Grant Permissions for the Data Collection Resources

3. Deploy the following CloudFormation Template into the Management Account. Update the Parameters below and leave all other values as they are:

4. Deploy the following CloudFormation Template into the Management Account as a StackSet that Target the Root OU ID with the Deployment Region being set to the AWS Control Tower Home Region. Update the Parameters below and leave all other values as they are:

Deploy the Data Collection Resources Required for all the Extra Dashboards Available

5. Deploy the following CloudFormation Template into the AWS Account that you have chosen in the Pre-Requisites Step 1. Update the Parameters below and leave all other values as they are:

6. In the Management Account, ensure that you have enable Cost Anomaly Detection and Rightsizing Recommendations within AWS Cost Management.
7. In the Management Account, ensure that you have enabled AWS Compute Optimiser for integration with AWS Organizations.

Note: Following the enablement of Cost Anomaly Detection, Rightsizing Recommendations and Compute Optimiser, this will take at least 24 hours for data to be generated.

Deploy the Dashboards

8. In the AWS Account that you have chosen in the Pre-Requisites Step 1, log in to Amazon QuickSight.

   • You will be asked to Sign up before you will be able to use it
   • After pressing the Sign up button you will be presented with 2 options, ensure you select the Enterprise Edition during this step
   • Select Continue and you will be presented with an option to add Paginated Reports. Choose No, Maybe Later.
   • You’ll then need to fill in a series of options in order to finish creating your account. Please select the appropriate Authentication method
   • Ensure you select the Region that is most appropriate based on where you plan to deploy the dashboards
   • Enter a name for your QuickSight account. This must be unique across all QuickSight accounts.
   • Enter an email address for notifications to be sent to. This email will be linked to your QuickSight user account so it can be your email.
   • Click Select S3 buckets and choose your Cost & Usage Reports Bucket (e.g., cid-[AWS Account Id that you have chosen in the Pre-Requisites Step 1]-shared)
   • Click Finish and wait for the congratulations screen to display

9. Deploy the following CloudFormation Template into the AWS Account that you have chosen in the Pre-Requisites Step 1. Update the Parameters below and leave all other values as they are:

The foundational Cloud Intelligence Dashboards have now been deployed, although there are some tweaks to configurations that need to be made.

Update the account_map Amazon Athena View

We need to update the account_map Amazon Athena view otherwise the Cloud Intelligence Dashboards only show the 12 Digit AWS Account Id of the Accounts as opposed to the Logical Account Names which can become overwhelming. Therefore we’ll update the view so that it pulls through the accurate account details that is collected via the Organization-Data-OptimizationDataCollectionStack Lambda Function.

10. In the AWS Account that you have chosen in the Pre-Requisites Step 1, test the Organization-Data-OptimizationDataCollectionStack Lambda Function by going into AWS Lambda, navigating to the Function and running a Test Event.

Once the Lambda Function has successfully executed, a Glue Crawler is triggered to crawl the data that it’s retrieved from AWS Organizations and written to Amazon S3. This in turn writes the data to a Glue Table called organization_data that is in a Glue Database named optimization_data.

11. In the AWS Account that you have chosen in the Pre-Requisites Step 1, execute the following SQL Query against the cid_cur database in Amazon Athena:

CREATE OR REPLACE VIEW account_map AS
SELECT DISTINCT
	"id" "account_id",
	"name" "account_name",
    "payer_id" "parent_account_id",
    "email" "account_email_id"
FROM
	"optimization_data"."organization_data"

Deploy the Cost Anomaly, Trends and Data Transfer Cost Analysis Dashboards

Cost Anomaly Dashboards

12. In the AWS Account that you have chosen in the Pre-Requisites Step 1, open up AWS CloudShell and execute the following:

python3 -m ensurepip --upgrade
pip3 install --upgrade cid-cmd
cid-cmd deploy --resources https://raw.githubusercontent.com/aws-samples/aws-cudos-framework-deployment/main/dashboards/cost-anomalies/cost-anomalies.yaml --dashboard-id aws-cost-anomalies --athena-database optimization_data

Trends Dashboards

13. In the AWS Account that you have chosen in the Pre-Requisites Step 1, open up AWS CloudShell and execute the following:

python3 -m ensurepip --upgrade
pip3 install --upgrade cid-cmd
cid-cmd deploy

• Select the Trends Dashboards
• Follow any instructions in the CLI

Data Transfer Cost Analysis Dashboards

14. In the AWS Account that you have chosen in the Pre-Requisites Step 1, open up AWS CloudShell and execute the following:

python3 -m ensurepip --upgrade
pip3 install --upgrade cid-cmd
cid-cmd deploy --resources https://github.com/aws-samples/aws-cudos-framework-deployment/raw/main/dashboards/data-transfer/DataTransfer-Cost-Analysis-Dashboard.yaml