Building a Modern Data Platform on AWS
Transaction Assurance Group (TAG) supports buying groups, distributors, retailers, and manufacturers by helping them to understand and manage their data and processes. TAG offers a full suite of cloud solutions, including: Vendor Portals, Vendor Scorecards, Item Management, Master Data Management, Data Warehousing, Data Analytics, Document Management, Contract Management, Rebate Management, Instant Rebate Reconciliation, and automated Request for Proposals. Having instant access to the full value of complex data sets provides a huge competitive advantage to their customers. Having instant access to the full value of complex data sets provides a huge competitive advantage to their customers.
Facing a massive increase in the amount of customer data it would have to handle, TAG needed a new data platform that was scalable, extensible, and easy to operate. To address this challenge, TAG engaged Symphonia to design and build a modern data platform using Amazon Web Services.
The resulting solution has enabled TAG to import customer data in less than 10% of the time it previously took and provide real time querying of the data sets that before required offline analysis, all while remaining secure, cost-efficient, and easy to operate with the existing team. Working with Symphonia, TAG have unlocked their ability to onboard new customers without technical limitations and without a proportional increase in costs.
TAG’s business is data. In late 2018, their pipeline of new customers was growing quickly, but the existing data ingestion and processing infrastructure was already under strain. The process for ingesting data for a new customer could take hours and would negatively impact the user experience of the application while that process was underway. Complex data analytics queries were either unacceptably slow or impossible to execute given the existing infrastructure. Each new customer brought more data into the system, and the team anticipated a 100x increase in the total amount of data within a year.
To maximize the value from the data, we need to Get It All, Get It Right, and Get It Now. Our previous ingestion process allowed us to get all of the data and process it to get the correct results, but it was too slow. It could take hours to complete the daily ingestion process for a customer, and the queries were too slow to allow access from our web app. This was not acceptable for us.Dr. Jonathan Graham, CTO of Transaction Assurance Group
TAG’s engineering team needed to stay focused on building customer-facing features, rather than be burdened by a high-touch data platform. TAG wanted a solution that minimized both operational overhead as well as infrastructure costs, so as not to dramatically increase the cost (of people and infrastructure) out of proportion with the number of customers.
In order to unblock their ability to onboard new customers, better service existing ones, and build for a successful future, TAG needed a modern data platform built on AWS.
TAG sought out a trusted partner to evaluate their current and future data needs, and to design and build a scalable modern data platform. TAG turned to Symphonia, an APN Select Consulting Partner with expertise in Serverless Computing and Big Data. Symphonia combines deep AWS technical expertise with extensive experience building scalable systems, drawing on their understanding of effective patterns and practices. This helps their customers level-up in understanding how to build and operate complex systems on AWS.
Data is central to everything that we do at TAG. We knew that we needed a platform that would scale with us, and that our data had to be secure. Symphonia came highly recommended from other industry experts that I respected, and after talking with them I quickly recognized that they knew what they were talking about and that they’d make a great partner.Dr. Jonathan Graham, CTO of Transaction Assurance Group
TAG’s new data platform makes use of several AWS services to deliver an easy-to-operate solution that will scale with TAG’s business needs.
The data warehouse itself is built on Amazon Redshift, which provides excellent query performance and the ability to handle the scale that TAG requires. Complex analytics queries that took minutes or hours to run on the legacy database now complete in seconds. Database performance has only improved over time, as the AWS service team responsible for Redshift continues to make improvements and add features.
Customer data is first loaded from S3 into an Aurora PostgreSQL database. That data is then loaded into Redshift via a data pipeline built using AWS Step Functions, Glue, Fargate, and Lambda. A Step Functions state machine is used to coordinate over 30 distinct data loading and processing steps. Data is ingested using Glue ETL scripts written in Python and PySpark, which make use of the Glue Data Catalog. That data is then post-processed and validated using additional Glue ETL, Fargate, and Lambda tasks. Common configuration parameters are stored using AWS Systems Manager Parameter Store, and secrets (like database passwords) are managed via Secrets Manager.
Tightly scoped IAM policies control access to AWS resources and data, ensuring that the various tasks have access to only the data and other components that they require. This “least privilege” security model not only reduces the likelihood of a security incident, but also keeps TAG’s data safe by limiting the “blast radius” of programming bugs or configuration errors.
From the Redshift database to the Lambda functions, all infrastructure is managed by AWS CloudFormation. Serverless components, like the data validation task, make use of the Serverless Application Model (SAM). This “infrastructure as code” approach ensures versioning, auditing, and the ability to automate release processes. It also makes it trivial to deploy development and test environments, which allow TAG developers to build new features and try out new AWS services without impacting the production data warehouse.
By building out a scalable, highly automated data platform on AWS, TAG has unlocked the ability to scale its processing of customer data, with no practical upper bound. Rather than having to carefully manage onboarding of new customers due to technical limitations, TAG is instead able to focus on customer-facing features and growing its business. The usage-based pricing for AWS services, combined with the “infrastructure as code” approach also means that TAG can expand its customer base and revenues without proportionally scaling its team, keeping costs down.
TAG’s backend data analytics processes now run in a fraction of the time that they used to, even as they perform far more complex calculations and analyses. Data ingestion is fast and repeatable, and has been separated and isolated from the main application. Whereas before the process for ingesting and processing a single customer’s daily data took upwards of 6 hours and severely impacted application performance, it now takes just minutes and is isolated from the main application. TAG’s users can also now search and export data from the data warehouse in near-real time - this was impossible before, even with much smaller datasets.
Another benefit of using managed AWS services like Glue ETL and Step Functions is that over time, the features and capabilities of those services are improved by the AWS service teams. For example, when Spark 2.4 support was added to Glue ETL, TAG was able to reduce the complexity of a data processing task and eliminate hundreds of lines of code.
TAG and Symphonia have continued to work together, refining and expanding not only the data platform, but also TAG’s usage of AWS services to solve other technical challenges and address new business requirements.
Partnering with Symphonia enabled us to build a secure and scalable data platform, allowing us to dramatically expand the value we can add to both existing and future customers. Furthermore, Symphonia’s expertise means that we can be confident that we will be able to implement and benefit from all the AWS changes and additions that happen as we move forward.Dr. Jonathan Graham, CTO of Transaction Assurance Group