The following is an article written by Trellance’s Associate Vice President of Enterprise Architecture, Bill Fogg. The article originally appeared on CUInsight.com
The importance of selecting the right cloud data warehouse in order to meet your enterprise analytics objectives and satisfy your stakeholders cannot be overstated. And given the costs associated with these data analytics implementation projects, it is even more critical to start out on the right path.
Different users across the organization have different analytical needs. A modern data architecture addresses those different needs by providing a way to manage all data types, workloads and analysis. It blends elements from analytical data architectures such as Operational Data Store (ODS), Data Warehouse, Data Mart, Data Lake, Data LakeHouse and Data Mesh.
Historically, data management and processing products aligned primarily with one of these data architectures; however, as technology vendors tried to meet all of the different analytical needs of the organization and support all data types and workloads, they started integrating elements of the various data architectures into their products.
A modern data architecture platform includes the following capabilities:
- A data repository that supports all data types
- Data ingestion and transformation services
- Support for all analytics workloads
- SQL, machine learning, graph and spatial processing capabilities
- Ability to bring the processing to the data instead of the data to the processing
- Automated provisioning, scaling and administration
There are multiple architecture, technology and product alternatives for each of these data platform capabilities. These alternatives can either be assembled into a composite data platform using a best of breed approach or used to evaluate and select a single vendor-integrated product.
You already know the benefits of moving to the cloud – but are you aware of the benefits of utilizing a data warehouse that is as cloud native as possible? Cloud native applications are optimized for cloud scale and performance, based on granular managed services and utilize continuous delivery. They include features and capabilities such as: managed cloud platform services, horizontal scalability, automatic scaling using proactive and reactive actions, fault tolerance without degradation, and non-blocking asynchronous communication in a loosely coupled architecture.
The following process provides guidance for identifying realistic alternatives, evaluating them and combining them into an implementation-ready specification and design. It is very likely that you will need to repeat this cycle multiple times before deciding on a solution and technical design that can be successfully implemented and that meets your business objectives.
Step 1: Define the Objectives
The first step in choosing a cloud data warehouse is defining your primary and secondary business objectives for your cloud data platform. Considering only primary objectives may restrict platform considerations, while incorporating secondary objectives will force consideration of other system characteristics. Cloud data platforms are multi-functional, but each has its own strengths and weaknesses. Remember to look for user functions beyond those covered by the primary objectives.
Examples of data analytics workloads and use cases are:
- Descriptive analytics
- Diagnostic analytics
- Predictive analytics (data science and ml)
- Prescriptive analytics
- Data warehouse
- Data lake
- Data engineering
- Data exploration – data wrangling
- Data sharing and collaboration
Additionally, make sure to consider who will be involved in performing the analysis and producing the analytics outputs and what levels of access and security will need to be supported.
Step 2: Identify Realistic Alternatives
In theory, there are a nearly unlimited set of alternative products and technologies that can be assembled to satisfy your cloud data platform objectives. Realistically however, there are far fewer alternative options and trade-offs. Alternatives should maximize the platform’s potential. Some questions to ask when considering alternatives include:
- How can the platform improve organizational functioning?
- Conversely, since platforms don’t operate themselves, how can the organization improve the platform functioning?
Cloud data platforms are multi-disciplinary and require input from all stakeholders in the formulation of the alternatives.
Step 3: Evaluate the Qualities of the Alternatives and Define an Integrated Solution
Assess each of the alternatives and combine them into one optimal integral design. The value of the integrated solution should be greater than the “sum-of-the-parts.” The assessment needs to consider both the implementation/acquisition and operations/maintenance costs. Innovative solutions can have higher risks and uncertainties, but these can be mitigated by using components that are based on industry standard patterns and practices. Continue to involve the stakeholders in the valuation and selection process.
Step 4: Elaborate on the Integrated Solution
Each of the selected alternatives that comprise the cloud data platform must provide value and meet the defined business objectives. Tell each of their stories within the integral design. Make sure that the elaboration provides confidence that it can be successfully implemented when considering the constraints of the organization and project, such as organizational capability, budget and timeline. Continue to involve stakeholders in the design and realization process.
Step 5: Prepare Integrated Solution for Implementation
Now that you have your elaborated integrated solution, translate it to a technical design and ensure that it can be implemented and operationalized within the organization.
Implementation needs are important, especially if the product or technology is new to the organization. Knowledgeable individuals are important, but do not underestimate the importance of organizational maturity and change management. Identify areas where external assistance will be required and develop statements of work that can be used for hiring, contracting and RFPs.
Develop operationalization maturity objectives and timelines with the stakeholders and identify gaps in the solution design and gaps in the organization that need to be addressed. It is never too early to analyze the impact of the new solution on the end users. Prepare risk analysis and contingency plans.
Finally, recruit an executive sponsor and champion, develop the project timeline(s) and acquire funding.
Following these steps may be a lengthy process and will require a lot of input from a lot of stakeholders. But selecting the correct cloud data warehouse for your credit union can make all the difference in meeting your organizational objectives. Give due consideration to each step; consult stakeholders and make sure you’re evaluating all possibilities. This is not a process to be rushed, so make sure you approach it with the right mindset.
Bill Fogg is the associate vice president of enterprise architecture at Trellance.