Moderator: Do you believe that the performance, scalability, and integration capabilities are different in different cloud offerings?
Anush: We generally do the benchmark to understand the performance, scalability, and integration. With benchmarking we try to understand the volume of data that we are dealing with, the level of complexity, and more. Once the benchmarking is done, we derive conclusions like what kinds of data should be moved to the cloud? What kind of data needs to be kept on-prem? And more. The choice of cloud provider stems from what you are trying to achieve with the data.
Ivan: All three big cloud providers (Microsoft Azure, Amazon AWS, and Google Cloud Provider) provide the option of scalability and performance. The question here should be how hard is it for users to keep their cloud infrastructure scaling efficiently without the need for highly specialized skills. That should be the key point to assess while evaluating different cloud providers without worrying about the number of systems, system architecture, or more. One should be able to focus on using the components rather than worrying about the tech behind them.
Moderator: What learnings or guidance do you have regarding solving the problem of siloed data. After moving to the cloud, what should be done, what should not be done? Any insights on why moving to the cloud is so difficult?
Anush: This is a problem that most clients face. For instance, a client’s sales team decided to use Salesforce, thinking that the same data could be useful for the marketing team. However, there can be a lot of mismatches and duplication in the data. At Sigmoid, our data engineering team follows best practices to make sure that duplication is quickly removed. We use technology and tools to streamline data. While this has solved the problem to a great extent, we are still working on bringing best practices to deal with data silos.
Nelson: The first point to keep in mind while dealing with siloed data is how the IT team can deliver solutions for business people. The second point is knowledge management. Several siloed data don’t have the business rules to reveal the solution. Recognizing siloed data at the beginning of the data migration process can be valuable. This data should then be shared in a secure way with the data user community. Our strategy is that we first catalog the siloed data. We then bring this data to our production environment to allow the IT guy to do reversal mapping and documentation. We then rebuild the whole solution using the correct technology in the correct data source. It is not only the point of the willingness of data owners to isolate or secure data but it is the data governance issue. It helps to take a user-first approach.
Moderator: Do you think that enterprises could have been better prepared to manage uncertainties, like COVID-19, faster if they had already moved to the cloud? Do you think that moving to the cloud now will help them set up for quicker response to crises?
Ivan: I wouldn’t want to say, yes, of course. It depends on the amount of data that each company has. In our case, we moved to the cloud a few months before COVID-19 hit us. It proved to be very useful. When the pandemic hit, the architecture that we had built enabled us to understand how consumer behavior was changing. We were able to make decisions to accordingly keep serving our customers. Moving to the cloud helped us reach the granularity of it, with the kind of analyses and the speed at which we could interact with the data. We wouldn’t have been able to achieve it in the previous architecture. So in our case, it absolutely helped.
Anush: Working with some of the companies in the CPG and E-commerce industries, we found that the processing of real-time data was minimal during the pre-pandemic state. It drastically changed during the pandemic. Our engineering team is now processing terabytes of data generated by these companies, especially in the E-commerce industry, to take big business decisions.
Nelson: Our business model is direct-selling and pandemic directly impacted our customer count. So, going digital was not an option but a necessity if we wanted to survive. But just deciding to go digital is not enough — we had to reorient our decisions and accelerate a data-driven culture. It had to be done quickly with no time to train models. Moving to the cloud helped us to make this transition quickly.