How Visa built a container security solution

Like many large companies, financial services giant Visa has adopted containerization technology, which enables companies to migrate from traditional monolithic applications to microservices-based application architectures, making it easier to maintain large-scale cloud infrastructure , Update and deploy. Nonetheless, the challenge of turning applications into microservices is to ensure that the various parts of the container hosting are properly monitored and protected from attacks.

Visa’s security team did not deploy commercial solutions or adjust resources for their environment, but instead returned to the basics and created its own continuous monitoring solution. The solution can handle the implementation of security policies, incident detection and remediation. Because of its excellent safety, this project helped the company win the CSO50 award. Visa’s solution is called MASHUP (Adaptive Security Enhancement and Usage Platform Based on Microservices), which is mainly built on open source tools and libraries and makes full use of the existing native functions on the container orchestration platform, such as cgroups, files System access control and SELinux strategy.

Self-built vs. purchased
Visa did not use commercial solutions from established vendors and chose to build its own security platform as a result of a combination of factors.

First, many vendors that provide security solutions for container-based infrastructure and containerized applications are emerging companies, so these products may not have reached the maturity expected by large enterprises.

Other products may include the monitoring and protection of containers, but this function is only part of the huge feature set, the company does not need other functions. Visa doesn’t want to buy the entire application just because it wants to use 10% of the functions.

Another important factor when making a self-build or purchase decision is the flexibility and agility of development. Full control of the platform means that Visa can quickly deploy new functions according to the needs of the internal team, and can also change the product roadmap according to the new priorities and strategies set by the management. At the same time, whether it is possible to quickly fix the errors that have been discovered is also an important consideration.

In addition, some of the available commercial products will lack certain functions in the company’s specific environment, and this missing part of the function may cause the threat to not be mitigated. If the company needs these functions, it can only wait for the supplier to add them back.

Sunil Seshadri, Visa ’s chief information security officer for technology and operations, said: “Visa decided to build itself because we have extensive internal expertise and understand the problems we need to solve. In addition, we found existing solutions on the market We cannot fully meet our needs. Therefore, we are determined to build our own products based on our usage and threat models. ”

Finally, by building their own solutions, Visa’s operations, security, and development teams can work closely together to support each other. With the rise of DevSecOps, this approach has become increasingly important in recent years.

Visa stated in the CSO50 award declaration: “We have considered security control when designing, rather than considering security control after all work has been completed, which can reduce the possibility of failing to mitigate threats in the system in use Future costs incurred. Visa ’s container security products (MASHUP) can help Visa provide critical container and Kubernetes security under build, deployment, and runtime security, while protecting their critical application stacks running in Visa ’s private cloud . ”

Step by step
The company initially relied on the native features of the host operating system (such as Linux), and then gradually added features. Finally, the product is platform independent and can be easily used with any host operating system (OS) or container orchestrator.

MASHUP can perform access control and monitoring at the kernel, SELinux, runtime, and container application levels, and can distinguish and correlate events and activities at the host and container levels. They can also enforce default security configurations to prevent vulnerabilities that attackers can exploit. This “left shift” verification means that only approved and secure configurations can enter the production environment.

The focus of large-scale development is to build a machine learning engine. By looking for differences between profiles automatically generated by a large number of workloads, the engine can perform continuous point anomaly detection. The machine learning engine uses open source libraries such as TensorFlow. If an event is detected, MASHUP will respond by applying an automated script created by Visa ’s security team and correct the problem. According to Visa, their incident response time has been shortened from a few days to a few minutes.

Seshadri said: “It took more than two years to develop the system. Over time, we have iterated on it. The good news is that the K8s (Kubernetes) technology is maturing and the iteration speed is accelerating. The same is true of technology. It is conducive to the realization of new functions and the discovery of new threats, which allows us to continuously improve our products. ”

Scalability and efficiency
The system can easily add new nodes and containers to the cluster and automatically expand. MASHUP runs in the background, and the monitored applications are completely transparent. This means that there are no hooks when the application is running, and there is no need to change the application code.

Both MASHUP servers and agents are packaged as container images for rolling upgrades without downtime. The solution developed by Visa itself provides many security features. If you want to implement the same feature set, you may need to run multiple commercial solutions if you use commercial solutions. In contrast, Visa’s self-built solution consumes very little resources and is acceptable.

Because the system is now integrated into the continuous integration and continuous delivery (CI / CD) pipeline, security controls can be verified in real time, and vulnerability scanning has been upgraded from weekly or monthly regular monitoring to continuous monitoring.

Deployment and results
In the first year of project development, Visa has integrated MASHUP into half of the container deployment. The deployment rate increased to 70% in the second year, and it is expected to achieve full coverage by the end of the second year. The company stated in the CSO50 project declaration report: “From the absence of MASHUP to the emergence of MASHUP, the automatic defense of most security-related events and attacks depends on the average detection time (about a few minutes).

Visa believes that compared with the deployment of commercial solutions from third-party vendors, their own solutions can save a lot of costs. The money saved mainly includes the infrastructure, labor, periodic and annual maintenance costs related to the deployment of the supplier ’s solution, hiring or training of personnel to operate the supplier ’s solution, and product licenses. These costs can be offset by using open source technology.

Can other companies do the same? If they have the ability to implement such a project and have a deep understanding of the environment and data to be protected, then they should be able to do it. It all starts with an excellent systemic threat model.

Seshadri said: “Although many companies have similarities in threat status and attack surface, there are still differences between them, and these differences are important. Factors that companies should consider include: attitudes toward self-build or purchase, engineering Talent, excellent operations and engineering management, iterative flexibility, and the ability to reselect business solutions, capital, labor, and the consequences of unsuccessful projects if self-build is unsuccessful. ”

In the field of security, there is a saying that never invent your own encryption algorithm. This is because there are relatively few crypto experts and crypto analysts who are proficient in encryption algorithms all over the world, and their work is subject to rigorous scrutiny. After the encryption algorithm is invented, it must be peer reviewed if it is to be promoted. If the same way of thinking is extended to a security system, then we will get the idea that systems created by experts are always better than those built by ourselves.

In the past five years, thanks to the advancement of machine learning, free resources and peer-reviewed open source code libraries have gained a wider range of possibilities, users can use these technologies to change some of the status quo. Behind the scenes, many commercial products are also using the same open source tools and native functions to capture and analyze data, and build statistical models on this basis. Today, these models are well defined in the standard library. As long as companies can avoid blind spots that need attention in all incidents and understand their data and threats, they can create rule-based anomaly detection engines and adapt them to their environment.

Seshadri warned: “Open source code bases can help companies develop rapidly, but they will not eliminate the other requirements needed to build successful anomaly detection models. For example, all intelligent systems require data and can handle massive amounts of data Computing platforms, including machine learning libraries, software stacks that can utilize parameters or trained models, infrastructure that can respond to and handle threats in near real time, and people who can do all of this. ”

Visa is currently evaluating some ways to contribute to the tools it uses, and the process of returning development to the community.