PIG: The platform that brings universities together to democratise access to scientific computing

Sharing resources to expand opportunities. That is the premise driving the Platform for GPU-based Research (PIG), an initiative of the CUDI Supercomputing Community that is demonstrating how collaboration between institutions can open up new possibilities for scientific research, artificial intelligence and the training of specialised talent in the region.

In a context where access to advanced computing infrastructure has become a prerequisite for conducting cutting-edge research, PIG proposes a simple yet transformative model: connecting resources distributed across different universities and making them available to a wider academic community.

The initiative brings together GPU (Graphics Processing Unit) infrastructure provided by various member institutions of CUDI – Mexico’s National Research and Education Network (RNIE) – the organisation that manages it, enabling researchers and students to access these resources via a shared platform, thereby reducing technical, financial and administrative barriers.

A solution born of a shared need

The story of PIG began in 2019, when CUDI was exploring the possibility of Mexico participating in what was then known as the Pacific Research Platform, now the National Research Platform (NRP), which was led by the University of California, San Diego. However, a lack of resources to acquire new infrastructure forced them to consider an alternative: “Due to a lack of funds to purchase the suggested equipment, we decided to undertake a similar project using infrastructure donated by various institutions,” recalls Diego Dávila, the project leader.

The proposal was as ambitious as it was pragmatic: to build a distributed platform based on Kubernetes (an open-source system that automates the deployment, scaling and management of containerised applications) that would enable specialised processing resources to be shared amongst universities. “The aim was, and still is, to demonstrate the advantages of sharing infrastructure under this model,” explains the PIG project leader.

Thus, what began as a collaborative experiment eventually became a functional platform that today connects computing capacities distributed across various Mexican institutions.

Greater access, more science

For CUDI, the true value of PIG lies not only in the technology, but in its ability to expand opportunities, as explained by its Director-General, Dr Moisés Torres: “The PIG project is strategic for the RNEI [sic] because it is a way of giving researchers, postgraduate and undergraduate students access to GPU resources to advance their research projects. These resources are made available to institutions and researchers who would otherwise not have access to this processing equipment.”

The importance of this mission is particularly evident in a landscape where artificial intelligence, machine learning and big data analytics demand ever-increasing computational capacity. “This project provides researchers with access to the GPUs they need for their artificial intelligence projects in order to advance their research,” reaffirms the director general of the Mexican academic network.

Trust as invisible infrastructure

Admittedly, the technological challenge was considerable, but building the trust necessary to share resources between institutions was key. “Inter-institutional collaboration is of the utmost importance, as the project aims to pool the GPU resources held by institutions and enable their researchers to access the combined resources of the entire project,” explains Dr Torres.

From a technical perspective, the path was not without obstacles. “Security and connectivity have been the biggest challenge,” says project leader Diego Dávila.

Much of the equipment integrated into PIG belongs to high-performance computing centres with strict security policies, protected networks and restricted access mechanisms. Even so, by developing secure interconnection mechanisms, the project succeeded in creating a shared infrastructure distributed across multiple universities.

And the results have exceeded expectations. “By sharing infrastructure automatically and continuously, we have established a significant level of trust between the participating institutions. This will open the door to other types of collaborative projects,” says Dávila.

A community for learning and experimentation

Beyond its processing capabilities, PIG has become a space for training new generations of researchers and data science specialists.

“We have built an ideal platform for experimentation and learning for future data scientists,” emphasises Dávila.

The experience of Bolivia Cuevas, a postdoctoral researcher at the UNAM Institute of Astronomy, illustrates this impact. Together with a team – comprising Juan Carlos Cuevas Tello, Luis León Anaya, Diego Dávila, Julio César Clemente, Octavio Valenzuela, Luciano Díaz, Victor Minjares, Edilberto Sánchez, Lizette Robles and Martha Ávila – she came to PIG through colleagues involved in the LSST (Legacy Survey of Space and Time) collaboration and soon found a solution to a recurring problem: the difficulty in accessing available GPUs when she needed them.

“Most of the cards on other infrastructure were occupied almost all the time. With PIG, I’ve had the opportunity to use GPUs and the infrastructure has been user-friendly,” says the researcher, whose work involves “testing classification algorithms for variable stars using LSST data”. Bolivia explains: “In astrophysics, we’re interested in studying variable stars (which change in brightness over time) as they allow us to calculate distances to the galaxies in which they are found”.

She is currently using the platform to carry out research related to classifiers of variable astronomical objects. “For now, the research is in its early stages and one of the next steps is to continue working with other users (some of whom are also PIG users) to make progress in finding the best algorithms for classifying variable stars in datasets such as those from the LSST survey at the Vera C. Rubin Observatory. “Up to this point, my main use of PIG has been for my research and in preparation for the hackathon. The team will now expand to include some of the hackathon participants who expressed an interest in continuing to work on this topic,” explains the UNAM postdoctoral researcher, referring to the supercomputing hackathon organised by CUDI and LAMOD last August.

Although adapting to a new technology involved a learning curve, the experience proved positive, says Bolivia: “Learning it was a bit tricky because I wasn’t very familiar with Kubernetes, but once I’d learnt it, everything was very straightforward to use. The truth is, it’s a very practical platform.” For her, the difference has been significant: “Having PIG has been a real game-changer, as having multiple cards allows us to run several jobs at the same time.”

From training to the development of a reproducible scientific pipeline

The experience of a student at the Autonomous University of San Luis Potosí (UASLP), shared by Dr Juan Carlos Cuevas Tello from the Centre for Research and Postgraduate Studies at the Faculty of Engineering, is another example of how PIG can support the training of new researchers. The student, Ernesto Vizcaíno Alvarado, was introduced to the platform during a visit by Diego Dávila, head of PIG, to the university, just as he was starting his community service placement and beginning to collaborate with researchers from the astronomy department at UNAM on a project to classify stars based on their light curves.

Before gaining access to PIG, the main barrier was the lack of specialised infrastructure: acquiring and maintaining a workstation with sufficient computing power proved costly, whilst computationally intensive analyses could consume all the resources of personal computers. The platform made it possible to offload scientific processing from local computers, store large datasets and run experiments with greater continuity and reproducibility.

Using this infrastructure, Ernesto Vizcaíno Alvarado developed a reproducible machine learning pipeline to classify periodic variable stars with multi-band light curves from the OGLE and ZTF astronomical surveys. The work encompassed data preparation and auditing, computational period search, the calculation of 51 features per light curve, the training and calibration of classifiers, and evaluation using separate test datasets. The system can distinguish eight physical subtypes and, in internal tests, achieved a macro F1 score of 0.988 with OGLE data and 0.877 with ZTF data. Rather than merely reducing execution times, the platform enabled the student to evaluate multiple models, study their stability and retain evidence of each stage with a view to producing a scientific paper.

Advancing astronomy with artificial intelligence

Another example of the platform’s potential comes from the SMART-Astro project, developed by researchers at UNAM’s Institute of Astronomy to accelerate the detection of asteroids using artificial intelligence and parallel processing.

Dr Benjamín Hernández, from the Department of Computational Astrophysics at UNAM’s Ensenada Institute of Astronomy, found in PIG an opportunity to overcome a specific limitation. “The main barrier was access to GPUs,” he notes, pointing out that, thanks to this processing capacity, his team has been able to make progress on the SMART-Astro-TR project, which aims to speed up the analysis of large volumes of astronomical images.

The results demonstrate the impact that access to shared infrastructure can have. “Currently, the GPU version is approximately 100 times faster than the CPU version,” says Dr Hernández, who acknowledges that the platform has enabled his team – comprising student Ricardo Celis Carmona, and Drs Benjamín Hernández Valencia, José Sergio Silva Cabrera and Mauricio Reyes Ruiz—not only to carry out experiments, but also to test architectures, validate libraries and better understand the computational requirements of future stages of research.

Towards a regional shared-infrastructure community

Five years after its inception, PIG continues to grow. The inclusion of new institutions, increased computing capacity and dedicated teams to run the project are all part of the roadmap for the coming years.

But the vision extends beyond Mexico. “We are now seeking the participation of more higher education institutions in Mexico and Latin America so that we can create a project offering access to these resources, thereby fostering an ecosystem of inter-institutional and international collaboration,” states Moisés Torres, inviting, on behalf of CUDI, the community associated with the National Research and Education Networks that are members of RedCLARA to discover and benefit from PIG.

In a region where equitable access to scientific infrastructure remains a challenge, PIG represents much more than a technological platform: it is a demonstration that cooperation between institutions can multiply capabilities, bridge gaps and generate new opportunities for science.

Because when universities share resources, they also share possibilities. And that, precisely, is what PIG is doing for Mexican research.

Find out more about PIG at: https://supercomputo.cudi.edu.mx/pig-plataforma-para-la-investigacion-con-gpus