Gina Cody School researchers unveil new AI approach to spot hidden patterns
“We wanted the algorithm to adapt to the data,” explains Yong Zeng.
Researchers working with complex datasets — whether from patient health records or self-driving car sensors — could uncover a wealth of insights hidden in patterns they never knew existed. For many AI experts, reaching that “aha” moment is difficult because most clustering tools rely on rigid setups or guesswork.
A team from Concordia University’s Gina Cody School of Engineering and Computer Science aims to solve that problem with a simpler method called Gauging-δ, now published in IEEE Transactions on Pattern Analysis and Machine Intelligence — a leading journal in artificial intelligence.
Rather than forcing data into predefined categories, the method invites the information itself to guide the clustering process from the ground up. Each data point starts off as its own group; the algorithm then systematically checks whether two groups genuinely fit together before merging them. This technique avoids guessing how many clusters might appear, while adapting naturally to data that’s either neatly separated or confusingly interwoven.
It’s an approach championed by Ph.D. student Jinli Yao, who collaborated with Jie Pan — a postdoctoral fellow at Concordia University during the study and now at the University of Calgary — and lead researcher Yong Zeng at the Concordia Institute for Information Systems Engineering. The team put their new method to the test on over one hundred datasets for the first time, spanning both synthetic scenarios and real-world examples.
“We wanted the algorithm to adapt to the data,” explains Zeng, who directs the Concordia Design & Entrepreneurship Centre. “When you impose too many rules from the start, you risk missing subtle connections. Our method helps you find those hidden relationships you didn’t even know to look for.”
Because it doesn’t require heavy fine-tuning or strict assumptions, the method could offer busy AI teams a more direct path to valuable discoveries. Alongside applications in health care and self-driving cars, the method might well open up new frontiers in finding novel concepts in any field overwhelmed by large, varied datasets — revealing patterns that standard techniques tend to miss.