Use record grouping over sources

Currently, the search results are provided to the user differentiated by the observing source. In most use cases, it makes more sense to group the otherwise same records coming from different sources, if all of the sources match the filtering criteria (currently we only process data from trusted recursors, so this is a tautology).
Two aspects need to be considered, first is the performance if source discrimination is ever used in the future (there are still bright-shining plans for that, albeit diminishing continually) and second whether this should be represented in the data structures. That could cut the overview tables sizes up to by a factor of source count, which is a significant improvement for scalability and resource efficiency. The associated growth of details tables is the main part of the first aspect, but that might be just solved by suitable partitioning scheme, as the Postgres’s abilities in this regard have almost reached the required levels, with further improvements expected.

Author: Radko Krkoš krkos@cesnet.cz