Use record grouping over sources
Currently, the search results are provided to the user differentiated by
the observing source. In most use cases, it makes more sense to group
the otherwise same records coming from different sources, if all of the
sources match the filtering criteria (currently we only process data
from trusted recursors, so this is a tautology).
Two aspects need to be considered, first is the performance if source
discrimination is ever used in the future (there are still
bright-shining plans for that, albeit diminishing continually) and
second whether this should be represented in the data structures. That
could cut the overview tables sizes up to by a factor of source count,
which is a significant improvement for scalability and resource
efficiency. The associated growth of details tables is the main part of
the first aspect, but that might be just solved by suitable partitioning
scheme, as the Postgres’s abilities in this regard have almost reached
the required levels, with further improvements expected.
Author: Radko Krkoš krkos@cesnet.cz