New complexity estimation on the Rainbow-Band-Separation attack

Multivariate public key cryptography is a candidate for post-quantum cryptography, and it allows generating particularly short signatures and fast verification. The Rainbow signature scheme proposed by Ding and Schmidt is such a multivariate cryptosystem, and it is considered secure against all known attacks. The Rainbow-Band-Separation attack recovers a secret key of Rainbow by solving certain systems of quadratic equations, and its complexity is estimated by the well-known theoretical value called the degree of regularity. However, the degree of regularity is generally larger than the solving degree in experiments, and an accurate estimation cannot be obtained. In this article, we propose a new theoretical value for the complexity of the Rainbow-Band-Separation attack using a Gröbner basis algorithm, which provides a more precise estimation compared to that using the degree of regularity. This theoretical value is deduced by the two-variable power series [Formula presented] Since the two-variable power series coincides with the one-variable power series at t₁=t₂ deriving the degree of regularity, the theoretical value is less than or equal to the degree of regularity under a certain condition. Moreover, we show a relation between the Rainbow-Band-Separation attack using the hybrid approach and the HighRank attack. By considering this relation and our theoretical value, we obtain a new complexity estimation for the Rainbow-Band-Separation attack. Furthermore, applying our theoretical value to the complexity formula used in the NIST PQC 2nd round, we show that a slight modification of the proposed Rainbow parameter sets is required. Consequently, we provide a new theoretical value for generally estimating the solving degree of a bi-graded polynomial system, which can influence the parameter selection of Rainbow in the NIST PQC standardization project.