Change #238875

Comments

factor: use 64-bit internal counters
* src/factor.c (factor_using_pollard_rho)
(factor_using_pollard_rho2, mp_factor_using_pollard_rho):
Use int_fast64_t for internal counters rather than int, as int
could overflow on some somewhat-practical examples.  Problem
discovered on a hypothetical platform where W_TYPE_SIZE is neither
32 nor 64, when factoring 0x7ffffffffffffeab7fffffffffff7369 ==
170141183460469225450570946617781744489, causing k to overflow in
mp_factor_using_pollard_rho.  Presumably a similar problem exists
in the previous stable coreutils 9.7, too, on 32-bit platforms
with somewhat-larger test cases, though I haven’t take the
somewhat-extensive CPU time to discover it.

Changed files

• src/factor.c