mtd: gpmi: do not set the default values for the extra clocks