Gitlab offers the possibility of disabling the cache for the jobs that do not need it. This can be done by using the configuration:
cache: {}