The category archive pages as well as the product category archive pages were the biggest challenge as I discovered a WPML bug where for these pages the wpml_object_id filter does not properly work.