I think most people in 30s or older in UK would. The term Jersey comes from the Channel Island Jersey. Until today, I did not know that Guernsey is also a woolly jumper, originally made for seaman and named after the island Geurnsey in the Channel Islands. It is traditionally navy blue.
The Channel Islands started producing and selling wool when laws allowed them to import wool from England in 16th century.