The militants accuse the neutral humanitarian agency of failing to meet its obligations to help Taliban prisoners in a jail in Kabul.
The ICRC monitors detention conditions and provides medical aid. It scaled down its presence in Afghanistan last year after seven staff were killed.
The group told the BBC it was concerned by the Taliban's move.
Separately on Thursday, a blast in a Shia Muslim area in western Kabul killed at least 48 people, the health ministry told the BBC. The Taliban denied it was responsible for the latest attack targeting civilians in the Afghan capital.
ICRC looking for solution
Spokeswoman Andrea Catta Preta said the ICRC (International Committee of the Red Cross) was in close contact with the Taliban following their announcement, and hoping to find a solution so the group's humanitarian work could continue.
The ICRC treats all parties harmed by warfare humanely and does not take sides. It has in the past given first aid training to Taliban members. It operates in Taliban-controlled areas with guarantees of safety and helps to repatriate bodies from both sides after fighting between the militants and the Afghan army.
In its statement on Monday, the Taliban said that many of its prisoners held in Pul-e-Charkhi jail in the capital Kabul were in a terrible state of health and that the ICRC would be responsible for whatever happened to them. Hundreds of fighters have been on hunger strike to demand better prison conditions.
The ICRC spokeswoman said the agency could not be involved in negotiations between prisoners and the authorities.
The ICRC has worked in Afghanistan for 30 years and has more than 1,000 staff in the country. Last October it took the "painful decision" to close two offices and scale down operations at a third in order to protect staff.
The move came after six employees were shot dead in northern Jowzjan province by suspected Islamic State group gunmen and a physiotherapist was killed by a patient in Mazar-i-Sharif, also in the north.
Many other humanitarian organisations have pulled out of Afghanistan in recent years as Taliban and Islamic State group militants have stepped up attacks.
The Taliban's announcement on the ICRC comes at a time when militants have been launching several deadly attacks across Afghanistan.
In the latest major assault, as many as 44 Afghan police and soldiers were killed at a military outpost in northern Baghlan province early on Wednesday.
Another attack in Shajoy district of southern Zabul province saw at least seven soldiers killed, the provincial governor said. Three civilians in a vehicle were also killed by a Taliban bomb in the same province.
Meanwhile, Taliban fighters have finally pulled out of the city of Ghazni, a strategic location on the highway leading south from Kabul.
The militants had stormed into the city last Friday, sparking a five-day battle with government forces that left hundreds dead or wounded. The UN has warned that up to 150 civilians may have been killed.