According to German prosecutors, the four men are between 39 and 51 years of age. They were arrested in Luebeck and Seevetal, towns in the state of Lower Saxony. Their apartments were also searched by police.
Three of the men are reportedly brothers, and investigators would not confirm whether the fourth man was also a family member, Germany's NDR broadcaster reported.
"Mustafa K., Abdullah K., Sultan K. and Ahmed K. are strongly suspected of being members of the foreign terrorist organization 'Jabhat al-Nusra' and of violating the War Weapons Control Act. Mustafa K. and Sultan K. are also to be charged under international humanitarian law with war crimes and looting," German federal prosecutors stated.
The men are alleged to have joined the al-Nusra Front, al-Qaeda's affiliate in Syria, no later than November 2012. They fought against Assad and Kurdish forces in the town of Ra's al-'Ain, in the al-Hasakah Governorate in northeastern Syria.
There is also evidence that Sultan and Abdullah fought the Kurdish People's Protection Units (YPG) in the north-east of the city.
The three brothers Mustafa, Abdullah and Ahmed all worked as drivers for Jabhat al-Nusra. Ahmed and other Jabhat al-Nusra members drove a pick-up truck, mounted with a machine gun, through the streets of north-east Ra's al-'Ain and told residents to leave the city.
The German authorities gave no information as to how the men came to be in Germany. On Tuesday, the suspects are to appear before a magistrate at the Federal Court of Justice, who will decide whether the men will remain in pre-trial custody.
Germany has been on high terror alert after a string of Islamist-linked attacks last year. Most recently, Daesh member Anis Amri drove a stolen truck into a crowd at a Berlin Christmas market in December, killing 12 people and injuring 48 others.