We aimed to investigate the immunoglobulin G response and neutralizing activity against severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) among primary health care workers (PHCW) in France and assess the association between the neutralizing activity and several factors, including the coronavirus disease 2019 (COVID-19) vaccination scheme. A cross-sectional survey was conducted between 10 May 2021 and 31 August 2021. Participants underwent capillary blood sampling and completed a questionnaire. Sera were tested for the presence of antibodies against the nucleocapsid (N) protein and the S-1 portion of the spike (S) protein and neutralizing antibodies. In total, 1612 PHCW were included. The overall seroprevalences were: 23.6% (95% confidence interval (CI) 21.6-25.7%) for antibodies against the N protein, 94.7% (93.6-95.7%) for antibodies against the S protein, and 81.3% (79.4-83.2%) for neutralizing antibodies. Multivariate regression analyses showed that detection of neutralizing antibodies was significantly more likely in PHCW with previous SARS-CoV-2 infection than in those with no such history among the unvaccinated (odds ratio (OR) 16.57, 95% CI 5.96-59.36) and those vaccinated with one vaccine dose (OR 41.66, 95% CI 16.05-120.78). Among PHCW vaccinated with two vaccine doses, the detection of neutralizing antibodies was not significantly associated with previous SARS-CoV-2 infection (OR 1.31, 95% CI 0.86-2.07), but was more likely in those that received their second vaccine dose within the three months before study entry than in those vaccinated more than three months earlier (OR 5.28, 95% CI 3.51-8.23). This study highlights that previous SARS-CoV-2 infection and the time since vaccination should be considered when planning booster doses and the design of COVID-19 vaccine strategies.