One Piece Chapter 45, page 10: Before the Storm

Sanji explains that the waiters fled out of fear, which is why Zeff is forcing Luffy to work. Luffy asks Sanji to join his crew again, but Sanji refuses, stating he has his own reasons for staying at the Baratie.