English is definitely the most difficult language to learn.
The grammar is disgustingly irregular, the orthography only pretends to be phonetic, and the vocabulary is the largest of any language. (Some 400,000+ words)
Comparitively, Mandarin (the most commonly spoken 'dialect' of Chinese) does not conjugate anything, as all words are basically monosyllabic. (There's some minor debate over this, but not about whether or not there's conjugation, just the monosyllable-ness of it.) The writing system, while seemingly complex, can be broken down into radicals, which make remembering the characters considerbly easier.
Japanese seems to difficult to most modern westerners, but I imagine a classical Latin speaker would not find terribly much difficulty with it. They are both inflective languages, unlike english, so all the conjugation takes place with word endings. The word ending changes in Japanese are pathologically regular, and phonetically predictable (mostly). The writing system, which was stolen wholesale from the Chinese (The two languages are almost, but not entirely, unlike one another.) is large, but, unlike mainland china, there is a (truely) phonetic syllabary, so even if you know NO kanji, you can still write something down and have someone else read it. (Although, once you know the Kanji, writing anything in Hiragana/Katakana is just plain irritating.)