During speech comprehension, the acoustic features of speech are mapped onto hierarchically organized linguistic units, including, e.g., syllables, words, and phrases. Here, I will present our recent studies on (1) the acoustic correlate of syllables and (2) rule-based encoding of phrases. I first will revisit the link between syllables and the acoustic speech envelope. Analyses based on large corpora confirm reliable phase locking between syllable onsets and speech envelope. Nevertheless, in the frequency domain, the mapping between the speech modulation spectrum and syllable rate is highly complex. Second, I will present our recent studies on neural encoding of “phrases” that are defined by artificial rules learned within an experiment, and compare the neural encoding of such artificial phrases and syntactically defined phrases. Finally, I will summarize some technique concerns on the analysis of neural encoding of speech units.