I am not a tough teacher. My courses are not exhausting in their workloads, I almost never give exams, and never feel compelled to grade on some sort of "curve." My grading is a matter of the excellence (or not) of the work, and that is almost always apparent and not subtle.
I am trying to teach people how to think about something, or how to do some craft with sufficient skill so that they can do interesting work. You might have to become acquainted with a literature, and learn to read it. The mastery is never a matter of being rigorous. It's a matter of can you do it well enough.
I try to teach people how to read a book (you try to figure out what's going on by reading the beginning, the end, the chapter beginnings) and how to take on a substantial project (write yourself notes about what you might do, and then one day just start on it). So the reading or the paper or the project need not be overwhelming. The trick in each case is to learn how to go back to your work and make it better, to reread to check your understanding of a book, ...
Trained in the natural sciences, it was a matter of doing the problem sets and trying to understand what was going on. And some courses were demanding. But I don't recall any that were exhausting or impossible to understand. Actually, I recall taking a first year graduate algebra course, as a junior, doing poorly, and realizing later that I did not know how to study mathematics or how to get help--so this course was beyond me, but I think it was in part that I did not know how to study, in part my mathematical imagination was too demanding of concreteness. And in my early undergraduate years, I did have some quite exhausting reading assignments, but I did not realize they were exhausting, nor did I realize how to read political theory or social theory or even novels and literature when you were reading for a course.
My point here is that whatever it was, my experience was never that courses were tough, per se. I seemed always to have missed the supposedly tough teachers--or perhaps their reputations were unearned?
Surely there are subjects which are difficult for some fraction of students, typically mathematical parts of social science, or subtle reading of texts, or mastery of a foreign language, or following a complex argument in theory or philosophy. My feeling is that these courses could do a much better job of giving away the secret handshakes, telling people what is going on in the formalism, in the text's argument, in the language or argument. And some of the time a course is difficult since you do not have the right background and prerequisites--and the instructor could make things better for you.
Soon after I received my PhD I discovered that what was most interesting about what I could do in a classroom or in a discussion with a student was to think out loud (even if that thinking was already written down in my lecture notes) and in response to a student. Watching me think through a problem was the best teaching I could offer. Since I could readily be encouraged to follow my nose (actually, going off topic), by a student's second question or something that came to mind, my meandering and high-jumping was part of the exhibition. That is, my lecturing was a matter of watching me think and watching me link up apparently disparate topics.
Now this all might be a vanity, an excuse for a lack of didactic focus. I do know how to teach physics and be didactic. But for what I teach, didactic would seem to be missing the point.