TIP 1: FOCUS
With character creation, many (myself included) have fallen into the trap of making a design more complicated than it needs to be. What I've learned over my years in character design is simple: You generally want to boil down a character to one defining trait. It may seem obvious but you'd be surprised how often it is overlooked.
You might think, "How can a character be realistic and compelling with only one defining trait?" (Or in this case, a monster) But if you look at some of the most popular characters in media, you can usually cut down their personality to a single adjective, or a single visual characteristic.
It is the suspension of disbelief (verisimilitude) that allows the human mind to either relate, hate, or fear a character, and generally that suspension of disbelief is triggered by one single trait that we latch onto.
For one of my more simple monsters, (his name is "Friend"!) you can tell just by a glance what his deal is. The lack of eyes and a simple shaped body is made to draw your eyes towards his best feature:
There are, of course, other things that make Friend a bit unnerving, like the contrasting color palette, or his abnormal body, but from the feedback I've gotten it is almost always the mouth that seems to pull it all together.
TIP 2: D E T A I L
Now, this one is a bit of an add on to the last, as I've already talked about an overuse of detail sometimes being detrimental to a final piece. But, this is an example where detail makes a drawing just that more creepy. Specifically, this works best when adding detail to an otherwise very simple design. Here's an example:
This is a drawing of myself! Cute, right? (I hope so...)
That is because of its simplicity. I'm sure you can think of plenty round cute characters with a simple design that just make your heart melt a little when you think of them.
But if you wanted to make this creepy, you don't have to sacrifice the overall simplicity. Here:
Now how am I looking? A bit unsettling?
That's because of focused detail put on a select few features, making the face creepy without changing the simple style.
Now let's apply this to monsters!
Here we have a wholesome, strapping young fish-lad. He looks friendly, no?
It is very easy to make him look evil.
Only enhancing already existing features to a point of absurdity, we can get this:
Which is far, far more disturbing. Turning the stylized realistic is probably the most effective way to make someone rethink whether or not they actually like their favorite characters, and introduces a new way to draw horror.
Good luck with your fiends!