This is part 3 of a series on creating custom monsters. This one covers creating monsters from scratch.
Creature Creation, Part 2: Modifications
Creature Creation, Appendix A: Traits
Creating a monster from scratch
There are two approaches here--numbers first and fiction first. I strongly recommend the fiction-first approach.
Here, pick a total CR and the pair of oCR and dCR that average to that CR. Generally, creatures have lower dCR than oCR; this accounts for the relative ease of adding defensive traits. On average, MM and VGtM creatures have around half the health indicated by their total CR but make up for it with other defensive traits and higher AC.
Use the table to extract HP, DPR, ATK, save DC, and AC values. Choose any combination of attacks, stats, etc. that give those numbers (less if you're going to add traits). Add mechanically-relevant traits and check their effect on the final CR.
There. You have a basic monster with no soul. For pure cannon-fodder, that can work but gets boring pretty fast.
First, figure out what kind of creature you want to make and give it a name. Good names are hard, but necessary. They shape how you'll see that creature. Consider its place in the world around you. Where does it live (terrain)? Does it go about in packs? What does it eat? Is it aggressive? Is it intelligent (more than an animal)? Figure out the size category (Tiny, Small, Medium, Large, Huge, or Gargantuan)
Second, figure out a CR target range. This can be very loose, but do you want it to threaten T1 characters (levels 1-4), T2 characters (levels 5-10), T3 characters (11-16), or T4 characters (17-20)? Note that monsters stay relevant for a while--the average threat faced by a T4 group will probably be in the CR 10-12 range. The high CRs (especially those over 20) are mostly reserved for boss-type monsters--legendary solos and non-legendary group fighters.
Now start deciding how the creature's ability scores should look. Is it a wall of meat that can absorb lots of punishment? If so, give it a high Constitution modifier. Is it nimble? Give it a high Dexterity. Is it smart? Give it a high Intelligence. Compare to various creatures--an ogre is STR 19, etc. The one hard cap here is that no ability score should be greater than 30, and only extremely-powerful creatures should break 20. 10 is average--if it's nothing special in either direction, give it a 10. Many (if not most) natural creatures have a decent Wisdom modifier (as in no lower than -1)--imperceptive creatures don't live long.
If a creature should be better than it seems at a certain saving throw, give it a bonus to that saving throw. This is usually the proficiency bonus (from the table for the CR), but just note that it has a bonus for now. You'll clean up the numbers later.
Figure out what the source of its armor class is--manufactured armors use the same table as PCs, but you can add natural armor (things like scales, being hard to hit because it's small, etc.) AC = (armor base or 10) + Dexterity modifier + shield (if present) + natural armor (if present). Adjust until it fits your image right, don't worry about the numbers.
Next is to determine HP: HP = HD x (HD value + Constitution modifier) on average. HD are arbitrary in number, but use the ones given by the size category. Remember that going up 1 size category increases the effect of each HD by 1 (except gargantuan, which adds 5/HD), so bigger (physically) things have fewer HD than smaller things of that same total HP. There is absolutely no direct connection between the number of HD and the CR--the Beholder (CR 13, 180 HP) has 19 HD, the Death Tyrant (CR 14, 187 HP) has 25. But the Beholder has a much higher CON (+4 instead of +2). Total HP is all that matters, not how it's calculated. Monsters don't take short rests (usually), so healing based on HD is rather irrelevant 99% of the time.
Now do the same sort of thing for attacks. Generally, cat-like creatures do both a bite and claws, while dog-like creatures only do a bite. Higher CR creatures tend to do more attacks rather than more damage per attack (at least within the same size category). Generally, a high CR creature should be making 3-4 attacks per round, a low CR creature 1 or 2. Remember that more attacks equals more rolls, and thus more table time. Monsters start making more than one attack much sooner than players do--there is no BAB-like forumla here. Go with what seems right for the creature. The attack bonus can come from either Strength or Dexerity (for melee weapon attacks)--ranged weapons use Dexterity and spell attacks (like a flameskull's fire ray) use one of the mental stats. Always add proficiency (using the table in the DMG).
Adding in non-attack abilities works similarly. I'd recommend looking for an analogous one in the MM or VGtM and copying the wording. This prevents stupid mistakes and loopholes and creates consistency. Note that if an ability doesn't do damage/affect survivability it will not affect CR but still may pose a significant threat. Non-combat abilities generally are free--sprinkle them in wherever appropriate.
Once you have an idea of what type of creature it is (both defenses and offenses), it's time to crunch the numbers. This works exactly the same as modifying another monster--go ahead and re-read the Calculating CR section. If the creature ends up too powerful or too weak for your purposes, adjust the health or the damage until it feels right.
Some traits are common throughout, others come on later.
Resistance to non-magical damage (as well as more than incidental resistances) are rare. These monsters are generally large at biggest, although there are a few huge beasts (dinosaurs, elephants, giant snakes, mostly). Lots of beasts, humanoids, and undead in this tier. At the upper reaches you start getting appreciable numbers of fiends. Wyrmling dragons are the dragons in this tier.
Resistance to non-magical becomes more common. A few legendaries, but still pretty rare. Significant primary spell-casters (other than the very weak acolyte/Priest NPCs) come online at the mid-top of this tier and low into the next one. This is where the vast majority of monsters sit, as these are the work-horse monsters for most campaigns. More large and huge monsters than small or tiny here. Young dragons, most fiends, major undead. "Standard" elementals are low T2.
Pretty much everything's online at this point. Many of the high T3 monsters (CRs 15-16) are starting to become more mini-bosses. Magic resistance is pretty common, as are legendary resistances/actions. Adult dragons, major fiends, and other boss-type monsters here. Beholders.
Just about everything here is legendary or a boss-type (things like the Balor are designed to have buddies around, which is why they don't have legendary actions). Bigger adult dragons, highest-rank fiends.
Epic Tier (21+)
Exclusively legendary, mostly ancient dragons. In fact, only Kraken does not have Legendary Resistances; it still has Legendary Actions.