Vitality Decay Damage is Vitality Damage over time. It is a fairly rare damage type and is reduced by Vitality Resistance.
Vitality Decay is effective against all types of living monsters, including Beasts and Beastmen, Insectoids, Demons and Plants. Undead monsters and Constructs are immune to this type of damage, though this immunity can be broken using -xx% Vitality (Damage) Resistance or -xx% to All Resistances modifiers.
Like Vitality Damage, Vitality Decay is affected by Intelligence. The formula to this is the same that is used to calculate any of the Elemental Damage over Time outputs (Burn Damage, Frostburn Damage, Electrical Burn Damage):
elementalDamageDV * ((intelligenceDV / 500) + 1)
This means, for every 50 points in Intelligence, Vitality Decay Damage is increased by +10%. 500 points in intelligence would add +100%.