3UL is the value of 3 considered an unsignedlongint. << is the bitshift left operator. 3UL<<30 shifts 3UL to the left by 30 bits. ~ is the bitwise "not" operator, which flips all bits.
... but don't forget the final result is: 0011 1111 1111 1111 1111 1111 1111 1111
which is then bitwise AND'd with something.
Seems like a lot of work for nothing. Is there a reason that people use the << instead of just saying the hex for the value they want?
It seems this particular programmer couldn't be bothered simply writing a constant.
Operators are most often used on values only known at runtime, the code you posted is a bad counter-example.