When you click on the displacement and go to face edit there's a displacement tab. There are 3 checkboxes for collisions.
No Physics Collision - Disables any physics objects colliding with the displacement. Useful for snow, mud, etc.
No Hull Collision - Disables any player or NPC collisions with the displacement.
No Ray Collision - Disables raycasts colliding with the displacement. Gunfire and bullets will not collide with the displacement surface.