The door pull should feel reasonably solid, so I suspect the card/pull isn't fitted correctly. Are there two screws in the pull itself? Two in the edge? One at the mirror corner? Is the card hooked over the window aperture properly? Are the bottom plastic trim fixings in place?
When you refit the card, the last thing is to fit the door pull. It should be quite hard to manoeuvre the pull pocket into place.
Regarding the material, it can be stuck back down with carpet glue or at worst you could recover the entire section.
HTH
David