To get points, you must hit the bar that is the same color as the player on the screen. If, on the other hand, you connect with a bar that is a different color than your own, you will die.
