Hi in this article i will try to tell you all about how to show custom error messages or popups on cartpage
in the below folder 
app/code/core/Mage/Checkout/controllers/CartController.php
near about line 537 you will find code something like this
 else {
                    $this->_getSession()->addError(
                        $this->__('Coupon code "%s" is not valid.', Mage::helper('core')->htmlEscape($couponCode))
                    );
                } 
which contains a error message for wrong coupon code apply
which i have change to 
else {
                    $this->_getSession()->addError(
                
                        $this->__('Coupon code "%s" is not valid.', Mage::helper('core')->htmlEscape($couponCode))
                    );
                    $myValue = 'Hello World';
                   Mage::getSingleton('customer/session')->setMyvalue($myValue);
                                               
                }
in this i have make use of
 magento sessions , i have set a session as setMyvalue
for the wrong code apply and than check the value of this in cart file
cart file path = under your theme /template/checkout/cart.phtm
i have retrieve the session value something like
 <?php $myValue=  Mage::getSingleton('customer/session')->getMyvalue(); ?> 
<?php if ($myValue!= ''){ ?>
do something:: you can do anything here like show your html ,js popups.....
<?php } else {?>
do somethindo something:: you can do anything here like show your html ,js popups.....
<?php }?>   
thats it ......