Your Button code should look something like this
Make sure you have window.js/default.css/magento.css added in page on which you are opening this popup
If not then add this like below in your custom layout update xml
dsvmaileditor will be define in js code
On above button onclick open first parameter should be ajax URL which return HTML and second is id anything i don’t know where it is used but it should be in parameter.
Now just add below JS code in your page.
My controller action just return phtml file html something like this
You can open these dialog using window.js
I have used Dialog.info
you can use as per your requirement.
For more info you can visit this site